RFC 7798
RTP Payload Format for High Efficiency Video Coding (HEVC)
Title Block 📄🎀
RFC 7798 is titled "RTP Payload Format for High Efficiency Video Coding (HEVC)" and it's from March 2016 🌸. It's an Internet Standards Track document, so it's a real, official standard, not just an idea. The ISSN is 2070-1721. It doesn't obsolete or update anything, since none is listed in the header.
The authors are a squad from Qualcomm (Y.-K. Wang), Fraunhofer HHI (Y. Sanchez and T. Schierl), Vidyo (S. Wenger) and Nokia (M. M. Hannuksela) 🫶. The document came from the IETF, the Internet Engineering Task Force, which is the group that writes the Internet's rulebook 📚.
Abstract 💖
Okay, so this memo explains how to pack HEVC video into RTP packets 🎬. HEVC is also called ITU-T Recommendation H.265 and ISO/IEC International Standard 23008-2, and it was developed by the Joint Collaborative Team on Video Coding (JCT-VC) ✨.
The format lets you put one or more Network Abstraction Layer (NAL) units into a single RTP packet payload 📦. It also lets you chop one NAL unit up across several RTP packets, which is called fragmentation 🧩. It works for one stream of HEVC bitstream, or for several RTP streams too, and when there are multiple streams you can use one transport or several transports 🚚.
It's giving versatile 💅. The format is useful for videoconferencing, internet video streaming, and high-bitrate entertainment-quality video, among other things 🍓.
Status of This Memo 🦋
This is an Internet Standards Track document 👑.
It's a product of the IETF and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) ✅. If you want more on Internet Standards, check out Section 2 of RFC 5741 🔍.
For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc7798 🌈.
Copyright Notice 🎀
Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved 💜.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at http://trustee.ietf.org/license-info and which were in effect on the date it was published 📜. Please actually read those, because they explain your rights and restrictions for this document 🙈. Code Components pulled out of this document must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License 🔥.
Table of Contents 🗂️
1. Introduction ....................................................3
1.1. Overview of the HEVC Codec .................................4
1.1.1. Coding-Tool Features ................................4
1.1.2. Systems and Transport Interfaces ....................6
1.1.3. Parallel Processing Support ........................11
1.1.4. NAL Unit Header ....................................13
1.2. Overview of the Payload Format ............................14
2. Conventions ....................................................15
3. Definitions and Abbreviations ..................................15
3.1. Definitions ...............................................15
3.1.1. Definitions from the HEVC Specification ...........15
3.1.2. Definitions Specific to This Memo .................17
3.2. Abbreviations .............................................19
4. RTP Payload Format .............................................20
4.1. RTP Header Usage ..........................................20
4.2. Payload Header Usage ......................................22
4.3. Transmission Modes ........................................23
4.4. Payload Structures ........................................24
4.4.1. Single NAL Unit Packets ............................24
4.4.2. Aggregation Packets (APs) ..........................25
4.4.3. Fragmentation Units ................................29
4.4.4. PACI Packets .......................................32
4.4.4.1. Reasons for the PACI Rules (Informative) ..34
4.4.4.2. PACI Extensions (Informative) .............35
4.5. Temporal Scalability Control Information ..................36
4.6. Decoding Order Number .....................................37
5. Packetization Rules ............................................39
6. De-packetization Process .......................................40
7. Payload Format Parameters ......................................42
7.1. Media Type Registration ...................................42
7.2. SDP Parameters ............................................64
7.2.1. Mapping of Payload Type Parameters to SDP ..........64
7.2.2. Usage with SDP Offer/Answer Model ..................65
7.2.3. Usage in Declarative Session Descriptions ..........73
7.2.4. Considerations for Parameter Sets ..................75
7.2.5. Dependency Signaling in Multi-Stream Mode ..........75
8. Use with Feedback Messages .....................................75
8.1. Picture Loss Indication (PLI) .............................75
8.2. Slice Loss Indication (SLI) ...............................76
8.3. Reference Picture Selection Indication (RPSI) .............77
8.4. Full Intra Request (FIR) ..................................77
9. Security Considerations ........................................78
10. Congestion Control ............................................79
11. IANA Considerations ...........................................80
12. References ....................................................80
12.1. Normative References .....................................80
12.2. Informative References ...................................82
Acknowledgments ...................................................85
Authors' Addresses ................................................861. Introduction ✨
Okay, so HEVC is the High Efficiency Video Coding spec 🎥. It's formally published as both ITU-T Recommendation H.265 [HEVC] and ISO/IEC International Standard 23008-2 [ISO23008-2]. The ITU-T ratified it in April 2013, and reportedly it gives a big boost in coding efficiency over H.264 [H.264] 🚀. Literally so much better!!
This memo is the RTP payload format for HEVC 💅. It shares its basic design with the RTP payload formats of [RFC6184] and [RFC6190] 🫶. It's similar to those earlier specs in terms of design philosophy, security, congestion control, and overall implementation complexity 💖. That's on purpose, because at least RFC 6184 is widely deployed and generally known in the relevant implementer communities 🌸. Mechanisms from RFC 6190 were brought in because HEVC version 1 supports temporal scalability 🧚.
To help the overlapping implementer community, the non-normative, explanatory parts of this memo often only highlight the differences between RFCs 6184 and 6190 and the HEVC payload format 🦋. Those parts assume you're basically familiar with both specs 📚. The normative parts, though, don't require you to study RFC 6184 or RFC 6190 at all ✨.
1.1. Overview of the HEVC Codec 🌈
H.264 and HEVC share a similar hybrid video codec design, so they're kind of like cousins 👯♀️. In this memo we give a very brief overview of the HEVC features that the payload format in this memo somehow deals with 🎀. Implementers have to read, understand, and apply the ITU-T/ISO/IEC specs about HEVC to get interoperable, well-performing implementations 🔥. Implementers should also consider testing their design, including how the payload format implementation works with the core video codec, using the tools ITU-T/ISO/IEC provide, like the conformance bitstreams specified in [H.265.1] 🧪. Historically, skipping that has led to systems that perform badly and aren't secure 💀. So please don't skip it, bestie 🙈.
Conceptually, both H.264 and HEVC include a Video Coding Layer (VCL), which is often used to mean the coding-tool features 🎞️. They also include a Network Abstraction Layer (NAL), which is often used to mean the systems and transport interface aspects of the codecs 📡. Two layers, one vibe, periodt 💜.
1.1.1. Coding-Tool Features ✨
Okay so, bestie, HEVC uses the same basic video coding design as earlier hybrid-video-coding standards, including H.264 🎀. First a prediction signal gets formed, either by intra- or motion-compensated prediction, and then the residual (the difference between the original and the prediction) gets coded. The glow-up is that HEVC redesigned and improved almost every part of the codec compared to the older ones, and it also has extra tools that make parallel implementations easier 💅.
Quad-tree block and transform structure 🌸
One of the big tools behind HEVC's coding efficiency is its flexible coding blocks and transforms, which are defined in a hierarchical quad-tree manner. H.264's basic coding block is a macroblock of fixed size 16x16, but HEVC defines a Coding Tree Unit (CTU) with a maximum size of 64x64. Each CTU can be split into smaller units in a hierarchical quad-tree manner, and it can represent blocks as small as 4x4. The transforms can also come in different sizes, from 4x4 all the way up to 32x32. Using large blocks and transforms is a major source of HEVC's gains, especially at high resolutions 🦋.
Entropy coding 🧚
HEVC uses a single entropy-coding engine, based on Context Adaptive Binary Arithmetic Coding (CABAC) [CABAC]. H.264 uses two distinct entropy coding engines, so this is a real change. CABAC in HEVC shares a lot with CABAC in H.264 but has several improvements, including better coding efficiency and lower implementation complexity, especially for parallel architectures ✨.
In-loop filtering 💜
H.264 has an in-loop adaptive deblocking filter that smooths out blocking artifacts around transform edges in the reconstructed picture, which improves picture quality and compression efficiency. HEVC uses a similar deblocking filter but with somewhat lower complexity. On top of that, pictures go through another filtering step called Sample Adaptive Offset (SAO), which is brand new in HEVC. SAO adds a pixel-level offset in an adaptive manner and usually acts as a de-ringing filter. It improves picture quality especially around sharp edges, and that's a big part of HEVC's visual quality wins 😭🔥.
Motion prediction and coding 🌈
Motion prediction got a lot of love here, and there are two main categories. The first is the motion merge and Advanced Motion Vector Prediction (AMVP) modes. The motion information of a prediction block can be inferred from spatially or temporally neighboring blocks. This is kinda like the DIRECT mode in H.264, but it's upgraded to work with the flexible quad-tree structure and to make parallel implementations smoother. The motion vector predictor can also be signaled for better efficiency. The second category is high-precision interpolation: the interpolation filter length goes from 6-tap to 8-tap, which improves coding efficiency but also adds complexity. The interpolation filter is also defined with higher precision and without any intermediate rounding operations, which pushes coding efficiency even further 👑.
Intra prediction and intra-coding 🍓
H.264 has 8 intra prediction modes, but HEVC supports angular intra prediction with 33 directions. That extra flexibility improves both objective coding efficiency and visual quality, since edges can be predicted better and ringing artifacts around edges can be reduced. The reference samples are also adaptively smoothed based on the prediction direction. To avoid contouring artifacts, a new interpolative prediction generation was added to boost visual quality. And for 4x4 intra-transform blocks, HEVC uses the Discrete Sine Transform (DST) instead of the traditional Discrete Cosine Transform (DCT) 🫶.
Other coding-tool features 🙈
HEVC also has tools for lossless coding and efficient screen-content coding, like skipping the transform for certain blocks. These are super useful when you're streaming a mobile device's user interface to a big display 📺💖.
1.1.2. Systems and Transport Interfaces 🎀
HEVC inherited its basic systems and transport interface designs from H.264, so it's giving familiar vibes 🥰. That includes the NAL-unit-based syntax structure, the hierarchical syntax and data unit structure, the Supplemental Enhancement Information (SEI) message mechanism, and the video buffering model based on the Hypothetical Reference Decoder (HRD). The hierarchical structure has sequence-level parameter sets, multi-picture-level or picture-level parameter sets, slice-level header parameters, and lower-level parameters. Here's a list of the differences from H.264 in these areas ✨.
Video parameter set 🧚
HEVC introduced a brand-new type of parameter set called the Video Parameter Set (VPS). For the first (2013) version of [HEVC], the VPS NAL unit is required to be available prior to its activation, but the information in the VPS isn't necessary for the operation of the decoding process. For future HEVC extensions, like the 3D or scalable extensions, the VPS is expected to include information necessary for the operation of the decoding process, such as decoding dependency or information for reference picture set construction of enhancement layers. The VPS gives you the "big picture" of a bitstream: what types of operation points are provided, the profile, tier, and level of those operation points, and other high-level properties that can be used as the basis for session negotiation and content selection, etc. (see Section 7.1). It's the main character of the bitstream 👑.
Profile, tier, and level 💜
The profile, tier, and level syntax structure can be included in both the VPS and the Sequence Parameter Set (SPS). It includes 12 bytes of data that describe the entire bitstream, including all temporally scalable layers (called sub-layers in the HEVC specification). It can also optionally include more profile, tier, and level information for individual temporally scalable layers. The profile indicator shows the "best viewed as" profile when the bitstream conforms to multiple profiles. This is similar to the major brand concept in the ISO Base Media File Format (ISOBMFF) [IS014496-12] [IS015444-12] and to file formats based on ISOBMFF, like the 3GPP file format [3GPPFF]. The structure also has indications for three things: 1) whether the bitstream is free of frame-packed content, 2) whether the bitstream is free of interlaced source content, and 3) whether the bitstream is free of field pictures. If the answer is yes for both 2) and 3), the bitstream contains only frame pictures of progressive source. Based on these, clients and players that don't support post-processing for frame-packed, interlaced source content, or field pictures can reject bitstreams that contain those pictures 🙅♀️.
Bitstream and elementary stream 🌸
HEVC defines an elementary stream, which is new compared to H.264 🆕. An elementary stream is a sequence of one or more bitstreams, and one with two or more bitstreams has typically been formed by splicing together two or more bitstreams (or parts of them). When an elementary stream contains more than one bitstream, the last NAL unit of the last access unit of a bitstream (except the last bitstream in the elementary stream) must contain an end of bitstream NAL unit. The first access unit of the next bitstream must be an Intra-Random Access Point (IRAP) access unit. That IRAP access unit may be a Clean Random Access (CRA), Broken Link Access (BLA), or Instantaneous Decoding Refresh (IDR) access unit 🦋.
Random access support 🍓
HEVC signals IRAP pictures beyond IDR pictures in the NAL unit header, through NAL unit types. Three types of IRAP pictures are supported: IDR, CRA, and BLA. IDR pictures are conventionally called closed group-of-pictures (closed-GOP) random access points, while CRA and BLA pictures are conventionally called open-GOP random access points. BLA pictures usually come from splicing two bitstreams, or part of one, at a CRA picture, for example during stream switching. To make IRAP pictures easier to use in systems, six different NAL units are defined to signal the properties of IRAP pictures. These can be used to better match the stream access point types defined in ISOBMFF [IS014496-12] [IS015444-12], which are used for random access support in both 3GP-DASH [3GPDASH] and MPEG DASH [MPEGDASH].
Pictures that follow an IRAP picture in decoding order but come before it in output order are called leading pictures associated with that IRAP picture. There are two types: Random Access Decodable Leading (RADL) pictures and Random Access Skipped Leading (RASL) pictures. RADL pictures can be decoded when decoding starts at the associated IRAP picture. RASL pictures can't be decoded when decoding starts at the associated IRAP picture, and they're usually discarded. HEVC has mechanisms for specifying the conformance of a bitstream in which the originally present RASL pictures have been discarded. So system components can discard RASL pictures when needed without worrying about making the bitstream non-compliant 🙌.
Temporal scalability support 🧚♀️
HEVC has improved temporal scalability support. It does this by including TemporalId signaling in the NAL unit header, restricting pictures of a particular temporal sub-layer from being used for inter prediction reference by pictures of a lower temporal sub-layer, and providing the sub-bitstream extraction process. It also requires that each sub-bitstream extraction output be a conforming bitstream. Media-Aware Network Elements (MANEs) can use the TemporalId in the NAL unit header for stream adaptation based on temporal scalability 🛰️✨.
Temporal sub-layer switching support 💖
Okay bestie, HEVC puts the signaling for Temporal Sub-layer Access (TSA) and Step-wise Temporal Sub-layer Access (STSA) right in the NAL unit header, through the NAL unit types 🦋✨. A TSA picture and every picture after it in decoding order do not use any picture from before the TSA picture (in decoding order) that has a TemporalId greater than or equal to the TSA picture's TemporalId for inter prediction reference 🙈. That's what makes a TSA picture so iconic: at that picture you can up-switch to the sub-layer it's in or any higher sub-layer, coming from the sub-layer right below it 🔥.
STSA is the cousin 💜. An STSA picture does not use pictures with the same TemporalId as itself for inter prediction reference. Pictures after it in decoding order that share its TemporalId also don't use pictures from before the STSA picture (in decoding order) with that same TemporalId for inter prediction reference. And an STSA picture lets you up-switch at that exact picture, into its own sub-layer, from the immediately lower sub-layer 🌸.
Sub-layer reference or non-reference pictures
Okay, this one's a plot twist 🙈. HEVC does reference and non-reference pictures differently than H.264 does. In H.264, if any other picture might use a picture for inter prediction reference, it's a reference picture; otherwise it's a non-reference picture, and that gets flagged with two bits in the NAL unit header. In HEVC, a picture is only called a reference picture when it's marked as "used for reference" 💅. HEVC also brought in the concept of a sub-layer reference picture. If a picture may be used by another picture with the same TemporalId for inter prediction reference, it's a sub-layer reference picture; otherwise it's a sub-layer non-reference picture. Whether it's one or the other gets signaled through NAL unit type values 👑.
Extensibility
HEVC also signals a six-bit layer ID in the NAL unit header, besides the TemporalId, and that layer ID must be equal to 0 for a single-layer bitstream 🧚. Extension mechanisms are built into the VPS, SPS, Picture Parameter Set (PPS), SEI NAL unit, slice headers, and so on 💖. All of these are designed to allow future extensions in a backward-compatible way, so bitstreams encoded under potential future HEVC extensions can be fed to then-legacy decoders (for example, HEVC version 1 decoders). Those legacy decoders can still decode and output the base-layer bitstream 🫶.
Bitstream extraction
HEVC has a bitstream-extraction process as an integral part of the overall decoding process 🌈. It's used in bitstream conformance tests, which are part of the HRD buffering model ✨.
Reference picture management
HEVC's reference picture management is a whole glow-up compared to H.264 💅. That covers reference picture marking, removal from the Decoded Picture Buffer (DPB), and Reference Picture List Construction (RPLC). Instead of H.264's sliding window plus adaptive Memory Management Control Operation (MMCO) marking, HEVC uses a Reference Picture Set (RPS), and RPLC is built on top of the RPS. An RPS is the set of reference pictures associated with a picture. It includes all reference pictures that come before the associated picture in decoding order and that may be used for inter prediction of the associated picture or of any picture after it in decoding order 🦋.
The reference picture set has five lists: RefPicSetStCurrBefore, RefPicSetStCurrAfter, RefPicSetStFoll, RefPicSetLtCurr, and RefPicSetLtFoll. RefPicSetStCurrBefore, RefPicSetStCurrAfter, and RefPicSetLtCurr hold all the reference pictures that may be used in inter prediction of the current picture and that may also be used in inter prediction of one or more pictures after the current picture in decoding order. RefPicSetStFoll and RefPicSetLtFoll hold all the reference pictures that are not used in inter prediction of the current picture but may be used in inter prediction of one or more pictures after it in decoding order 🌸. RPS gives you "intra-coded" signaling of the DPB status instead of "inter-coded" signaling, mainly for better error resilience 🛡️. The RPLC process is based on the RPS and signals an index to an RPS subset for each reference index, which makes it simpler than the RPLC process in H.264 🎀.
Ultra-low delay support
HEVC specifies a sub-picture-level HRD operation to support so-called ultra-low delay ⚡. This mechanism is a standard-compliant way to reduce delay below a one-picture interval 💨. CPB and DPB parameters at the sub-picture level may be signaled, and the spec defines how that information is used to derive CPB timing (where the CPB removal time corresponds to decoding time) and DPB output timing (display time) 🍓. Decoders are allowed to operate the HRD at the conventional access-unit level, even when sub-picture-level HRD parameters are present 💅.
New SEI messages
HEVC inherits a lot of H.264 SEI messages, with changes in syntax and/or semantics that make them apply to HEVC 🌈. There are also a few brand-new SEI messages, which I'll go through quickly 💜.
The display orientation SEI message tells the decoder about a transformation that is recommended to be applied to the cropped decoded picture before display, so the pictures can be properly displayed, for example in an upside-up manner 🙃.
The structure of pictures SEI message gives info on the NAL unit types, picture-order count values, and prediction dependencies of a sequence of pictures 🔍. It can be used, for example, to figure out what impact a lost picture has on other pictures 💀.
The decoded picture hash SEI message provides a checksum derived from the sample values of a decoded picture 🧁. It can be used to detect whether a picture was correctly received and decoded ✅.
The active parameter sets SEI message includes the IDs of the active video parameter set and the active sequence parameter set, and it can be used to activate VPSs and SPSs 👑. It also includes these indications: 1) an indication of whether "full random accessibility" is supported (when it is, all parameter sets needed for decoding the remaining bitstream when random accessing from the beginning of the current CVS, by completely discarding all access units earlier in decoding order, are present in the remaining bitstream, and all coded pictures in the remaining bitstream can be correctly decoded) 🦋; and 2) an indication of whether there is no parameter set within the current CVS that updates another parameter set of the same type preceding it in decoding order 🫶. An update of a parameter set means using the same parameter set ID but with some other parameters changed. If this property is true for all CVSs in the bitstream, then all parameter sets can be sent out-of-band before session start 🚀.
The decoding unit information SEI message provides information about coded picture buffer removal delay for a decoding unit, and it can be used in very-low-delay buffering operations 🏃♀️.
The region refresh information SEI message can be used together with the recovery point SEI message (present in both H.264 and HEVC) for better support of gradual decoding refresh 🌷. This supports random access from inter-coded pictures, where complete pictures can be correctly decoded or recovered after an indicated number of pictures in output/display order 🎉.
1.1.3. Parallel Processing Support ✨
Okay bestie, so HEVC's encoding workload is reportedly way heavier than H.264's 💀, and meanwhile the market keeps demanding higher video resolution, both spatially and temporally 📈. So HEVC got VCL coding tools built specifically so the work can be split up at the sub-picture level 🧩. The parallelization happens at the finest granularity of an integer number of CTUs 💅. The targets are multicore CPUs, DSPs, and multiprocessor systems 🖥️. For these tools to actually be useful in a system design, they need signaling support, which is provided in Section 7 of this memo 🎀. This section gives a quick tour of the tools available in [HEVC] 🦋.
Many HEVC tools were designed with parallel implementations on multicore and multiprocessor architectures in mind 👑. For parallelization, there are four picture partition strategies, and we're going to walk through each one 🌸.
Slices are segments of the bitstream that can be reconstructed independently of other slices within the same picture (though there may still be interdependencies through loop filtering operations) 🧵. Slices are the only parallelization tool that also shows up, in virtually identical form, in H.264 🙈. Slice-based parallelization doesn't need much inter-processor or inter-core communication, except for data sharing for motion compensation when decoding a predictively coded picture, which is typically way heavier than the sharing caused by in-picture prediction 🤏. Slices are designed to be independently decodable, but for that exact same reason they can carry some coding overhead 😭. Slices also (unlike some of the other tools below) serve as the key mechanism for bitstream partitioning to match Maximum Transfer Unit (MTU) size requirements. That's because slices are independent within the picture and each regular slice is encapsulated in its own NAL unit 📦. Sometimes the goal of parallelization and the goal of MTU size matching fight each other over the slice layout in a picture 🥊. That clash is what led to the development of the more advanced tools below 💖.
Dependent slice segments let a coded slice be fragmented into pieces at CTU boundaries without breaking any in-picture prediction mechanisms 🧩. They're complementary to the fragmentation mechanism described in this memo, but they need the cooperation of the encoder to work 🤝. A dependent slice segment always contains an integer number of CTUs, so a decoder using multiple cores operating on CTUs can process a dependent slice segment without passing parts of that slice segment's bitstream to other cores 🧚. Fragmentation as specified in this memo is different: it does not guarantee that a fragment contains an integer number of CTUs 🙅♀️.
In Wavefront Parallel Processing (WPP), the picture is partitioned into rows of CTUs 🌈. Entropy decoding and prediction are allowed to use data from CTUs in other partitions 👀. Parallel processing happens by decoding CTU rows in parallel, where the start of decoding each row is delayed by two CTUs. This makes sure the data for the CTU above and to the right of the subject CTU is available before the subject CTU gets decoded ⏳. That staggered start looks like a wavefront when you draw it, and it means parallelization can use up to as many processors or cores as the picture has CTU rows 🌊.
Because in-picture prediction between neighboring CTU rows is allowed, the inter-processor/inter-core communication needed to make that work can get pretty substantial 📡. WPP partitioning doesn't create more NAL units than when it isn't applied. So WPP can't be used for MTU size matching, though slices can be combined with it for that purpose 🚫.
Tiles define horizontal and vertical boundaries that partition a picture into tile columns and tile rows 🧱. The CTU scan order is changed to be local within a tile (following a CTU raster scan of that tile) before the top-left CTU of the next tile is decoded, following the tile raster scan order of the picture 🔄. Like slices, tiles break in-picture prediction dependencies, including entropy decoding dependencies 💅. But tiles don't have to be put into individual NAL units (same as WPP in this regard), so tiles can't be used for MTU size matching either, though slices can be combined with them for that 🙈. Each tile can be processed by one processor or core. The inter-processor/inter-core communication needed for in-picture prediction between processing units decoding neighboring tiles is limited to conveying the shared slice header when a slice spans more than one tile, plus loop-filtering-related sharing of reconstructed samples and metadata. Because of the in-picture independence between two neighboring partitions, tiles need less inter-processor communication bandwidth than WPP does, which is lowkey iconic 👑🫶.
1.1.4. NAL Unit Header 🎀
HEVC keeps the NAL unit concept from H.264, with some modifications 🦋. HEVC uses a two-byte NAL unit header, shown in Figure 1 👀. The payload of a NAL unit is the NAL unit excluding the NAL unit header 🌸.
+---------------+---------------+
|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| Type | LayerId | TID |
+-------------+-----------------+Figure 1: The Structure of the HEVC NAL Unit Header
The semantics of the NAL unit header fields are as specified in [HEVC], and I'll give you the quick version below for convenience 💖. For each field I'm also listing the matching syntax element name from [HEVC], so you can cross-check 🔍.
F: 1 bit 🚩 The forbidden_zero_bit. It's required to be zero in [HEVC] 🙅♀️. This bit was included in the NAL unit header so HEVC video could be transported over MPEG-2 transport systems, avoiding start code emulations [MPEG2S] 🧠. In the context of this memo, the value 1 may be used to indicate a syntax violation. For example, that happens for a NAL unit that resulted from aggregating a number of fragmented units of a NAL unit but is missing the last fragment, as described in Section 4.4.3 💀.
Type: 6 bits 🧩 The nal_unit_type. This field specifies the NAL unit type as defined in Table 7-1 of [HEVC] 📋. If the most significant bit of this field is equal to 0 (in other words, the value of this field is less than 32), the NAL unit is a VCL NAL unit. Otherwise, the NAL unit is a non-VCL NAL unit 🔥. For a reference of all currently defined NAL unit types and their semantics, check Section 7.4.2 in [HEVC] 📚.
LayerId: 6 bits 🌈 The nuh_layer_id. It's required to be equal to zero in [HEVC] 🙃. It's anticipated that future scalable or 3D video coding extensions of this specification will use this syntax element to identify additional layers that may be present in the CVS. A layer could be, for example, a spatial scalable layer, a quality scalable layer, a texture view, or a depth view ✨.
TID: 3 bits ⭐ The nuh_temporal_id_plus1. This field specifies the temporal identifier of the NAL unit plus 1. The value of TemporalId is equal to TID minus 1 🧚. A TID value of 0 is illegal. That guarantees there's at least one bit in the NAL unit header equal to 1, which lets start code emulations in the NAL unit header and in the NAL unit payload data be considered independently 🫶.
1.2. Overview of the Payload Format 🌸
This payload format defines the following processes, which are required for transporting HEVC coded data over RTP [RFC3550] 🚀:
- Usage of the RTP header with this payload format 🎀
- Packetization of HEVC coded NAL units into RTP packets using three types of payload structures: a single NAL unit packet, an aggregation packet, and a fragment unit 📦
- Transmission of HEVC NAL units of the same bitstream within a single RTP stream or multiple RTP streams (within one or more RTP sessions). Within an RTP stream, transmission of NAL units may be either non-interleaved (the transmission order of NAL units is the same as their decoding order) or interleaved (the transmission order of NAL units is different from the decoding order) 🔀
- Media type parameters to be used with the Session Description Protocol (SDP) [RFC4566] 📝
- A payload header extension mechanism and data structures for enhanced support of temporal scalability based on that extension mechanism 🦋
2. Conventions 💅
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] 📜.
In this document, the above key words convey that interpretation only when they're in ALL CAPS. Lowercase uses of these words are not to be interpreted as carrying the significance described in RFC 2119 🙈.
This specification uses the notion of setting and clearing a bit when bit fields are handled. Setting a bit is the same as assigning that bit the value of 1 (On). Clearing a bit is the same as assigning that bit the value of 0 (Off) 🔛.
3. Definitions and Abbreviations 🧠
3.1. Definitions ✨
This document uses the terms and definitions of [HEVC]. Section 3.1.1 lists relevant definitions from [HEVC] for convenience. Section 3.1.2 provides definitions specific to this memo 💖.
3.1.1. Definitions from the HEVC Specification 💖
okay so this bit is the HEVC spec's own vocab list, and we're borrowing it bc it's the source of truth 🫶
access unit: a set of NAL units that are linked by a specified classification rule, that sit consecutively in decoding order, and that contain exactly one coded picture 📦
BLA access unit: an access unit whose coded picture is a BLA picture 🌸
BLA picture: an IRAP picture where every VCL NAL unit has nal_unit_type equal to BLA_W_LP, BLA_W_RADL, or BLA_N_LP 💅
Coded Video Sequence (CVS): a sequence of access units that goes, in decoding order, like this: an IRAP access unit with NoRaslOutputFlag equal to 1, then zero or more access units that aren't IRAP access units with NoRaslOutputFlag equal to 1, including every one of them up to (but not including) the next IRAP access unit with NoRaslOutputFlag equal to 1 ✨
Informative note: An IRAP access unit can be an IDR access unit, a BLA access unit, or a CRA access unit.NoRaslOutputFlagis equal to 1 for each IDR access unit, each BLA access unit, and each CRA access unit that is the first access unit in the bitstream in decoding order, is the first access unit that follows an end of sequence NAL unit in decoding order, or hasHandleCraAsBlaFlagequal to 1 🧠
CRA access unit: an access unit whose coded picture is a CRA picture 🦋
CRA picture: a RAP picture where every VCL NAL unit has nal_unit_type equal to CRA_NUT 🎀
IDR access unit: an access unit whose coded picture is an IDR picture 🍓
IDR picture: a RAP picture where every VCL NAL unit has nal_unit_type equal to IDR_W_RADL or IDR_N_LP 🌈
IRAP access unit: an access unit whose coded picture is an IRAP picture 👑
IRAP picture: a coded picture where every VCL NAL unit has nal_unit_type in the range of BLA_W_LP (16) to RSV_IRAP_VCL23 (23), inclusive 🔥
layer: a set of VCL NAL units that all have a particular value of nuh_layer_id, plus the associated non-VCL NAL units, or one of a set of syntactical structures that have a hierarchical relationship 🧚
operation point: a bitstream made from another bitstream by running the sub-bitstream extraction process, which takes that other bitstream, a target highest TemporalId, and a target-layer identifier list as its inputs 🎯
random access: starting the decoding process for a bitstream at some point other than the beginning of the bitstream 🚪
sub-layer: a temporal scalable layer of a temporal scalable bitstream, made up of VCL NAL units with a particular value of the TemporalId variable, plus the associated non-VCL NAL units 🌸
sub-layer representation: a subset of the bitstream made up of NAL units from a particular sub-layer and the lower sub-layers 💜
tile: a rectangular region of coding tree blocks within a particular tile column and a particular tile row in a picture 🧩
tile column: a rectangular region of coding tree blocks that has a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set 🏛️
tile row: a rectangular region of coding tree blocks that has a height specified by syntax elements in the picture parameter set and a width equal to the width of the picture 🛤️
3.1.2. Definitions Specific to This Memo 🎀
dependee RTP stream: an RTP stream that another RTP stream depends on. All RTP streams in a Multiple RTP streams on a Single media Transport (MRST) or Multiple RTP streams on Multiple media Transports (MRMT), except for the highest RTP stream, are dependee RTP streams 🫶
highest RTP stream: the RTP stream that no other RTP stream depends on. The RTP stream in a Single RTP stream on a Single media Transport (SRST) is the highest RTP stream 👑
Media-Aware Network Element (MANE): a network element, such as a middlebox, selective forwarding unit, or application-layer gateway, that can parse certain aspects of the RTP payload headers or the RTP payload and react to their contents 🕵️♀️
Informative note: A MANE is more than a normal router or gateway. It has to be aware of the signaling (e.g., to learn about the payload type mappings of the media streams), and it has to be trusted when working with Secure RTP (SRTP). The advantage of MANEs is that they let packets be dropped according to the needs of the media coding. For example, if a MANE has to drop packets due to congestion on a certain link, it can identify and remove the packets whose elimination produces the least adverse effect on the user experience. After dropping packets, MANEs must rewrite RTCP packets to match the changes to the RTP stream, as specified in Section 7 of [RFC3550] 😭✨
Media Transport: as used in the MRST, MRMT, and SRST definitions below, this is the transport of packets over a transport association identified by a 5-tuple (source address, source port, destination address, destination port, transport protocol). See also Section 2.1.13 of [RFC7656] 🚚
Informative note: The term "bitstream" in this document is equivalent to the term "encoded stream" in [RFC7656] 🧠
Multiple RTP streams on a Single media Transport (MRST): multiple RTP streams carrying a single HEVC bitstream on a single transport. See also Section 3.5 of [RFC7656] 🌊
Multiple RTP streams on Multiple media Transports (MRMT): multiple RTP streams carrying a single HEVC bitstream on multiple transports. See also Section 3.5 of [RFC7656] 🌈
NAL unit decoding order: a NAL unit order that conforms to the constraints on NAL unit order given in Section 7.4.2.4 in [HEVC] 📋
NAL unit output order: a NAL unit order in which NAL units of different access units are in the output order of the decoded pictures that correspond to the access units, as specified in [HEVC], and in which NAL units within an access unit are in their decoding order 🎬
NAL-unit-like structure: a data structure that's similar to NAL units in that it also has a NAL unit header and a payload. The difference is that the payload does not follow the start code emulation prevention mechanism required for the NAL unit syntax as specified in Section 7.3.1.1 of [HEVC]. Examples of NAL-unit-like structures defined in this memo are the packet payloads of Aggregation Packet (AP), PAyload Content Information (PACI), and Fragmentation Unit (FU) packets 🧸
NALU-time: the value the RTP timestamp would have if the NAL unit were transported in its own RTP packet ⏰
RTP stream: see [RFC7656]. Within the scope of this memo, one RTP stream is used to transport one or more temporal sub-layers 🎶
Single RTP stream on a Single media Transport (SRST): a single RTP stream carrying a single HEVC bitstream on a single (media) transport. See also Section 3.5 of [RFC7656] 💫
transmission order: the order of packets in ascending RTP sequence number order (in modulo arithmetic). Within an aggregation packet, the NAL unit transmission order is the same as the order in which the NAL units appear in the packet 📬
3.2. Abbreviations 🔤
AP Aggregation Packet
BLA Broken Link Access
CRA Clean Random Access
CTB Coding Tree Block
CTU Coding Tree Unit
CVS Coded Video Sequence
DPH Decoded Picture Hash
FU Fragmentation Unit
HRD Hypothetical Reference Decoder
IDR Instantaneous Decoding Refresh
IRAP Intra Random Access Point
MANE Media-Aware Network Element
MRMT Multiple RTP streams on Multiple media Transports
MRST Multiple RTP streams on a Single media Transport
MTU Maximum Transfer Unit
NAL Network Abstraction Layer
NALU Network Abstraction Layer Unit
PACI PAyload Content Information
PHES Payload Header Extension Structure
PPS Picture Parameter Set
RADL Random Access Decodable Leading (Picture)
RASL Random Access Skipped Leading (Picture)
RPS Reference Picture Set
SEI Supplemental Enhancement Information
SPS Sequence Parameter Set
SRST Single RTP stream on a Single media Transport
STSA Step-wise Temporal Sub-layer Access
TSA Temporal Sub-layer Access
TSCI Temporal Scalability Control Information
VCL Video Coding Layer
VPS Video Parameter Set4. RTP Payload Format ✨
4.1. RTP Header Usage ✨
okay so the RTP header format is defined in [RFC3550], and I'm reprinting it as Figure 2 so you don't have to go hunting 🧚 this payload format uses the header fields in a way that matches that spec, no notes 🫶
The RTP payload (plus the settings for some RTP header bits) for aggregation packets and fragmentation units lives in Sections 4.4.2 and 4.4.3, respectively 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2: RTP Header According to [RFC3550] 🎀
so here's how the RTP header bits get set for this format 🦋
Marker bit (M): 1 bit 💜
Set for the last packet of the access unit, carried in the current RTP stream. This is totally in line with how the M bit is normally used in video formats, so playout buffers can be handled efficiently ✨ When MRST or MRMT is in use, if an access unit shows up in multiple RTP streams, the marker bit gets set on each RTP stream's last packet of the access unit 🫶
Informative note: The content of a NAL unit does not tell whether or not the NAL unit is the last NAL unit, in decoding order, of an access unit. An RTP sender implementation may obtain this information from the video encoder. If, however, the implementation cannot obtain this information directly from the encoder, e.g., when the bitstream was pre-encoded, and also there is no timestamp allocated for each NAL unit, then the sender implementation can inspect subsequent NAL units in decoding order to determine whether or not the NAL unit is the last NAL unit of an access unit as follows. A NAL unit is determined to be the last NAL unit of an access unit if it is the last NAL unit of the bitstream. A NAL unit naluX is also determined to be the last NAL unit of an access unit if both the following conditions are true: 1) the next VCL NAL unit naluY in decoding order has the high-order bit of the first byte after its NAL unit header equal to 1, and 2) all NAL units between naluX and naluY, when present, have nal_unit_type in the range of 32 to 35, inclusive, equal to 39, or in the ranges of 41 to 44, inclusive, or 48 to 55, inclusive. 🙈
Payload Type (PT): 7 bits 🌸
The assignment of an RTP payload type for this new packet format is outside the scope of this document, so it isn't specified here. A payload type has to be assigned either through the profile being used or dynamically 💅
Informative note: It is not required to use different payload type values for different RTP streams in MRST or MRMT. 🍓
Sequence Number (SN): 16 bits 🦋
Set and used in accordance with [RFC3550] ✨
Timestamp: 32 bits ⭐
The RTP timestamp is set to the sampling timestamp of the content. A 90 kHz clock rate MUST be used 🌈
If the NAL unit has no timing properties of its own (e.g., parameter set and SEI NAL units), the RTP timestamp MUST be set to the RTP timestamp of the coded picture of the access unit in which the NAL unit (according to Section 7.4.2.4.4 of [HEVC]) is included. 🧚
Receivers MUST use the RTP timestamp for the display process, even when the bitstream contains picture timing SEI messages or decoding unit information SEI messages as specified in [HEVC]. However, this does not mean that picture timing SEI messages in the bitstream should be discarded, because picture timing SEI messages may contain frame-field information that is important in appropriately rendering interlaced video. 👑
Synchronization source (SSRC): 32 bits 💖
Used to identify the source of the RTP packets. When using SRST, by definition a single SSRC is used for all parts of a single bitstream. In MRST or MRMT, different SSRCs are used for each RTP stream containing a subset of the sub-layers of the single (temporally scalable) bitstream. A receiver is required to correctly associate the set of SSRCs that are included parts of the same bitstream. 🔥
4.2. Payload Header Usage 🎀
The first two bytes of the payload of an RTP packet are called the payload header. It has the same fields (F, Type, LayerId, and TID) as the NAL unit header shown in Section 1.1.4, no matter what kind of payload structure it is 🫶
The TID value tells you (among other things) how important an RTP packet is. For example, NAL units in higher temporal sub-layers aren't used for decoding lower temporal sub-layers, so they matter less. A lower TID value means higher importance 👑 More-important NAL units MAY be better protected against transmission losses than less-important NAL units 🛡️
4.3. Transmission Modes 🦋
This memo lets you send an HEVC bitstream over:
- a Single RTP stream on a Single media Transport (SRST),
- Multiple RTP streams over a Single media Transport (MRST), or
- Multiple RTP streams on Multiple media Transports (MRMT).
Informative note: While this specification enables the use of MRST within the H.265 RTP payload, the signaling of MRST within SDP offer/answer is not fully specified at the time of this writing. See [RFC5576] and [RFC5583] for what is supported today as well as [RTP-MULTI-STREAM] and [SDP-NEG] for future directions. 🔮
When in MRMT, the dependency of one RTP stream on another RTP stream is typically indicated as specified in [RFC5583]. [RFC5583] can also be used to specify dependencies within MRST, but only if the RTP streams use distinct payload types 💜
SRST or MRST SHOULD be used for point-to-point unicast scenarios. MRMT SHOULD be used for point-to-multipoint multicast scenarios where different receivers need different operation points of the same HEVC bitstream, to improve bandwidth utilization efficiency 🌸
Informative note: A multicast may degrade to a unicast after all but one receivers have left (this is a justification of the first "SHOULD" instead of "MUST"), and there might be scenarios where MRMT is desirable but not possible, e.g., when IP multicast is not deployed in certain network (this is a justification of the second "SHOULD" instead of "MUST"). 🙈
The transmission mode is indicated by the tx-mode media parameter (see Section 7.1). If tx-mode is equal to "SRST", SRST MUST be used. Otherwise, if tx-mode is equal to "MRST", MRST MUST be used. Otherwise (tx-mode is equal to "MRMT"), MRMT MUST be used. 🍓
Informative note: When an RTP stream does not depend on other RTP streams, any of SRST, MRST, or MRMT may be in use for the RTP stream. ✨
Receivers MUST support all of SRST, MRST, and MRMT. 🌈
Informative note: The required support of MRMT by receivers does not imply that multicast must be supported by receivers. 🧚
4.4. Payload Structures 💅
There are four different types of RTP packet payload structures. A receiver can identify the type of an RTP packet payload through the Type field in the payload header 🫶
The four payload structures are:
- Single NAL unit packet: Contains a single NAL unit in the payload, and the NAL unit header of the NAL unit also serves as the payload header. This payload structure is specified in Section 4.4.1.
- Aggregation Packet (AP): Contains more than one NAL unit within one access unit. This payload structure is specified in Section 4.4.2.
- Fragmentation Unit (FU): Contains a subset of a single NAL unit. This payload structure is specified in Section 4.4.3.
- PACI carrying RTP packet: Contains a payload header (that differs from other payload headers for efficiency), a Payload Header Extension Structure (PHES), and a PACI payload. This payload structure is specified in Section 4.4.4. 💖
4.4.1. Single NAL Unit Packets 🦋
A single NAL unit packet contains exactly one NAL unit. It's made up of a payload header (called PayloadHdr), a conditional 16-bit DONL field (in network byte order), and the NAL unit payload data (the NAL unit minus its NAL unit header) of the contained NAL unit, as shown in Figure 3 ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr | DONL (conditional) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NAL unit payload data |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3: The Structure of a Single NAL Unit Packet 🌈
The payload header SHOULD be an exact copy of the NAL unit header of the contained NAL unit. However, the Type (i.e., nal_unit_type) field MAY be changed, e.g., when it is desirable to handle a CRA picture to be a BLA picture [JCTVC-J0107] 🧚
The DONL field, when present, specifies the value of the 16 least significant bits of the decoding order number of the contained NAL unit. If sprop-max-don-diff is greater than 0 for any of the RTP streams, the DONL field MUST be present, and the variable DON for the contained NAL unit is derived as equal to the value of the DONL field. Otherwise (sprop-max-don-diff is equal to 0 for all the RTP streams), the DONL field MUST NOT be present. 👑
4.4.2. Aggregation Packets (APs) ✨
okay bestie, APs exist so we stop wasting bytes on tiny NAL units, like most of the non-VCL ones that are often only a few octets 🥺🎀 nobody wants a whole packet just to carry a crumb of data
An AP aggregates NAL units within one access unit 🧚 each NAL unit going into an AP gets wrapped in its own aggregation unit, and the NAL units inside one AP stay in NAL unit decoding order. it's giving organized 💅
An AP is a payload header (called PayloadHdr) followed by two or more aggregation units, as you can see in Figure 4 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=48) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| two or more aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4: The Structure of an Aggregation Packet 🌸
The payload header fields get set like this, and this part is super important 🔥 The F bit MUST be equal to 0 if the F bit of each aggregated NAL unit is equal to zero; otherwise, it MUST be equal to 1. The Type field MUST be equal to 48. The value of LayerId MUST be equal to the lowest value of LayerId of all the aggregated NAL units. The value of TID MUST be the lowest value of TID of all the aggregated NAL units. no skipping these, periodt 💅
Informative note (the fine print, but make it cute): all VCL NAL units in an AP have the same TID value since they belong to the same access unit. however, an AP may contain non-VCL NAL units for which the TID value in the NAL unit header may be different than the TID value of the VCL NAL units in the same AP. so yeah, don't assume they all match 🙈
An AP MUST carry at least two aggregation units, and it can carry as many as it needs 💖 BUT the total amount of data in an AP obviously MUST fit into an IP packet, and the size SHOULD be chosen so that the resulting IP packet is smaller than the MTU size, so you avoid IP layer fragmentation. the rules keep going: an AP MUST NOT contain FUs specified in Section 4.4.3. APs MUST NOT be nested, i.e., an AP must not contain another AP. no AP inception allowed 🌀
The first aggregation unit in an AP starts with a conditional 16-bit DONL field (in network byte order), then a 16-bit unsigned size field (also in network byte order) that tells you the size of the NAL unit in bytes. that size excludes these two octets but includes the NAL unit header. after that comes the NAL unit itself, header and all 🧁 Figure 5 shows the layout ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: DONL (conditional) | NALU size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU size | |
+-+-+-+-+-+-+-+-+ NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5: The Structure of the First Aggregation Unit in an AP 🍓
The DONL field, when it's present, gives you the value of the 16 least significant bits of the decoding order number of the aggregated NAL unit 🌈
If sprop-max-don-diff is greater than 0 for any of the RTP streams, the DONL field MUST be present in an aggregation unit that is the first aggregation unit in an AP, and the variable DON for the aggregated NAL unit is derived as equal to the value of the DONL field. otherwise (sprop-max-don-diff is equal to 0 for all the RTP streams), the DONL field MUST NOT be present in an aggregation unit that is the first aggregation unit in an AP. so it's either present or it isn't, depending on that check 👑
An aggregation unit that is not the first one in an AP starts with a conditional 8-bit DOND field, then a 16-bit unsigned size field (network byte order) that tells you the size of the NAL unit in bytes. same deal as before: that size excludes these two octets but includes the NAL unit header. then the NAL unit itself, header included, as Figure 6 shows 🦋✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: DOND (cond) | NALU size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NAL unit |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: The Structure of an Aggregation Unit That Is Not the First Aggregation Unit in an AP 💜
When it's present, the DOND field plus 1 tells you the difference between the decoding order number values of the current aggregated NAL unit and the preceding aggregated NAL unit in the same AP 🫶
If sprop-max-don-diff is greater than 0 for any of the RTP streams, the DOND field MUST be present in an aggregation unit that is not the first aggregation unit in an AP, and the variable DON for the aggregated NAL unit is derived as equal to the DON of the preceding aggregated NAL unit in the same AP plus the value of the DOND field plus 1 modulo 65536. otherwise (sprop-max-don-diff is equal to 0 for all the RTP streams), the DOND field MUST NOT be present in an aggregation unit that is not the first aggregation unit in an AP, and in this case the transmission order and decoding order of NAL units carried in the AP are the same as the order the NAL units appear in the AP. the vibes are very "what you see is what you get" here 🧚
Figure 7 is an example of an AP with two aggregation units, labeled 1 and 2 in the figure, and no DONL or DOND fields present 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=48) | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 HDR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NALU 1 Data |
| . . . |
| |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| . . . | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 HDR | |
+-+-+-+-+-+-+-+-+ NALU 2 Data |
| . . . |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7: An Example of an AP Packet Containing Two Aggregation Units without the DONL and DOND Fields 🎀
Figure 8 is an example of an AP with two aggregation units, labeled 1 and 2 in the figure, with the DONL and DOND fields present 💅
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=48) | NALU 1 DONL |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NALU 1 Data . . . |
| |
+ . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 DOND | NALU 2 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 HDR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NALU 2 Data |
| |
| . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8: An Example of an AP Containing Two Aggregation Units with the DONL and DOND Fields 🧚
4.4.3. Fragmentation Units 🧩
okay so Fragmentation Units (FUs) exist so you can chop one single NAL unit into multiple RTP packets, and the HEVC encoder doesn't even have to be in on it 💅 A fragment is just an integer number of consecutive octets from that NAL unit. Fragments of the same NAL unit MUST be sent in consecutive order with ascending RTP sequence numbers, and no other RTP packets from the same RTP stream can sneak in between the first and last fragment 🫶
When a NAL unit gets fragmented and carried inside FUs, we call it a fragmented NAL unit 🦋 APs MUST NOT be fragmented. FUs MUST NOT be nested, i.e., an FU must not contain a subset of another FU 🙈
The RTP timestamp of an RTP packet carrying an FU is set to the NALU-time of the fragmented NAL unit ⏰
An FU is made of a payload header (PayloadHdr), a one-octet FU header, a conditional 16-bit DONL field (in network byte order), and an FU payload, as shown in Figure 9 ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=49) | FU header | DONL (cond) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| DONL (cond) | |
|-+-+-+-+-+-+-+-+ |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 9: The Structure of an FU
The fields in the payload header are set like this. The Type field MUST be equal to 49. The fields F, LayerId, and TID MUST be equal to the fields F, LayerId, and TID, respectively, of the fragmented NAL unit 💖
The FU header is made of an S bit, an E bit, and a 6-bit FuType field, as shown in Figure 10 🎀
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E| FuType |
+---------------+Figure 10: The Structure of FU Header
Here's the tea on each FU header field 👑
S: 1 bit When set to 1, the S bit indicates the start of a fragmented NAL unit, i.e., the first byte of the FU payload is also the first byte of the payload of the fragmented NAL unit. When the FU payload is not the start of the fragmented NAL unit payload, the S bit MUST be set to 0 🌸
E: 1 bit When set to 1, the E bit indicates the end of a fragmented NAL unit, i.e., the last byte of the payload is also the last byte of the fragmented NAL unit. When the FU payload is not the last fragment of a fragmented NAL unit, the E bit MUST be set to 0 🌷
FuType: 6 bits The field FuType MUST be equal to the field Type of the fragmented NAL unit 🍓
The DONL field, when present, specifies the value of the 16 least significant bits of the decoding order number of the fragmented NAL unit 🧚
Now the DONL drama 🎭 If sprop-max-don-diff is greater than 0 for any of the RTP streams, and the S bit is equal to 1, the DONL field MUST be present in the FU, and the variable DON for the fragmented NAL unit is derived as equal to the value of the DONL field. Otherwise (sprop-max-don-diff is equal to 0 for all the RTP streams, or the S bit is equal to 0), the DONL field MUST NOT be present in the FU 😭
A non-fragmented NAL unit MUST NOT be transmitted in one FU. Basically, the Start bit and End bit must not both be set to 1 in the same FU header 🙅♀️
The FU payload is made of fragments of the fragmented NAL unit's payload, so that if the FU payloads of consecutive FUs (starting with an FU with the S bit equal to 1 and ending with an FU with the E bit equal to 1) get concatenated in order, the payload of the fragmented NAL unit can be rebuilt 🧩 The NAL unit header of the fragmented NAL unit is not included as such in the FU payload. Instead, its information rides along in the F, LayerId, and TID fields of the FU payload headers and in the FuType field of the FU header 🎀 An FU payload MUST NOT be empty 🚫
If an FU gets lost, the receiver SHOULD discard all following fragmentation units in transmission order that belong to the same fragmented NAL unit, unless the decoder in the receiver is known to be prepared to gracefully handle incomplete NAL units 💀
A receiver in an endpoint or in a MANE MAY aggregate the first n-1 fragments of a NAL unit into an (incomplete) NAL unit, even if fragment n of that NAL unit is not received. In this case, the forbidden_zero_bit of the NAL unit MUST be set to 1 to indicate a syntax violation 🚨
4.4.4. PACI Packets 🦋
This section specifies the PACI packet structure. The basic payload header specified in this memo is intentionally limited to the 16 bits of the NAL unit header, so that packetization overhead stays at a minimum 🍃 However, cases have been identified where it's advisable to put control information in an easily accessible spot in the packet header, even with the extra overhead. One such piece of control information is the TSCI, as specified in Section 4.5. PACI packets carry this and future, similar structures 💜
The PACI packet structure is based on a payload header extension mechanism that is generic and extensible, so it can carry payload header extensions. In this section, the focus is on use within this specification. Section 4.4.4.2 gives guidance for specification designers on how to employ the extension mechanism in future specifications 🧠
A PACI packet is made of a payload header (denoted as PayloadHdr), whose structure follows what is described in Section 4.2. After the payload header come the fields A, cType, PHSsize, F[0..2], and Y 🎀
Figure 11 shows a PACI packet that complies with this memo, i.e., one without any extensions ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=50) |A| cType | PHSsize |F0..2|Y|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Payload Header Extension Structure (PHES) |
|=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=|
| |
| PACI payload: NAL unit |
| . . . |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 11: The Structure of a PACI
The fields in the payload header are set like this. The F bit MUST be equal to 0. The Type field MUST be equal to 50. The value of LayerId MUST be a copy of the LayerId field of the PACI payload NAL unit or NAL-unit-like structure. The value of TID MUST be a copy of the TID field of the PACI payload NAL unit or NAL-unit-like structure 💖
Here's what the other fields mean 🌈
A: 1 bit Copy of the F bit of the PACI payload NAL unit or NAL-unit-like structure.
cType: 6 bits Copy of the Type field of the PACI payload NAL unit or NAL-unit-like structure.
PHSsize: 5 bits Indicates the length of the PHES field. The value is limited to be less than or equal to 32 octets, to simplify encoder design for MTU size matching 📏
F0: This field, when equal to 1, specifies the presence of a temporal scalability support extension in the PHES 🦋
F1, F2: MUST be 0, available for future extensions, see Section 4.4.4.2. Receivers compliant with this version of the HEVC payload format MUST ignore F1=1 and/or F2=1, and also ignore any information in the PHES indicated as present by F1=1 and/or F2=1 🙈
Informative note: The receiver can do that by first decoding information associated with F0=1, and then skipping over any remaining bytes of the PHES based on the value of PHSsize.
Y: 1 bit MUST be 0, available for future extensions, see Section 4.4.4.2. Receivers compliant with this version of the HEVC payload format MUST ignore Y=1, and also ignore any information in the PHES indicated as present by Y 🫶
PHES: variable number of octets A variable number of octets, as indicated by the value of PHSsize 🧁
PACI Payload: The single NAL unit packet or NAL-unit-like structure (such as an FU or an AP) to be carried, not including the first two octets 🎁
Informative note: The first two octets of the NAL unit or NAL-unit-like structure carried in the PACI payload are not included in the PACI payload. Rather, the respective values are copied into the locations of the PayloadHdr of the RTP packet. This design offers two advantages. First, the overall structure of the payload header is preserved, i.e., there is no special case of payload header structure that needs to be implemented for PACI. Second, no additional overhead is introduced.
A PACI payload MAY be a single NAL unit, an FU, or an AP. PACIs MUST NOT be fragmented or aggregated. The following subsection documents the reasons for these design choices 🧠
4.4.4.1. Reasons for the PACI Rules (Informative) ✨
A PACI can't be fragmented, and here's why 💅 If a PACI got fragmented and a fragment that wasn't the first one went missing, the info inside the PACI would be totally out of reach 😭 So a PACI must not be fragmented. Put another way, an FU must not carry (fragments of) a PACI.
A PACI can't be aggregated either. Stacking PACIs together is kinda inadvisable from a compression point of view, because a bunch of NAL units being aggregated would often share the exact same PACI fields and values, and those would get repeated for no reason 🙄 Most, if not all, of what PACI aggregation would get you can be done by aggregating the NAL units and bundling them with a PACI (more on that below) 🫶 So a PACI must not be aggregated. In other words, an AP must not contain a PACI.
The payload of a PACI can be a fragment! 🧩 Middleboxes and senders with inflexible (often hardware-based) encoders sometimes end up in a spot where a PACI plus its headers is bigger than the MTU size 😬 In that case, the middlebox or sender can fragment the NAL unit and wrap the fragment in a PACI. This keeps the payload header extension info for every fragment, so downstream middleboxes and the receiver can use it too ✨ Therefore, a sender may place a fragment into a PACI, and a receiver must be able to handle such a PACI.
The payload of a PACI can also be an aggregation NAL unit 🎀 HEVC bitstreams can have unevenly sized and/or small (compared to the MTU size) NAL units. To pack those tiny ones efficiently, APs were introduced. The benefits of APs don't depend on needing a payload header extension, so they're their own thing 💖 Therefore, a sender may place an AP into a PACI, and a receiver must be able to handle such a PACI.
4.4.4.2. PACI Extensions (Informative) 🧚
Okay, this part is for future spec designers who want to extend the PACI syntax 🌸 Designers are totally free to specify whatever makes sense for them at the time. BUT a lot of thought went into the extension mechanism below, so if you're going to deviate from it, you'd better have a really good explanation 💅
This memo only defines a single payload header extension (TSCI, described in Section 4.5), so only the F0 bit carries semantics. F1 and F2 are already named (not just marked as reserved, like a typical video spec designer would do 🙄) and they're meant to signal two additional extensions. The Y bit lets you recursively add further F and Y bits to extend the mechanism beyond three possible payload header extensions. If you want to give F1, F2, or Y different semantics than the ones suggested here, it's suggested that you define a new packet type (using a different value for Type) instead 🦋
When a Y bit is set, an 8-bit flag-extension gets inserted after the Y bit. A flag-extension consists of 7 flags F[n..n+6], and another Y bit.
The basic PACI header already includes F0, F1, and F2. So the Fx bits in the first flag-extension are numbered F3, F4, ..., F9, and the F bits in the second flag-extension are numbered F10, F11, ..., F16, and so on 🔢 At least three Fx bits are always in the PACI, but you can increase the number of Fx bits (and their associated types of extensions) by setting the next Y bit and adding an octet of flag-extensions, carrying seven flags and another Y bit. The size of this list of flags is subject to the limits in Section 4.4.4 (32 octets for all flag-extensions and the PHES information combined) 🫣
Each of the F bits can indicate either the presence or the absence of certain information in the Payload Header Extension Structure (PHES).
When a spec developer creates a new syntax that takes advantage of the PACI extension mechanism, they have to follow the constraints below, or the extension mechanism may break 💀
- The fields added for a particular Fx bit MUST be fixed in length and not depend on what other Fx bits are set (no parsing dependency).
- The Fx bits must be assigned in order.
- An implementation that supports the n-th Fn bit for any value of n must understand the syntax (though not necessarily the semantics) of the fields Fk (with k < n), so as to be able to either use those bits when present, or at least be able to skip over them.
4.5. Temporal Scalability Control Information
This section describes the single payload header extension defined in this specification, known as TSCI 🌈 If extra payload header extensions become necessary in the future, they could be specified in this section of an updated version of this document, or in their own documents.
When F0 is set to 1 in a PACI, it means the PHES field includes the TSCI fields TL0PICIDX, IrapPicID, S, and E, like so:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=50) |A| cType | PHSsize |F0..2|Y|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TL0PICIDX | IrapPicID |S|E| RES | |
|-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| .... |
| PACI payload: NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 12: The Structure of a PACI with a PHES Containing a TSCITL0PICIDX (8 bits) When present, the TL0PICIDX field MUST be set to equal to temporal_sub_layer_zero_idx as specified in Section D.3.22 of [HEVC] for the access unit containing the NAL unit in the PACI 🎯
IrapPicID (8 bits) When present, the IrapPicID field MUST be set to equal to irap_pic_id as specified in Section D.3.22 of [HEVC] for the access unit containing the NAL unit in the PACI 🎯
S (1 bit) The S bit MUST be set to 1 if any of the following conditions is true, and MUST be set to 0 otherwise:
- The NAL unit in the payload of the PACI is the first VCL NAL unit, in decoding order, of a picture.
- The NAL unit in the payload of the PACI is an AP, and the NAL unit in the first contained aggregation unit is the first VCL NAL unit, in decoding order, of a picture.
- The NAL unit in the payload of the PACI is an FU with its S bit equal to 1 and the FU payload containing a fragment of the first VCL NAL unit, in decoding order, of a picture.
E (1 bit) The E bit MUST be set to 1 if any of the following conditions is true, and MUST be set to 0 otherwise:
- The NAL unit in the payload of the PACI is the last VCL NAL unit, in decoding order, of a picture.
- The NAL unit in the payload of the PACI is an AP and the NAL unit in the last contained aggregation unit is the last VCL NAL unit, in decoding order, of a picture.
- The NAL unit in the payload of the PACI is an FU with its E bit equal to 1 and the FU payload containing a fragment of the last VCL NAL unit, in decoding order, of a picture.
RES (6 bits) MUST be equal to 0. Reserved for future extensions 🔒
The value of PHSsize MUST be set to 3. Receivers MUST allow other values of the fields F0, F1, F2, Y, and PHSsize, and MUST ignore any additional fields, when present, than specified above in the PHES 💜
4.6. Decoding Order Number
For each NAL unit, the variable AbsDon is derived, representing the decoding order number that indicates the NAL unit decoding order 🦋
Let NAL unit n be the n-th NAL unit in transmission order within an RTP stream.
If sprop-max-don-diff is equal to 0 for all the RTP streams carrying the HEVC bitstream, AbsDon[n], the value of AbsDon for NAL unit n, is derived as equal to n 🍓
Otherwise (sprop-max-don-diff is greater than 0 for any of the RTP streams), AbsDon[n] is derived as follows, where DON[n] is the value of the variable DON for NAL unit n 🧮
- If n is equal to 0 (i.e., NAL unit n is the very first NAL unit in transmission order),
AbsDon[0]is set equal toDON[0]. - Otherwise (n is greater than 0), the following applies for derivation of
AbsDon[n]:
If DON[n] == DON[n-1],
AbsDon[n] = AbsDon[n-1]
If (DON[n] > DON[n-1] and DON[n] - DON[n-1] < 32768),
AbsDon[n] = AbsDon[n-1] + DON[n] - DON[n-1]
If (DON[n] < DON[n-1] and DON[n-1] - DON[n] >= 32768),
AbsDon[n] = AbsDon[n-1] + 65536 - DON[n-1] + DON[n]
If (DON[n] > DON[n-1] and DON[n] - DON[n-1] >= 32768),
AbsDon[n] = AbsDon[n-1] - (DON[n-1] + 65536 -
DON[n])
If (DON[n] < DON[n-1] and DON[n-1] - DON[n] < 32768),
AbsDon[n] = AbsDon[n-1] - (DON[n-1] - DON[n])For any two NAL units m and n, the following applies:
AbsDon[n]greater thanAbsDon[m]indicates that NAL unit n follows NAL unit m in NAL unit decoding order.- When
AbsDon[n]is equal toAbsDon[m], the NAL unit decoding order of the two NAL units can be in either order. AbsDon[n]less thanAbsDon[m]indicates that NAL unit n precedes NAL unit m in decoding order.
Informative note: When two consecutive NAL units in the NAL unit decoding order have different values of AbsDon, the absolute difference between the two AbsDon values may be greater than or equal to 1. 🌟
Informative note: There are multiple reasons to allow for the absolute difference of the values of AbsDon for two consecutive NAL units in the NAL unit decoding order to be greater than one. An increment by one is not required, as at the time of associating values of AbsDon to NAL units, it may not be known whether all NAL units are to be delivered to the receiver. For example, a gateway may not forward VCL NAL units of higher sub-layers or some SEI NAL units when there is congestion in the network. In another example, the first intra-coded picture of a pre-encoded clip is transmitted in advance to ensure that it is readily available in the receiver, and when transmitting the first intra-coded picture, the originator does not exactly know how many NAL units will be encoded before the first intra-coded picture of the pre-encoded clip follows in decoding order. Thus, the values of AbsDon for the NAL units of the first intra-coded picture of the pre-encoded clip have to be estimated when they are transmitted, and gaps in values of AbsDon may occur. Another example is MRST or MRMT with sprop-max-don-diff greater than 0, where the AbsDon values must indicate cross-layer decoding order for NAL units conveyed in all the RTP streams. 🧠
5. Packetization Rules 🎀
Okay bestie, the packetization rules are next, and they're giving "follow these or the packets get messy" 💅
- If
sprop-max-don-diffis greater than 0 for any of the RTP streams, the transmission order of NAL units carried in the RTP stream MAY be different than the NAL unit decoding order and the NAL unit output order. Otherwise (sprop-max-don-diffis equal to 0 for all the RTP streams), the transmission order of NAL units carried in the RTP stream MUST be the same as the NAL unit decoding order and, whentx-modeis equal to"MRST"or"MRMT", MUST also be the same as the NAL unit output order. 🔥 - A NAL unit of a small size SHOULD be encapsulated in an aggregation packet together with one or more other NAL units, so you don't waste packetization overhead on tiny NAL units 🧚 For example, non-VCL NAL units such as access unit delimiters, parameter sets, or SEI NAL units are typically small and can often be aggregated with VCL NAL units without violating MTU size constraints. Small things, big vibes ✨
- Each non-VCL NAL unit SHOULD, when possible from an MTU size match viewpoint, be encapsulated in an aggregation packet together with its associated VCL NAL unit. Why? A non-VCL NAL unit is typically meaningless without the associated VCL NAL unit being available, so they really belong together 🫶
- For carrying exactly one NAL unit in an RTP packet, a single NAL unit packet MUST be used. Simple, clean, no drama 🌸
6. De-packetization Process 🦋
Okay, de-packetization. The big idea is to pull the NAL units out of the RTP packets in an RTP stream (and every RTP stream that stream depends on, if there are any), then hand them to the decoder in NAL unit decoding order. Iconic 👑
The de-packetization process is implementation dependent. So the description below is just an example of a suitable implementation. Other schemes are fine too, as long as the output for the same input is the same as the process described below. The output counts as the same when the set of output NAL units and their order are both identical. Optimizations relative to the described algorithms are possible 💜
All normal RTP mechanisms related to buffer management apply. In particular, duplicated or outdated RTP packets (as indicated by the RTP sequence number and the RTP timestamp) are removed. 🗑️ To figure out the exact time for decoding, factors such as a possible intentional delay to allow for proper inter-stream synchronization must be factored in ⏰
NAL units with NAL unit type values in the range of 0 to 47, inclusive, may be passed to the decoder. NAL-unit-like structures with NAL unit type values in the range of 48 to 63, inclusive, MUST NOT be passed to the decoder. Nope, not those 🙅♀️
The receiver includes a receiver buffer. It's used to compensate for transmission delay jitter within individual RTP streams and across RTP streams, to reorder NAL units from transmission order to the NAL unit decoding order, and to recover the NAL unit decoding order in MRST or MRMT, when applicable. 🎀 In this section, the receiver operation is described under the assumption that there is no transmission delay jitter within an RTP stream and across RTP streams. To tell it apart from a practical receiver buffer (which also handles transmission delay jitter), this section calls the receiver buffer the de-packetization buffer from here on. Receivers should also prepare for transmission delay jitter. That means either reserving separate buffers for transmission delay jitter buffering and de-packetization buffering, or using a receiver buffer for both transmission delay jitter and de-packetization. Moreover, receivers should take transmission delay jitter into account in the buffering operation, e.g., by additional initial buffering before starting of decoding and playback. Plan ahead, babe 🧠
When sprop-max-don-diff is equal to 0 for all the received RTP streams, the de-packetization buffer size is zero bytes, and the process in the rest of this paragraph applies. When only one RTP stream is received, the NAL units carried in that single RTP stream are passed straight to the decoder in their transmission order, which is identical to their decoding order. When more than one RTP stream is received, the NAL units carried in the multiple RTP streams are passed to the decoder in their NTP timestamp order. When several NAL units from different RTP streams share the same NTP timestamp, they're passed to the decoder in their dependency order, so NAL units of a dependee RTP stream go before the NAL units of the dependent RTP stream. When several NAL units of the same RTP stream share the same NTP timestamp, they're passed to the decoder in their transmission order. It's all about the order, bestie 📋
Informative note: The mapping between RTP and NTP timestamps is conveyed in RTCP SR packets. In addition, the mechanisms for faster media timestamp synchronization discussed in [RFC6051] may be used to speed up the acquisition of the RTP-to-wall-clock mapping. ⏱️
When sprop-max-don-diff is greater than 0 for any of the received RTP streams, the process described in the rest of this section applies. This is where it gets spicy 🌶️
There are two buffering states in the receiver: initial buffering and buffering while playing. Initial buffering starts when the reception is initialized. Once initial buffering is done, decoding and playback start, and the receiver switches to buffering-while-playing mode 🎬
No matter which buffering state it's in, the receiver stores incoming NAL units in reception order into the de-packetization buffer. Each NAL unit carried in an RTP packet is stored individually, and its AbsDon value is calculated and stored along with it. When MRST or MRMT is in use, the NAL units of all RTP streams of a bitstream go into the same de-packetization buffer. When NAL units carried in any two RTP streams are ready to be placed into the de-packetization buffer, the NAL units from the RTP stream lower in the dependency tree go in first. For example, if RTP stream A depends on RTP stream B, then the NAL units carried in RTP stream B are placed into the buffer first. 🌳
Initial buffering keeps going until either condition A or condition B is true. Condition A is that the difference between the greatest and smallest AbsDon values of the NAL units in the de-packetization buffer is greater than or equal to the value of sprop-max-don-diff of the highest RTP stream. Condition B is that the number of NAL units in the de-packetization buffer is greater than the value of sprop-depack-buf-nalus. 📦
After initial buffering, whenever condition A or condition B is true, the following operation is applied again and again until both condition A and condition B are false:
- The NAL unit in the de-packetization buffer with the smallest value of AbsDon is removed from the de-packetization buffer and passed to the decoder. 🚀
When no more NAL units are flowing into the de-packetization buffer, all NAL units still in the buffer are removed and passed to the decoder in order of increasing AbsDon values. Everybody out, in order 🏁
7. Payload Format Parameters 💖
This section lists the parameters that MAY be used to pick optional features of the payload format, and certain features or properties of the bitstream or the RTP stream. These parameters are defined here as part of the media type registration for the HEVC codec. A mapping of the parameters into the Session Description Protocol (SDP) [RFC4566] is also provided for apps that use SDP. Equivalent parameters could be defined elsewhere for use with control protocols that don't use SDP 🦋
7.1. Media Type Registration 🌈
The media subtype for the HEVC codec is allocated from the IETF tree. Official, very serious, very real 👑
The receiver MUST ignore any unrecognized parameter. If it doesn't recognize it, it just scrolls past 🙈
Type name: video
Subtype name: H265
Required parameters: none
OPTIONAL parameters:The OPTIONAL parameters are profile-space, tier-flag, profile-id, profile-compatibility-indicator, interop-constraints, and level-id. These describe the profile, tier, default level, and some constraints of the bitstream carried by the RTP stream and all RTP streams the RTP stream depends on. They can also describe a specific set of profile, tier, default level, and some constraints that the receiver supports. ✨
The profile and some constraints are signaled together by profile-space, profile-id, profile-compatibility-indicator, and interop-constraints. The profile specifies the subset of coding tools that may have been used to generate the bitstream, or that the receiver supports. 🧰
Informative note: There are 32 values ofprofile-id, and there are 32 flags inprofile-compatibility-indicator, with each flag corresponding to one value ofprofile-id. According to HEVC version 1 in [HEVC], when more than one of the 32 flags is set for a bitstream, the bitstream would comply with all the profiles corresponding to the set flags. However, in a draft of HEVC version 2 in [HEVCv2], Subclause A.3.5, 19 Format Range Extensions profiles have been specified. They all use the same value ofprofile-id(4) and are told apart by some of the 48 bits ininterop-constraints. This (rather unexpected way of signaling profiles) means that one of the 32 flags may correspond to multiple profiles. To be able to support whatever HEVC extension profile might be specified and indicated usingprofile-space,profile-id,profile-compatibility-indicator, andinterop-constraintsin the future, it would be safe to require symmetric use of these parameters in SDP offer/answer, unlessrecv-sub-layer-idis included in the SDP answer for choosing one of the sub-layers offered. 🤓
The tier is signaled by tier-flag. The default level is signaled by level-id. The tier and the default level specify the limits on values of syntax elements, or arithmetic combinations of values of syntax elements, that are followed when generating the bitstream or that the receiver supports. 📏
A set of profile-space, tier-flag, profile-id, profile-compatibility-indicator, interop-constraints, and level-id parameters ptlA is said to be consistent with another set of these parameters ptlB if any decoder that conforms to the profile, tier, level, and constraints indicated by ptlB can decode any bitstream that conforms to the profile, tier, level, and constraints indicated by ptlA. Consistency is everything 🫶
In SDP offer/answer, when the SDP answer does not include a recv-sub-layer-id parameter that is less than the sprop-sub-layer-id parameter in the SDP offer, the following applies:
- The
profile-space,tier-flag,profile-id,profile-compatibility-indicator, andinterop-constraintsparameters MUST be used symmetrically. In other words, the value of each of these parameters in the offer MUST be the same as that in the answer, either explicitly signaled or implicitly inferred. Mirror, mirror 🪞💅
okay so the level-id parameter is totally changeable 🦋 as long as the highest level the answer indicates is either equal to or lower than the offer's. ngl the highest level is worked out from level-id and max-recv-level-id together 💅
In SDP offer/answer, when the SDP answer includes a recv-sub-layer-id that's less than the sprop-sub-layer-id in the offer 🎀, the set of profile-space, tier-flag, profile-id, profile-compatibility-indicator, interop-constraints, and level-id parameters in the answer MUST be consistent with that of the chosen sub-layer representation as indicated in the SDP offer. The one exception is level-id: the answer's level-id can change as long as the highest level the answer indicates is lower than or equal to the offer's. 💖
more specs for these parameters, including how they line up with the profile, tier, and level syntax elements from [HEVC], are coming up below 👀
profile-space, profile-id:
the value of profile-space MUST be in the range of 0 to 3, inclusive, and profile-id MUST be in the range of 0 to 31, inclusive 🌈
when profile-space isn't there, a value of 0 MUST be inferred. when profile-id isn't there, a value of 1 (aka the Main profile) MUST be inferred 🧚
when these are used to describe a bitstream, profile-space and profile-id come from the profile, tier, and level syntax elements in SPS or VPS NAL units like this, where general_profile_space, general_profile_idc, sub_layer_profile_space[j], and sub_layer_profile_idc[j] are defined in [HEVC] 🍓
- if the RTP stream is the highest RTP stream:
profile-space = general_profile_spaceprofile-id = general_profile_idc
- otherwise (the RTP stream is a dependee RTP stream), with j being the value of the sprop-sub-layer-id parameter:
profile-space = sub_layer_profile_space[j]profile-id = sub_layer_profile_idc[j]
tier-flag, level-id:
the value of tier-flag MUST be in the range of 0 to 1, inclusive, and the value of level-id MUST be in the range of 0 to 255, inclusive 🍭
if tier-flag and level-id are used to describe a bitstream, they tell you the tier and the highest level the bitstream complies with 💅
if they're used for capability exchange, here's the tea ☕ if max-recv-level-id isn't present, the default level set by level-id is the highest level the codec wants to support. otherwise, max-recv-level-id is the highest level the codec supports for receiving. either way, for receiving or sending, every level lower than the highest supported level MUST also be supported 🫶
if no tier-flag is present, a value of 0 MUST be inferred. if no level-id is present, a value of 93 (i.e., level 3.1) MUST be inferred 😭
when tier-flag and level-id describe a bitstream, they're derived from the profile, tier, and level syntax elements in SPS or VPS NAL units like this, where general_tier_flag, general_level_idc, sub_layer_tier_flag[j], and sub_layer_level_idc[j] are specified in [HEVC] ✨
- if the RTP stream is the highest RTP stream:
tier-flag = general_tier_flaglevel-id = general_level_idc
- otherwise (the RTP stream is a dependee RTP stream), with j being the value of the sprop-sub-layer-id parameter:
tier-flag = sub_layer_tier_flag[j]level-id = sub_layer_level_idc[j]
interop-constraints:
this is a base16 [RFC4648] (hexadecimal) representation of six bytes of data, made up of progressive_source_flag, interlaced_source_flag, non_packed_constraint_flag, frame_only_constraint_flag, and reserved_zero_44bits 🎀
if interop-constraints isn't present, these MUST be inferred:
progressive_source_flag = 1interlaced_source_flag = 0non_packed_constraint_flag = 1frame_only_constraint_flag = 1reserved_zero_44bits = 0
when interop-constraints is used to describe a bitstream, it works like this, where general_progressive_source_flag, general_interlaced_source_flag, general_non_packed_constraint_flag, general_non_packed_constraint_flag, general_frame_only_constraint_flag, general_reserved_zero_44bits, sub_layer_progressive_source_flag[j], sub_layer_interlaced_source_flag[j], sub_layer_non_packed_constraint_flag[j], sub_layer_frame_only_constraint_flag[j], and sub_layer_reserved_zero_44bits[j] are specified in [HEVC] 🌸
- if the RTP stream is the highest RTP stream:
progressive_source_flag = general_progressive_source_flaginterlaced_source_flag = general_interlaced_source_flagnon_packed_constraint_flag = general_non_packed_constraint_flagframe_only_constraint_flag = general_frame_only_constraint_flagreserved_zero_44bits = general_reserved_zero_44bits
- otherwise (the RTP stream is a dependee RTP stream), with j being the value of the sprop-sub-layer-id parameter:
progressive_source_flag = sub_layer_progressive_source_flag[j]interlaced_source_flag = sub_layer_interlaced_source_flag[j]non_packed_constraint_flag = sub_layer_non_packed_constraint_flag[j]frame_only_constraint_flag = sub_layer_frame_only_constraint_flag[j]reserved_zero_44bits = sub_layer_reserved_zero_44bits[j]
using interop-constraints for capability exchange means any bitstream has to comply with the interop-constraints 🔥
profile-compatibility-indicator:
this is a base16 [RFC4648] representation of four bytes of data 🦋
when profile-compatibility-indicator is used to describe a bitstream, it works like this, where general_profile_compatibility_flag[j] and sub_layer_profile_compatibility_flag[i][j] are specified in [HEVC] 👑
the profile-compatibility-indicator here lists extra profiles on top of the profile set by profile-space, profile-id, and interop-constraints that the bitstream conforms to. a decoder that conforms to any one of all the profiles the bitstream conforms to can decode the bitstream. these extra profiles are defined by profile-space, each set bit of profile-compatibility-indicator, and interop-constraints 🧚
- if the RTP stream is the highest RTP stream, the following applies for each value of j in the range of 0 to 31, inclusive:
- bit j of profile-compatibility-indicator = general_profile_compatibility_flag[j]
- otherwise (the RTP stream is a dependee RTP stream), the following applies for i equal to sprop-sub-layer-id and for each value of j in the range of 0 to 31, inclusive:
- bit j of profile-compatibility-indicator = sub_layer_profile_compatibility_flag[i][j]
using profile-compatibility-indicator for capability exchange means any bitstream has to comply with the profile-compatibility-indicator. this is meant to cover cases where a future HEVC profile gets defined as the intersection of two or more profiles 🍓
if this parameter isn't present, it defaults to this: bit j, with j equal to profile-id, of profile-compatibility-indicator is inferred to be equal to 1, and all other bits are inferred to be equal to 0 🌈
sprop-sub-layer-id:
this parameter MAY be used to indicate the highest allowed value of TID in the bitstream. when it's not present, the value of sprop-sub-layer-id is inferred to be equal to 6 🍭
the value of sprop-sub-layer-id MUST be in the range of 0 to 6, inclusive 💅
recv-sub-layer-id:
this parameter MAY be used to signal a receiver's pick of the offered or declared sub-layer representations in the sprop-vps. the value of recv-sub-layer-id is the TID of the highest sub-layer of the bitstream that a receiver supports. when it's not present, the value of recv-sub-layer-id is inferred to be equal to the sprop-sub-layer-id value in the SDP offer 🫶
the value of recv-sub-layer-id MUST be in the range of 0 to 6, inclusive 🎀
max-recv-level-id:
this parameter MAY be used to indicate the highest level a receiver supports. that highest level is equal to the value of max-recv-level-id divided by 30 ✨
the value of max-recv-level-id MUST be in the range of 0 to 255, inclusive 💖
when max-recv-level-id isn't present, the value is inferred to be equal to level-id 🦋
max-recv-level-id MUST NOT be present when the highest level the receiver supports isn't higher than the default level 🙈
tx-mode:
this parameter tells you whether the transmission mode is SRST, MRST, or MRMT 🌸
the value of tx-mode MUST be equal to "SRST", "MRST" or "MRMT". when it's not present, the value of tx-mode is inferred to be equal to "SRST" 🧚
if the value is equal to "MRST", MRST MUST be in use. otherwise, if the value is equal to "MRMT", MRMT MUST be in use. otherwise (the value is equal to "SRST"), SRST MUST be in use 🔥
the value of tx-mode MUST be equal to "MRST" for all RTP streams in an MRST 👑
the value of tx-mode MUST be equal to "MRMT" for all RTP streams in an MRMT 🍓
sprop-vps:
this parameter MAY be used to carry any video parameter set NAL unit of the bitstream for out-of-band transmission of video parameter sets. it MAY also be used for capability exchange and to indicate sub-stream characteristics (i.e., properties of sub-layer representations as defined in [HEVC]) 🌈
the value of the parameter is a comma-separated (',') list of base64 [RFC4648] representations of the video parameter set NAL units, as specified in Section 7.3.2.1 of [HEVC] 💜
Okay so this is where the sprop-vps rules get juicy 💅
The sprop-vps parameter MAY contain one or more than one video parameter set NAL unit 🎀 BUT every other video parameter set in there MUST be consistent with the first one. 🫶 So what does "consistent" even mean? A video parameter set vpsB is consistent with vpsA if any decoder that handles the profile, tier, level, and constraints from the first profile_tier_level( ) in vpsA can also decode any bitstream that matches the profile, tier, level, and constraints from the first profile_tier_level( ) in vpsB. Those are the 12 bytes from general_profile_space all the way down to general_level_idc, inclusive 🧚 It's basically a compatibility check, bestie. Main character energy for the first VPS ✨
sprop-sps
This parameter MAY be used to carry sequence parameter set NAL units of the bitstream for out-of-band transmission of sequence parameter sets 🌸 The value is a comma-separated (,) list of base64 [RFC4648] representations of those sequence parameter set NAL units, as described in Section 7.3.2.2 of [HEVC]. Lowkey the whole point is getting them to the receiver without stuffing them in the stream 💜
sprop-pps
Same energy here 🦋 This parameter MAY be used to carry picture parameter set NAL units of the bitstream for out-of-band transmission of picture parameter sets. The value is a comma-separated (,) list of base64 [RFC4648] representations of the picture parameter set NAL units, as described in Section 7.3.2.3 of [HEVC].
sprop-sei
This parameter MAY be used to convey one or more SEI messages that describe bitstream characteristics 🔥 When it's present, a decoder can rely on the bitstream characteristics described in those SEI messages for the entire duration of the session. That's true no matter what the persistence scopes of the SEI messages are, as specified in [HEVC]. Iconic 👑
The value is a comma-separated (,) list of base64 [RFC4648] representations of SEI NAL units, as specified in Section 7.3.2.4 of [HEVC].
Informative note: Intentionally, there's no list of applicable or inapplicable SEI messages here 🙈 Sending certain SEI messages in sprop-sei can be sensible in some app scenarios and totally meaningless in others. A few examples though:
- In an environment where the bitstream was made from film-based source material, and no splicing is going to happen during the session, the film grain characteristics SEI message or the tone mapping information SEI message are likely meaningful 🎞️ Sending them in
sprop-seiinstead of in the bitstream at each entry point may help save bits, and lets you configure the renderer only once, avoiding unwanted artifacts. Efficient AND aesthetic 🌈 - The structure of pictures information SEI message in
sprop-seican tell a decoder about the NAL unit types, picture-order count values, and prediction dependencies of a sequence of pictures 🧩 Knowing that is super helpful for error recovery. - Examples of SEI messages that would be meaningless to convey in
sprop-seiinclude: the decoded picture hash SEI message (it's close to impossible that all decoded pictures have the same hashtag 💀), the display orientation SEI message when the device is a handheld device (because the display orientation may change when the handheld is turned around 📱), or the filler payload SEI message (there's no point in just having more bits in SDP 🙃).
max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, max-tc
These parameters MAY be used to signal the capabilities of a receiver implementation 🎀 They MUST NOT be used for any other purpose, so no sneaky side quests 🚫 The highest level (specified by max-recv-level-id) MUST be the highest that the receiver is fully capable of supporting. max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, and max-tc MAY be used to indicate capabilities of the receiver that extend the required capabilities of the highest level, as specified below.
When more than one parameter from that set (max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, max-tc) is present, the receiver MUST support all signaled capabilities simultaneously 🫶 For example, if both max-lsr and max-br are present, the highest level with the extension of both the picture rate and bitrate is supported. So the receiver can decode bitstreams where the luma sample rate is up to max-lsr (inclusive), the bitrate is up to max-br (inclusive), the coded picture buffer size is derived as described in the semantics of the max-br parameter below, and everything else complies with the highest level given by max-recv-level-id. Full combo unlocked 🔓
Informative note: When the OPTIONAL media type parameters are used to signal the properties of a bitstream, and max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, and max-tc are not present, the values of profile-space, tier-flag, profile-id, profile-compatibility-indicator, interop-constraints, and level-id must always be such that the bitstream complies fully with the specified profile, tier, and level. Consistency is everything, periodt 💅
max-lsr
The value of max-lsr is an integer indicating the maximum processing rate in units of luma samples per second 🌸 The max-lsr parameter signals that the receiver can decode video at a higher rate than the highest level requires.
When max-lsr is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with one exception: the MaxLumaSR value in Table A-2 of [HEVC] for the highest level is replaced with the value of max-lsr ✨ Senders MAY use this knowledge to send pictures of a given size at a higher picture rate than the highest level indicates.
When it's not present, the value of max-lsr is inferred to be equal to the MaxLumaSR value given in Table A-2 of [HEVC] for the highest level.
The value of max-lsr MUST be in the range of MaxLumaSR to 16 * MaxLumaSR, inclusive, where MaxLumaSR is given in Table A-2 of [HEVC] for the highest level 🔢
max-lps
The value of max-lps is an integer indicating the maximum picture size in units of luma samples 🖼️ The max-lps parameter signals that the receiver can decode larger picture sizes than the highest level requires. When max-lps is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with one exception: the MaxLumaPS value in Table A-1 of [HEVC] for the highest level is replaced with the value of max-lps. Senders MAY use this knowledge to send larger pictures at a proportionally lower picture rate than the highest level indicates 🦋
When it's not present, the value of max-lps is inferred to be equal to the MaxLumaPS value given in Table A-1 of [HEVC] for the highest level.
The value of max-lps MUST be in the range of MaxLumaPS to 16 * MaxLumaPS, inclusive, where MaxLumaPS is given in Table A-1 of [HEVC] for the highest level 🧮
max-cpb
The value of max-cpb is an integer indicating the maximum coded picture buffer size in units of CpbBrVclFactor bits for the VCL HRD parameters, and in units of CpbBrNalFactor bits for the NAL HRD parameters 🧠 CpbBrVclFactor and CpbBrNalFactor are defined in Section A.4 of [HEVC]. The max-cpb parameter signals that the receiver has more memory than the minimum coded picture buffer memory required by the highest level. When max-cpb is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with one exception: the MaxCPB value in Table A-1 of [HEVC] for the highest level is replaced with the value of max-cpb. Senders MAY use this knowledge to build coded bitstreams with greater variation of bitrate than the MaxCPB value in Table A-1 of [HEVC] allows 🌊
When it's not present, the value of max-cpb is inferred to be equal to the MaxCPB value given in Table A-1 of [HEVC] for the highest level.
The value of max-cpb MUST be in the range of MaxCPB to 16 * MaxCPB, inclusive, where MaxLumaCPB is given in Table A-1 of [HEVC] for the highest level 💜
Informative note: The coded picture buffer is used in the hypothetical reference decoder (Annex C of [HEVC]) 🤓 The hypothetical reference decoder is recommended in HEVC encoders to verify that the produced bitstream conforms to the standard and to control the output bitrate. So the coded picture buffer is conceptually independent of any other potential buffers in the receiver, including de-packetization and de-jitter buffers. The coded picture buffer doesn't need to be implemented in decoders as specified in Annex C of [HEVC]. Standard-compliant decoders can have any buffering arrangements, as long as they can decode standard-compliant bitstreams 🎉 In practice, that means the input buffer for a video decoder can be integrated with the receiver's de-packetization and de-jitter buffers.
max-dpb
The value of max-dpb is an integer indicating the maximum decoded picture buffer size, in units of decoded pictures at the MaxLumaPS for the highest level. In other words, it's the number of decoded pictures at the maximum picture size defined by the highest level 📚 The value of max-dpb MUST be in the range of 1 to 16, respectively. The max-dpb parameter signals that the receiver has more memory than the minimum decoded picture buffer memory required by default, which is MaxDpbPicBuf as defined in [HEVC] (equal to 6) 🧁 When max-dpb is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with one exception: the MaxDpbPicBuff value defined in [HEVC] as 6 is replaced with the value of max-dpb. Consequently, a receiver that signals max-dpb MUST be capable of storing the following number of decoded pictures (MaxDpbSize) in its decoded picture buffer:
okay bestie, the max-dpb math is giving "the buffer scales with how tiny the picture is" 💖 here's the whole formula, exactly as it goes:
if( PicSizeInSamplesY <= ( MaxLumaPS >> 2 ) )
MaxDpbSize = Min( 4 * max-dpb, 16 )
else if ( PicSizeInSamplesY <= ( MaxLumaPS >> 1 ) )
MaxDpbSize = Min( 2 * max-dpb, 16 )
else if ( PicSizeInSamplesY <= ( ( 3 * MaxLumaPS ) >> 2
) )
MaxDpbSize = Min( (4 * max-dpb) / 3, 16 )
else
MaxDpbSize = max-dpbwhere MaxLumaPS comes from Table A-1 of [HEVC] for the highest level, and PicSizeInSamplesY is the current size of each decoded picture in luma samples, as defined in [HEVC] ✨
The value of max-dpb MUST be greater than or equal to the value of MaxDpbPicBuf (i.e., 6) as defined in [HEVC]. Senders MAY use this knowledge to build coded bitstreams with better compression 🫶
When it's not present, max-dpb is inferred to be equal to MaxDpbPicBuf (i.e., 6) as defined in [HEVC] 🌸
Informative note: this parameter was added mainly to match a similar codepoint in the ITU-T Recommendation H.245, so signaling gateways are easier to design 🎀 The decoded picture buffer stores reconstructed samples. It has no relationship to the buffers used in RTP, especially the de-packetization and de-jitter buffers 🙈
max-br: okay this one is a whole mood 🔥 The value is an integer for the maximum video bitrate, in units of CpbBrVclFactor bits per second for the VCL HRD parameters and in units of CpbBrNalFactor bits per second for the NAL HRD parameters. Both factors are defined in Section A.4 of [HEVC] 💅
The max-br parameter says the receiver's video decoder can handle a higher bitrate than the highest level asks for ✨
When max-br is signaled, the video codec of the receiver MUST be able to decode bitstreams that conform to the highest level, with these exceptions inside the highest level's limits:
- The value of
max-brreplaces the MaxBR value in Table A-2 of [HEVC] for the highest level. - When the
max-cpbparameter is not present, this formula replaces the MaxCPB value in Table A-1 of [HEVC]:
(MaxCPB of the highest level) * max-br / (MaxBR of the highest level)For example, if a receiver signals Main profile Level 2 with max-br equal to 2000, that means a maximum video bitrate of 2000 kbits/sec for VCL HRD parameters, a maximum video bitrate of 2200 kbits/sec for NAL HRD parameters, and a CPB size of 2000000 bits (2000000 / 1500000 * 1500000) 🧚
Senders MAY use this knowledge to send higher-bitrate video, as the level definition in Annex A of [HEVC] allows, for better video quality 🌈
When it's not present, max-br is inferred to be equal to the MaxBR value given in Table A-2 of [HEVC] for the highest level 💜
The value of max-br MUST be in the range of MaxBR to 16 * MaxBR, inclusive, where MaxBR is given in Table A-2 of [HEVC] for the highest level.
Informative note: this parameter was also added mainly to match a similar codepoint in H.245, for signaling gateway designs. You can't assume from this value that the network can handle these bitrates at any given time. You also can't conclude from it that the signaled bitrate is possible under congestion control constraints 👀
max-tr: the value is an integer indicating the maximum number of tile rows 🦋 The max-tr parameter says the receiver can decode video with more tile rows than the highest level allows.
When max-tr is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with one exception: the MaxTileRows value in Table A-1 of [HEVC] for the highest level gets replaced by the value of max-tr 🍓
Senders MAY use this knowledge to send pictures with more tile rows than the highest level allows 🌷
When it's not present, max-tr is inferred to be equal to the MaxTileRows value given in Table A-1 of [HEVC] for the highest level 🫧
The value of max-tr MUST be in the range of MaxTileRows to 16 * MaxTileRows, inclusive, where MaxTileRows is given in Table A-1 of [HEVC] for the highest level.
max-tc: same energy as max-tr but for columns 👑 The value is an integer indicating the maximum number of tile columns. The max-tc parameter says the receiver can decode video with more tile columns than the highest level allows ✨
When max-tc is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with one exception: the MaxTileCols value in Table A-1 of [HEVC] for the highest level gets replaced by the value of max-tc 💖
Senders MAY use this knowledge to send pictures with more tile columns than the highest level allows 🎀
When it's not present, max-tc is inferred to be equal to the MaxTileCols value given in Table A-1 of [HEVC] for the highest level 🌸
The value of max-tc MUST be in the range of MaxTileCols to 16 * MaxTileCols, inclusive, where MaxTileCols is given in Table A-1 of [HEVC] for the highest level.
max-fps: this one is a constraint, not a boost 🙈 The value is an integer for the maximum picture rate, in units of pictures per 100 seconds, that the receiver can effectively process. The max-fps parameter MAY be used to signal that the receiver can't process video effectively at the full picture rate implied by the highest level and, when present, by one or more of the parameters max-lsr, max-lps, and max-br 🍃
The value of max-fps is not necessarily the picture rate at which the maximum picture size can be sent. It's a limit on the maximum picture rate for all resolutions 🧃
Informative note:max-fpsis semantically different frommax-lsr,max-lps,max-cpb,max-dpb,max-br,max-tr, andmax-tc, becausemax-fpssignals a constraint that lowers the maximum picture rate below what the other parameters imply 😭
The encoder MUST use a picture rate equal to or less than this value. When max-fps is absent, the encoder is free to pick any picture rate allowed by the highest level and any signaled optional parameters 🦄
The value of max-fps MUST be smaller than or equal to the full picture rate implied by the highest level and, when present, by one or more of the parameters max-lsr, max-lps, and max-br 💅
sprop-max-don-diff: okay this is the decoding-order-number drama 🎭 If tx-mode is equal to "SRST" and there is no NAL unit naluA that is followed in transmission order by any NAL unit preceding naluA in decoding order (so the transmission order is the same as the decoding order), the value of this parameter MUST be equal to 0 🌈
Otherwise, if tx-mode is equal to "MRST" or "MRMT" and the decoding order of the NAL units of all the RTP streams is the same as the NAL unit transmission order and the NAL unit output order, the value of this parameter MUST be equal to either 0 or 1 🍓
Otherwise, if tx-mode is equal to "MRST" or "MRMT" and the decoding order of the NAL units of all the RTP streams is the same as the NAL unit transmission order but not the same as the NAL unit output order, the value of this parameter MUST be equal to 1 🫶
Otherwise, this parameter specifies the maximum absolute difference between the decoding order number (i.e., AbsDon) values of any two NAL units naluA and naluB, where naluA follows naluB in decoding order and precedes naluB in transmission order 🧠
The value of sprop-max-don-diff MUST be an integer in the range of 0 to 32767, inclusive. When it's not present, sprop-max-don-diff is inferred to be equal to 0 🌸
sprop-depack-buf-nalus: this one counts the NAL units that are sitting in line 🚦 It specifies the maximum number of NAL units that precede a NAL unit in transmission order and follow that same NAL unit in decoding order.
The value of sprop-depack-buf-nalus MUST be an integer in the range of 0 to 32767, inclusive. When it's not present, the value is inferred to be equal to 0 💖
When sprop-max-don-diff is present and greater than 0, this parameter MUST be present, and the value MUST be greater than 0 ✨
sprop-depack-buf-bytes: the size of the de-packetization buffer, in bytes 🧺 The value of the parameter MUST be greater than or equal to the maximum buffer occupancy (in bytes) of the de-packetization buffer, as specified in Section 6 🎀
The value of sprop-depack-buf-bytes MUST be an integer in the range of 0 to 4294967295, inclusive.
When sprop-max-don-diff is present and greater than 0, this parameter MUST be present, and the value MUST be greater than 0. When it's not present, sprop-depack-buf-bytes is inferred to be equal to 0 🌷
Informative note: sprop-depack-buf-bytes covers only the required size of the de-packetization buffer. If network jitter can happen, you'll also need a properly sized jitter buffer 🫧
depack-buf-cap: this is the receiver's own capability report 🦋 It gives the amount of de-packetization buffer space, in bytes, that the receiver has available for putting the NAL unit decoding order back together from NAL units carried in one or more RTP streams. A receiver can handle any RTP stream, and all RTP streams it depends on (when present), where the sprop-depack-buf-bytes value is smaller than or equal to this parameter 🍓
When it's not present, depack-buf-cap is inferred to be equal to 4294967295. The value of depack-buf-cap MUST be an integer in the range of 1 to 4294967295, inclusive.
Informative note: depack-buf-cap is the maximum possible size of the receiver's de-packetization buffer only, without allowing for network jitter 💜
sprop-segmentation-id:
Okay bestie, so sprop-segmentation-id 🧩 is a MAY param, meaning you're allowed to send it 💅. It tells the receiver which segmentation tools are in the bitstream and can be used for parallelization ✨. Its value MUST be an integer from 0 to 3, inclusive. If it's not there, it's inferred to be 0.
When it's 0, no info about segmentation tools is given 🌸. When it's 1, slices are in the bitstream 🍓. When it's 2, tiles are in there 🧚. And when it's 3, WPP is being used in the bitstream 👑. (WPP is the one that's been stealing the show, iykyk 💀.)
sprop-spatial-segmentation-idc 🎀 is a base16 [RFC4648] representation of the syntax element min_spatial_segmentation_idc from [HEVC]. This one MAY be used to describe how parallelizable the bitstream is 🦋.
dec-parallel-cap 💜 is the next big one, and it's giving main character energy. It MAY be used to signal the decoder's extra decoding capabilities when parallel decoding tools (slices, tiles, and WPP) are in the bitstream 🌈. Your decoding capability can change depending on which of those tools are set up, like how big the tiles are 🫶. So you can send multiple capability points, each one showing the minimum decoding capability tied to a parallelism requirement (a requirement on the bitstream that lets it be decoded in parallel) 🔥.
Each capability point is a combo of 1) a parallelism requirement, 2) a profile (set by profile-space and profile-id), 3) a highest level, and 4) a max processing rate, a max picture size, and a max video bitrate that may be equal to or greater than what the highest level sets 😭. The ABNF [RFC5234] for it looks like this:
dec-parallel-cap = "dec-parallel-cap={" cap-point *(","
cap-point) "}"
cap-point = ("w" / "t") ":" spatial-seg-idc 1*(";"
cap-parameter)
spatial-seg-idc = 1*4DIGIT ; (1-4095)
cap-parameter = tier-flag / level-id / max-lsr
/ max-lps / max-br
tier-flag = "tier-flag" EQ ("0" / "1")
level-id = "level-id" EQ 1*3DIGIT ; (0-255)
max-lsr = "max-lsr" EQ 1*20DIGIT ; (0-
18,446,744,073,709,551,615)
max-lps = "max-lps" EQ 1*10DIGIT ; (0-4,294,967,295)
max-br = "max-br" EQ 1*20DIGIT ; (0-
18,446,744,073,709,551,615)
EQ = "="The whole set of capability points lives inside curly braces {} ✨. Two capability points in a row are split by a comma ,. Inside each capability point, every pair of parameter and value is split by a semicolon ; 💅.
The profile is the same for all capability points, and it's set by profile-space and profile-id, which live outside dec-parallel-cap 🦋.
Each capability point starts with its parallelism requirement: a parallel tool type (w or t) plus a decimal spatial-seg-idc value 🌸. If the type is w, that point only works for H.265 bitstreams with WPP in use, meaning entropy_coding_sync_enabled_flag equal to 1 🍓. If the type is t, it only works for H.265 bitstreams with WPP not in use (so entropy_coding_sync_enabled_flag equal to 0) 🧚. Either way, the point only works for H.265 bitstreams whose min_spatial_segmentation_idc is equal to or greater than spatial-seg-idc 👑.
After that, each capability point continues with one or more parameter-and-value pairs, in any order, for any of these:
tier-flaglevel-idmax-lsrmax-lpsmax-br
Each of those five can show up at most once per capability point 💖.
For a capability point, tier-flag and level-id tell you the highest level of that point 🌈. max-lsr, max-lps, and max-br give the max processing rate (in luma samples per second), the max picture size (in luma samples), and the max video bitrate. The bitrate is in units of CpbBrVclFactor bits per second for the VCL HRD parameters, and in units of CpbBrNalFactor bits per second for the NAL HRD parameters. Both of those factors are defined in Section A.4 of [HEVC] 🤓.
If any of these is missing, it gets a default 🫶. A missing dec-parallel-cap.tier-flag takes the value of tier-flag outside dec-parallel-cap. A missing dec-parallel-cap.level-id takes the value of max-recv-level-id outside the parameter. And a missing dec-parallel-cap.max-lsr, dec-parallel-cap.max-lps, or dec-parallel-cap.max-br takes the value of max-lsr, max-lps, or max-br outside the parameter, respectively 💅.
The general decoding capability is what you get from the parameters outside dec-parallel-cap ✨. It's the capability point made from: 1) the parallelism requirement for sprop-segmentation-id equal to 0 for a bitstream, 2) the profile set by profile-space, profile-id, profile-compatibility-indicator, and interop-constraints, 3) the tier and highest level set by tier-flag and max-recv-level-id, and 4) the max processing rate, max picture size, and max video bitrate set by that highest level 🔥. The general decoding capability MUST NOT be included as one of the capability points in dec-parallel-cap 🙈. Don't double-dip on it, bestie.
Here's an example: this expresses the general decoding capability of 720p30 (Level 3.1), plus extra decoding capability for 1080p30 (Level 4), as long as the spatially largest tile or slice in the bitstream is equal to or less than 1/3 of the picture size 🎀:
a=fmtp:98 level-id=93;dec-parallel-cap={t:8;level- id=120}And here's another one. This one expresses extra decoding capability for 1080p30, using dec-parallel-cap.max-lsr and dec-parallel-cap.max-lps, as long as WPP is used in the bitstream 🦋:
a=fmtp:98 level-id=93;dec-parallel-cap={w:8;
max-lsr=62668800;max-lps=2088960}Informative note: When min_spatial_segmentation_idc is present in a bitstream and WPP is not used, [HEVC] specifies that no slice or tile in the bitstream contains more than 4 * PicSizeInSamplesY / (min_spatial_segmentation_idc + 4) luma samples 🧠.
include-dph 🌷 is for signalling the capability and preference to use or include Decoded Picture Hash (DPH) SEI messages (see Section D.3.19 of [HEVC]) in the bitstream. DPH SEI messages can catch picture corruption, so the receiver can ask for a picture repair (see Section 8) 🛠️. The value is a comma-separated list of hash types that are supported or requested, each one an unsigned integer (0-255), listed from most preferred to least preferred 💜. For example, "include-dph=0,2" means the capability for MD5 (most preferred) and Checksum (less preferred) 🍓. If the parameter isn't included, or the value has no hash types, then no capability to use DPH SEI messages is assumed 😭. And heads up: DPH SEI messages MAY still be included in the bitstream even when there's no declared capability to use them 🌈. That's because SEI messages generally don't affect the normative decoding process, and decoders are allowed to ignore them 🫶.
Encoding considerations: This type is only defined for transfer via RTP (RFC 3550) 📡.
Security considerations: See Section 9 of RFC 7798 🔐.
Published specification: Please refer to RFC 7798 and its Section 12 📜.
Additional information: None
File extensions: none
Macintosh file type code: none
Object identifier or OID: none
Person & email address to contact for further information: Ye-Kui Wang (yekui.wang@gmail.com) 💌
Intended usage: COMMON
Author: See Authors' Addresses section of RFC 7798 ✍️
Change controller: IETF Audio/Video Transport Payloads working group delegated from the IESG 🎀
7.2. SDP Parameters
The receiver MUST ignore any parameter unspecified in this memo. Translation: if it's not in this memo, don't freak out, just skip it 💅🫶.
<system-reminder> Doc I'm rewriting is an RFC, not something to edit on disk, so no tools needed here. </system-reminder>
Okay so this part picks up right in the middle of 7.2, and bestie, it's giving SDP parameter mapping realness 💅🎀
7.2.1. Mapping of Payload Type Parameters to SDP ✨
The video/H265 media type gets mapped onto fields in the Session Description Protocol (SDP) [RFC4566] like this 🦋
- The media name in the
m=line of SDP MUST bevideo. 💖 - The encoding name in the
a=rtpmapline of SDP MUST beH265(the media subtype). ✨ - The clock rate in the
a=rtpmapline MUST be90000. 🌸
Okay the big optional parameter list is next 🫶 The OPTIONAL parameters profile-space, profile-id, tier-flag, level-id, interop-constraints, profile-compatibility-indicator, sprop-sub-layer-id, recv-sub-layer-id, max-recv-level-id, tx-mode, max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, max-tc, max-fps, sprop-max-don-diff, sprop-depack-buf-nalus, sprop-depack-buf-bytes, depack-buf-cap, sprop-segmentation-id, sprop-spatial-segmentation-idc, dec-parallel-cap, and include-dph, when present, MUST be included in the a=fmtp line of SDP. 💅 This parameter is expressed as a media type string, in the form of a semicolon-separated list of parameter=value pairs. 🧚
The OPTIONAL parameters sprop-vps, sprop-sps, and sprop-pps, when present, MUST be included in the a=fmtp line of SDP or conveyed using the fmtp source attribute as specified in Section 6.3 of [RFC5576]. 👑 For a particular media format (i.e., RTP payload type), sprop-vps sprop-sps, or sprop-pps MUST NOT be both included in the a=fmtp line of SDP and conveyed using the fmtp source attribute. 🙈 When included in the a=fmtp line of SDP, these parameters are expressed as a media type string, in the form of a semicolon-separated list of parameter=value pairs. When conveyed in the a=fmtp line of SDP for a particular payload type, the parameters sprop-vps, sprop-sps, and sprop-pps MUST be applied to each SSRC with the payload type. 🦋 When conveyed using the fmtp source attribute, these parameters are only associated with the given source and payload type as parts of the fmtp source attribute. 🌈
Informative note: Conveyingsprop-vps,sprop-sps, andsprop-ppsusing thefmtpsource attribute allows for out-of-band transport of parameter sets in topologies like Topo-Video-switch-MCU as specified in [RFC7667]. 🫶
An example of media representation in SDP is as follows 🍓
m=video 49170 RTP/AVP 98
a=rtpmap:98 H265/90000
a=fmtp:98 profile-id=1;
sprop-vps=<video parameter sets data>7.2.2. Usage with SDP Offer/Answer Model ✨
When HEVC is offered over RTP using SDP in an offer/answer model [RFC3264] for negotiation for unicast usage, the following limitations and rules apply 💖
- The parameters identifying a media format configuration for HEVC are
profile-space,profile-id,tier-flag,level-id,interop-constraints,profile-compatibility-indicator, andtx-mode. These media configuration parameters, exceptlevel-id, MUST be used symmetrically when the answerer does not includerecv-sub-layer-idin the answer for the media format (payload type) or the includedrecv-sub-layer-idis equal tosprop-sub-layer-idin the offer. The answerer MUST: 🔥- maintain all configuration parameters with the values remaining the same as in the offer for the media format (payload type), with the exception that the value of
level-idis changeable as long as the highest level indicated by the answer is not higher than that indicated by the offer; - include in the answer the
recv-sub-layer-idparameter, with a value less than thesprop-sub-layer-idparameter in the offer, for the media format (payload type), and maintain all configuration parameters with the values being the same as signaled in thesprop-vpsfor the chosen sub-layer representation, with the exception that the value oflevel-idis changeable as long as the highest level indicated by the answer is not higher than the level indicated by thesprop-vpsin offer for the chosen sub-layer representation; or - remove the media format (payload type) completely (when one or more of the parameter values are not supported).
- maintain all configuration parameters with the values remaining the same as in the offer for the media format (payload type), with the exception that the value of
Informative note: The above requirement for symmetric use does not apply for level-id, and does not apply for the other bitstream or RTP stream properties and capability parameters. 🧚
- The
profile-compatibility-indicator, when offered assendonly, describes bitstream properties. The answerer MAY accept an RTP payload type even if the decoder is not capable of handling the profile indicated by theprofile-space,profile-id, andinterop-constraintsparameters, but capable of any of the profiles indicated by theprofile-space,profile-compatibility-indicator, andinterop-constraints. 🦋 However, when theprofile-compatibility-indicatoris used in arecvonlyorsendrecvmedia description, the bitstream using this RTP payload type is required to conform to all profiles indicated byprofile-space,profile-compatibility-indicator, andinterop-constraints. 🙈 - To simplify handling and matching of these configurations, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [RFC3264]. 🌸
- The same RTP payload type number used in the offer for the media subtype H265 MUST be used in the answer when the answer includes
recv-sub-layer-id. When the answer does not includerecv-sub-layer-id, the answer MUST NOT contain a payload type number used in the offer for the media subtype H265 unless the configuration is exactly the same as in the offer or the configuration in the answer only differs from that in the offer with a different value oflevel-id. The answer MAY contain therecv-sub-layer-idparameter if an HEVC bitstream contains multiple operation points (using temporal scalability and sub-layers) andsprop-vpsis included in the offer where information of sub-layers are present in the first video parameter set contained insprop-vps. If thesprop-vpsis provided in an offer, an answerer MAY select a particular operation point indicated in the first video parameter set contained insprop-vps. When the answer includes arecv-sub-layer-idthat is less than asprop-sub-layer-idin the offer, all video parameter sets contained in thesprop-vpsparameter in the SDP answer and all video parameter sets sent in-band for either the offerer-to-answerer direction or the answerer-to-offerer direction MUST be consistent with the first video parameter set in thesprop-vpsparameter of the offer (see the semantics ofsprop-vpsin Section 7.1 of this document on one video parameter set being consistent with another video parameter set), and the bitstream sent in either direction MUST conform to the profile, tier, level, and constraints of the chosen sub-layer representation as indicated by the firstprofile_tier_level( )syntax structure in the first video parameter set in thesprop-vpsparameter of the offer. 💅
Informative note: When an offerer receives an answer that does not includerecv-sub-layer-id, it has to compare payload types not declared in the offer based on the media type (i.e.,video/H265) and the above media configuration parameters with any payload types it has already declared. This will enable it to determine whether the configuration in question is new or if it is equivalent to configuration already offered, since a different payload type number may be used in the answer. The ability to perform operation point selection enables a receiver to utilize the temporal scalable nature of an HEVC bitstream. 🧚
- The parameters
sprop-max-don-diff,sprop-depack-buf-nalus, andsprop-depack-buf-bytesdescribe the properties of an RTP stream, and all RTP streams the RTP stream depends on, when present, that the offerer or the answerer is sending for the media format configuration. This differs from the normal usage of the offer/answer parameters: normally such parameters declare the properties of the bitstream or RTP stream that the offerer or the answerer is able to receive. When dealing with HEVC, the offerer assumes that the answerer will be able to receive media encoded using the configuration being offered. 🌈
Informative note: The above parameters apply for any RTP stream and all RTP streams the RTP stream depends on, when present, sent by a declaring entity with the same configuration. In other words, the applicability of the above parameters to RTP streams depends on the source endpoint. Rather than being bound to the payload type, the values may have to be applied to another payload type when being sent, as they apply for the configuration. 🦋
- The capability parameters
max-lsr,max-lps,max-cpb,max-dpb,max-br,max-tr, andmax-tcMAY be used to declare further capabilities of the offerer or answerer for receiving. These parameters MUST NOT be present when the direction attribute issendonly. 💖 - The capability parameter
max-fpsMAY be used to declare lower capabilities of the offerer or answerer for receiving. The parameters MUST NOT be present when the direction attribute issendonly. ✨ - The capability parameter
dec-parallel-capMAY be used to declare additional decoding capabilities of the offerer or answerer for receiving. Upon receiving such a declaration of a receiver, a sender MAY send a bitstream to the receiver utilizing those capabilities under the assumption that the bitstream fulfills the parallelism requirement. A bitstream that is sent based on choosing a capability point with parallel tool type 'w' fromdec-parallel-capMUST haveentropy_coding_sync_enabled_flagequal to 1 andmin_spatial_segmentation_idcequal to or larger thandec-parallel-cap.spatial-seg-idcof the capability point. A bitstream that is sent based on choosing a capability point with parallel tool type 't' fromdec-parallel-capMUST haveentropy_coding_sync_enabled_flagequal to 0 andmin_spatial_segmentation_idcequal to or larger thandec-parallel-cap.spatial-seg-idcof the capability point. 🌸 - An offerer has to include the size of the de-packetization buffer,
sprop-depack-buf-bytes, as well assprop-max-don-diffandsprop-depack-buf-nalus, in the offer for an interleaved HEVC bitstream or for the MRST or MRMT transmission mode whensprop-max-don-diffis greater than 0 for at least one of the RTP streams. To enable the offerer and answerer to inform each other about their capabilities for de-packetization buffering in receiving RTP streams, both parties are RECOMMENDED to includedepack-buf-cap. For interleaved RTP streams or in MRST or MRMT, it is also RECOMMENDED to consider offering multiple payload types with different buffering requirements when the capabilities of the receiver are unknown. 🫶 - The capability parameter
include-dphMAY be used to declare the capability to utilize decoded picture hash SEI messages and which types of hashes in any HEVC RTP streams received by the offerer or answerer. 👑
okay so the answerer gets a say in how its parameter sets travel 🎀 out-of-band or in-band, its call! 🦋
sprop-vps,sprop-sps, orsprop-pps, when present (included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute as specified in Section 6.3 of [RFC5576]), are used for out-of-band transport of the parameter sets (VPS, SPS, or PPS, respectively). 📦✨- The answerer MAY use either out-of-band or in-band transport of parameter sets for the bitstream it is sending, regardless of whether out-of-band parameter sets transport has been used in the offerer-to-answerer direction. 💅 Parameter sets in an answer are totally independent of the ones in the offer, because they decode two different bitstreams, one from the answerer to the offerer and one going the other way 🔄 In case some RTP streams are sent before the SDP offer/answer settles down, in-band parameter sets MUST be used for those RTP stream parts sent before the SDP offer/answer. 🚨
- The following rules apply to transport of parameter sets in the offerer-to-answerer direction. 📨
- An offer MAY include
sprop-vps,sprop-sps, and/orsprop-pps. If none of these parameters is present in the offer, then only in-band transport of parameter sets is used. 🌸 - If the level to use in the offerer-to-answerer direction is equal to the default level in the offer, the answerer MUST be prepared to use the parameter sets included in
sprop-vps,sprop-sps, andsprop-pps(either included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute) for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. Otherwise, the answerer MUST ignoresprop-vps,sprop-sps, andsprop-pps(either included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute) and the offerer MUST transmit parameter sets in-band. 🙈 - In MRST or MRMT, the answerer MUST be prepared to use the parameter sets out-of-band transmitted for the RTP stream and all RTP streams the RTP stream depends on, when present, for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. 🧚
- An offer MAY include
- The following rules apply to transport of parameter sets in the answerer-to-offerer direction. 📬
- An answer MAY include
sprop-vps,sprop-sps, and/orsprop-pps. If none of these parameters is present in the answer, then only in-band transport of parameter sets is used. 🍓 - The offerer MUST be prepared to use the parameter sets included in
sprop-vps,sprop-sps, andsprop-pps(either included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute) for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. 👑 - In MRST or MRMT, the offerer MUST be prepared to use the parameter sets out-of-band transmitted for the RTP stream and all RTP streams the RTP stream depends on, when present, for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. 💜
- An answer MAY include
- When
sprop-vps,sprop-sps, and/orsprop-ppsare conveyed using thefmtpsource attribute as specified in Section 6.3 of [RFC5576], the receiver of the parameters MUST store the parameter sets included insprop-vps,sprop-sps, and/orsprop-ppsand associate them with the source given as part of thefmtpsource attribute. Parameter sets associated with one source (given as part of thefmtpsource attribute) MUST only be used to decode NAL units conveyed in RTP packets from the same source (given as part of thefmtpsource attribute). When this mechanism is in use, SSRC collision detection and resolution MUST be performed as specified in [RFC5576]. 🔐
For bitstreams being delivered over multicast, the following rules apply: 📡
- The media format configuration is identified by
profile-space,profile-id,tier-flag,level-id,interop-constraints,profile-compatibility-indicator, andtx-mode. These media format configuration parameters, includinglevel-id, MUST be used symmetrically; that is, the answerer MUST either maintain all configuration parameters or remove the media format (payload type) completely. Note that this implies that thelevel-idfor offer/answer in multicast is not changeable. 🌈 - To simplify the handling and matching of these configurations, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [RFC3264]. An answer MUST NOT contain a payload type number used in the offer unless the configuration is the same as in the offer. 🎯
- Parameter sets received MUST be associated with the originating source and MUST only be used in decoding the incoming bitstream from the same source. 🦋
- The rules for other parameters are the same as above for unicast as long as the three above rules are obeyed. ✨
Table 1 lists the interpretation of all the parameters that MUST be used for the various combinations of offer, answer, and direction attributes. Note that the two columns wherein the recv-sub-layer-id parameter is used only apply to answers, whereas the other columns apply to both offers and answers. 📊
Table 1. Interpretation of parameters for various combinations of
offers, answers, direction attributes, with and without recv-sub-
layer-id. Columns that do not indicate offer or answer apply to
both.
sendonly --+
answer: recvonly, recv-sub-layer-id --+ |
recvonly w/o recv-sub-layer-id --+ | |
answer: sendrecv, recv-sub-layer-id --+ | | |
sendrecv w/o recv-sub-layer-id --+ | | | |
| | | | |
profile-space C D C D P
profile-id C D C D P
tier-flag C D C D P
level-id D D D D P
interop-constraints C D C D P
profile-compatibility-indicator C D C D P
tx-mode C C C C P
max-recv-level-id R R R R -
sprop-max-don-diff P P - - P
sprop-depack-buf-nalus P P - - P
sprop-depack-buf-bytes P P - - P
depack-buf-cap R R R R -
sprop-segmentation-id P P P P P
sprop-spatial-segmentation-idc P P P P P
max-br R R R R -
max-cpb R R R R -
max-dpb R R R R -
max-lsr R R R R -
max-lps R R R R -
max-tr R R R R -
max-tc R R R R -
max-fps R R R R -
sprop-vps P P - - P
sprop-sps P P - - P
sprop-pps P P - - P
sprop-sub-layer-id P P - - P
recv-sub-layer-id X O X O -
dec-parallel-cap R R R R -
include-dph R R R R - Legend:
C: configuration for sending and receiving bitstreams
D: changeable configuration, same as C except possible
to answer with a different but consistent value (see the
semantics of the six parameters related to profile, tier,
and level on these parameters being consistent)
P: properties of the bitstream to be sent
R: receiver capabilities
O: operation point selection
X: MUST NOT be present
-: not usable, when present MUST be ignoredParameters used for declaring receiver capabilities are, in general, downgradable; i.e., they express the upper limit for a sender's possible behavior. Thus, a sender MAY select to set its encoder using only lower/lesser or equal values of these parameters. 📉💖
When the answer does not include a recv-sub-layer-id that is less than the sprop-sub-layer-id in the offer, parameters declaring a configuration point are not changeable, with the exception of the level-id parameter for unicast usage, and these parameters express values a receiver expects to be used and MUST be used verbatim in the answer as in the offer. 🔒
When a sender's capabilities are declared with the configuration parameters, these parameters express a configuration that is acceptable for the sender to receive bitstreams. In order to achieve high interoperability levels, it is often advisable to offer multiple alternative configurations. It is impossible to offer multiple configurations in a single payload type. Thus, when multiple configuration offers are made, each offer requires its own RTP payload type associated with the offer. However, it is possible to offer multiple operation points using one configuration in a single payload type by including sprop-vps in the offer and recv-sub-layer-id in the answer. 🦋🎀
A receiver SHOULD understand all media type parameters, even if it only supports a subset of the payload format's functionality. This ensures that a receiver is capable of understanding when an offer to receive media can be downgraded to what is supported by the receiver of the offer. 🫶
An answerer MAY extend the offer with additional media format configurations. However, to enable their usage, in most cases a second offer is required from the offerer to provide the bitstream property parameters that the media sender will use. This also has the effect that the offerer has to be able to receive this media format configuration, not only to send it. 🌸✨
7.2.3. Usage in Declarative Session Descriptions
okay so bestie, this part is about when HEVC over RTP gets offered with SDP in a declarative style 🎀 like in Real Time Streaming Protocol (RTSP) [RFC2326] or Session Announcement Protocol (SAP) [RFC2974]. the following considerations are necessary 💅
- All parameters that could mean either bitstream properties OR receiver capabilities only indicate bitstream properties here 🌸 🌸 so like, profile-tier-level-id tells you what the bitstream is actually using, not what the receiver can handle 🧚 the following interpretation of the parameters MUST be used:
- Declaring actual configuration or bitstream properties: ✨
profile-spaceprofile-idtier-flaglevel-idinterop-constraintsprofile-compatibility-indicatortx-modesprop-vpssprop-spssprop-ppssprop-max-don-diffsprop-depack-buf-nalussprop-depack-buf-bytessprop-segmentation-idsprop-spatial-segmentation-idc
- Not usable (when present, they MUST be ignored): 🙈
max-lpsmax-lsrmax-cpbmax-dpbmax-brmax-trmax-tcmax-fpsmax-recv-level-iddepack-buf-capsprop-sub-layer-iddec-parallel-capinclude-dph
- Declaring actual configuration or bitstream properties: ✨
- A receiver of the SDP is REQUIRED to support all parameters and values of the parameters provided 👑; otherwise, the receiver MUST reject (RTSP) or not participate in (SAP) the session. ngl it falls on the creator of the session to use values that are expected to be supported by the receiving application 💀
7.2.4. Considerations for Parameter Sets
okay this one is about out-of-band parameter sets 🦋 when out-of-band transport is used, parameter sets MAY still be additionally transported in-band unless an application explicitly disallows it, and some of those extra in-band sets may update some of the out-of-band ones ✨ "update" here means sending a parameter set of the same type with the same parameter set ID but with different values for at least one other parameter of that set 🍓
7.2.5. Dependency Signaling in Multi-Stream Mode
ok so if MRST or MRMT is used, the rules on signaling media decoding dependency in SDP from [RFC5583] apply 🌈 the rules on "hierarchical or layered encoding" with multicast in Section 5.7 of [RFC4566] do NOT apply, lowkey plot twist 🫶 this means the notation for Connection Data c= SHALL NOT be used with more than one address 😭 i.e., the sub-field <number of addresses> in the sub-field <connection-address> of the c= field, described in [RFC4566], must not be present 💅 and the order of session dependency goes from the RTP stream with the lowest temporal sub-layer up to the RTP stream with the highest temporal sub-layer ✨
8. Use with Feedback Messages ✨
okay this section is all about feedback messages 🎀 the following subsections define how to use the Picture Loss Indication (PLI), Slice Lost Indication (SLI), Reference Picture Selection Indication (RPSI), and Full Intra Request (FIR) messages with HEVC 💖 the PLI, SLI, and RPSI messages are defined in [RFC4585], and the FIR message is defined in [RFC5104] 🦋
8.1. Picture Loss Indication (PLI) 🌸
as specified in RFC 4585, Section 6.3.1, getting a PLI tells a media sender that there was "the loss of an undefined amount of coded video data belonging to one or more pictures" 🫶 without any specific knowledge of the bitstream setup (like use and location of in-band parameter sets, non-IDR decoder refresh points, picture structures, and so forth), the reaction to a PLI from an HEVC sender SHOULD be to send an IDR picture and relevant parameter sets 🔥 potentially with enough redundancy to make sure they arrive correctly 💅
sometimes though, info about the bitstream structure is already known 👑 for example, state could have been set up outside this document so that parameter sets are only conveyed out of band and stay static for the whole session 🧚 in that case it's obviously unnecessary to send them in-band because of a PLI 🙈 other examples could be devised based on a priori knowledge of different aspects of the bitstream structure ✨ in all cases, the timing and congestion control mechanisms of RFC 4585 MUST be observed 💜
8.2. Slice Loss Indication (SLI) 🦋
the SLI described in RFC 4585 can be used to tell a sender about the loss of a number of Coded Tree Blocks (CTBs) in a CTB raster scan order of a picture 🌈 in the SLI's Feedback Control Indication (FCI) field, the subfield "First" MUST be set to the CTB address of the first lost CTB 💖 note that the CTB address is in CTB-raster-scan order of a picture 🍓 for the first CTB of a slice segment, the CTB address is the value of slice_segment_address when present, or 0 when the value of first_slice_segment_in_pic_flag is equal to 1 🎀 both syntax elements are in the slice segment header 🌸 the subfield "Number" MUST be set to the number of consecutive lost CTBs, again in CTB-raster-scan order of a picture ✨
note that because both "First" and "Number" are counted in CTBs in CTB-raster-scan order of a picture, NOT in tile-scan order (which is the bitstream order of CTBs), you might need multiple SLI messages to report the loss of one tile that covers multiple CTB rows but is less wide than the picture 😭💅
the subfield "PictureID" MUST be set to the 6 least significant bits of a binary representation of the value of PicOrderCntVal, as defined in [HEVC], of the picture the lost CTBs are indicated for 🫶 note that for IDR pictures the syntax element slice_pic_order_cnt_lsb is not present, but then the value is inferred to be equal to 0 🧚
as described in RFC 4585, an encoder in a media sender can use this info to "clean up" the corrupted picture by sending intra information, while observing the constraints in RFC 4585, for example with respect to congestion control 💜 in lots of cases, error tracking is needed to find the corrupted region in the receiver's state (reference pictures), because errors leak into uncorrupted regions of the picture through motion compensation 🔥 reference-picture selection can also be used to "clean up" the corrupted picture, and that's usually more efficient and less likely to cause congestion than sending intra information 👑
in contrast to the video codecs contemplated in RFCs 4585 and 5104 [RFC5104], in HEVC the "macroblock size" is NOT fixed to 16x16 luma samples, it's variable 🌈 that doesn't create a conceptual problem for SLI, though, because the CTB size is set as a sequence-level thing, and using a slice loss indication across CVS boundaries is meaningless since there's no prediction across sequence boundaries 💖 however, using SLI messages properly isn't as straightforward as it was with older fixed-macroblock-sized codecs 🍓 the state of the sequence parameter set (where the CTB size lives) has to be taken into account when interpreting the "First" subfield in the FCI 🎀
8.3. Reference Picture Selection Indication (RPSI) 🌸
feedback-based reference picture selection has been shown to be a powerful tool for stopping temporal error propagation and improving error resilience [Girod99][Wang05] ✨ in one approach, the decoder side tracks errors in decoded pictures and tells the encoder side that a particular picture decoded relatively earlier is correct and still present in the decoded picture buffer 😭 it asks the encoder to use that correct picture-availability info when encoding the next picture, so further temporal error propagation stops 💅 for this approach, the decoder side should use the RPSI feedback message 🫶
encoders can encode some long-term reference pictures as specified in H.264 or HEVC for the purposes in the previous paragraph without needing a huge decoded picture buffer 🧚 as shown in [Wang05], with a flexible reference picture management scheme like H.264 and HEVC have, even a decoded picture buffer size of two picture storage buffers would work for that approach 🦋
the field "Native RPSI bit string defined per codec" is a base16 [RFC4648] representation of the 8 bits made of the 2 most significant bits equal to 0 and 6 bits of nuh_layer_id, as defined in [HEVC], followed by the 32 bits representing the value of PicOrderCntVal (in network byte order), as defined in [HEVC], for the picture that the RPSI feedback message indicates 💖
the use of the RPSI feedback message as a positive acknowledgement with HEVC is deprecated 🥺 in other words, the RPSI feedback message MUST only be used as a reference picture selection request, so that it can also be used in multicast 👑
8.4. Full Intra Request (FIR) 💜
the whole point of the FIR message is to force an encoder to send an independent decoder refresh point as soon as possible 🔥 (observing, for example, the congestion-control-related constraints in RFC 5104 🌈)
upon getting a FIR, a sender MUST send an IDR picture 🎀 parameter sets MUST also be sent, EXCEPT when there's a priori knowledge that the parameter sets were already set up correctly 🫶 a typical example is an understanding between sender and receiver, set up outside this document, that parameter sets are only ever sent out-of-band 🍓
9. Security Considerations 🔐
Okay bestie, security time, so let's be real about it 🫶 This section is only about the payload format itself, plus one HEVC feature that can get really dangerous if you implement it carelessly 😭 The payload format on its own isn't a full system. Read the security stuff for RTP, especially the Security Considerations section in [RFC3550], and think about the security of whatever call-control stack you pick (it might use this memo's media type registration). Also watch out for known security bugs in video coding and decoding implementations in general and just avoid those 💀
Apart from the user data SEI message (more on that in a sec), no security threats beyond the usual RTP payload format stuff are known here 💅 So neither the media-plane mechanisms nor the signaling part of this memo seems to add any risk beyond what every RTP-based system already has ✨
RTP packets using this payload format are covered by the RTP security considerations in [RFC3550], plus whatever applicable RTP profile you're using, like RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/SAVPF [RFC5124]. Now, "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] explains that a payload format isn't supposed to pick the solution for basic security goals like confidentiality, integrity, and source authenticity. That job belongs to whoever is using RTP in their app 🙈 If you need help, "Options for Securing RTP Sessions" [RFC7201] has guidance on the security options and what to think about. Applications SHOULD use one or more appropriate strong security mechanisms 🔥 The rest of this section is about the security-relevant properties of the payload format itself, so let's go 🎀
Because the compression used with this payload format is applied end-to-end, any encryption has to happen after compression 🧚 Heads up for denial-of-service drama: encodings that use compression with uneven receiver-side computational load can be attacked. An attacker can slip pathological datagrams into the bitstream that are super expensive to decode and overload the receiver 💀 H.265 is especially vulnerable here, because it's super easy to make datagrams with NAL units that mess with the decoding of lots of future NAL units 😭 So, to keep things safe, using data origin authentication and data integrity protection on at least the RTP packet is RECOMMENDED, for example with SRTP [RFC3711] 🛡️✨
Like [H.264], HEVC has a user data Supplemental Enhancement Information (SEI) message 👀 This one lets you stick any arbitrary bitstring inside the video bitstream. That bitstring could contain JavaScript, machine code, or other active content, yikes 🙈 HEVC just leaves it up to the receiving system to decide what to do with it. To avoid harmful side effects, decoder implementations cannot naively trust what's inside. For example, forwarding any JavaScript a decoder finds over to a web browser would be a bad, insecure implementation practice, so please don't do that 💅 The safest move is to simply discard user data SEI messages, but that can hurt the user's quality of experience 🥺
⚠️ Heads up: End-to-end security with authentication, integrity, or confidentiality protection stops a MANE from doing media-aware stuff, except for discarding whole packets. If confidentiality protection is on, the MANE can't even discard packets in a media-aware way anymore 😭 For a MANE to be allowed to do those operations, it is required to be a trusted entity that's included in the security context establishment 👑
10. Congestion Control 🌊
Congestion control for RTP SHALL be used according to RTP [RFC3550] and any applicable RTP profile, for example AVP [RFC3551] 💖 If you're using best-effort service, there's one more thing: users of this payload format MUST monitor packet loss to make sure the loss rate stays in an acceptable range 👀 Packet loss counts as acceptable when a TCP flow on the same network path, dealing with the same network conditions, would get an average throughput (measured on a reasonable timescale) that's not less than what all your RTP streams combined are getting 🔥 You can make that happen by adding congestion-control mechanisms that adapt the transmission rate, adapt how many layers you subscribe to in a layered multicast session, or by having a receiver leave the session if the loss rate gets unacceptably high 🦋
Bitrate adaptation for congestion control is super easy when you're doing real-time encoding, for example by tuning the quantization parameter well 🧚✨
BUT when you're sending pre-encoded content, the pre-coded bitstream has to be made for adaptivity first 😭 The key tool HEVC gives you is temporal scalability 💜 A media sender can drop NAL units from higher temporal sub-layers (the ones with a high TID value) until the sending bitrate falls into an acceptable range 🌸 HEVC also has mechanisms for lightweight identification of switching points in temporal enhancement layers, as discussed in Section 1.1.2 of this memo 🎀 An HEVC media sender can start sending packets from NAL units of temporal enhancement layers at these switching points, to probe for available bandwidth and to use bandwidth that has been shown to be available 💅
The mechanisms above generally work within a defined profile and level, so no renegotiation of the channel is needed 🙌 Only when non-downgradable parameters (like profile) have to change does the RTP stream(s) need to be terminated and restarted. You can do that by using different RTP payload types 🍓
MANEs MAY remove certain unusable packets from the RTP stream when that stream got damaged by earlier packet losses 🫧 This can cut network load in some special cases 💫 For example, MANES can drop FUs when the leading FUs of the same NAL unit were lost, or drop dependent slice segments when the leading slice segments of the same slice were lost, because the trailing FUs or dependent slice segments are pointless to most decoders 😭 MANES can also drop higher temporal scalable layers if the outbound transmission (from the MANE's point of view) is congested 🌈
11. IANA Considerations 🗂️
A new media type, as specified in Section 7.1 of this memo, has been registered with IANA 🎉✨
12. References 📚
12.1. Normative References 💜
[H.264] ITU-T, "Advanced video coding for generic audiovisual
services", ITU-T Recommendation H.264, April 2013.
[HEVC] ITU-T, "High efficiency video coding", ITU-T Recommendation
H.265, April 2013.
[ISO23008-2]
ISO/IEC, "Information technology -- High efficiency coding
and media delivery in heterogeneous environments -- Part 2:
High efficiency video coding", ISO/IEC 23008-2, 2013.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model
with Session Description Protocol (SDP)", RFC 3264,
DOI 10.17487/RFC3264, June 2002,
<http://www.rfc-editor.org/info/rfc3264>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, July
2003, <http://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003,
<http://www.rfc-editor.org/info/rfc3551>.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, DOI 10.17487/RFC3711, March 2004,
<http://www.rfc-editor.org/info/rfc3711>.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, DOI 10.17487/RFC4566, July
2006, <http://www.rfc-editor.org/info/rfc4566>.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585,
DOI 10.17487/RFC4585, July 2006,
<http://www.rfc-editor.org/info/rfc4585>.
[RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data
Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006,
<http://www.rfc-editor.org/info/rfc4648>.
[RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman,
"Codec Control Messages in the RTP Audio-Visual Profile
with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104,
February 2008, <http://www.rfc-editor.org/info/rfc5104>.
[RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for
Real-time Transport Control Protocol (RTCP)-Based Feedback
(RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February
2008, <http://www.rfc-editor.org/info/rfc5124>.
[RFC5234] Crocker, D., Ed., and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234,
DOI 10.17487/RFC5234, January 2008,
<http://www.rfc-editor.org/info/rfc5234>.
[RFC5576] Lennox, J., Ott, J., and T. Schierl, "Source-Specific Media
Attributes in the Session Description Protocol (SDP)",
RFC 5576, DOI 10.17487/RFC5576, June 2009,
<http://www.rfc-editor.org/info/rfc5576>.
[RFC5583] Schierl, T. and S. Wenger, "Signaling Media Decoding
Dependency in the Session Description Protocol (SDP)",
RFC 5583, DOI 10.17487/RFC5583, July 2009,
<http://www.rfc-editor.org/info/rfc5583>.12.2. Informative References 📚✨
okay bestie, these are the extra reads that back up the whole spec 🤓 none of them are required to implement this, but they're where the receipts live 💅
[3GPDASH] 3GPP, "Transparent end-to-end Packet-switched Streaming
Service (PSS); Progressive Download and Dynamic Adaptive
Streaming over HTTP (3GP-DASH)", 3GPP TS 26.247 12.1.0,
December 2013.
[3GPPFF] 3GPP, "Transparent end-to-end packet switched streaming
service (PSS); 3GPP file format (3GP)", 3GPP TS 26.244
12.20, December 2013.
[CABAC] Sole, J., Joshi, R., Nguyen, N., Ji, T., Karczewicz, M.,
Clare, G., Henry, F., and Duenas, A., "Transform
coefficient coding in HEVC", IEEE Transactions on Circuts
and Systems for Video Technology, Vol. 22, No. 12,
pp. 1765-1777, DOI 10.1109/TCSVT.2012.2223055, December
2012.
[Girod99] Girod, B. and Faerber, F., "Feedback-based error control
for mobile video transmission", Proceedings of the IEEE,
Vol. 87, No. 10, pp. 1707-1723, DOI 10.1109/5.790632,
October 1999.
[H.265.1] ITU-T, "Conformance specification for ITU-T H.265 high
efficiency video coding", ITU-T Recommendation H.265.1,
October 2014.
[HEVCv2] Flynn, D., Naccari, M., Rosewarne, C., Sharman, K., Sole,
J., Sullivan, G. J., and T. Suzuki, "High Efficiency Video
Coding (HEVC) Range Extensions text specification: Draft
7", JCT-VC document JCTVC-Q1005, 17th JCT-VC meeting,
Valencia, Spain, March/April 2014.
[IS014496-12]
IS0/IEC, "Information technology - Coding of audio-visual
objects - Part 12: ISO base media file format", IS0/IEC
14496-12, 2015.
[IS015444-12]
IS0/IEC, "Information technology - JPEG 2000 image coding
system - Part 12: ISO base media file format", IS0/IEC
15444-12, 2015.
[JCTVC-J0107]
Wang, Y.-K., Chen, Y., Joshi, R., and Ramasubramonian, K.,
"AHG9: On RAP pictures", JCT-VC document JCTVC-L0107, 10th
JCT-VC meeting, Stockholm, Sweden, July 2012.
[MPEG2S] ISO/IEC, "Information technology - Generic coding of moving
pictures and associated audio information - Part 1:
Systems", ISO International Standard 13818-1, 2013.
[MPEGDASH] ISO/IEC, "Information technology - Dynamic adaptive
streaming over HTTP (DASH) -- Part 1: Media presentation
description and segment formats", ISO International
Standard 23009-1, 2012.
[RFC2326] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time
Streaming Protocol (RTSP)", RFC 2326, DOI 10.17487/RFC2326,
April 1998, <http://www.rfc-editor.org/info/rfc2326>.
[RFC2974] Handley, M., Perkins, C., and E. Whelan, "Session
Announcement Protocol", RFC 2974, DOI 10.17487/RFC2974,
October 2000, <http://www.rfc-editor.org/info/rfc2974>.
[RFC6051] Perkins, C. and T. Schierl, "Rapid Synchronisation of RTP
Flows", RFC 6051, DOI 10.17487/RFC6051, November 2010,
<http://www.rfc-editor.org/info/rfc6051>.
[RFC6184] Wang, Y.-K., Even, R., Kristensen, T., and R. Jesup, "RTP
Payload Format for H.264 Video", RFC 6184,
DOI 10.17487/RFC6184, May 2011,
<http://www.rfc-editor.org/info/rfc6184>.
[RFC6190] Wenger, S., Wang, Y.-K., Schierl, T., and A. Eleftheriadis,
"RTP Payload Format for Scalable Video Coding", RFC 6190,
DOI 10.17487/RFC6190, May 2011,
<http://www.rfc-editor.org/info/rfc6190>.
[RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP
Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014,
<http://www.rfc-editor.org/info/rfc7201>.
[RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP Framework:
Why RTP Does Not Mandate a Single Media Security Solution",
RFC 7202, DOI 10.17487/RFC7202, April 2014,
<http://www.rfc-editor.org/info/rfc7202>.
[RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and
B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms for
Real-Time Transport Protocol (RTP) Sources", RFC 7656,
DOI 10.17487/RFC7656, November 2015,
<http://www.rfc-editor.org/info/rfc7656>.
[RFC7667] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667,
DOI 10.17487/RFC7667, November 2015,
<http://www.rfc-editor.org/info/rfc7667>.
[RTP-MULTI-STREAM]
Lennox, J., Westerlund, M., Wu, Q., and C. Perkins,
"Sending Multiple Media Streams in a Single RTP Session",
Work in Progress, draft-ietf-avtcore-rtp-multi-stream-11,
December 2015.
[SDP-NEG] Holmberg, C., Alvestrand, H., and C. Jennings, "Negotiating
Medai Multiplexing Using Session Description Protocol
(SDP)", Work in Progress,
draft-ietf-mmusic-sdp-bundle-negotiation-25, January 2016.
[Wang05] Wang, Y.-K., Zhu, C., and Li, H., "Error resilient video
coding using flexible reference fames", Visual
Communications and Image Processing 2005 (VCIP 2005),
Beijing, China, July 2005.Acknowledgements 🫶
Big shoutout to Muhammed Coban and Marta Karczewicz 💖 they talked things through with the author about how to spec the use with feedback messages, plus a bunch of other bits in this memo 🗣️
Jonathan Lennox and Jill Boyce get a hug 🤗 for their work on the PACI design that made it into here 🎀
Rickard Sjoberg, Arild Fuldseth, Bo Burman, Magnus Westerlund, and Tom Kristensen are thanked for their contributions to the signaling around parallel processing ⚡
And a whole squad of reviewers made valuable comments that made the doc better 🌈 Magnus Westerlund, Jonathan Lennox, Bernard Aboba, Jonatan Samuelsson, Roni Even, Rickard Sjoberg, Sachin Deshpande, Woo Johnman, Mo Zanaty, Ross Finlayson, Danny Hong, Bo Burman, Ben Campbell, Brian Carpenter, Qin Wu, Stephen Farrell, and Min Wang 👑✨ literally the dream team
Authors' Addresses 📮
here's who to slide into the DMs of if you've got questions 💌
Ye-Kui Wang
Qualcomm Incorporated
5775 Morehouse Drive
San Diego, CA 92121
United States
Phone: +1-858-651-8345
Email: yekui.wang@gmail.com
Yago Sanchez
Fraunhofer HHI
Einsteinufer 37
D-10587 Berlin
Germany
Phone: +49 30 31002-663
Email: yago.sanchez@hhi.fraunhofer.de
Thomas Schierl
Fraunhofer HHI
Einsteinufer 37
D-10587 Berlin
Germany
Phone: +49-30-31002-227
Email: thomas.schierl@hhi.fraunhofer.de
Stephan Wenger
Vidyo, Inc.
433 Hackensack Ave., 7th floor
Hackensack, NJ 07601
United States
Phone: +1-415-713-5473
Email: stewe@stewe.org
Miska M. Hannuksela
Nokia Corporation
P.O. Box 1000
33721 Tampere
Finland
Phone: +358-7180-08000
Email: miska.hannuksela@nokia.com📦 where this page comes from
- ietf: rfc7798 2016-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms