RFC 9328
RTP Payload Format for Versatile Video Coding (VVC)
1. Introduction ✨
Okay bestie, this is RFC 9328, and it's all about getting VVC video through the Internet 🎀 Here's the title block tea so you know who's behind it:
- Title: RTP Payload Format for Versatile Video Coding (VVC) 🦋
- Authors: S. Zhao (Intel), S. Wenger (Tencent), Y. Sanchez (Fraunhofer HHI), Y.-K. Wang (Bytedance Inc.), and M. M Hannuksela (Nokia Technologies) 👑
- Category: Standards Track, from the Internet Engineering Task Force (IETF) 💜
- Date: December 2022
- ISSN: 2070-1721
- Obsoletes / Updates: the title block doesn't list any RFCs this one obsoletes or updates, so nothing gets replaced or amended here ✨
Abstract
Okay so, this memo describes an RTP payload format for the Versatile Video Coding (VVC) spec 🎬 That spec was published as both ITU-T Recommendation H.266 and ISO/IEC International Standard 23090-3, and it was developed by the Joint Video Experts Team (JVET) 💅
The payload format lets you pack one or more Network Abstraction Layer (NAL) units into a single RTP packet payload. It also lets you split one NAL unit across multiple RTP packets (fragmentation), which is giving flexible packing energy 📦✨ It's super applicable for videoconferencing, Internet video streaming, and high-bitrate entertainment-quality video, among other things 🌈
Status of This Memo
This is an Internet Standards Track document 🎀
This document is a product of the Internet Engineering Task Force (IETF), and it represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check Section 2 of RFC 7841 🫶
Info about the current status of this document, any errata, and how to give feedback is at https://www.rfc-editor.org/info/rfc9328 🔥
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. 💖
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they explain your rights and restrictions with respect to this document 🧚
Code Components extracted from this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Revised BSD License. Don't skip this part, it's the fine print that keeps you legal 💀
Table of Contents
1. Introduction
1.1. Overview of the VVC Codec
1.1.1. Coding-Tool Features (Informative)
1.1.2. Systems and Transport Interfaces (Informative)
1.1.3. High-Level Picture Partitioning (Informative)
1.1.4. NAL Unit Header
1.2. Overview of the Payload Format
2. Conventions
3. Definitions and Abbreviations
3.1. Definitions
3.1.1. Definitions from the VVC Specification
3.1.2. Definitions Specific to This Memo
3.2. Abbreviations
4. RTP Payload Format
4.1. RTP Header Usage
4.2. Payload Header Usage
4.3. Payload Structures
4.3.1. Single NAL Unit Packets
4.3.2. Aggregation Packets (APs)
4.3.3. Fragmentation Units
4.4. Decoding Order Number
5. Packetization Rules
6. De-packetization Process
7. Payload Format Parameters
7.1. Media Type Registration
7.2. Optional Parameters Definition
7.3. SDP Parameters
7.3.1. Mapping of Payload Type Parameters to SDP
7.3.2. Usage with SDP Offer/Answer Model
7.3.3. Multicast
7.3.4. Usage in Declarative Session Descriptions
7.3.5. Considerations for Parameter Sets
8. Use with Feedback Messages
8.1. Picture Loss Indication (PLI)
8.2. Full Intra Request (FIR)
9. Security Considerations
10. Congestion Control
11. IANA Considerations
12. References
12.1. Normative References
12.2. Informative References
Acknowledgements
Authors' Addresses1. Introduction ✨
The Versatile Video Coding spec was formally published as both ITU-T Recommendation H.266 [VVC] and ISO/IEC International Standard 23090-3 [ISO23090-3]. VVC is reported to be a major glow-up over High Efficiency Video Coding [HEVC] (aka H.265) and other earlier video codecs, with significant coding efficiency gains 📈💖
This memo specifies an RTP payload format for VVC. It shares its basic design with the NAL-unit-based RTP payload formats of Advanced Video Coding (AVC) [RFC6184], Scalable Video Coding (SVC) [RFC6190], and High Efficiency Video Coding (HEVC) [RFC7798], plus their respective predecessors 🌸 In terms of design philosophy, security, congestion control, and overall implementation complexity, it has similar properties to those earlier payload format specs. That's a deliberate choice, because at least [RFC6184] is widely deployed and generally known in the implementer communities, so it's giving familiar vibes 🫶
Some scalability mechanisms from [RFC6190] also made it into this document, since VVC version 1 supports temporal, spatial, and signal-to-noise ratio (SNR) scalability 🧚
1.1. Overview of the VVC Codec 💅
VVC and HEVC share a similar hybrid video codec design, so they're basically cousins 👯 In this memo, we give a very brief overview of the VVC features that the payload format in this document actually deals with. Implementers have to read, understand, and apply the ITU-T/ISO/IEC specs for VVC to end up with interoperable, well-performing implementations 🙈
Conceptually, both VVC and HEVC include a Video Coding Layer (VCL), which is often used to refer to the coding-tool features. They also include a NAL, which is often used to refer to the systems and transport interface aspects of the codecs. Two layers, one slay 👑
Part 1 of 16 done, bestie! 💜 More parts are coming, so stay tuned 🦋
1.1.1. Coding-Tool Features (Informative) ✨
okay bestie, let's get into the coding tools 🎀 we'll sometimes compare them to HEVC, which everyone in the community already knows.
VVC uses the same basic recipe as earlier hybrid-video-coding standards, HEVC included 💅. First you build a prediction signal, either with intra prediction or with motion-compensated prediction. Then you code the residual, which is just the difference between the original and the prediction. The coding-efficiency gains come from redesigning and improving almost every part of the codec, not just one piece 🔥. VVC also has several tools that make it easier to implement on parallel architectures 🧚. And it supports temporal, spatial, and SNR scalability, plus multiview coding 🌈.
Coding blocks and transform structure. The biggest upgrade from HEVC is the coding tree structure, which is way more flexible in VVC. It's called the multi-type tree 🦋. On top of the quadtree, VVC also supports binary and ternary trees, and that adds a lot of coding efficiency. The maximum coding tree unit (CTU) size also went up from 64x64 to 128x128 👑. For better chroma coding, luma-chroma-separated trees at the CTU level may be used for intra slices. The square transforms from HEVC became non-square transforms for the rectangular blocks that come out of binary and ternary splits 📐. VVC also supports multiple transform sets (MTSs), which include DCT-2, DST-7, and DCT-8, plus the non-separable secondary transform 💜. The transform sizes are different too, with bigger sizes allowed. DCT-2 goes from 2x2 to 64x64, and DST-7 and DCT-8 go from 4x4 to 32x32 ✨. VVC also has sub-block transform for both intra-coded and inter-coded blocks. For intra blocks, intra sub-partitioning (ISP) may be used for sub-block-based intra prediction and transform. For inter blocks, sub-block transform may be used when only part of the block has non-zero transform coefficients 🍓.
Entropy coding. Like HEVC, VVC uses one entropy-coding engine. It's based on context adaptive binary arithmetic coding [CABAC], but it adds support for multiple window sizes. Those windows can be initialized differently for different context models, so adaptation is faster and coding efficiency is better 🌸. There's also a joint chroma residual coding scheme that takes advantage of the correlation between the residuals of the two color components 🫶. And VVC uses different residual coding schemes for regular transform coefficients and for residual samples produced in transform-skip mode 💅.
In-loop filtering. VVC has more loop-filter features than HEVC 😭. Its deblocking filter is similar to HEVC's, but it works on a smaller grid. After deblocking and sample adaptive offset (SAO), an adaptive loop filter (ALF) may be applied. ALF is a Wiener filter, and it reduces distortion in decoded pictures 🔍. VVC also adds a brand-new module called luma mapping with chroma scaling. It makes full use of the signal's dynamic range, which improves rate-distortion performance for both Standard Dynamic Range (SDR) and High Dynamic Range (HDR) content 🌈.
Motion prediction and coding. There's a lot going on here, bestie 😭. First is adaptive motion vector resolution (AMVR), which saves bits on motion vectors by signaling the motion vector resolution adaptively. Then there's affine motion compensation, which captures complex motion like zooming and rotation 🌀. Prediction refinement with optical flow (PROF) in affine mode is used to mimic affine motion at the pixel level 🪄. Decoder-side motion vector refinement (DMVR) derives the motion vector on the decoder side using block matching, so fewer bits have to go to motion vectors 💖. Bidirectional optical flow (BDOF) is similar to PROF. It adds a sample-wise offset at the 4x4 sub-block level, derived from gradients of the prediction samples and a motion difference relative to the coding-unit (CU) motion vectors 🧠. Merge with motion vector difference (MMVD) is a special mode that signals a limited set of motion vector differences on top of merge mode 🎀. There are also three more special merge modes: sub-block merge, triangle, and combined intra/inter prediction (CIIP). The sub-block merge list holds one sub-block temporal motion vector prediction (SbTMVP) candidate and up to four affine motion vector candidates. Triangle uses triangular block motion compensation 🔺. CIIP blends intra and inter predictions with weighting ⚖️. Finally, bi-prediction with CU-based weighting (BCW) is a block-level tool that can use adaptive weighting, and it gives you more flexibility than HEVC does 🦋.
Intra prediction and intra coding. To capture local texture directions with finer detail, VVC supports 65 angular directions, while HEVC only had 33 💅. The intra mode coding uses a 6-most-probable-modes scheme, and those 6 most probable modes are derived from the neighboring intra prediction directions 🧚. VVC also has wide-angle intra prediction (WAIP), which handles the different intra angle distributions you get with different block aspect ratios. It does that by including intra prediction angles beyond the ones HEVC has 📐. HEVC only uses the closest line of reference samples for intra prediction, but VVC can also use two further reference lines. That's called multi-reference-line (MRL) intra prediction 😭. The extra reference lines can only be used with the 6 most probable intra prediction modes. To capture the strong correlation between color components, VVC uses a cross-component linear mode (CCLM), which assumes a linear relationship between luma sample values and their associated chroma samples 🌸. VVC also applies position-dependent prediction combination (PDPC) to refine prediction samples close to the intra prediction block boundary 🫶. And there are matrix-based intra prediction (MIP) modes, which create an intra prediction block of up to 8x8 by taking a weighted sum of downsampled neighboring reference samples. The weights are hard-coded constants 🔢.
Other coding-tool features. VVC introduces dependent quantization (DQ), which reduces quantization error by switching between two quantizers based on state 💜.
1.1.2. Systems and Transport Interfaces (Informative) 🛰️
VVC inherits its basic systems and transport interface designs from HEVC and AVC, which is very iconic of it 👑. That covers the NAL-unit-based syntax structure, the hierarchical syntax and data unit structure, the supplemental enhancement information (SEI) message mechanism, and the video buffering model based on the hypothetical reference decoder (HRD) 💅. VVC's scalability features are conceptually similar to SHVC, which is the scalable extension of HEVC 🌈. The hierarchical syntax and data unit structure has parameter sets at several levels: decoder, sequence (pertaining to all), sequence (pertaining to a single), and picture. It also has picture-level header parameters, slice-level header parameters, and lower-level parameters 🧩.
A bunch of key components shaped the VVC network abstraction layer design, and they also shaped this memo. Here they are, bestie 💖
Decoding capability information. The decoding capability information (DCI) holds parameters that stay constant for the whole life of a VVC bitstream. Think a video conference, a continuous video stream, or anything else a decoder processes between setup and teardown 🎬. For streaming, those constant parameters still apply through splicing. DCI includes profile, level, and sub-profile information. Together these set a maximum capability interop point that is guaranteed to never be exceeded, even if video sequences get spliced within a session 🔒. It also has constraint fields, mostly flags, which can optionally be set to show that the bitstream is limited in its use of certain features, as the field values indicate. So a bitstream can be labeled as not using certain tools, and that helps with things like resource allocation in a decoder implementation 🍰.
Video parameter set. The video parameter set (VPS) applies to one or more coded video sequences (CVSs) with multiple layers that cover the same range of access units. Among other things, it includes decoding dependency info, expressed as info for building reference picture lists for enhancement layers 🧠. The VPS gives you the "big picture" of a scalable sequence. That covers which operation points are offered, the profile, tier, and level of those operation points, and some other high-level bitstream properties. Those can be used for session negotiation and content selection, and so on 📺. One VPS may be referenced by one or more sequence parameter sets 💜.
Sequence parameter set. The sequence parameter set (SPS) has syntax elements for a coded layer video sequence (CLVS). A CLVS is a group of pictures from the same layer. It starts with a random access point and is followed by pictures that may depend on each other, up until the next random access point picture 🎞️. In MPEG-2, the equivalent of a CVS was a group of pictures (GOP), which usually started with an I frame and then had P and B frames. VVC has more complex random access point options, but it keeps that basic idea 🥰. One big difference is that a CLVS can start with a Gradual Decoding Refresh (GDR) picture. It doesn't need a traditional random access point in the bitstream, like an instantaneous decoding refresh (IDR) or clean random access (CRA) picture 🤯. In many TV-like applications, a CVS is a few hundred milliseconds to a few seconds of video 📺. In video conferencing, where no switching Multipoint Control Units (MCUs) are involved, a CVS can last as long as the whole session 📞💖
Picture and adaptation parameter set 🎀
okay so the picture parameter set (PPS) and the adaptation parameter set (APS) carry info for zero or more pictures and zero or more slices, respectively. 💅 The PPS holds stuff that's likely to stay the same from picture to picture, at least for pictures of a certain type. The APS is for stuff that's likely to change from picture to picture or even within a picture, like adaptive loop filter coefficients ✨. A single APS gets referenced by all slices of the same picture if that APS has info about luma mapping with chroma scaling (LMCS) or a scaling list. Different APSs containing ALF parameters can be referenced by slices of the same picture, which is lowkey iconic 🦋.
Picture header 🌸
a picture header (PH) holds info that's common to all slices in the same picture. bestie, being able to send that as its own NAL unit when a picture is split into several slices saves bitrate, because you don't have to repeat the same info in every slice 🍓. BUT, and this is the plot twist, low-bitrate video might use just one slice per picture 🙈. For that case, having a separate NAL unit is extra overhead, so the picture header syntax structure gets included directly in the slice header instead. Whether the PH sits in its own NAL unit can only be switched on or off for an entire CLVS, and it can only be switched off when, across the entire CLVS, each picture contains only one slice. not me being this specific about it 💀
Profile, tier, and level 👑
the profile, tier, and level syntax structures in DCI, VPS, and SPS carry profile, tier, and level info for all layers that refer to the DCI, for layers associated with one or more output layer sets specified by the VPS, and for any layer that refers to the SPS, respectively. it's giving organized 🧚
Sub-profiles 🦋
okay this one is a whole vibe ✨. within the VVC specification, a sub-profile is a 32-bit number, coded according to ITU-T Recommendation T.35, and it does not carry semantics. it gets carried in the profile_tier_level structure, so it's (potentially) present in the DCI, VPS, and SPS. external registration bodies can register a T.35 codepoint with ITU-T registration authorities and attach to their registration a description of bitstream restrictions beyond the profiles defined by ITU-T and ISO/IEC. this lets encoder manufacturers label their bitstreams as complying with that sub-profile 🏷️. upstream standardization orgs (like Digital Video Broadcasting (DVB) and Advanced Television Systems Committee (ATSC)), plus walled-garden video services, are expected to take advantage of this labeling system 💜. compared to "normal" profiles, sub-profiles may indicate encoder choices that were traditionally left open in the (decoder-centric) video coding specs, like GOP structures, minimum and maximum Quantizer Parameter (QP) values, and the mandatory use of certain tools or SEI messages. literally so much control, I'm obsessed 😭
General constraint fields 🌈
the profile_tier_level structure carries a considerable number of constraint fields (most of them are flags) that an encoder can use to tell a decoder it will not use a certain tool or technology 🙅♀️. they were added in reaction to a perceived market need to label a bitstream as not exercising a tool that has become commercially unviable. ngl that's a very real-world origin story 💅
Temporal scalability support ⭐
VVC supports temporal scalability in a few ways ✨. first, the TemporalId is signaled in the NAL unit header. second, pictures of a particular temporal sublayer cannot be used for inter prediction reference by pictures of a lower temporal sublayer. third, there's the sub-bitstream extraction process, and fourth, each sub-bitstream extraction output is required to be a conforming bitstream. Media-Aware Network Elements (MANEs) can use the TemporalId in the NAL unit header for stream adaptation based on temporal scalability, which is so smart 🧠🫶
Reference picture resampling (RPR) 🔥
okay SO. in AVC and HEVC, the spatial resolution of pictures can't change unless a new sequence using a new SPS starts with an intra random access point (IRAP) picture 😤. VVC lets the picture resolution change within a sequence, at a position, without encoding an IRAP picture (which is always intra coded). this feature is sometimes called reference picture resampling (RPR), because it needs to resample a reference picture used for inter prediction when that reference has a different resolution than the current picture being decoded 🎀. RPR means resolution can change without coding an IRAP picture, which avoids a momentary bit rate spike that an IRAP picture would cause in streaming or video conferencing, like when network conditions change 🌸. RPR can also be used for zooming the entire video region or just some region of interest. it's a main character feature, no cap 👑
Spatial, SNR, and multiview scalability 🧚
VVC supports spatial, SNR, and multiview scalability 🌈. scalable video coding is widely considered to have technical benefits and to enrich services for all kinds of video apps. until recently, though, that functionality wasn't included in the first version of the video codec specs. in VVC, all those forms of scalability are supported natively in the first version, through the signaling of the nuh_layer_id in the NAL unit header, the VPS associating layers with their given nuh_layer_id values, reference picture selection, reference picture resampling for spatial scalability, and a number of other mechanisms not relevant for this memo 💅✨
Spatial scalability 🦋
with reference picture resampling (RPR) in place, the extra burden for scalability support is just a modification of the high-level syntax (HLS) 🎀. inter-layer prediction is used in a scalable system to improve the coding efficiency of the enhancement layers. on top of the spatial and temporal motion-compensated predictions available in a single-layer codec, VVC's inter-layer prediction uses the possibly resampled video data of the reconstructed reference picture from a reference layer to predict the current enhancement layer 🫶. the resampling process for inter-layer prediction (when it's used) happens at the block level and reuses the existing interpolation process for motion compensation in single-layer coding. so no extra resampling process is needed to support spatial scalability. efficiency bestie 🙌
SNR scalability 🌸
SNR scalability is similar to spatial scalability, except the resampling factors are 1:1 🍓. in other words, there's no change in resolution, but there is inter-layer prediction. it's giving "same size, better quality" energy ✨
Multiview scalability 💜
the first version of VVC also supports multiview scalability 👀. a multi-layer bitstream carries layers representing multiple views, and one or more of those represented views can be output at the same time. multiple POVs, one stream, iykyk 🦋
SEI messages 🧚
okay, supplemental enhancement information (SEI) messages are info in the bitstream that does not influence the decoding process as specified in the VVC specification 🙅♀️. instead they address representation and rendering issues for the decoded bitstream, label the bitstream for certain applications, and do other similar tasks. the overall concept of SEI messages, and many of the messages themselves, were inherited from the AVC and HEVC specifications 💜. except for SEI messages that affect the specification of the hypothetical reference decoder (HRD), the other SEI messages for use in the VVC environment (the ones generally useful in other video coding technologies too) aren't in the main VVC specification. they live in a companion specification, [VSEI] ✨🌈
1.1.3. High-Level Picture Partitioning (Informative) ✨
Okay bestie, VVC kept a lot of the good stuff from HEVC, like tiles and WPP (wavefront parallel processing) 🧩, with some small-to-medium glow-ups. Slices are still here too, but they got a whole redesign 💅. The big new thing is subpictures. VVC is the first video coding standard to have them 🔥, and they do the same job as HEVC motion-constrained tile sets (MCTSs), just built differently for better coding efficiency and to play nicer with application systems. The details are below 👇
Tiles and WPP
Same as HEVC, a picture can be split into tile rows and tile columns in VVC 🧱. In-picture prediction across tile boundaries is disallowed, etc. The big change is that the tile partitioning syntax got simpler, because VVC uses one unified syntax for both the uniform and the non-uniform mode 🙌. Also, signaling entry point offsets for tiles in the slice header is optional in VVC, while HEVC makes it mandatory 🫶. WPP has two differences from HEVC: i) the CTU row delay dropped from two CTUs to one CTU ⚡, and ii) signaling entry point offsets for WPP in the slice header is optional in VVC, while HEVC makes it mandatory.
Slices
Okay, so the conventional slices based on CTUs (like HEVC) or macroblocks (like AVC) are gone in VVC 💀. Here's the tea. The advances in video coding since 2003 (the publication year of AVC v1) got so good that slice-based error concealment is basically impossible now 😭, because there are more and more efficient in-picture and inter-picture prediction mechanisms. An error-concealed picture is what you get when a coded picture decodes with some data loss (like losing some slices) of the coded picture or a reference picture, since at least some part of it isn't error-free (like when that reference picture was itself error-concealed). For example, if one of several slices in a picture gets lost, it can be error-concealed by interpolating the neighboring slices 🩹. Fancy prediction gives way better coding efficiency, but it also makes it harder for machines to estimate how good an error-concealed picture looks, and that was already hard with simpler prediction 🙈. Advanced in-picture prediction also makes the coding efficiency loss from splitting a picture into multiple slices way more noticeable. Plus, network conditions are much better now, and packet loss techniques have leveled up too 📶. So very few implementations have recently used slices for maximum-transmission-unit-size matching 🤷♀️. Instead, basically all low-delay error resilience apps (like video telephony and video conferencing) rely on system/transport-level error resilience (like retransmission or forward error correction) and/or picture-based error resilience tools (like feedback-based error resilience, inserting IRAPs, scalability with a higher protection level for the base layer, and so on) 🛡️. Putting it all together, it's now very rare for a picture that can't be decoded correctly to reach the decoder. And when that rare case does happen, the system can afford to wait for an error-free picture to be decoded and ready for display, without causing frequent and long freezes that end users would see 🧊.
Slices in VVC have two modes: rectangular slices and raster-scan slices 🎀. The rectangular slice does what its name says and covers a rectangular region of the picture 📐. Typically, a rectangular slice contains several complete tiles. But it can also be a subset of a tile, made of one or more consecutive, complete CTU rows within that tile. A raster-scan slice is one or more complete tiles in tile raster-scan order 🧩. So the region covered by raster-scan slices doesn't need to be rectangular, though it could be, and it can also just happen to be a rectangle. Basically, slices in VVC are tied to tiles rather than CTUs (like HEVC) or macroblocks (like AVC) 💜.
Subpictures
VVC is the first video coding standard with subpictures as a feature 🦋. Each subpicture is made of one or more complete rectangular slices that together cover a rectangular region of the picture. A subpicture can be either extractable (meaning it's coded independently of the other subpictures in the same picture and of earlier pictures in decoding order) or not extractable 🍓. No matter which one it is, the encoder can control, for each subpicture individually, whether in-loop filtering (including deblocking, SAO, and ALF) is applied across the subpicture boundaries 🎛️.
Functionally, subpictures are similar to HEVC MCTSs 👯. They both allow independent coding and extraction of a rectangular subset of a sequence of coded pictures. That's super useful for stuff like viewport-dependent 360-degree video streaming optimization and region of interest (ROI) applications 🌈.
Now, subpictures and MCTSs are different in a few important ways, so let's go 🧠. First, subpictures let motion vectors of a coding block point outside of the subpicture, even when the subpicture is extractable. This works by applying sample padding at the subpicture boundaries, similar to what happens at picture boundaries 🙌. Second, VVC adds extra changes to how motion vectors are selected and derived in merge mode and in the decoder-side motion vector refinement process. This gives higher coding efficiency than the non-normative motion constraints applied at the encoder side for MCTSs 📈. Third, rewriting slice headers (SHs) (and PH NAL units, when present) isn't needed when you extract one or more extractable subpictures from a sequence of pictures to make a sub-bitstream that is a conforming bitstream ✨. With HEVC MCTS-based sub-bitstream extraction, SH rewriting is needed 🙃. In both cases, rewriting SPSs and PPSs is needed. But there are usually only a few parameter sets in a bitstream, while each picture has at least one slice, so rewriting SHs can be a real burden for application systems 😩. Fourth, slices of different subpictures within a picture are allowed to have different NAL unit types 🫧. Fifth, VVC specifies HRD and level definitions for subpicture sequences, so encoders can ensure the conformance of the sub-bitstream of each extractable subpicture sequence 👑.
1.1.4. NAL Unit Header
VVC keeps the NAL unit concept from HEVC, with some upgrades 🎁. VVC uses a two-byte NAL unit header, shown in Figure 1 below 👀. The payload of a NAL unit is the NAL unit minus the NAL unit header.
+---------------+---------------+
|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|Z| LayerID | Type | TID |
+---------------+---------------+Figure 1: The Structure of the VVC NAL Unit Header
Here's what each field in the NAL unit header means. It's the same as in VVC, and I'm summarizing it briefly for convenience 💖. For each field, you get its name and size, plus the matching VVC syntax element name.
- F: 1 bit
forbidden_zero_bit. This field is required to be zero in VVC. Fun fact: this bit was included in the NAL unit header so VVC video could travel over MPEG-2 transport systems without start code emulations [MPEG2S] 🧃. In the context of this payload format, the value 1 may be used to indicate a syntax violation. For example, that could happen with a NAL unit made by aggregating a number of fragmented units of a NAL unit that's missing the last fragment, as described in the last sentence of Section 4.3.3 🚨.
- Z: 1 bit
nuh_reserved_zero_bit. This field is required to be zero in VVC, and it's reserved for future extensions by ITU-T and ISO/IEC 🔮. This memo doesn't overload the "Z" bit for local extensions, for two reasons: a) overloading the "F" bit is already enough, and b) it keeps this memo useful for possible future versions of [VVC] 🧠.
- LayerId: 6 bits
nuh_layer_id. This field identifies the layer a NAL unit belongs to. A layer could be, for example, a spatial scalable layer, a quality scalable layer, a layer with a different view, and so on 🌸.
- Type: 5 bits
nal_unit_type. This field specifies the NAL unit type, as defined in Table 5 of [VVC] 🗂️. For a reference of all the NAL unit types currently defined and what they mean, check Section 7.4.2.2 in [VVC] 📖.
- TID: 3 bits
nuh_temporal_id_plus1. This field specifies the temporal identifier of the NAL unit plus 1. The value of TemporalId is equal to TID minus 1. A TID value of 0 is illegal 🙅♀️. The reason is to make sure there's at least one bit in the NAL unit header equal to 1, so start code emulations in the NAL unit payload data can be considered independently of the NAL unit header 🦄.
1.2. Overview of the Payload Format
This payload format defines the following processes, which are required for transporting VVC coded data over RTP [RFC3550] 🚀:
- usage of the RTP header with this payload format
- packetization of VVC coded NAL units into RTP packets using three types of payload structures: a single NAL unit packet, aggregation packet, and fragment unit 📦
- transmission of VVC NAL units of the same bitstream within a single RTP stream
- media type parameters to be used with the Session Description Protocol (SDP) [RFC8866] 📝
- usage of RTCP feedback messages 💬
2. Conventions 🎀
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
3. Definitions and Abbreviations 💅
3.1. Definitions
This document uses the terms and definitions of VVC 📚. Section 3.1.1 lists relevant definitions from [VVC] for convenience. Section 3.1.2 provides definitions specific to this memo. All the used terms and definitions in this memo are verbatim copies from the [VVC] specification 🫶.
3.1.1. Definitions from the VVC Specification 💖
okay so these are the VVC definitions we're borrowing, bestie. 🎀
Access unit (AU): a set of PUs from different layers, all holding coded pictures that belong to the same time for output from the DPB. 🌸
Adaptation parameter set (APS): a syntax structure with syntax elements that apply to zero or more slices, as decided by zero or more syntax elements in slice headers. 🧚
Bitstream: a sequence of bits, either as a NAL unit stream or a byte stream, that represents a sequence of AUs forming one or more coded video sequences (CVSs). 🦋
Coded picture: a coded representation of a picture made of VCL NAL units with one particular nuh_layer_id value inside an AU, and it holds all the CTUs of that picture. ✨
Clean random access (CRA) PU: a PU whose coded picture is a CRA picture. 🍓
Clean random access (CRA) picture: an IRAP picture where every VCL NAL unit has nal_unit_type equal to CRA_NUT. 💜
Coded video sequence (CVS): a sequence of AUs that, in decoding order, starts with a CVSS AU, then has zero or more AUs that are not CVSS AUs. It keeps going until just before the next AU that is a CVSS AU. 🎀
Coded video sequence start (CVSS) AU: an AU with a PU for each layer in the CVS, where the coded picture in each PU is a CLVSS picture. 👑
Coded layer video sequence (CLVS): a sequence of PUs with the same nuh_layer_id value. In decoding order it starts with a CLVSS PU, then has zero or more PUs that are not CLVSS PUs, and it keeps going until just before the next CLVSS PU. 🌈
Coded layer video sequence start (CLVSS) PU: a PU whose coded picture is a CLVSS picture. 🫶
Coded layer video sequence start (CLVSS) picture: a coded picture that is either an IRAP picture with NoOutputBeforeRecoveryFlag equal to 1, or a GDR picture with NoOutputBeforeRecoveryFlag equal to 1. 🔥
Coding Tree Block (CTB): an NxN block of samples, for some value of N, such that dividing a component into CTBs is a partitioning. 🧩
Coding tree unit (CTU): a CTB of luma samples. It can also be two matching CTBs of chroma samples for a picture with three sample arrays, or a CTB of samples for a monochrome picture, or a CTB of samples for a picture coded with three separate colour planes. The syntax structures used to code those samples come along with it. 🌟
Coding Unit (CU): a coding block of luma samples plus two matching coding blocks of chroma samples, for a picture with three sample arrays in the single tree mode. It can also be a coding block of luma samples for a picture with three sample arrays in the dual tree mode, or two coding blocks of chroma samples for a picture with three sample arrays in the dual tree mode, or a coding block of samples for a monochrome picture. Syntax structures used to code the samples are included too. 🧁
Decoding Capability Information (DCI): a syntax structure with syntax elements that apply to the entire bitstream. 🌷
Decoded picture buffer (DPB): a buffer that holds decoded pictures for reference, output reordering, or output delay, as specified for the hypothetical reference decoder. 📦
Gradual decoding refresh (GDR) picture: a picture where every VCL NAL unit has nal_unit_type equal to GDR_NUT. 🌱
Instantaneous decoding refresh (IDR) PU: a PU whose coded picture is an IDR picture. 💫
Instantaneous decoding refresh (IDR) picture: an IRAP picture where every VCL NAL unit has nal_unit_type equal to IDR_W_RADL or IDR_N_LP. 🔄
Intra random access point (IRAP) AU: an AU with a PU for each layer in the CVS, where the coded picture in each PU is an IRAP picture. 🎯
Intra random access point (IRAP) PU: a PU whose coded picture is an IRAP picture. 🍭
Intra random access point (IRAP) picture: a coded picture where all VCL NAL units have the same nal_unit_type value, which falls in the range from IDR_W_RADL to CRA_NUT, inclusive. 🌺
Layer: a set of VCL NAL units that all share one particular nuh_layer_id value, along with the non-VCL NAL units that go with them. 🥞
Network abstraction layer (NAL) unit: a syntax structure that holds an indication of the type of data coming next, plus the bytes of that data in the form of an RBSP, with emulation prevention bytes put in wherever needed. 📨
Network abstraction layer (NAL) unit stream: a sequence of NAL units. 🧵
Output Layer Set (OLS): a set of layers where one or more of those layers are specified as output layers. 🎞️
Operation point (OP): a temporal subset of an OLS, identified by an OLS index and a highest value of TemporalId. 🎈
Picture Header (PH): a syntax structure with syntax elements that apply to all slices of a coded picture. 🧸
Picture parameter set (PPS): a syntax structure with syntax elements that apply to zero or more entire coded pictures, as determined by a syntax element found in each slice header. 🪄
Picture unit (PU): a set of NAL units that are associated with each other under a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture. 🗂️
Random access: starting the decoding process for a bitstream at a point other than the beginning of that bitstream. 🚀
Raw Byte Sequence Payload (RBSP): a syntax structure containing an integer number of bytes, encapsulated in a NAL unit. It is either empty, or it looks like a string of data bits holding syntax elements, followed by an RBSP stop bit and zero or more subsequent bits equal to 0. 🔢
Sequence parameter set (SPS): a syntax structure with syntax elements that apply to zero or more entire CLVSs, as determined by the content of a syntax element in the PPS that is referred to by a syntax element in each picture header. 📜
Slice: an integer number of complete tiles, or an integer number of consecutive complete CTU rows within a tile of a picture, that are exclusively contained in a single NAL unit. 🍰
Slice header (SH): the part of a coded slice that holds the data elements pertaining to all tiles, or all CTU rows within a tile, represented in the slice. 📋
Sublayer: a temporal scalable layer of a temporal scalable bitstream, made of VCL NAL units with one particular TemporalId value, plus the associated non-VCL NAL units. 🪜
Subpicture: a rectangular region of one or more slices within a picture. 🖼️
Sublayer representation: a subset of the bitstream made of NAL units from one particular sublayer and the lower sublayers. 🧃
Tile: a rectangular region of CTUs within one particular tile column and one particular tile row in a picture. 🧱
Tile column: a rectangular region of CTUs with a height equal to the height of the picture, and a width specified by syntax elements in the picture parameter set. 🏛️
Tile row: a rectangular region of CTUs with a height specified by syntax elements in the picture parameter set, and a width equal to the width of the picture. 🛤️
Video coding layer (VCL) NAL unit: a collective term for coded slice NAL units, plus the subset of NAL units with reserved nal_unit_type values that this Specification classifies as VCL NAL units. 🎬
3.1.2. Definitions Specific to This Memo 🦋
Media-Aware Network Element (MANE): a network element, such as a middlebox, selective forwarding unit, or application-layer gateway, that can parse certain parts of the RTP payload headers or the RTP payload and react to what it finds. 🛡️
Informative note: a MANE is a bit more than a normal router or gateway, because it has to be aware of the signaling (for example, to learn the payload type mappings of the media streams). It also has to be trusted when it works with Secure RTP (SRTP). The good part is that MANEs can drop packets based on what the media coding needs. If a MANE has to drop packets because of congestion on a certain link, it can spot and remove the packets whose loss does the least damage to the user experience. After dropping packets, MANEs must rewrite RTCP packets to match the changes to the RTP stream, as specified in Section 7 of [RFC3550]. 💅
NAL unit decoding order: a NAL unit order that conforms to the constraints on NAL unit order given in Section 7.4.2.4 in [VVC]. It follows the order of NAL units in the bitstream. 🧭
RTP stream (see [RFC7656]): within the scope of this memo, one RTP stream is used to transport a VVC bitstream. That bitstream may contain one or more layers, and each layer may contain one or more temporal sublayers. 📡
Transmission order: the order of packets in ascending RTP sequence number order, using modulo arithmetic. Inside an aggregation packet, the NAL unit transmission order is the same as the order in which those NAL units appear in the packet. 🚚
3.2. Abbreviations 🌈
AU Access Unit
AP Aggregation Packet
APS Adaptation Parameter Set
CTU Coding Tree Unit
CVS Coded Video Sequence
DPB Decoded Picture Buffer
DCI Decoding Capability Information
DON Decoding Order Number
FIR Full Intra Request
FU Fragmentation Unit
GDR Gradual Decoding Refresh
HRD Hypothetical Reference Decoder
IDR Instantaneous Decoding Refresh
IRAP Intra Random Access Point
MANE Media-Aware Network Element
MTU Maximum Transfer Unit
NAL Network Abstraction Layer
NALU Network Abstraction Layer Unit
OLS Output Layer Set
PLI Picture Loss Indication
PPS Picture Parameter Set
RPSI Reference Picture Selection Indication
SEI Supplemental Enhancement Information
SLI Slice Loss Indication
SPS Sequence Parameter Set
VCL Video Coding Layer
VPS Video Parameter Set4. RTP Payload Format ✨
4.1. RTP Header Usage
The RTP header format comes from [RFC3550] (reprinted as Figure 2 for convenience, bestie 🎀). This payload format uses the header fields in a way that matches that spec, so no drama there ✨.
The RTP payload (plus the settings for some RTP header bits) for aggregation packets and fragmentation units lives in Sections 4.3.2 and 4.3.3, respectively 🦋.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: RTP Header According to RFC 3550So here's how the RTP header fields get set for this format 💅:
Marker bit (M): 1 bit 🍓 Set for the last packet, in transmission order, among each set of packets that carry NAL units from one access unit. This matches how the M bit normally works in video formats, so the playout buffer can handle things efficiently 🧚.
Payload Type (PT): 7 bits 🌸 The assignment of an RTP payload type for this new packet format is outside the scope of this document, so it isn't specified here 🙈. The PT has to be assigned either through the profile being used or dynamically.
Sequence Number (SN): 16 bits 🎀 Set and used according to [RFC3550]. Easy.
Timestamp: 32 bits ⏰ The RTP timestamp is set to the sampling timestamp of the content. A 90 kHz clock rate MUST be used 🔥. If the NAL unit has no timing properties of its own (like parameter set and SEI NAL units), the RTP timestamp MUST be set to the RTP timestamp of the coded pictures of the access unit that the NAL unit is included in (according to Section 7.4.2.4 of [VVC]) 💖. Receivers MUST use the RTP timestamp for the display process, even when the bitstream contains picture timing SEI messages or decoding unit information SEI messages, as specified in [VVC] 👑.
Informative note: When picture timing SEI messages are present, the RTP sender is responsible for making sure the RTP timestamps are consistent with the timing information carried in those picture timing SEI messages 🫶.
Synchronization source (SSRC): 32 bits 🦋 Used to identify the source of the RTP packets. A single SSRC is used for all parts of a single bitstream 💜.
4.2. Payload Header Usage
The first two bytes of the payload of an RTP packet are called the payload header 🌈. It has the same fields (F, Z, LayerId, Type, and TID) as the NAL unit header in Section 1.1.4, no matter which type of payload structure it is 💅.
The TID value tells you (among other things) how important an RTP packet is. For example, NAL units in higher temporal sublayers aren't used to decode lower temporal sublayers, so that's a whole vibe 🦋. A lower TID value means higher importance. More important NAL units MAY be better protected against transmission losses than less-important ones, ngl 🛡️.
4.3. Payload Structures
There are three different types of RTP packet payload structures. A receiver can tell which type an RTP packet payload is by looking at the Type field in the payload header 🔍.
The three payload structures are:
- Single NAL unit packet: Contains a single NAL unit in the payload, and the NAL unit header of that NAL unit also serves as the payload header 🎀. This structure is specified in Section 4.3.1.
- Aggregation Packet (AP): Contains more than one NAL unit within one access unit 🌷. This structure is specified in Section 4.3.2.
- Fragmentation Unit (FU): Contains a subset of a single NAL unit 🧩. This structure is specified in Section 4.3.3.
4.3.1. Single NAL Unit Packets
A single NAL unit packet contains exactly one NAL unit. It's made of a payload header, as defined in Table 5 of [VVC] (we call it PayloadHdr here) 💅, followed by a conditional 16-bit DONL field (in network byte order), and then the NAL unit payload data (the NAL unit minus its NAL unit header) of the contained NAL unit. Check out Figure 3 💖.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr | DONL (conditional) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NAL unit payload data |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: The Structure of a Single NAL Unit PacketThe DONL field, when it's present, holds the value of the 16 least significant bits of the decoding order number of the contained NAL unit 🧮. If sprop-max-don-diff (defined in Section 7.2) is greater than 0, the DONL field MUST be present, and the variable DON for the contained NAL unit is derived as equal to the value of the DONL field 🔥. Otherwise (sprop-max-don-diff is equal to 0), the DONL field MUST NOT be present 🚫.
4.3.2. Aggregation Packets (APs) ✨
okay so aggregation packets (APs) are the ultimate packing bestie 🧳 Lots of non-VCL NAL units are only a few octets each, so sending them one by one would be a lot of wrapping for not much stuff. APs fix that by bundling them together and cutting down on packetization overhead. it's giving efficiency 💅
An AP aggregates NAL units of one access unit, and it MUST NOT contain NAL units from more than one AU. Each NAL unit that goes in an AP gets wrapped in its own aggregation unit 🎀 The NAL units aggregated in one AP are included in NAL-unit-decoding order, so the order is always preserved 🦋
An AP is made of a payload header, as defined in Table 5 of [VVC] (we'll call it PayloadHdr with Type=28 here), followed by two or more aggregation units. Check out the layout in Figure 4 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=28) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| two or more aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: The Structure of an Aggregation PacketNow let's talk about the payload header fields of an AP 👑 The F bit MUST be equal to 0 if the F bit of each aggregated NAL unit is equal to zero; otherwise, it MUST be equal to 1. The Type field MUST be equal to 28.
The value of LayerId MUST be equal to the lowest value of LayerId of all the aggregated NAL units. The value of TID MUST be the lowest value of TID of all the aggregated NAL units ✨
Informative note: All VCL NAL units in an AP have the same TID value, since they belong to the same access unit. However, an AP may contain non-VCL NAL units whose TID value in the NAL unit header is different from the TID value of the VCL NAL units in the same AP. 🙈
Informative note: If a system is planning subpicture-level or picture-level changes, like removing subpictures or pictures of a particular layer, a smart move on the sender's side is to aggregate only NAL units that belong to the same subpicture or picture of a particular layer. 🧚
Size check time 📏 An AP MUST carry at least two aggregation units and can carry as many aggregation units as necessary. However, the total amount of data in an AP obviously MUST fit into an IP packet, and the size SHOULD be chosen so that the resulting IP packet is smaller than the MTU size, in order to avoid IP layer fragmentation. An AP MUST NOT contain the FUs specified in Section 4.3.3. And APs MUST NOT be nested, i.e., an AP cannot contain another AP. no nesting dolls here 🪆
The first aggregation unit in an AP is made up of a conditional 16-bit DONL field (in network byte order), followed by 16 bits of unsigned size information (also in network byte order). That size tells you how many bytes the NAL unit is, not counting these two octets but including the NAL unit header. After that comes the NAL unit itself, header and all. See Figure 5 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : DONL (conditional) | NALU size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU size | |
+-+-+-+-+-+-+-+-+ NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 5: The Structure of the First Aggregation Unit in an APInformative note: The first octet of Figure 5 (the one marked by the first colon) belongs to a previous aggregation unit. It's drawn there to show that aggregation units are only octet aligned. Similarly, the NAL unit carried in the aggregation unit can end on an octet boundary. 🦋
The DONL field, when it's present, specifies the value of the 16 least significant bits of the decoding order number of the aggregated NAL unit 🔢
Here's the tea on when DONL shows up. If sprop-max-don-diff is greater than 0, the DONL field MUST be present in an aggregation unit that is the first aggregation unit in an AP. The variable DON for that aggregated NAL unit is derived as equal to the value of the DONL field. For an aggregation unit that is not the first one in an AP, the variable DON for its aggregated NAL unit is derived as equal to the DON of the preceding aggregated NAL unit in the same AP plus 1 modulo 65536. Otherwise (when sprop-max-don-diff is equal to 0), the DONL field MUST NOT be present in an aggregation unit that is the first aggregation unit in an AP. Plot twist, but it makes sense 🌈
An aggregation unit that is not the first one in an AP will be followed immediately by 16 bits of unsigned size information (in network byte order). That size is the NAL unit size in bytes, again not counting these two octets but including the NAL unit header. Then comes the NAL unit itself, header included, as shown in Figure 6 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : NALU size | NAL unit |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 6: The Structure of an Aggregation Unit That Is Not the First
Aggregation Unit in an APInformative note: The first octet of Figure 6 (the one marked by the first colon) belongs to a previous aggregation unit. It's drawn there to show that aggregation units are only octet aligned. Similarly, the NAL unit carried in the aggregation unit can end on an octet boundary. 🦋
Let's look at a couple of examples! 🍓 Figure 7 shows an AP with two aggregation units, labeled 1 and 2, and no DONL field 🧸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=28) | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 HDR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NALU 1 Data |
| . . . |
| |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| . . . | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 HDR | |
+-+-+-+-+-+-+-+-+ NALU 2 Data |
| . . . |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7: An Example of an AP Packet Containing Two Aggregation
Units without the DONL FieldFigure 8 is the same vibe, but this time with two aggregation units, labeled 1 and 2, and the DONL field present 🌷
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=28) | NALU 1 DONL |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NALU 1 Data . . . |
| |
+ . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : NALU 2 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 HDR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NALU 2 Data |
| |
| . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 8: An Example of an AP Containing Two Aggregation Units
with the DONL Field4.3.3. Fragmentation Units 🎀
okay so Fragmentation Units (FUs) are the move when one NAL unit is too big for a single packet 💅 They let you chop a single NAL unit into multiple RTP packets, and the [VVC] encoder doesn't even have to cooperate or know about it 🫶 A fragment is just an integer number of consecutive octets of that NAL unit. Fragments of the same NAL unit MUST be sent in consecutive order with ascending RTP sequence numbers, and no other RTP packets from the same RTP stream can sneak in between the first and last fragment 🚫
When a NAL unit gets fragmented and carried in FUs, it's called a fragmented NAL unit. APs MUST NOT be fragmented ✋ And FUs MUST NOT be nested, i.e., an FU can't contain a subset of another FU. no nesting allowed, bestie 🙅♀️
The RTP timestamp of an RTP packet carrying an FU is set to the NALU-time of the fragmented NAL unit ⏰
An FU is made of a payload header as defined in Table 5 of [VVC] (called PayloadHdr with Type=29 here), an FU header of one octet, a conditional 16-bit DONL field (in network byte order), and an FU payload (as shown in Figure 9) 🧩
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=29) | FU header | DONL (cond) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| DONL (cond) | |
|-+-+-+-+-+-+-+-+ |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 9: The Structure of an FU
The fields in the payload header are set like this. The Type field MUST be equal to 29. The fields F, LayerId, and TID MUST be equal to the fields F, LayerId, and TID, respectively, of the fragmented NAL unit 🌸
The FU header is made of an S bit, an E bit, an R bit, and a 5-bit FuType field, as shown in Figure 10 🦋
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|P| FuType |
+---------------+Figure 10: The Structure of the FU Header
Here's what the FU header fields mean 👀
S: 1 bit When set to 1, the S bit indicates the start of a fragmented NAL unit, i.e., the first byte of the FU payload is also the first byte of the payload of the fragmented NAL unit. When the FU payload is not the start of the fragmented NAL unit payload, the S bit MUST be set to 0.
E: 1 bit When set to 1, the E bit indicates the end of a fragmented NAL unit, i.e., the last byte of the payload is also the last byte of the fragmented NAL unit. When the FU payload is not the last fragment of a fragmented NAL unit, the E bit MUST be set to 0.
P: 1 bit When set to 1, the P bit indicates the last FU of the last VCL NAL unit of a coded picture, i.e., the last byte of the FU payload is also the last byte of the last VCL NAL unit of the coded picture. When the FU payload is not the last fragment of the last VCL NAL unit of a coded picture, the P bit MUST be set to 0.
FuType: 5 bits The field FuType MUST be equal to the field Type of the fragmented NAL unit.
The DONL field, when it's there, holds the value of the 16 least significant bits of the decoding order number of the fragmented NAL unit 🔢
If sprop-max-don-diff is greater than 0, and the S bit is equal to 1, the DONL field MUST be present in the FU, and the variable DON for the fragmented NAL unit is derived as equal to the value of the DONL field. Otherwise (sprop-max-don-diff is equal to 0, or the S bit is equal to 0), the DONL field MUST NOT be present in the FU 💅
A non-fragmented NAL unit MUST NOT be transmitted in one FU. the Start bit and End bit can't both be set to 1 in the same FU header 🚫
The FU payload is made of fragments of the payload of the fragmented NAL unit, so that if the FU payloads of consecutive FUs (starting with an FU whose S bit is 1 and ending with an FU whose E bit is 1) get concatenated one after another, the payload of the fragmented NAL unit can be rebuilt ✨ The NAL unit header of the fragmented NAL unit isn't included as such in the FU payload. Instead, its info rides along in the F, LayerId, and TID fields of the FU payload headers and in the FuType field of the FU header 🧠 An FU payload MUST NOT be empty.
If an FU is lost, the receiver SHOULD discard all following fragmentation units in transmission order that belong to the same fragmented NAL unit, unless the decoder in the receiver is known to be prepared to gracefully handle incomplete NAL units 🙈
A receiver in an endpoint or in a MANE MAY aggregate the first n-1 fragments of a NAL unit into an (incomplete) NAL unit, even if fragment n of that NAL unit is not received. In this case, the forbidden_zero_bit of the NAL unit MUST be set to 1 to indicate a syntax violation 🚨
4.4. Decoding Order Number 🔢
For each NAL unit, the variable AbsDon is derived. It represents the decoding order number that tells you the NAL unit decoding order 🧮
Let NAL unit n be the n-th NAL unit in transmission order within an RTP stream.
If sprop-max-don-diff is equal to 0, AbsDon[n], the value of AbsDon for NAL unit n, is derived as equal to n.
Otherwise (sprop-max-don-diff is greater than 0), AbsDon[n] is derived as follows, where DON[n] is the value of the variable DON for NAL unit n:
- If n is equal to 0 (i.e., NAL unit n is the very first NAL unit in transmission order), AbsDon[0] is set equal to DON[0].
- Otherwise (n is greater than 0), the following applies for derivation of AbsDon[n]:
If DON[n] == DON[n-1],
AbsDon[n] = AbsDon[n-1]
If (DON[n] > DON[n-1] and DON[n] - DON[n-1] < 32768),
AbsDon[n] = AbsDon[n-1] + DON[n] - DON[n-1]
If (DON[n] < DON[n-1] and DON[n-1] - DON[n] >= 32768),
AbsDon[n] = AbsDon[n-1] + 65536 - DON[n-1] + DON[n]
If (DON[n] > DON[n-1] and DON[n] - DON[n-1] >= 32768),
AbsDon[n] = AbsDon[n-1] - (DON[n-1] + 65536 - DON[n])
If (DON[n] < DON[n-1] and DON[n-1] - DON[n] < 32768),
AbsDon[n] = AbsDon[n-1] - (DON[n-1] - DON[n])For any two NAL units (m and n), the following applies:
- When AbsDon[n] is greater than AbsDon[m], this indicates that NAL unit n follows NAL unit m in NAL unit decoding order.
- When AbsDon[n] is equal to AbsDon[m], the NAL unit decoding order of the two NAL units can be in either order.
- When AbsDon[n] is less than AbsDon[m], this indicates that NAL unit n precedes NAL unit m in decoding order.
Informative note: When two consecutive NAL units in the NAL unit decoding order have different values of AbsDon, the absolute difference between the two AbsDon values may be greater than or equal to 1.
Informative note: There are multiple reasons to allow for the absolute difference of the values of AbsDon for two consecutive NAL units in the NAL unit decoding order to be greater than one. An increment by one is not required, as at the time of associating values of AbsDon to NAL units, it may not be known whether all NAL units are to be delivered to the receiver. For example, a gateway might not forward VCL NAL units of higher sublayers or some SEI NAL units when there is congestion in the network. In another example, the first intra-coded picture of a pre-encoded clip is transmitted in advance to ensure that it is readily available in the receiver, and when transmitting the first intra-coded picture, the originator does not exactly know how many NAL units will be encoded before the first intra-coded picture of the pre-encoded clip follows in decoding order. Thus, the values of AbsDon for the NAL units of the first intra-coded picture of the pre-encoded clip have to be estimated when they are transmitted, and gaps in values of AbsDon may occur.
5. Packetization Rules 📦
The following packetization rules apply 💖
- If sprop-max-don-diff is greater than 0, the transmission order of NAL units carried in the RTP stream MAY be different than the NAL unit decoding order. Otherwise (sprop-max-don-diff is equal to 0), the transmission order of NAL units carried in the RTP stream MUST be the same as the NAL unit decoding order.
- A NAL unit of a small size SHOULD be encapsulated in an aggregation packet together with one or more other NAL units, in order to avoid the unnecessary packetization overhead for small NAL units. For example, non-VCL NAL units, such as access unit delimiters, parameter sets, or SEI NAL units, are typically small and can often be aggregated with VCL NAL units without violating MTU size constraints 🌈
- Each non-VCL NAL unit SHOULD, when possible from an MTU size match viewpoint, be encapsulated in an aggregation packet together with its associated VCL NAL unit, as typically a non-VCL NAL unit would be meaningless without the associated VCL NAL unit being available 🧚
- For carrying exactly one NAL unit in an RTP packet, a single NAL unit packet MUST be used 🎯
6. De-packetization Process 📦✨
Okay so de-packetization is basically unwrapping presents 🎁. The big idea is to pull the NAL units out of the RTP packets in an RTP stream and hand them to the decoder in NAL unit decoding order. Iconic, right?
The de-packetization process is implementation dependent, so the description below is just one example of a solid implementation. Other schemes are totally fine too, as long as the output for the same input is the same as what this process gives you. The output counts as the same when the set of output NAL units and their order are both identical. Optimizations on top of these algorithms are possible 💅
All the normal RTP buffer management stuff applies here. Duplicated or outdated RTP packets (as indicated by the RTP sequence number and the RTP timestamp) get removed 🗑️ When it comes to figuring out the exact time for decoding, things like an intentional delay to allow for proper inter-stream synchronization MUST be factored in.
NAL units with NAL unit type values from 0 to 27, inclusive, may be passed to the decoder. NAL-unit-like structures with NAL unit type values from 28 to 31, inclusive, MUST NOT be passed to the decoder. 🚫
The receiver has a receiver buffer 🪣 that compensates for transmission delay jitter within individual RTP streams and reorders NAL units from transmission order to NAL unit decoding order. In this section, we assume there is no transmission delay jitter within an RTP stream, so the buffer gets called the de-packetization buffer here, just to keep things clear. Receivers should also prepare for jitter, either by reserving separate buffers for transmission delay jitter buffering and de-packetization buffering, or by using one receiver buffer for both transmission delay jitter and de-packetization. Receivers should also take jitter into account in the buffering operation, e.g., with extra initial buffering before decoding and playback start. Main character preparation energy ✨
Now the extraction part, which is where the magic happens 🪄
When an RTP packet carries a single NAL unit packet, the payload of that RTP packet is extracted as a single NAL unit. The DONL field (the third and fourth bytes) gets excluded when sprop-max-don-diff is greater than 0.
When an RTP packet carries an aggregation packet, several NAL units get extracted from the payload. Each NAL unit is the part of the payload of each aggregation unit that follows the NALU size field, as described in Section 4.3.2.
When an RTP packet carries a Fragmentation Unit (FU), you collect every RTP packet from the first FU (with the S field equal to 1) of the fragmented NAL unit, all the way up to the last FU (with the E field equal to 1) of that fragmented NAL unit. Then you rebuild the NAL unit by concatenating all the FU payloads in the same order as their RTP packets, and appending a NAL unit header. That header's F, LayerId, and TID fields get set to the values of those same fields in the payload header of the FUs, and its NAL unit type gets set to the value of the FuType field in the FU header of the FUs, as described in Section 4.3.3. It's giving reassembly 🧩
When sprop-max-don-diff is equal to 0, the de-packetization buffer size is zero bytes. The NAL units carried in the single RTP stream get passed straight to the decoder in their transmission order, which is identical to their decoding order. Simple and clean 🌸
When sprop-max-don-diff is greater than 0, the process in the rest of this section applies.
There are two buffering states in the receiver: initial buffering and buffering while playing. Initial buffering starts when the reception is initialized. Once initial buffering is done, decoding and playback start, and the receiver switches to buffering-while-playing mode 🎬
No matter which buffering state you're in, the receiver stores incoming NAL units in reception order into the de-packetization buffer. NAL units carried in RTP packets are stored individually, and the value of AbsDon is calculated and stored for each NAL unit.
Initial buffering lasts until the difference between the greatest and smallest AbsDon values of the NAL units in the de-packetization buffer is greater than or equal to the value of sprop-max-don-diff. ⏳
After initial buffering, whenever the difference between the greatest and smallest AbsDon values of the NAL units in the de-packetization buffer is greater than or equal to the value of sprop-max-don-diff, the following operation gets repeated until that difference is smaller than sprop-max-don-diff:
The NAL unit in the de-packetization buffer with the smallest value of AbsDon is removed from the de-packetization buffer and passed to the decoder.
When no more NAL units are flowing into the de-packetization buffer, all the NAL units still sitting in the buffer get removed and passed to the decoder in order of increasing AbsDon values. Clean finish 🫶
7. Payload Format Parameters 🎀
This section specifies the optional parameters. There's also a mapping of the parameters to Session Description Protocol (SDP) [RFC8866], for applications that use SDP 💜
Parameters starting with the string "sprop", which stands for stream properties, can be used by a sender to tell a receiver about the properties of the stream that is or will be sent. Here's the tea ☕ The media sender, not the receiver, decides whether and with what values "sprop" parameters are sent. This is an uncommon thing, and it may not feel intuitive in the context of some signaling protocol concepts, especially offer/answer. Check out Section 7.3.2 for guidance on using sprop parameters in the offer/answer case.
7.1. Media Type Registration 🦋
The receiver MUST ignore any parameter unspecified in this memo.
Here's the registration template, exactly as it's written:
Type name: video
Subtype name: H266
Required parameters: N/A
Optional parameters: profile-id, tier-flag, sub-profile-id, interop-
constraints, level-id, sprop-sublayer-id, sprop-ols-id, recv-
sublayer-id, recv-ols-id, max-recv-level-id, sprop-dci, sprop-vps,
sprop-sps, sprop-pps, sprop-sei, max-lsr, max-fps, sprop-max-don-
diff, sprop-depack-buf-bytes, depack-buf-cap (refer to Section 7.2
for definitions).
Encoding considerations: This type is only defined for transfer via
RTP [RFC3550].
Security considerations: See Section 9 of RFC 9328.
Interoperability considerations: N/A
Published specification: Please refer to RFC 9328 and VVC coding
specification [VVC].
Applications that use this media type: Any application that relies
on VVC-based video services over RTP
Fragment identifier considerations: N/A
Additional information: N/A
Person & email address to contact for further information:
Stephan Wenger (stewe@stewe.org)
Intended usage: COMMON
Restrictions on usage: N/A
Author: See Authors' Addresses section of RFC 9328.
Change controller: IETF <avtcore@ietf.org>7.2. Optional Parameters Definition 💖
profile-id, tier-flag, sub-profile-id, interop-constraints, and level-id: These parameters tell you the profile, the tier, the default level, the sub-profile, and some constraints of the bitstream carried by the RTP stream. They can also describe a specific set of those things that the receiver supports.
The subset of coding tools that may have been used to generate the bitstream, or that the receiver supports, plus some extra constraints, gets indicated together by profile-id, sub-profile-id, and interop-constraints 🧰
Informative note: There are 128 values ofprofile-id. The subset of coding tools identified byprofile-idcan be constrained further with up to 255 instances ofsub-profile-id. On top of that, 68 bits ininterop-constraintscan be extended up to 324 bits, and they let you restrict tools from existing profiles even more. To support this fine-grained signaling of coding-tool subsets withprofile-id,sub-profile-id, andinterop-constraints, it would be safe to require symmetric use of these parameters in SDP offer/answer, unlessrecv-ols-idis included in the SDP answer for choosing one of the layers offered. 🤓
The tier is indicated by tier-flag, and the default level is indicated by level-id. The tier and the default level set the limits on values of syntax elements, or on arithmetic combinations of values of syntax elements, that get followed when generating the bitstream or that the receiver supports 📏
In SDP offer/answer, when the SDP answer does not include a recv-ols-id parameter that is less than the sprop-ols-id parameter in the SDP offer, the following applies:
- The
tier-flag,profile-id,sub-profile-id, andinterop-constraintsparameters MUST be used symmetrically, i.e., the value of each of these parameters in the offer MUST be the same as that in the answer, either explicitly signaled or implicitly inferred. - The
level-idparameter is changeable as long as the highest level indicated by the answer is either equal to or lower than that in the offer. Note that the highest level higher thanlevel-idin the offer for receiving can be included asmax-recv-level-id.
In SDP offer/answer, when the SDP answer does include a recv-ols-id parameter that is less than the sprop-ols-id parameter in the SDP offer, the set of tier-flag, profile-id, sub-profile-id, interop-constraints, and level-id parameters included in the answer MUST be consistent with that for the chosen output layer set as indicated in the SDP offer. The one exception is that the level-id parameter in the SDP answer is changeable as long as the highest level indicated by the answer is either lower than or equal to that in the offer. 🦄
There are more specifications of these parameters, including how they relate to syntax elements specified in [VVC], coming up below 👀
profile-id: When profile-id is not present, a value of 1 (i.e., the Main 10 profile) MUST be inferred 💫
When it's used to indicate properties of a bitstream, profile-id is derived from the general_profile_idc syntax element that applies to the bitstream, in an instance of the profile_tier_level( ) syntax structure. 🧠
VVC bitstreams transported over RTP using the technologies of this memo SHOULD contain only a single profile_tier_level( ) structure in the DCI, unless the sender can assure that a receiver can correctly decode the VVC bitstream regardless of which profile_tier_level( ) structure contained in the DCI was used for deriving profile-id and other parameters for the SDP offer/answer exchange. Not me being careful about this stuff, but it's literally the whole point 🙈
As specified in [VVC], a profile_tier_level( ) syntax structure can live in an SPS NAL unit, and one or more of them can live in a VPS NAL unit and in a DCI NAL unit 🌸. Only one of these three cases applies to the container NAL unit of the profile_tier_level( ) structure that has the syntax elements used to derive profile-id, tier-flag, level-id, sub-profile-id, or interop-constraints ✨:
- The container NAL unit is an SPS, the bitstream is a single-layer bitstream, and the
profile_tier_level( )syntax structures in all SPSs referenced by the CVSs in the bitstream have the same values, respectively, for thoseprofile_tier_level( )syntax elements. 🍓 - The container NAL unit is a VPS, the
profile_tier_level( )syntax structure is the one in the VPS that applies to the OLS corresponding to the bitstream, and theprofile_tier_level( )syntax structures applicable to the OLS corresponding to the bitstream in all VPSs referenced by the CVSs in the bitstream have the same values, respectively, for thoseprofile_tier_level( )syntax elements. 🧚 - The container NAL unit is a DCI NAL unit, and the
profile_tier_level( )syntax structures in all DCI NAL units in the bitstream have the same values, respectively, for thoseprofile_tier_level( )syntax elements. 💖
[VVC] lets a DCI NAL unit hold multiple profile_tier_level( ) structures, and those can have different values for these syntax elements across the entries 🦋. But the only thing defined here is a single profile-id, tier-flag, level-id, sub-profile-id, or interop-constraints 🫶. So when you're signaling these and a DCI NAL unit has multiple profile_tier_level( ) structures, these values SHOULD be the same as the first profile_tier_level structure in the DCI, unless the sender has made sure the receiver can decode the bitstream when a different value is chosen 👑.
tier-flag, level-id:
The value of tier-flag MUST be in the range of 0 to 1, inclusive. The value of level-id MUST be in the range of 0 to 255, inclusive. 🌈
If the tier-flag and level-id parameters are used to indicate properties of a bitstream, they tell you the tier and the highest level the bitstream complies with. ✨
If the tier-flag and level-id parameters are used for capability exchange, here's the deal 💅. If max-recv-level-id is not present, the default level defined by level-id indicates the highest level the codec wishes to support. Otherwise, max-recv-level-id indicates the highest level the codec supports for receiving. For either receiving or sending, all levels that are lower than the highest level supported MUST also be supported.
If no tier-flag is present, a value of 0 MUST be inferred. If no level-id is present, a value of 51 (i.e., level 3.1) MUST be inferred. 🫶
🎀 Informative note: The level values currently defined in the VVC specification are in the form of"majorNum.minorNum", and the value of thelevel-idfor each of the levels is equal tomajorNum * 16 + minorNum * 3. It is expected that, if any levels are defined in the future, the same convention will be used, but this cannot be guaranteed.
When used to indicate properties of a bitstream, the tier-flag and level-id parameters are derived, respectively, from the syntax element general_tier_flag, and the syntax element general_level_idc or sub_layer_level_idc[j], that apply to the bitstream in an instance of the profile_tier_level( ) syntax structure. 🦋
If the tier-flag and level-id are derived from the profile_tier_level( ) syntax structure in a DCI NAL unit, the following applies:
tier-flag = general_tier_flaglevel-id = general_level_idc
Otherwise, if the tier-flag and level-id are derived from the profile_tier_level( ) syntax structure in an SPS or VPS NAL unit, and the bitstream contains the highest sublayer representation in the OLS corresponding to the bitstream, the following applies:
tier-flag = general_tier_flaglevel-id = general_level_idc
Otherwise, if the tier-flag and level-id are derived from the profile_tier_level( ) syntax structure in an SPS or VPS NAL unit, and the bitstream does not contain the highest sublayer representation in the OLS corresponding to the bitstream, the following applies, with j being the value of the sprop-sublayer-id parameter:
tier-flag = general_tier_flaglevel-id = sub_layer_level_idc[j]
sub-profile-id:
The value of the parameter is a comma-separated (',') list of data using base64 encoding (Section 4 of [RFC4648]) representation without "==" padding. 🧁
When used to indicate properties of a bitstream, sub-profile-id is derived from each of the ptl_num_sub_profiles general_sub_profile_idc[i] syntax elements that apply to the bitstream in a profile_tier_level( ) syntax structure. 💜
interop-constraints:
A base64 encoding (Section 4 of [RFC4648]) representation of the data that includes the ptl_frame_only_constraint_flag syntax element, the ptl_multilayer_enabled_flag syntax element, and the general_constraints_info( ) syntax structure that apply to the bitstream in an instance of the profile_tier_level( ) syntax structure. 🌸
If the interop-constraints parameter is not present, the following MUST be inferred:
ptl_frame_only_constraint_flag = 1ptl_multilayer_enabled_flag = 0gci_present_flag in the general_constraints_info( ) syntax structure = 0
Using interop-constraints for capability exchange results in a requirement on any bitstream to be compliant with the interop-constraints. 🔥
sprop-sublayer-id:
This parameter MAY be used to indicate the highest allowed value of TID in the bitstream. When not present, the value of sprop-sublayer-id is inferred to be equal to 6. 🧚
The value of sprop-sublayer-id MUST be in the range of 0 to 6, inclusive. 🍓
sprop-ols-id:
This parameter MAY be used to indicate the OLS that the bitstream applies to. When not present, the value of sprop-ols-id is inferred to be equal to TargetOlsIdx, as specified in Section 8.1.1 of [VVC]. If this optional parameter is present, sprop-vps MUST also be present, or its content MUST be known a priori at the receiver. 🙈
The value of sprop-ols-id MUST be in the range of 0 to 256, inclusive.
✨ Informative note: VVC allows having up to 257 output layer sets indicated in the VPS, as the number of output layer sets minus 2 is indicated with a field of 8 bits.
recv-sublayer-id:
This parameter MAY be used to signal a receiver's choice of the offered or declared sublayer representations in sprop-vps and sprop-sps. The value of recv-sublayer-id indicates the TID of the highest sublayer that a receiver supports. When not present, the value of recv-sublayer-id is inferred to be equal to the value of the sprop-sublayer-id parameter in the SDP offer. 🎀
The value of recv-sublayer-id MUST be in the range of 0 to 6, inclusive.
recv-ols-id:
This parameter MAY be used to signal a receiver's choice of the offered or declared output layer sets in sprop-vps. The value of recv-ols-id indicates the OLS index of the bitstream that a receiver supports. When not present, the value of recv-ols-id is inferred to be equal to the value of the sprop-ols-id parameter inferred from or indicated in the SDP offer. When present, the value of recv-ols-id must be included only when sprop-ols-id was received, and must refer to an output layer set in the VPS that includes no layers other than all or a subset of the layers of the OLS referred to by sprop-ols-id. If this optional parameter is present, sprop-vps must have been received, or its content must be known a priori at the receiver. 🦋
The value of recv-ols-id MUST be in the range of 0 to 256, inclusive.
max-recv-level-id:
This parameter MAY be used to indicate the highest level a receiver supports. 👑
The value of max-recv-level-id MUST be in the range of 0 to 255, inclusive.
When max-recv-level-id is not present, the value is inferred to be equal to level-id.
max-recv-level-id MUST NOT be present when the highest level the receiver supports is not higher than the default level. 💀
sprop-dci:
This parameter MAY be used to convey a decoding capability information NAL unit of the bitstream for out-of-band transmission. The parameter MAY also be used for capability exchange. The value of the parameter is a base64 encoding (Section 4 of [RFC4648]) representation of the decoding capability information NAL unit, as specified in Section 7.3.2.1 of [VVC]. 🌈
sprop-vps:
This parameter MAY be used to convey any video parameter set to the NAL unit of the bitstream for out-of-band transmission of video parameter sets. The parameter MAY also be used for capability exchange and to indicate substream characteristics (i.e., properties of output layer sets and sublayer representations, as defined in [VVC]). The value of the parameter is a comma-separated (',') list of base64 encoding (Section 4 of [RFC4648]) representations of the video parameter set NAL units, as specified in Section 7.3.2.3 of [VVC]. 🧁
The sprop-vps parameter MAY contain one or more than one video parameter set NAL unit. However, all other video parameter sets contained in the sprop-vps parameter MUST be consistent with the first video parameter set in the sprop-vps parameter. A video parameter set vpsB is said to be consistent with another video parameter set vpsA if the number of OLSs in vpsA and vpsB are the same, and any decoder that conforms to the profile, tier, level, and constraints indicated by the data starting from the syntax element general_profile_idc to the syntax structure general_constraints_info(), inclusive, in the profile_tier_level( ) syntax structure corresponding to any OLS with index olsIdx in vpsA can decode any CVS(s) referencing vpsB when TargetOlsIdx is equal to olsIdx that conforms to the profile, tier, level, and constraints indicated by the data starting from the syntax element general_profile_idc to the syntax structure general_constraints_info(), inclusive, in the profile_tier_level( ) syntax structure corresponding to the OLS with index TargetOlsIdx in vpsB. 🔥
sprop-sps:
This parameter MAY be used to convey sequence parameter set NAL units of the bitstream for out-of-band transmission of sequence parameter sets. The value of the parameter is a comma-separated (',') list of base64 encoding (Section 4 of [RFC4648]) representations of the sequence parameter set NAL units, as specified in Section 7.3.2.4 of [VVC]. 💖
A sequence parameter set spsB is said to be consistent with another sequence parameter set spsA if any decoder that conforms to the profile, tier, level, and constraints indicated by the data starting from the syntax element general_profile_idc to the syntax structure general_constraints_info(), inclusive, in the profile_tier_level( ) syntax structure in spsA can decode any CLVS(s) referencing spsB that conforms to the profile, tier, level, and constraints indicated by the data starting from the syntax element general_profile_idc to the syntax structure general_constraints_info(), inclusive, in the profile_tier_level( ) syntax structure in spsB. 🦋
sprop-pps 💖
This parameter MAY be used to convey picture parameter set NAL units of the bitstream for out-of-band transmission of picture parameter sets. The value is a comma-separated (',') list of base64 encoding (Section 4 of [RFC4648]) representations of the picture parameter set NAL units, as specified in Section 7.3.2.5 of [VVC]. 🎀
sprop-sei 🌸
This parameter MAY be used to convey one or more SEI messages that describe bitstream characteristics. When it's present, a decoder can rely on the bitstream characteristics described in the SEI messages for the entire duration of the session, independently from the persistence scopes of the SEI messages, as specified in [VSEI]. Iconic. 👑
The value is a comma-separated (',') list of base64 encoding (Section 4 of [RFC4648]) representations of SEI NAL units, as specified in [VSEI].
📝 Informative note: Intentionally, no list of applicable or inapplicable SEI messages is specified here. Conveying certain SEI messages in sprop-sei may be sensible in some application scenarios and meaningless in others. However, a few examples are described below: In an environment where the bitstream was created from film-based source material, and no splicing is going to occur during the lifetime of the session, the film grain characteristics SEI message is likely meaningful, and sending it in sprop-sei, rather than in the bitstream at each entry point, may help with saving bits and allows one to configure the renderer only once, avoiding unwanted artifacts. Examples for SEI messages that would be meaningless to be conveyed in sprop-sei include the decoded picture hash SEI message (it is close to impossible that all decoded pictures have the same hashtag) or the filler payload SEI message (as there is no point in just having more bits in SDP). 💀
max-lsr 🧚
The max-lsr MAY be used to signal the capabilities of a receiver implementation and MUST NOT be used for any other purpose. The value of max-lsr is an integer indicating the maximum processing rate in units of luma samples per second. The max-lsr parameter signals that the receiver is capable of decoding video at a higher rate than is required by the highest level. ✨
📝 Informative note: When the OPTIONAL media type parameters are used to signal the properties of a bitstream, and max-lsr is not present, the values of tier-flag, profile-id, sub-profile-id, interop-constraints, and level-id must always be such that the bitstream complies fully with the specified profile, sub-profile, tier, level, and interop-constraints.
When max-lsr is signaled, the receiver MUST be able to decode bitstreams that conform to the highest level, with the exception that the MaxLumaSr value in Table A.3 of [VVC] for the highest level is replaced with the value of max-lsr. Senders MAY use this knowledge to send pictures of a given size at a higher picture rate than is indicated in the highest level. 🚀
When not present, the value of max-lsr is inferred to be equal to the value of MaxLumaSr given in Table A.3 of [VVC] for the highest level.
The value of max-lsr MUST be in the range of MaxLumaSr to 16 * MaxLumaSr, inclusive, where MaxLumaSr is given in Table A.3 of [VVC] for the highest level. 🔥
max-fps 🌈
The value of max-fps is an integer indicating the maximum picture rate in units of pictures per 100 seconds that can be effectively processed by the receiver. The max-fps parameter MAY be used to signal that the receiver has a constraint in that it is not capable of processing video effectively at the full picture rate that is implied by the highest level and, when present, max-lsr. 😭
The value of max-fps is not necessarily the picture rate at which the maximum picture size can be sent. It's a constraint on maximum picture rate for all resolutions. Main character energy. 💅
📝 Informative note: Themax-fpsparameter is semantically different frommax-lsrin thatmax-fpsis used to signal a constraint, lowering the maximum picture rate from what is implied by other parameters.
The encoder MUST use a picture rate equal to or less than this value. When the max-fps parameter is absent, the encoder is free to choose any picture rate according to the highest level and any signaled optional parameters. 🍓
The value of max-fps MUST be smaller than or equal to the full picture rate that is implied by the highest level and, when present, max-lsr. 🫶
sprop-max-don-diff 🦋
If there is no NAL unit naluA that is followed in transmission order by any NAL unit preceding naluA in decoding order (i.e., the transmission order of the NAL units is the same as the decoding order), the value of this parameter MUST be equal to 0. 🙈
Otherwise, this parameter specifies the maximum absolute difference between the decoding order number (i.e., AbsDon) values of any two NAL units naluA and naluB, where naluA follows naluB in decoding order and precedes naluB in transmission order. Okay, that one's a little spicy, but it's worth it. 🌸
The value of sprop-max-don-diff MUST be an integer in the range of 0 to 32767, inclusive.
When not present, the value of sprop-max-don-diff is inferred to be equal to 0.
sprop-depack-buf-bytes 🎀
This parameter signals the required size of the de-packetization buffer in units of bytes. The value of the parameter MUST be greater than or equal to the maximum buffer occupancy (in units of bytes) of the de-packetization buffer, as specified in Section 6. ✨
The value of sprop-depack-buf-bytes MUST be an integer in the range of 0 to 4294967295, inclusive.
When sprop-max-don-diff is present and greater than 0, this parameter MUST be present and the value MUST be greater than 0. When not present, the value of sprop-depack-buf-bytes is inferred to be equal to 0. 🧚
📝 Informative note: The value of sprop-depack-buf-bytes indicates the required size of the de-packetization buffer only. When network jitter can occur, an appropriately sized jitter buffer has to be available as well. 💜
depack-buf-cap 🌷
This parameter signals the capabilities of a receiver implementation and indicates the amount of de-packetization buffer space in units of bytes that the receiver has available for reconstructing the NAL unit decoding order from NAL units carried in the RTP stream. A receiver is able to handle any RTP stream for which the value of the sprop-depack-buf-bytes parameter is smaller than or equal to this parameter. 🫧
When not present, the value of depack-buf-cap is inferred to be equal to 4294967295. The value of depack-buf-cap MUST be an integer in the range of 1 to 4294967295, inclusive.
📝 Informative note: depack-buf-cap indicates the maximum possible size of the de-packetization buffer of the receiver only, without allowing for network jitter. 🌟
7.3. SDP Parameters ✨
The receiver MUST ignore any parameter unspecified in this memo. 🎀
7.3.1. Mapping of Payload Type Parameters to SDP ✨
okay so bestie, here's how the video/H266 media type maps onto the Session Description Protocol [RFC8866] fields 💅
- The media name in the
m=line of SDP MUST bevideo. 🎬 - The encoding name in the
a=rtpmapline of SDP MUST beH266(the media subtype). 🦋 - The clock rate in the
a=rtpmapline MUST be90000. ⏰ - The OPTIONAL parameters
profile-id,tier-flag,sub-profile-id,interop-constraints,level-id,sprop-sublayer-id,sprop-ols-id,recv-sublayer-id,recv-ols-id,max-recv-level-id,max-lsr,max-fps,sprop-max-don-diff,sprop-depack-buf-bytes, anddepack-buf-cap, when present, MUST be included in thea=fmtpline of SDP. The fmtp line is written as a media type string, a semicolon-separated list of parameter=value pairs. 🧁 - The OPTIONAL parameters
sprop-vps,sprop-sps,sprop-pps,sprop-sei, andsprop-dci, when present, MUST be included in thea=fmtpline of SDP, or conveyed using the "fmtp" source attribute as specified in Section 6.3 of [RFC5576]. For a particular media format (i.e., RTP payload type),sprop-vps,sprop-sps,sprop-pps,sprop-sei, orsprop-dciMUST NOT be both included in thea=fmtpline of SDP and conveyed using the "fmtp" source attribute. When they're in thea=fmtpline, those parameters are written as a media type string, a semicolon-separated list of parameter=value pairs. When they're conveyed in thea=fmtpline for a particular payload type,sprop-vps,sprop-sps,sprop-pps,sprop-sei, andsprop-dciMUST be applied to each SSRC with that payload type. When they're conveyed using the "fmtp" source attribute, they're only associated with the given source and payload type as parts of that attribute. 🫶
Informative note: Conveyingsprop-vps,sprop-sps, andsprop-ppsusing the "fmtp" source attribute lets you move parameter sets out-of-band in topologies like Topo-Video-switch-MCU, as specified in [RFC7667]. 🌈
A general usage of media representation in SDP looks like this 📝
m=video 49170 RTP/AVP 98
a=rtpmap:98 H266/90000
a=fmtp:98 profile-id=1;
sprop-vps=<video parameter sets data>;
sprop-sps=<sequence parameter set data>;
sprop-pps=<picture parameter set data>;A SIP offer/answer exchange where both parties are expected to send and receive could look like this. Only the media codec-specific parts of the SDP are shown, and some lines are wrapped because of text constraints 🌸
Offerer->Answerer:
m=video 49170 RTP/AVP 98
a=rtpmap:98 H266/90000
a=fmtp:98 profile-id=1; level_id=83;That's an offer for symmetric video communication using [VVC] and its payload specification at the main profile and level 5.1 (and since levels are downgradable, all lower levels too). Basically the offer is saying "I can receive up to 4Kp60 at the maximum bitrates specified in [VVC]!" 🤩 And if the offer got accepted "as is", the offerer could expect the answerer to receive and properly decode H.266 media up to and including level 5.1. 👑
Answerer->Offerer:
m=video 49170 RTP/AVP 98
a=rtpmap:98 H266/90000
a=fmtp:98 profile-id=1; level_id=67So this answer is the receiving system going "nah, I can't handle H.266 at level 5.1, but I can decode 1080p60" 🙈 H.266 video codecs must support decoding at all levels below the maximum level they implement, so the vibe is probably both systems sending video at 1080p60. Still, nothing stops an encoder from dialing its sending down to something like 720p30 if it's short on cycles or bandwidth or has other reasons. 🍓
7.3.2. Usage with SDP Offer/Answer Model 🦋
This section covers negotiating unicast messages with the offer/answer model described in [RFC3264] and its updates. It's split into subsections, covering a) media format configurations that don't involve non-temporal scalability; b) scalable media format configurations; c) how to use the parameters that don't describe the media configuration itself but rather the design of the payload format; and d) multicast. 🧚
7.3.2.1. Non-scalable Media Format Configuration 💜
A non-scalable VVC media configuration is one where no non-temporal scalability mechanisms are allowed. In [VVC] version 1, it's implied that general_profile_idc indicates one of these profiles: Main 10, Main 10 Still Picture, Main 10 4:4:4, or Main 10 4:4:4 Still Picture, with general_profile_idc values of 1, 65, 33, and 97, respectively. Heads up that non-scalable media configurations still include temporal scalability, inline with VVC's design philosophy and profile structure. ✨
The following limitations and rules about the media configuration apply 👑
- The parameters that identify a media format configuration for VVC are
profile-id,tier-flag,sub-profile-id,level-id, andinterop-constraints. These media configuration parameters, exceptlevel-id, MUST be used symmetrically.
The answerer MUST structure its answer according to one of the following three options:
- maintain all configuration parameters with the values remaining the same as in the offer for the media format (payload type), with the exception that the value of
level-idis changeable as long as the highest level indicated by the answer is not higher than that indicated by the offer; - include in the answer the
recv-sublayer-idparameter, with a value less than thesprop-sublayer-idparameter in the offer, for the media format (payload type), and maintain all configuration parameters with the values remaining the same as in the offer for the media format (payload type), with the exception that the value oflevel-idis changeable as long as the highest level indicated by the answer is not higher than the level indicated bysprop-spsorsprop-vpsin the offer for the chosen sublayer representation; or - remove the media format (payload type) completely (when one or more of the parameter values are not supported). 🙅♀️
Informative note: The above requirement for symmetric use does not apply to level-id, and it doesn't apply to the other bitstream or RTP stream properties and capability parameters, as described in Section 7.3.2.3 below. 🎀
- To simplify handling and matching of these configurations, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [RFC3264]. 🧩
- The same RTP payload type number used in the offer for the media subtype H266 MUST be used in the answer when the answer includes
recv-sublayer-id. When the answer does not includerecv-sublayer-id, the answer MUST NOT contain a payload type number used in the offer for the media subtype H266, unless the configuration is exactly the same as in the offer, or the configuration in the answer only differs from the offer by a different value oflevel-id. The answer MAY contain therecv-sublayer-idparameter if a VVC bitstream contains multiple operation points (using temporal scalability and sublayers) andsprop-spsorsprop-vpsis included in the offer, where information about sublayers is present in the first sequence parameter set or video parameter set contained insprop-spsorsprop-vps, respectively. Ifsprop-spsorsprop-vpsis provided in an offer, an answerer MAY select a particular operation point indicated in the first sequence parameter set or video parameter set contained insprop-spsorsprop-vps, respectively. When the answer includes arecv-sublayer-idthat's less than asprop-sublayer-idin the offer, the following applies:- When the
sprop-spsparameter is present, all sequence parameter sets contained in thesprop-spsparameter in the SDP answer, and all sequence parameter sets sent in-band for either the offerer-to-answerer direction or the answerer-to-offerer direction, MUST be consistent with the first sequence parameter set in thesprop-spsparameter of the offer (see the semantics ofsprop-spsin Section 7.1 of this document for one sequence parameter set being consistent with another sequence parameter set). 🌷 - When the
sprop-vpsparameter is present, all video parameter sets contained in thesprop-vpsparameter in the SDP answer, and all video parameter sets sent in-band for either the offerer-to-answerer direction or the answerer-to-offerer direction, MUST be consistent with the first video parameter set in thesprop-vpsparameter of the offer (see the semantics ofsprop-vpsin Section 7.1 of this document for one video parameter set being consistent with another video parameter set). 🌺 - The bitstream sent in either direction MUST conform to the profile, tier, level, and constraints of the chosen sublayer representation, as indicated by the
profile_tier_level( )syntax structure in the first sequence parameter set in thesprop-spsparameter, or by the firstprofile_tier_level( )syntax structure in the first video parameter set in thesprop-vpsparameter of the offer. 🔥
- When the
Informative note: When an offerer receives an answer that does not include recv-sublayer-id, it has to compare payload types not declared in the offer, based on the media type (i.e., video/H266) and the above media configuration parameters, with any payload types it has already declared. That lets it figure out whether the configuration is new or equivalent to one already offered, since a different payload type number may be used in the answer. Being able to do operation point selection lets a receiver take advantage of the temporal scalable nature of a VVC bitstream. 🌈
7.3.2.2. Scalable Media Format Configuration ✨
okay so a scalable VVC media configuration is any config where non-temporal scalability mechanisms are allowed 🦋 it's giving layers, bestie. In [VVC] version 1, it's implied that general_profile_idc is one of two profiles: Multilayer Main 10 (value 17) or Multilayer Main 10 4:4:4 (value 49) 💅
Here are the limitations and rules for this media configuration, listed in an order that's logical for an implementation to follow 🎀
- Identifying parameters. The parameters that identify a scalable VVC media format configuration are
profile-id,tier-flag,sub-profile-id,level-id,interop-constraints, andsprop-vps. All of these exceptlevel-idMUST be used symmetrically, except as noted below. 🫶 - Answerer's level-id. The answerer MAY include a
level-idthat MUST be lower than or equal to thelevel-idindicated in the offer. That can be either expressed bylevel-idin the offer or implied by the default level, as specified in Section 7.1. 📈 - sprop-vps is required with sprop-ols-id. When
sprop-ols-idis present in an offer,sprop-vpsMUST also be present in the same offer, and it MUST include at least one valid VPS. This lets the answerer meaningfully interpretsprop-ols-idand selectrecv-ols-id(see below) 🔍 - recv-ols-id rules. The answerer MUST NOT include
recv-ols-idunless the offer includessprop-ols-id. When it's present,recv-ols-idMUST indicate a supported output layer set in the VPS. That set includes no layers other than all or a subset of the layers of the OLS referred to bysprop-ols-id. If the answerer is unable to do that, the answerer MUST remove the media format 🙈
Informative note: If an offerer wants to offer more than one output layer set, it can do so by offering multiple VVC media with different payload types. ✨
- Sublayer parameters. The offerer MAY include
sprop-sublayer-id, which indicates the highest allowed value of TID in the bitstream. The answerer MAY includerecv-sublayer-id, which can be used to reduce the number of sublayers from the value ofsprop-sublayer-id🌈 - Operating point after recv-ols-id. When the answerer includes
recv-ols-idalong withprofile-id,tier-flag,sub-profile-id,level-id, andinterop-constraints, it MUST use the configuration parameter values signaled insprop-vpsfor the operating point with the largest number of sublayers for the chosen output layer set. The one exception islevel-id, which can change, as long as the highest level in the answer is not higher than the level indicated bysprop-vpsin the offer for that same operating point 👑
7.3.2.3. Payload Format Configuration
okay so this part is all about the buffer management stuff and how the offer/answer dance works for parameter sets 🎀 it applies to both scalable and non-scalable VVC, so it's giving everyone gets a seat 💖
- The parameters sprop-max-don-diff and sprop-depack-buf-bytes describe the properties of an RTP stream that the offerer or the answerer is sending for the media format configuration. This is kinda different from normal offer/answer params 👀 normally they declare what the offerer or answerer is able to receive. With VVC, the offerer assumes the answerer will be able to receive media encoded using the configuration being offered 🦋
Informative note: The above parameters apply for any RTP stream, when present, sent by a declaring entity with the same configuration. In other words, the applicability of the above parameters to RTP streams depends on the source endpoint. Rather than being bound to the payload type, the values may have to be applied to another payload type when being sent, as they apply for the configuration. 🧚
- The capability parameter max-lsr MAY be used to declare further capabilities of the offerer or answerer for receiving. It MUST NOT be present when the direction attribute is sendonly. 🫶
- The capability parameter max-fps MAY be used to declare lower capabilities of the offerer or answerer for receiving. It MUST NOT be present when the direction attribute is sendonly. 🌸
- When an offerer offers an interleaved stream, indicated by the presence of sprop-max-don-diff with a value larger than zero, the offerer MUST include the size of the de-packetization buffer sprop-depack-buf-bytes. 🍓
- To enable the offerer and answerer to inform each other about their capabilities for de-packetization buffering in receiving RTP streams, both parties are RECOMMENDED to include depack-buf-cap. 💜
- The parameters sprop-dci, sprop-vps, sprop-sps, or sprop-pps, when present (included in the "a=fmtp" line of SDP or conveyed using the "fmtp" source attribute, as specified in Section 6.3 of [RFC5576]), are used for out-of-band transport of the parameter sets (DCI, VPS, SPS, or PPS, respectively). ✨
- The answerer MAY use either out-of-band or in-band transport of parameter sets for the bitstream it is sending, regardless of whether out-of-band parameter sets transport has been used in the offerer-to-answerer direction. Parameter sets included in an answer are independent of those included in the offer, because they are used for decoding two different bitstreams: one from the answerer to the offerer and the other in the opposite direction. In case some RTP packets are sent before the SDP offer/answer settles down, in-band parameter sets MUST be used for those RTP stream parts sent before the SDP offer/answer. 🔥
- The following rules apply to transport of parameter sets in the offerer-to-answerer direction. 👑
- An offer MAY include sprop-dci, sprop-vps, sprop-sps, and/or sprop-pps. If none of these parameters are present in the offer, then only in-band transport of parameter sets is used. 💅
- If the level to use in the offerer-to-answerer direction is equal to the default level in the offer, the answerer MUST be prepared to use the parameter sets included in sprop-vps, sprop-sps, and sprop-pps (either included in the "a=fmtp" line of SDP or conveyed using the "fmtp" source attribute) for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. Otherwise, the answerer MUST ignore sprop-vps, sprop-sps, and sprop-pps (either included in the "a=fmtp" line of SDP or conveyed using the "fmtp" source attribute) and the offerer MUST transmit parameter sets in-band. 🙈
- The following rules apply to transport of parameter sets in the answerer-to-offerer direction. 🌈
- An answer MAY include sprop-dci, sprop-vps, sprop-sps, and/or sprop-pps. If none of these parameters are present in the answer, then only in-band transport of parameter sets is used. 🦋
- The offerer MUST be prepared to use the parameter sets included in sprop-vps, sprop-sps, and sprop-pps (either included in the "a=fmtp" line of SDP or conveyed using the "fmtp" source attribute) for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. 🎀
- When sprop-dci, sprop-vps, sprop-sps, and/or sprop-pps are conveyed using the "fmtp" source attribute, as specified in Section 6.3 of [RFC5576], the receiver of the parameters MUST store the parameter sets included in sprop-dci, sprop-vps, sprop-sps, and/or sprop-pps and associate them with the source given as part of the "fmtp" source attribute. Parameter sets associated with one source (given as part of the "fmtp" source attribute) MUST only be used to decode NAL units conveyed in RTP packets from the same source (given as part of the "fmtp" source attribute). When this mechanism is in use, SSRC collision detection and resolution MUST be performed as specified in [RFC5576]. 🔐
Figure 11 lists the interpretation of all the parameters that MAY be used for the various combinations of offer, answer, and direction attributes. 📋
sendonly --+
answer: recvonly, recv-ols-id --+ |
recvonly w/o recv-ols-id --+ | |
answer: sendrecv, recv-ols-id --+ | | |
sendrecv w/o recv-ols-id --+ | | | |
| | | | |
profile-id C D C D P
tier-flag C D C D P
level-id D D D D P
sub-profile-id C D C D P
interop-constraints C D C D P
max-recv-level-id R R R R -
sprop-max-don-diff P P - - P
sprop-depack-buf-bytes P P - - P
depack-buf-cap R R R R -
max-lsr R R R R -
max-fps R R R R -
sprop-dci P P - - P
sprop-sei P P - - P
sprop-vps P P - - P
sprop-sps P P - - P
sprop-pps P P - - P
sprop-sublayer-id P P - - P
recv-sublayer-id O O O O -
sprop-ols-id P P - - P
recv-ols-id X O X O -Legend:
C: configuration for sending and receiving bitstreams
D: changeable configuration, same as C, except possible
to answer with a different but consistent value (see the
semantics of the six parameters related to profile, tier,
and level on these parameters being consistent)
P: properties of the bitstream to be sent
R: receiver capabilities
O: operation point selection
X: MUST NOT be present
-: not usable, when present MUST be ignored Figure 11: Interpretation of Parameters for Various Combinations
of Offers, Answers, and Direction Attributes, with and without
recv-ols-id.Parameters used for declaring receiver capabilities are, in general, downgradable, i.e., they express the upper limit for a sender's possible behavior. Thus, a sender MAY select to set its encoder using only lower/lesser or equal values of these parameters. 🍰
When the answer does not include a recv-ols-id that is less than the sprop-ols-id in the offer, parameters declaring a configuration point are not changeable, with the exception of the level-id parameter for unicast usage, and these parameters express values a receiver expects to be used and MUST be used verbatim in the answer as in the offer. 🔒
When a sender's capabilities are declared with the configuration parameters, these parameters express a configuration that is acceptable for the sender to receive bitstreams. In order to achieve high interoperability levels, it is often advisable to offer multiple alternative configurations. It is impossible to offer multiple configurations in a single payload type. Thus, when multiple configuration offers are made, each offer requires its own RTP payload type associated with the offer. However, it is possible to offer multiple operation points using one configuration in a single payload type by including sprop-vps in the offer and recv-ols-id in the answer. 🌟
An implementation SHOULD be able to understand all media type parameters (including all optional media type parameters), even if it doesn't support the functionality related to the parameter. This, in conjunction with proper application logic in the implementation, allows the implementation, after having received an offer, to create an answer by potentially downgrading one or more of the optional parameters to the point where the implementation can cope, leading to higher chances of interoperability beyond the most basic interop points (for which, as described above, no optional parameters are necessary). 🎉
Informative note: In implementations of previous H.26x payload formats, it was occasionally observed that implementations were incapable of parsing most (or all) of the optional parameters. As a result, the offer/answer exchange resulted in a baseline performance (using the default values for the optional parameters) with the resulting suboptimal user experience. However, there are valid reasons to forego the implementation complexity of implementing the parsing of some or all of the optional parameters, for example, when there is predetermined knowledge, not negotiated by an SDP-based offer/answer process, of the capabilities of the involved systems (walled gardens, baseline requirements defined in application standards higher up in the stack, and similar). 🫠
An answerer MAY extend the offer with additional media format configurations. However, to enable their usage, in most cases, a second offer is required from the offerer to provide the bitstream property parameters that the media sender will use. This also has the effect that the offerer has to be able to receive this media format configuration, not only to send it. 💌
7.3.3. Multicast ✨
okay bestie, so when the bitstream is getting delivered over multicast, these rules apply 🫶
- The media format configuration is identified by
profile-id,tier-flag,sub-profile-id,level-id, andinterop-constraints. These media format configuration parameters, includinglevel-id, MUST be used symmetrically; that is, the answerer MUST either maintain all configuration parameters or remove the media format (payload type) completely. Note that this implies that thelevel-idfor offer/answer in multicast is not changeable. 💅 - To simplify the handling and matching of these configurations, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [RFC3264]. An answer MUST NOT contain a payload type number used in the offer unless the configuration is the same as in the offer. 🎀
- Parameter sets received MUST be associated with the originating source and MUST only be used in decoding the incoming bitstream from the same source. 🔒
- The rules for other parameters are the same as above for unicast as long as the three above rules are obeyed. 🦋
7.3.4. Usage in Declarative Session Descriptions
when VVC over RTP is offered with SDP in a declarative style, like in Real Time Streaming Protocol (RTSP) [RFC7826] or Session Announcement Protocol (SAP) [RFC2974], there are some considerations you need to know about 📝
- All parameters capable of indicating both bitstream properties and receiver capabilities are used to indicate only bitstream properties. For example, in this case, the parameters
profile-id,tier-id, andlevel-iddeclare the values used by the bitstream, not the capabilities for receiving bitstreams. As a result, the following interpretation of the parameters MUST be used:- Declaring actual configuration or bitstream properties:
profile-idtier-flaglevel-idinterop-constraintssub-profile-idsprop-dcisprop-vpssprop-spssprop-ppssprop-max-don-diffsprop-depack-buf-bytessprop-sublayer-idsprop-ols-idsprop-sei
- Not usable (when present, they MUST be ignored):
max-lsrmax-fpsmax-recv-level-iddepack-buf-caprecv-sublayer-idrecv-ols-id
- A receiver of the SDP is required to support all parameters and values of the parameters provided; otherwise, the receiver MUST reject (RTSP) or not participate in (SAP) the session. It falls on the creator of the session to use values that are expected to be supported by the receiving application. 🫠
- Declaring actual configuration or bitstream properties:
7.3.5. Considerations for Parameter Sets
when out-of-band transport of parameter sets is used, parameter sets MAY still be additionally transported in-band unless explicitly disallowed by an application, and some of these additional parameter sets may update some of the out-of-band transported parameter sets. an update of a parameter set refers to the sending of a parameter set of the same type using the same parameter set ID but with different values for at least one other parameter of the parameter set. 🌸
8. Use with Feedback Messages 💌
the following subsections define the use of the Picture Loss Indication (PLI) and Full Intra Request (FIR) feedback messages with [VVC]. The PLI is defined in [RFC4585], and the FIR message is defined in [RFC5104]. In accordance with this memo, unlike [HEVC], a sender MUST NOT send Slice Loss Indication (SLI) or Reference Picture Selection Indication (RPSI), and a receiver SHOULD ignore RPSI and treat a received SLI as a PLI. 🙅♀️
8.1. Picture Loss Indication (PLI)
As specified in Section 6.3.1 of [RFC4585], the reception of a PLI by a media sender indicates "the loss of an undefined amount of coded video data belonging to one or more pictures". Without having any specific knowledge of the setup of the bitstream (such as use and location of in-band parameter sets, non-IRAP decoder refresh points, picture structures, and so forth), a reaction to the reception of a PLI by a VVC sender SHOULD be to send an IRAP picture and relevant parameter sets, potentially with sufficient redundancy so to ensure correct reception. However, sometimes information about the bitstream structure is known. For example, such information can be parameter sets that have been conveyed out of band through mechanisms not defined in this document and that are known to stay static for the duration of the session. In that case, it is obviously unnecessary to send them in-band as a result of the reception of a PLI. Other examples could be devised based on a priori knowledge of different aspects of the bitstream structure. In all cases, the timing and congestion control mechanisms of [RFC4585] MUST be observed. ⏱️
8.2. Full Intra Request (FIR)
the purpose of the FIR message is to force an encoder to send an independent decoder refresh point as soon as possible while observing applicable congestion-control-related constraints, such as those set out in [RFC8082]. 🚀
Upon reception of a FIR, a sender MUST send an IDR picture. Parameter sets MUST also be sent, except when there is a priori knowledge that the parameter sets have been correctly established. A typical example for that is an understanding between the sender and receiver, established by means outside this document, that parameter sets are exclusively sent out of band. 🤝
9. Security Considerations 🔐
the scope of this section is limited to the payload format itself and to one feature of [VVC] that may pose a particularly serious security risk if implemented naively. The payload format, in isolation, does not form a complete system. Implementers are advised to read and understand relevant security-related documents, especially those pertaining to RTP (see the Security Considerations section in [RFC3550]) and the security of the call-control stack chosen (that may make use of the media type registration of this memo). Implementers should also consider known security vulnerabilities of video coding and decoding implementations in general and avoid those. 😬
Within this RTP payload format, and with the exception of the user data SEI message as described below, no security threats other than those common to RTP payload formats are known. In other words, neither the various media-plane-based mechanisms nor the signaling part of this memo seem to pose a security risk beyond those common to all RTP-based systems. ✨
RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550] and in any applicable RTP profile, such as RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/SAVPF [RFC5124]. However, as "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] discusses, it is not an RTP payload format's responsibility to discuss or mandate what solutions are used to meet the basic security goals, like confidentiality, integrity, and source authenticity for RTP in general. This responsibility lays on anyone using RTP in an application. They can find guidance on available security mechanisms and important considerations in "Options for Securing RTP Sessions" [RFC7201]. The rest of this section discusses the security impacting properties of the payload format itself. 🧐
Because the data compression used with this payload format is applied end to end, any encryption needs to be performed after compression. A potential denial-of-service threat exists for data encodings using compression techniques that have non-uniform receiver-end computational load. The attacker can inject pathological datagrams into the bitstream that are complex to decode and that cause the receiver to be overloaded. [VVC] is particularly vulnerable to such attacks, as it is extremely simple to generate datagrams containing NAL units that affect the decoding process of many future NAL units. Therefore, the usage of data origin authentication and data integrity protection of at least the RTP packet is RECOMMENDED but NOT REQUIRED based on the thoughts of [RFC7202]. 🚨
Like HEVC [RFC7798], [VVC] includes a user data Supplemental Enhancement Information (SEI) message. This SEI message allows inclusion of an arbitrary bitstring into the video bitstream. Such a bitstring could include JavaScript, machine code, and other active content. [VVC] leaves the handling of this SEI message to the receiving system. In order to avoid harmful side effects of the user data SEI message, decoder implementations cannot naively trust its content. For example, it would be a bad and insecure implementation practice to forward any JavaScript a decoder implementation detects to a web browser. The safest way to deal with user data SEI messages is to simply discard them, but that can have negative side effects on the quality of experience by the user. 🙈
End-to-end security with authentication, integrity, or confidentiality protection will prevent a MANE from performing media-aware operations other than discarding complete packets. In the case of confidentiality protection, it will even be prevented from discarding packets in a media-aware way. To be allowed to perform such operations, a MANE is required to be a trusted entity that is included in the security context establishment. This on-path inclusion of the MANE forgoes end-to-end security guarantees for the end points. 💀
10. Congestion Control 🚦
Okay bestie, congestion control is the whole vibe of not flooding the network 🌊. Using it for RTP SHALL follow [RFC3550] and any RTP profile that applies, like AVP [RFC3551] or AVPF [RFC4585]. If you're riding best-effort service, there's one extra thing you MUST do: keep an eye on packet loss so the loss rate stays in a range that's acceptable 👀. Loss counts as acceptable when a TCP flow on the same network path, dealing with the same network conditions, would get an average throughput (measured over a reasonable timescale) that's no less than what all your RTP streams combined are getting. You can meet this by adapting your transmission rate, by adjusting how many layers you subscribe to in a layered multicast session, or by having a receiver leave the session if the loss rate gets unacceptably high 🙅♀️.
Adapting bitrate is super doable when you're encoding in real time. You can just tune the quantization parameter and you're good 🎀. Pre-encoded content is more of a challenge, though, because the pre-coded bitstream has to be built for adaptivity in the first place 😭. The key tools in [VVC] are temporal scalability and spatial/SNR scalability ✨. A media sender can drop NAL units from higher temporal sublayers (the ones with a high TID value) or from higher spatio-SNR layers until the sending bitrate falls into an acceptable range 💅.
These mechanisms generally work inside a defined profile and level, so you don't need to renegotiate the channel 🦋. Only when non-downgradable parameters, like the profile, have to change do you need to end and restart the RTP stream(s). Using different RTP payload types can make that happen 🧚.
MANEs MAY remove certain unusable packets from an RTP stream that got damaged by earlier packet losses 🧹. This can cut network load in some special cases. For example, a MANE can remove FUs whose leading FUs from the same NAL unit were lost, or dependent slice segments whose leading slice segments from the same slice were lost, since the trailing FUs or dependent slice segments are meaningless to most decoders 🙈. A MANE can also remove higher temporal scalable layers if its outbound transmission is experiencing congestion 💜.
11. IANA Considerations 📝
Quick one: a new media type has been registered with IANA. See Section 7.1 🎀.
12. References 📚
12.1. Normative References
Here are the normative refs, sis, bookmark these 🔖:
[ISO23090-3]
International Organization for Standardization,
"Information technology - Coded representation of
immersive media - Part 3: Versatile video coding", ISO/
IEC 23090-3:2022, September 2022,
<https://www.iso.org/standard/73022.html>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model
with Session Description Protocol (SDP)", RFC 3264,
DOI 10.17487/RFC3264, June 2002,
<https://www.rfc-editor.org/info/rfc3264>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
DOI 10.17487/RFC3551, July 2003,
<https://www.rfc-editor.org/info/rfc3551>.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)",
RFC 3711, DOI 10.17487/RFC3711, March 2004,
<https://www.rfc-editor.org/info/rfc3711>.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585,
DOI 10.17487/RFC4585, July 2006,
<https://www.rfc-editor.org/info/rfc4585>.
[RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data
Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006,
<https://www.rfc-editor.org/info/rfc4648>.
[RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman,
"Codec Control Messages in the RTP Audio-Visual Profile
with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104,
February 2008, <https://www.rfc-editor.org/info/rfc5104>.
[RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for
Real-time Transport Control Protocol (RTCP)-Based Feedback
(RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February
2008, <https://www.rfc-editor.org/info/rfc5124>.
[RFC5576] Lennox, J., Ott, J., and T. Schierl, "Source-Specific
Media Attributes in the Session Description Protocol
(SDP)", RFC 5576, DOI 10.17487/RFC5576, June 2009,
<https://www.rfc-editor.org/info/rfc5576>.
[RFC8082] Wenger, S., Lennox, J., Burman, B., and M. Westerlund,
"Using Codec Control Messages in the RTP Audio-Visual
Profile with Feedback with Layered Codecs", RFC 8082,
DOI 10.17487/RFC8082, March 2017,
<https://www.rfc-editor.org/info/rfc8082>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8866] Begen, A., Kyzivat, P., Perkins, C., and M. Handley, "SDP:
Session Description Protocol", RFC 8866,
DOI 10.17487/RFC8866, January 2021,
<https://www.rfc-editor.org/info/rfc8866>.
[VSEI] ITU-T, "Versatile supplemental enhancement information
messages for coded video bitstreams", ITU-T
Recommendation H.274, May 2022,
<https://www.itu.int/rec/T-REC-H.274>.
[VVC] ITU-T, "Versatile Video Coding", ITU-T
Recommendation H.266, April 2022,
<http://www.itu.int/rec/T-REC-H.266>.12.2. Informative References
And now the informative ones, the nice-to-read extras 📖✨:
[CABAC] Sole, J., et al., "Transform coefficient coding in HEVC",
IEEE Transactions on Circuits and Systems for Video
Technology, DOI 10.1109/TCSVT.2012.2223055, December 2012,
<https://doi.org/10.1109/TCSVT.2012.2223055>.
[HEVC] ITU-T, "High efficiency video coding", ITU-T
Recommendation H.265, August 2021,
<https://www.itu.int/rec/T-REC-H.265>.
[MPEG2S] International Organization for Standardization,
"Information technology - Generic coding of moving
pictures and associated audio information - Part 1:
Systems", ISO/IEC 13818-1:2022, September 2022.
[RFC2974] Handley, M., Perkins, C., and E. Whelan, "Session
Announcement Protocol", RFC 2974, DOI 10.17487/RFC2974,
October 2000, <https://www.rfc-editor.org/info/rfc2974>.
[RFC6184] Wang, Y.-K., Even, R., Kristensen, T., and R. Jesup, "RTP
Payload Format for H.264 Video", RFC 6184,
DOI 10.17487/RFC6184, May 2011,
<https://www.rfc-editor.org/info/rfc6184>.
[RFC6190] Wenger, S., Wang, Y.-K., Schierl, T., and A.
Eleftheriadis, "RTP Payload Format for Scalable Video
Coding", RFC 6190, DOI 10.17487/RFC6190, May 2011,
<https://www.rfc-editor.org/info/rfc6190>.
[RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP
Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014,
<https://www.rfc-editor.org/info/rfc7201>.
[RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP
Framework: Why RTP Does Not Mandate a Single Media
Security Solution", RFC 7202, DOI 10.17487/RFC7202, April
2014, <https://www.rfc-editor.org/info/rfc7202>.
[RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and
B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms
for Real-Time Transport Protocol (RTP) Sources", RFC 7656,
DOI 10.17487/RFC7656, November 2015,
<https://www.rfc-editor.org/info/rfc7656>.
[RFC7667] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667,
DOI 10.17487/RFC7667, November 2015,
<https://www.rfc-editor.org/info/rfc7667>.
[RFC7798] Wang, Y.-K., Sanchez, Y., Schierl, T., Wenger, S., and M.
M. Hannuksela, "RTP Payload Format for High Efficiency
Video Coding (HEVC)", RFC 7798, DOI 10.17487/RFC7798,
March 2016, <https://www.rfc-editor.org/info/rfc7798>.
[RFC7826] Schulzrinne, H., Rao, A., Lanphier, R., Westerlund, M.,
and M. Stiemerling, Ed., "Real-Time Streaming Protocol
Version 2.0", RFC 7826, DOI 10.17487/RFC7826, December
2016, <https://www.rfc-editor.org/info/rfc7826>.Acknowledgements 🫶
Huge thanks to Dr. Byeongdoo Choi for the video-codec technical discussion and everything else in this memo 💖. Xin Zhao and Dr. Xiang Li contributed the descriptive content for the [VVC] specification, so they get major credit 👑. Spencer Dawkins gave valuable review comments that made this memo so much better 🌸. Some parts of this specification share text with the RTP payload format for HEVC [RFC7798], and we're grateful to its authors for their excellent work 🙌.
Authors' Addresses 💌
Here's who wrote this, with their contact details 💜:
Shuai Zhao
Intel
2200 Mission College Blvd
Santa Clara, 95054
United States of America
Email: shuai.zhao@ieee.org
Stephan Wenger
Tencent
2747 Park Blvd
Palo Alto, 94588
United States of America
Email: stewe@stewe.org
Yago Sanchez
Fraunhofer HHI
Einsteinufer 37
10587 Berlin
Germany
Email: yago.sanchez@hhi.fraunhofer.de
Ye-Kui Wang
Bytedance Inc.
8910 University Center Lane
San Diego, 92122
United States of America
Email: yekui.wang@bytedance.com
Miska M. Hannuksela
Nokia Technologies
Hatanpään valtatie 30
FI-33100 Tampere
Finland
Email: miska.hannuksela@nokia.com📦 where this page comes from
- ietf: rfc9328 2022-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms