RFC 9584
RTP Payload Format for Essential Video Coding (EVC)
RFC 9584 ✨: RTP Payload Format for Essential Video Coding (EVC) 🎀
Okay bestie, so this is the spec that tells RTP how to carry EVC video 🎬💖. It's an Internet Standards Track doc from the IETF, published June 2024, and it's giving "approved by the IESG, reviewed by the community, fully official" 💅.
The authors are S. Zhao (Intel) 🌸, S. Wenger (Tencent) 🦋, and Y. Lim (Samsung Electronics) 🧚. The status is ISSN 2070-1721 ✨. It doesn't obsolete or update any other RFCs, so it's a clean fresh start 🌈.
Abstract
Okay so the abstract is literally the vibe check 💅. EVC is the video codec published as ISO/IEC International Standard 23094-1, and it was developed by MPEG 🎞️. This payload format lets you pack one or more Network Abstraction Layer (NAL) units into a single RTP packet payload 📦, and it also lets you chop a single NAL unit up across multiple RTP packets when it's too big 🔪. It's basically built for videoconferencing 📹, internet video streaming 📺, and high-bitrate entertainment-quality video 🍿. Iconic range 🔥.
Status of This Memo
This is an Internet Standards Track document 💜.
It's a product of the IETF 🌈, it represents the community consensus ✨, it went through public review 👀, and the IESG approved it for publication 🫶. If you want more on Internet Standards, check Section 2 of RFC 7841 📚.
For the current status, any errata, and how to give feedback, go to https://www.rfc-editor.org/info/rfc9584 🔗.
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the document authors 💅. All rights reserved 🔒.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info), as in effect on the publication date 📜. Please read these carefully, because they spell out your rights and restrictions 🙈. Any Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Revised BSD License describes 💀.
Table of Contents 🗺️
- 1. Introduction
- 1.1. Overview of the EVC Codec
- 1.1.1. Coding-Tool Features (Informative)
- 1.1.2. Systems and Transport Interfaces
- 1.1.3. Parallel Processing Support (Informative)
- 1.1.4. NAL Unit Header
- 1.2. Overview of the Payload Format
- 1.1. Overview of the EVC Codec
- 2. Conventions
- 3. Definitions and Abbreviations
- 3.1. Definitions
- 3.1.1. Definitions from the EVC Standard
- 3.1.2. Definitions Specific to This Document
- 3.2. Abbreviations
- 3.1. Definitions
- 4. RTP Payload Format
- 4.1. RTP Header Usage
- 4.2. Payload Header Usage
- 4.3. Payload Structures
- 4.3.1. Single NAL Unit Packets
- 4.3.2. Aggregation Packets (APs)
- 4.3.3. Fragmentation Units (FUs)
- 4.4. Decoding Order Number
- 5. Packetization Rules
- 6. De-packetization Process
- 7. Payload Format Parameters
- 7.1. Media Type Registration
- 7.2. Optional Parameters Definition
- 7.3. SDP Parameters
- 7.3.1. Mapping of Payload Type Parameters to SDP
- 7.3.2. Usage with SDP Offer/Answer Model
- 7.3.3. Multicast
- 7.3.4. Usage in Declarative Session Descriptions
- 7.3.5. Considerations for Parameter Sets
- 8. Use with Feedback Messages
- 8.1. Picture Loss Indication (PLI)
- 8.2. Full Intra Request (FIR)
- 9. Security Considerations
- 10. Congestion Control
- 11. IANA Considerations
- 12. References
- 12.1. Normative References
- 12.2. Informative References
Acknowledgements
Authors' Addresses
1. Introduction 🌸
So the Essential Video Coding [EVC] standard, formally known as ISO/IEC International Standard 23094-1 [EVC], came out in 2020 🎉. One of MPEG's big goals is keeping EVC's Baseline profile essentially royalty-free 💸. It does that by using technology that was published more than 20 years ago, or that's otherwise known to be usable without paying royalties 🙌. The more advanced profiles follow a reasonable and non-discriminatory licensing terms policy 📝. Both the Baseline profile and the higher profiles are reported to beat High Efficiency Video Coding [HEVC] and Advanced Video Coding [AVC] on coding efficiency under certain configurations 📈✨. Lowkey impressive 😭.
This document describes an RTP payload format for EVC 🎀. It shares its basic design with the NAL unit-based RTP payload formats for H.264 Video Coding [RFC6184], Scalable Video Coding (SVC) [RFC6190], High Efficiency Video Coding (HEVC) [RFC7798], and Versatile Video Coding (VVC) [RFC9328] 🦋. In terms of design philosophy, security, congestion control, and overall implementation complexity, it has similar properties to those earlier specs 💅. That's a conscious choice 🫶, because the RTP Payload Format for H.264 video in [RFC6184] is widely deployed and generally known to implementers 🙌. Some mechanisms from [RFC6190] were brought in, since EVC supports temporal scalability ⏳. EVC doesn't offer higher forms of scalability yet 🙈.
1.1. Overview of the EVC Codec 🎬
The codings described in [EVC], [AVC], [HEVC], and [VVC] all share a similar hybrid video codec design 🧩. This document gives a very brief overview of the EVC features that the payload format addresses in some form 🍓. Implementers have to read, understand, and apply the ISO/IEC standard for EVC [EVC] to get interoperable, well-performing implementations 🔧. EVC has a Baseline profile and a Main profile, and the Main profile is a superset of Baseline that adds more advanced features 🌟. EVC also has still image variants of both Baseline and Main profiles, where the bitstream is restricted to a single IDR picture 🖼️. EVC supports certain walled garden implementations under intellectual property constraints by including syntax elements that let encoders mark which of the many independent coding tools a bitstream uses, similar in spirit to the general_constraint_info of [VVC] 🔐💜.
Conceptually, EVC, AVC, HEVC, and VVC all include a Video Coding Layer (VCL), which people often use to refer to the coding-tool features 🛠️. They also include a Network Abstraction Layer (NAL), which usually refers to the systems and transport interface aspects of the codecs 🌐. Iykyk 💀.
Part 1 of 13 done 🎀. Say the word and I'll keep going with Section 1.1.1 and onward 💖.
1.1.1. Coding-Tool Features (Informative)
Coding blocks and transform structure 🧱
okay so bestie, EVC uses a traditional block-based coding structure 🎀 where the encoded image gets split into blocks of up to 64x64 luma samples for the Baseline profile and 128x128 luma samples for the Main profile, and those blocks can be recursively divided into smaller blocks 💅. The Baseline profiles use HEVC-like quad-tree-blocks partitioning, so a block can be divided horizontally and vertically into four smaller square blocks 🦋. The Main profile adds two advanced coding structure tools ✨: 1) Binary Ternary Tree (BTT) partitioning, which allows non-square coding units, and 2) Split Unit Coding Order segmentation, which changes the processing order of the blocks from the traditional left-to-right and top-to-bottom scanning order to an alternative right-to-left and bottom-to-top scanning order 🔥. In the Main profile, the picture can be divided into slices and tiles, which can be independently encoded and/or decoded in parallel 👑.
EVC also uses a traditional video codec prediction model 🌈 with two general types of predictions: Intra (spatial) and Inter (temporal) predictions 🧚. A residue block is calculated by subtracting predicted data from the original (encoded) one 🍓. The Baseline profile allows only discrete cosine transform (DCT-2) and scalar quantization to transform and quantize residue data 🫶, whereas the Main profile additionally has options to use discrete sine transform (DST-7) and another type of discrete cosine transform (DCT-8) 💜. Also, for the Main profile, Improved Quantization and Transform (IQT) uses a different mapping or clipping function for quantization ⭐. An inverse zig-zag scanning order is used for coefficient coding 🌸. Advanced Coefficient Coding (ADCC) in the Main profile can code coefficient values more efficiently, for example, indicated by the last non-zero coefficient 😭. The Baseline profile uses a straightforward RLE-based approach to encode the quantized coefficients 💀.
Entropy coding 🫧
EVC uses a similar binary arithmetic coding mechanism as HEVC CABAC (context adaptive binary arithmetic coding) and VVC 💖. The mechanism includes a binarization step and a probability update defined by a lookup table 🙈. In the Main profile, the derivation process of syntax elements based on adjacent blocks makes the context modeling and initialization process more efficient 🎀.
In-loop filtering 🪞
The Baseline profile of EVC uses the deblocking filter defined in H.263 Annex J [VIDEO-CODING] 🌷. In the Main profile, an Advanced Deblocking Filter (ADDB) can be used as an alternative, which can further reduce undesirable compression artifacts ✨. The Main profile also defines two additional in-loop filters that can be used to improve the quality of decoded pictures before output and/or for Inter prediction 👑. A Hadamard Transform Domain Filter (HTDF) is applied to the luma samples before deblocking, and a lookup table is used to determine four adjacent samples for filtering 🔥. An adaptive Loop Filter (ALF) allows signals of up to 25 different filters to be sent for the luma components; the best filter can be selected through the classification process for each 4x4 block 🦋. Similarly to VVC, the filter parameters of ALF are signaled in the Adaptation Parameter Set (APS) 🧚.
Inter prediction 🚀
The basis of EVC's Inter prediction is motion compensation using interpolation filters with a quarter sample resolution 🌈. In the Baseline profile, a motion vector is transmitted using one of three spatially neighboring motion vectors and a temporally collocated motion vector as a predictor 🍓. A motion vector difference may be signaled relative to the selected predictor, but there is a case where no motion vector difference is signaled, and there is no remaining data in the block 🙈. This mode is called a "skip" mode 💅. The Main profile includes six additional tools to provide improved Inter prediction 💖. With Advanced Motion Vectors Prediction (ADMVP), adjacent blocks can be conceptually merged to indicate that they use the same motion, but more advanced schemes can also be used to create predictions from the basic model list of candidate predictors 🫶. The Merge with Motion Vector Difference (MMVD) tool uses a process similar to the concept of merging neighboring blocks, but also allows the use of expressions that include a starting point, motion amplitude, and direction of motion to send a motion vector signal 🌸. Using Advanced Motion Vector Prediction (AMVP), candidate motion vector predictions for the block can be derived from its neighboring blocks in the same picture and collocated blocks in the reference picture 💜. The Adaptive Motion Vector Resolution (AMVR) tool provides a way to reduce the accuracy of a motion vector from a quarter sample to half sample, full sample, double sample, or quad sample, which provides an efficiency advantage, such as when sending large motion vector differences 😭. The Main profile also includes the Decoder-side Motion Vector Refinement (DMVR), which uses a bilateral template matching process to refine the motion vectors without additional signaling 👑.
Intra prediction and intra coding 🎨
Intra prediction in EVC is performed on adjacent samples of coding units in a partitioned structure 🌷. For the Baseline profile, when all coding units are square, there are five different prediction modes: DC (mean value of the neighborhood), horizontal, vertical, and two different diagonal directions 🧚. In the Main profile, intra prediction can be applied to any rectangular coding unit, and 28 additional direction modes are available in the Enhanced Intra Prediction Directions (EIPDs) ✨. Also in the Main profile, an encoder can use Intra Block Copy (IBC), where previously decoded sample blocks of the same picture are used as a predictor 🔥. A displacement vector in integer sample precision is signaled to indicate where the prediction block in the current picture is used for this mode 💖.
Reference frames management 📚
In EVC, decoded pictures can be stored in a decoded picture buffer (DPB) for predicting pictures that follow them in the decoding order 🫶. In the Baseline profile, the management of the DPB (i.e., the process of adding and deleting reference pictures) is controlled by a straightforward AVC-like sliding window approach with very few parameters from the sequence parameter set (SPS) 🌸. For the Main profile, DPB management can be handled much more flexibly using explicitly signaled Reference Picture Lists (RPLs) in the SPS or slice level 👑.
1.1.2. Systems and Transport Interfaces ✨
okay so EVC basically borrowed the systems and transport interface designs from AVC and HEVC, and bestie, it's giving familiar 💅 We get NAL-unit-based syntax, a hierarchical syntax and data unit structure, and the Supplemental Enhancement Information (SEI) message mechanism. The hierarchy goes: one sequence-level parameter set (SPS), two picture-level parameter sets (PPS and APS, where each one can apply to one or more pictures), then slice-level header parameters, then lower-level parameters. Layers all the way down 🧁
Some key components that shaped the NAL design of EVC, and this document, are below 👇
Sequence parameter set The Sequence Parameter Set (SPS) holds syntax elements for a Coded Video Sequence (CVS). A CVS is a group of pictures that starts with a random access point picture and is followed by zero or more pictures that may depend on each other and on that random access point picture. In MPEG-2, the equivalent is a Group of Pictures (GOP), which usually starts with an I frame and then has P and B frames. EVC has more options for random access points, but it keeps this basic idea 🎀 In a lot of TV-like apps, a CVS is a few hundred milliseconds to a few seconds of video. In video conferencing (without switching Multipoint Control Units (MCUs) involved), a CVS can last as long as the whole session. Long CVS, who dis? 🙈
Picture and adaptation parameter set The Picture Parameter Set (PPS) and the Adaptation Parameter Set (APS) carry information about a single picture. The PPS has info that's likely to stay the same from picture to picture, at least for pictures of a certain type. The APS has stuff that's likely to change from picture to picture, like adaptive loop filter coefficients 🦋
Profile, level, and toolsets Profiles and levels follow the same design ideas we know from AVC, HEVC, and video codecs as old as MPEG-1 Video, so no surprises here 🌸 A profile is a set of tools (not the same thing as the "toolset" we'll get to in a sec) that a decoder compliant with that profile has to support. In EVC, profiles are defined in Annex A of [EVC]. Formally, they're a set of constraints that a bitstream needs to conform to. The Baseline profile is way more heavily constrained than the Main profile, which cuts down implementation complexity. Levels are about bitstream complexity in things like maximum sample decoding rate, maximum picture size, and similar parameters that directly relate to computational complexity and/or memory demands 💜
Profiles and levels are signaled in the highest parameter set available, which is the SPS.
EVC also has another mechanism for coding tools, called the toolset syntax elements. These are toolset_idc_h and toolset_idc_l (located in the SPS), and they're bitmasks that let encoders indicate which coding tools they're using within the menu of profiles offered by the profile that's also signaled. Here's the tea ☕ No decoder conformance point is tied to the toolset. But a bitstream that uses a coding tool marked as not used in the toolset syntax element would be non-compliant. MPEG specifically rules out using the toolset syntax element as a conformance point. Walled garden implementations could still do it without the interoperability problems MPEG is worried about, and they could make bitstreams and decoders that don't support one or more given tools. That, in turn, may be useful for mitigating certain intellectual property-related risks. Strategic, honestly iconic 👑
Bitstream and elementary stream Above the Coded Video Sequence (CVS), EVC defines a video bitstream that can be used as an elementary stream in the MPEG systems context. For this document, the video bitstream syntax level is not relevant 🧚
Random access support EVC supports random access mechanisms based on IDR and clean random access (CRA) access units ✨
Temporal scalability support EVC supports temporal scalability through the generalized reference picture selection approach known since AVC/SVC. Up to six temporal layers are supported. The temporal layer is signaled in the NAL unit header (which co-serves as the payload header in this document), in the nuh_temporal_id field. Layers on layers, obsessed 🌈
Reference picture management EVC's reference picture management is POC-based, similar to HEVC. In the Main profile, substantially all reference picture list manipulations available in HEVC are specified, including explicit transmissions or updates of reference picture lists. EVC uses a modern VVC-like RPL approach for reference pictures management, though, and it's conceptually simpler than the HEVC one 🫶 In the Baseline profile, reference picture management is more restricted, and only allows a comparatively simple group of picture structures.
SEI Message EVC inherits many of HEVC's SEI messages. Some have syntax and/or semantics changes so they apply to EVC. On top of that, some of the codec-agnostic SEI messages from the VSEI specification [VSEI] are also mapped 🍓
1.1.3. Parallel Processing Support (Informative) 🌸
okay parallel processing, let's gooo 🏃♀️ The Baseline profile has no tools that specifically address parallel-processing support. The Main profile does include independently decodable slices for parallel processing. A slice is any rectangular region within a picture. Slices can be encoded to have coding dependencies with other slices from the previous picture, but not with other slices in the same picture. No specific support for parallel processing is specified in this RTP payload format, so keep that in mind 🙈
1.1.4. NAL Unit Header 💖
EVC keeps the NAL unit concept of [VVC] with different parameter options. EVC also uses a two-byte NAL unit header, shown in Figure 1. The payload of a NAL unit is the NAL unit minus the NAL unit header. Simple, cute, done 🎀
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| Type | TID | Reserve |E|
+-------------+-----------------+Figure 1: The Structure of the EVC NAL Unit Header
The fields in the NAL unit header mean what EVC says they mean, and I'll briefly describe them below for convenience 💅 For each field I'll give its size and the matching syntax element name in EVC.
F: 1 bit
forbidden_zero_bit: Required to be zero in EVC. Fun fact: this bit is in the NAL unit header so EVC video can be carried over MPEG-2 transport systems without start code emulations [MPEG2S]. In this document, the value 1 may be used to indicate a syntax violation. For example, that could be a NAL unit that came from aggregating a number of fragmented units of a NAL unit but is missing the last fragment, as described in Section 4.3.3. Yikes, that's a red flag 🚩
Type: 6 bits
nal_unit_type_plus1: This field lets you compute the NAL Unit Type. The NAL Unit Type (NalUnitType) equals the value in this field minus 1. In other words:
NalUnitType = nal_unit_type_plus1 - 1.The NAL unit type is detailed in Table 4 of [EVC]. If the value of NalUnitType is less than or equal to 23, the NAL unit is a VCL NAL unit. Otherwise, it's a non-VCL NAL unit. For a reference of all currently defined NAL unit types and their semantics, check Section 7.4.2.2 of [EVC]. Also, nal_unit_type_plus1 MUST NOT be zero. Don't skip this one, bestie 🙈
TID: 3 bits
nuh_temporal_id: This field specifies the temporal identifier of the NAL unit. The value of TemporalId is equal to TID. TemporalId shall be equal to 0 if it is an IDR NAL unit type (NAL unit type 1) 🧚
Reserve: 5 bits
nuh_reserved_zero_5bits: This field shall be equal to the version of the EVC standard. Values of nuh_reserved_zero_5bits greater than 0 are reserved for future use by ISO/IEC. Decoders conforming to a profile specified in Annex A of [EVC] shall ignore (i.e., remove from the bitstream and discard) all NAL units with values of nuh_reserved_zero_5bits greater than 0. Reserved means hands off, babe ✋
E: 1 bit
nuh_extension_flag: This field shall be equal to the version of the EVC standard. The value of nuh_extension_flag equal to 1 is reserved for future use by ISO/IEC. Decoders conforming to a profile specified in Annex A of [EVC] shall ignore (i.e., remove from the bitstream and discard) all NAL units with values of nuh_extension_flag equal to 1. Same energy as the reserved bits, periodt 🌸
1.2. Overview of the Payload Format 🦋
okay so this payload format defines the processes you need to transport EVC-coded data over RTP [RFC3550]. Here's what's covered:
- usage of RTP header with this payload format
- packetization of EVC-coded NAL units into RTP packets using three types of payload structures: a single NAL unit, aggregation, and fragment unit
- transmission of EVC NAL units of the same bitstream within a single RTP stream
- usage of media type parameters to be used with the Session Description Protocol (SDP) [RFC8866]
- usage of RTCP feedback messages
2. Conventions 🍓
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. So capital letters = real rules, lowercase = just vibes 💜
3. Definitions and Abbreviations 🎀
3.1. Definitions ✨
This document uses the terms and definitions of EVC. Section 3.1.1 lists relevant definitions from [EVC] for convenience. Section 3.1.2 provides definitions specific to this document 💖
3.1.1. Definitions from the EVC Standard ✨
okay so this part is basically the glossary bestie 💅 all the official EVC terms, and they're all borrowed from the EVC standard itself, so no cap, these are the real definitions 🎀
Access Unit (AU): A set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture. 🦋
Adaptation Parameter Set (APS): A syntax structure containing syntax elements that apply to zero or more slices as determined by zero or more syntax elements found in slice headers. 🌸
Bitstream: A sequence of bits, in the form of a NAL unit stream or a byte stream, that forms the representation of coded pictures and associated data forming one or more CVSs. 💜
Coded Picture: A coded representation of a picture containing all CTUs of the picture. ✨
Coded Video Sequence (CVS): A sequence of access units that consists, in decoding order, of an IDR access unit, followed by zero or more access units that are not IDR access units, including all subsequent access units up to but not including any subsequent access unit that is an IDR access unit. 👑
Coding Tree Block (CTB): An NxN block of samples for some value of N such that the division of a component into CTBs is a partitioning. 🧚
Coding Tree Unit (CTU): A CTB of luma samples, two corresponding CTBs of chroma samples of a picture that has three sample arrays, or a CTB of samples of a monochrome picture or a picture that is coded using three separate color planes and syntax structures used to code the samples. 🌈
Decoded Picture: A decoded picture is derived by decoding a coded picture. 🍓
Decoded Picture Buffer (DPB): A buffer holding decoded pictures for reference, output reordering, or output delay specified for the hypothetical reference decoder in Annex C of the [EVC] standard. 🫶
Dynamic Range Adjustment (DRA): A mapping process that is applied to the decoded picture prior to cropping and output as part of the decoding process; it is controlled by parameters conveyed in an Adaptation Parameter Set (APS). 🔥
Hypothetical Reference Decoder (HRD): A hypothetical decoder model that specifies constraints on the variability of conforming NAL unit streams or conforming byte streams that an encoding process may produce. 🙈
IDR Access Unit: An access unit in which the coded picture is an IDR picture. 🦋
IDR Picture: The coded picture for which each VCL NAL unit has NalUnitType equal to IDR_NUT. 🌸
Level: A defined set of constraints on the values that may be taken by the syntax elements and variables of this document, or the value of a transform coefficient prior to scaling. 💅
Network Abstraction Layer (NAL) Unit: A syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary. 🎀
Network Abstraction Layer (NAL) Unit Stream: A sequence of NAL units. 💖
Non-IDR Picture: A coded picture that is not an IDR picture. ✨
Non-VCL NAL Unit: A NAL unit that is not a VCL NAL unit. 💀
Picture Parameter Set (PPS): A syntax structure containing syntax elements that apply to zero or more entire coded pictures as determined by a syntax element found in each slice header. 👑
Picture Order Count (POC): A variable that is associated with each picture, uniquely identifies the associated picture among all pictures in the CVS, and (when the associated picture is to be output from the DPB) indicates the position of the associated picture in output order relative to the output order positions of the other pictures in the same CVS that are to be output from the DPB. 🌈
Raw Byte Sequence Payload (RBSP): A syntax structure containing an integer number of bytes that is encapsulated in a NAL unit and that is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and zero or more subsequent bits equal to 0. 🧚
Sequence Parameter Set (SPS): A syntax structure containing syntax elements that apply to zero or more entire CVSs as determined by the content of a syntax element found in the PPS referred to by a syntax element found in each slice header. 🍓
Slice: An integer number of tiles of a picture in the tile scan of the picture, exclusively contained in a single NAL unit. 💜
Tile: A rectangular region of CTUs within a particular tile column and a particular tile row in a picture. 🦋
Tile Column: A rectangular region of CTUs having a height equal to the height of the picture and width specified by syntax elements in the PPS. 🌸
Tile Row: A rectangular region of CTUs having a height specified by syntax elements in the PPS and a width equal to the width of the picture. 💅
Tile Scan: A specific sequential ordering of CTUs partitioning a picture in which the CTUs are ordered consecutively in CTU raster scan in a tile, whereas tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. 🫶
Video Coding Layer (VCL) NAL Unit: A collective term for coded slice NAL units and the subset of NAL units that have reserved values of NalUnitType that are classified as VCL NAL units in this document. 🔥
3.1.2. Definitions Specific to This Document ✨
okay now we're getting into the stuff that's specific to this RFC, so pay attention bestie 👀
Media-Aware Network Element (MANE): A network element, such as a middlebox, selective forwarding unit, or application-layer gateway, that is capable of parsing certain aspects of the RTP payload headers or the RTP payload and reacting to their contents. 🙈
Informative note: The concept of a MANE goes beyond normal routers or gateways in that a MANE has to be aware of the signaling (e.g., to learn about the payload type mappings of the media streams), and in that it has to be trusted when working with Secure RTP (SRTP). The advantage of using MANEs is that they allow packets to be dropped according to the needs of the media coding. For example, if a MANE has to drop packets due to congestion on a certain link, it can identify and remove those packets whose elimination produces the least adverse effect on the user experience. After dropping packets, MANEs must rewrite RTCP packets to match the changes to the RTP stream, as specified in Section 7 of [RFC3550]. 🧚
NAL unit decoding order: A NAL unit order that conforms to the constraints on NAL unit order given in Section 7.4.2.3 of [EVC] and follows the order of NAL units in the bitstream. 💖
NALU-time: The value that the RTP timestamp would have if the NAL unit would be transported in its own RTP packet. ⏰
NAL unit output order: A NAL unit order in which NAL units of different access units are in the output order of the decoded pictures corresponding to the access units, as specified in [EVC], and in which NAL units within an access unit are in their decoding order. 🌸
RTP stream: See [RFC7656]. Within the scope of this document, one RTP stream is utilized to transport an EVC bitstream, which may contain one or more temporal sub-layers. 🦋
Transmission order: The order of packets in ascending RTP sequence number order (in modulo arithmetic). Within an Aggregation Packet (AP), the NAL unit transmission order is the same as the order of appearance of NAL units in the packet. 🎀
3.2. Abbreviations 🌈
Here's the full list of abbreviations, no skipping, bestie 🫶
AU Access Unit
AP Aggregation Packet
APS Adaptation Parameter Set
ATS Adaptive Transform Selection
B Bi-predictive
CBR Constant Bit Rate
CPB Coded Picture Buffer
CTB Coding Tree Block
CTU Coding Tree Unit
CVS Coded Video Sequence
DPB Decoded Picture Buffer
HRD Hypothetical Reference Decoder
HSS Hypothetical Stream Scheduler
I Intra
IDR Instantaneous Decoding Refresh
LSB Least Significant Bit
LTRP Long-Term Reference Picture
MMVD Merge with Motion Vector Difference
MSB Most Significant Bit
NAL Network Abstraction Layer
P Predictive
POC Picture Order Count
PPS Picture Parameter Set
QP Quantization Parameter
RBSP Raw Byte Sequence Payload
RGB Red, Green, and Blue
SAR Sample Aspect Ratio
SEI Supplemental Enhancement Information
SODB String Of Data Bits
SPS Sequence Parameter Set
STRP Short-Term Reference Picture
VBR Variable Bit Rate
VCL Video Coding Layer4. RTP Payload Format 🎀
4.1. RTP Header Usage 💖
okay so the RTP header format is specified in [RFC3550], and this payload format just uses its fields in a way that matches that spec 💅 the picture is included below as Figure 2 for convenience 🦋
The RTP payload (and the settings for some RTP header bits) for APs and Fragmentation Units (FUs) are specified in Sections 4.3.2 and 4.3.3, respectively. ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: RTP Header According to RFC 3550The RTP header information to be set according to this RTP payload format is set as follows: 🌸
Marker bit (M): 1 bit
Set for the last packet of the access unit and carried in the current RTP stream. This is in line with the normal use of the M bit in video formats to allow an efficient playout buffer handling. 🫶
Payload Type (PT): 7 bits
The assignment of an RTP payload type for this new payload format is outside the scope of this document and will not be specified here. The assignment of a payload type has to be performed either through the profile used or in a dynamic way. 🧚
Sequence Number (SN): 16 bits
Set and used in accordance with [RFC3550]. 💜
Timestamp: 32 bits
The RTP timestamp is set to the sampling timestamp of the content. A 90 kHz clock rate MUST be used. If the NAL unit has no timing properties of its own (e.g., parameter sets or certain SEI NAL units), the RTP timestamp MUST be set to the RTP timestamp of the coded picture of the access unit in which the NAL unit is included. For SEI messages, this information is specified in Annex D of [EVC]. Receivers MUST use the RTP timestamp for the display process, even when the bitstream contains picture timing SEI messages or decoding unit information SEI messages as specified in [EVC]. 🔥
Heads up bestie: the 90 kHz clock rate MUST be used, and receivers MUST use the RTP timestamp for display, no exceptions 😭
Synchronization source (SSRC): 32 bits
Used to identify the source of the RTP packets. According to this document, a single SSRC is used for all parts of a single bitstream. 👑
4.2. Payload Header Usage 💖
Okay bestie, so the first two bytes of an RTP packet's payload are called the payload header 🎀 It has the exact same fields (F, TID, Reserve, and E) as the NAL unit header from Section 1.1.4, no matter what kind of payload structure you're dealing with ✨
The TID value tells you (among other things) how important an RTP packet is. Like, NAL units with bigger TID values aren't used to decode the ones with smaller TID values 🦋 A lower TID value means a higher importance 👑 More important NAL units MAY be better protected against transmission losses than less important NAL units 🫶
4.3. Payload Structures 🧚
Three different types of RTP packet payload structures are specified 💅 A receiver can identify which type it's dealing with just by looking at the Type field in the payload header 🔍
The three payload structures are:
- Single NAL unit packet: Contains a single NAL unit in the payload, and the NAL unit header of the NAL unit also serves as the payload header. This payload structure is specified in Section 4.3.1.
- Aggregation Packet (AP): Contains more than one NAL unit within one access unit. This payload structure is specified in Section 4.3.2.
- Fragmentation Unit (FU): Contains a subset of a single NAL unit. This payload structure is specified in Section 4.3.3.
4.3.1. Single NAL Unit Packets 🍓
Single NAL unit packets are the simplest vibe 😭 Each one contains exactly one NAL unit and is built from a payload header as defined in Table 4 of [EVC] (we'll call it PayloadHdr), then a conditional 16-bit DONL field (in network byte order), and then the NAL unit payload data (that's the NAL unit minus its NAL unit header) 💖 Check out Figure 3 for the full layout 👀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr | DONL (conditional) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NAL unit payload data |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: The Structure of a Single NAL Unit PacketNow the DONL field, the main drama 🎭 When it's present, it holds the value of the 16 least significant bits of the decoding order number of the contained NAL unit. If sprop-max-don-diff (defined in Section 7.2) is greater than 0, the DONL field MUST be present, and the variable DON for the contained NAL unit is derived as equal to the value of the DONL field 🔥 Otherwise (where sprop-max-don-diff is equal to 0), the DONL field MUST NOT be present 🙈
4.3.2. Aggregation Packets (APs) 🎀
okay so Aggregation Packets (APs) exist to cut down on packetization overhead for tiny NAL units 🫶 like most of the non-VCL ones, which are often only a few octets big 😭 packing them together is giving efficiency ✨
An AP gathers NAL units from one access unit, and it MUST NOT contain NAL units from more than one AU 💅 Each NAL unit going into an AP gets wrapped in its own aggregation unit, and they're lined up in NAL-unit-decoding order 🦋
An AP is made of a payload header, as defined in Table 4 of [EVC] (we're calling it PayloadHdr with Type=56 here 🧚), followed by two or more aggregation units, like Figure 4 shows 👑
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=56) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| two or more aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4: The Structure of an Aggregation Packet
Now the payload header fields of an AP are set like this 🌸 The F bit MUST be equal to 0 if the F bit of each aggregated NAL unit is equal to zero; otherwise, it MUST be equal to 1. The Type field MUST be equal to 56 ✨
The value of TID MUST be the smallest value of TID of all the aggregated NAL units. The value of Reserve and E MUST be equal to 0 for this specification 💜
✨ Informative note: All VCL NAL units in an AP have the same TID value since they belong to the same access unit. However, an AP may contain non-VCL NAL units for which the TID value in the NAL unit header may be different from the TID value of the VCL NAL units in the same AP.
An AP MUST carry at least two aggregation units, and it can carry as many as it needs 🍓 However, the total amount of data in an AP obviously MUST fit into an IP packet 💀 The size SHOULD be picked so the resulting IP packet is smaller than the path MTU size, so IP layer fragmentation gets avoided 🌈 An AP MUST NOT contain FUs specified in Section 4.3.3 🙈 And APs MUST NOT be nested, i.e., an AP cannot contain another AP 🚫
🧚 Informative note: If a receiver encounters nested APs, which is against the aforementioned requirement, it has several options, listed in order of ease of implementation: 1. ignore the nested AP; 2. ignore the nested AP and report a "packet loss" to the decoder, if such functionality exists in the API; and 3. implement support for nested APs and extract the NAL units from these nested APs.
The first aggregation unit in an AP is a conditional 16-bit DONL field (in network byte order), then a 16-bit unsigned size field (in network byte order) saying how big the NAL unit is in bytes. That size excludes these two octets but includes the NAL unit header. After that comes the NAL unit itself, header and all, as shown in Figure 5 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : DONL (conditional) | NALU size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU size | |
+-+-+-+-+-+-+-+-+ NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5: The Structure of the First Aggregation Unit in an AP
🦋 Informative note: The first octet of Figure 5 (indicated by the first colon) belongs to a previous aggregation unit. It is depicted to emphasize that aggregation units are octet aligned only. Similarly, the NAL unit carried in the aggregation unit can terminate at the octet boundary.
The DONL field, when it's present, specifies the value of the 16 least significant bits of the decoding order number of the aggregated NAL unit 🔥
Here's the tricky bit 🙈 If sprop-max-don-diff is greater than 0, the DONL field MUST be present in an aggregation unit that is the first aggregation unit in an AP. The variable DON for the aggregated NAL unit is derived as equal to the value of the DONL field. The variable Decoding Order Number (DON) for an aggregation unit that is not the first aggregation unit in an AP-aggregated NAL unit is derived as equal to the DON of the preceding aggregated NAL unit in the same AP plus 1 modulo 65536. Otherwise (where sprop-max-don-diff is equal to 0), the DONL field MUST NOT be present in an aggregation unit that is the first aggregation unit in an AP 🧠
An aggregation unit that is not the first one in an AP gets followed right away by a 16-bit unsigned size field (in network byte order). That size says how big the NAL unit is in bytes, again excluding these two octets but including the NAL unit header. Then the NAL unit itself follows, header included, as shown in Figure 6 🌟
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : NALU size | NAL unit |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: The Structure of an Aggregation Unit That Is Not the First Aggregation Unit in an AP
🌸 Informative note: The first octet of Figure 6 (indicated by the first colon) belongs to a previous aggregation unit. It is depicted to emphasize that aggregation units are octet aligned only. Similarly, the NAL unit carried in the aggregation unit can terminate at the octet boundary.
Figure 7 is an example of an AP with two aggregation units, labeled "NALU 1" and "NALU 2", and no DONL field 🍓
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=56) | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 HDR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NALU 1 Data |
| . . . |
| |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| . . . | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 HDR | |
+-+-+-+-+-+-+-+-+ NALU 2 Data |
| . . . |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7: An Example of an AP Packet Containing Two Aggregation Units without the DONL Field
Figure 8 is another example, with the same two aggregation units ("NALU 1" and "NALU 2"), but this time the DONL field IS present 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=56) | NALU 1 DONL |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| NALU 1 Data . . . |
| |
+ . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| : NALU 2 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 HDR | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NALU 2 Data |
| |
| . . . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8: An Example of an AP Containing Two Aggregation Units with the DONL Field
4.3.3. Fragmentation Units (FUs) ✨
okay bestie, so FUs exist to chop one big NAL unit into multiple RTP packets 🔪🎀. The wild part is that the sender doesn't even need the EVC encoder to cooperate or know about it 🧚. A fragment is just an integer number of consecutive octets from that NAL unit. Fragments of the same NAL unit MUST be sent in consecutive order with ascending RTP sequence numbers, and no other RTP packets from the same RTP stream can sneak in between the first and last fragment 🚫💅.
When a NAL unit gets fragmented and carried in FUs, we call it a fragmented NAL unit 💖. APs MUST NOT be fragmented, and FUs MUST NOT be nested, meaning an FU must not contain a subset of another FU. no nesting drama here 🙅♀️🌸.
The RTP timestamp of an RTP packet carrying an FU is set to the NALU-time of the fragmented NAL unit ⏰✨.
An FU is made up of a payload header as defined in Table 4 of [EVC] (denoted as PayloadHdr with Type=57), a one-octet FU header, a conditional 16-bit DONL field (in network byte order), and an FU payload 🎀. Check out the layout in Figure 9:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PayloadHdr (Type=57) | FU header | DONL (cond) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| DONL (cond) | |
|-+-+-+-+-+-+-+-+ |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 9: The Structure of an FUhere's the payload header setup 🧁. The Type field MUST be equal to 57 💜. The fields F, TID, Reserve, and E MUST be equal to the fields F, TID, Reserve, and E, respectively, of the fragmented NAL unit. so they basically copy the NAL unit's own vibes 🪞✨.
The FU header is giving three parts: an S bit, an E bit, and a 6-bit FuType field, all shown in Figure 10 🦋:
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|S|E| FuType |
+---------------+
Figure 10: The Structure of FU Headerhere's the tea on each FU header field 👑:
- S: 1 bit. When set to 1, the S bit marks the start of a fragmented NAL unit. That means the first byte of the FU payload is also the first byte of the fragmented NAL unit's payload 🌈. If the FU payload isn't the start of that payload, the S bit MUST be set to 0 😭.
- E: 1 bit. When set to 1, the E bit marks the end of a fragmented NAL unit, so the last byte of the payload is also the last byte of the fragmented NAL unit 🏁. If the FU payload isn't the last fragment, the E bit MUST be set to 0 🫶.
- FuType: 6 bits. The FuType field MUST be equal to the Type field of the fragmented NAL unit 🍓.
the DONL field, when it's there, holds the 16 least significant bits of the decoding order number of the fragmented NAL unit 🔢💖.
Now for the DONL rules, which are a whole mood 🙈. If sprop-max-don-diff is greater than 0 and the S bit is equal to 1, the DONL field MUST be present in the FU, and the variable DON for the fragmented NAL unit is derived as equal to the value of the DONL field. Otherwise (where sprop-max-don-diff is equal to 0, or where the S bit is equal to 0), the DONL field MUST NOT be present in the FU 🚫🎀.
Also, a non-fragmented NAL unit MUST NOT be transmitted in one FU. Basically the S-bit and E-bit MUST NOT both be set to 1 in the same FU header, because that would be a whole contradiction 💀.
The FU payload is made of fragments of the fragmented NAL unit's payload. If you take the FU payloads of consecutive FUs, starting with an FU whose S bit is 1 and ending with an FU whose E bit is 1, and concatenate them in order, you get the payload back 🧩✨. The NAL unit header of the fragmented NAL unit is NOT copied into the FU payload as such. Instead, its info lives in the F, TID, Reserve, and E fields of the FU payload headers, plus the FuType field of the FU header 🎁. An FU payload MUST NOT be empty 🙅♀️.
if an FU gets lost, the receiver SHOULD discard all the following fragmentation units, in transmission order, that belong to the same fragmented NAL unit 🗑️💅. The only exception is if the decoder in the receiver is known to gracefully handle incomplete NAL units 🦸♀️.
A receiver in an endpoint or a MANE MAY aggregate the first n-1 fragments of a NAL unit into an (incomplete) NAL unit, even if fragment n is not received 🧃. In that case, the forbidden_zero_bit of the NAL unit MUST be set to 1 to signal a syntax violation 🚨. Dramatic, but it's how we flag it 😤🔥.
4.4. Decoding Order Number
okay so every NAL unit gets a variable called AbsDon 🔢🌸. It represents the decoding order number, which tells you the NAL unit decoding order 🧭✨.
Let NAL unit n be the n-th NAL unit in transmission order within an RTP stream 🦋.
If sprop-max-don-diff is equal to 0, then AbsDon[n] (the value of AbsDon for NAL unit n) is simply equal to n 💖.
Otherwise (where sprop-max-don-diff is greater than 0), AbsDon[n] gets derived step by step, where DON[n] is the value of the variable DON for NAL unit n 🧮:
- If n is equal to 0 (i.e., NAL unit n is the very first NAL unit in transmission order), AbsDon[0] is set equal to DON[0] 🥇.
- Otherwise (where n is greater than 0), the following applies:
If DON[n] == DON[n-1],
AbsDon[n] = AbsDon[n-1]
If (DON[n] > DON[n-1] and DON[n] - DON[n-1] < 32768),
AbsDon[n] = AbsDon[n-1] + DON[n] - DON[n-1]
If (DON[n] < DON[n-1] and DON[n-1] - DON[n] >= 32768),
AbsDon[n] = AbsDon[n-1] + 65536 - DON[n-1] + DON[n]
If (DON[n] > DON[n-1] and DON[n] - DON[n-1] >= 32768),
AbsDon[n] = AbsDon[n-1] - (DON[n-1] + 65536 - DON[n])
If (DON[n] < DON[n-1] and DON[n-1] - DON[n] < 32768),
AbsDon[n] = AbsDon[n-1] - (DON[n-1] - DON[n])for any two NAL units, m and n, here's how the order works 🪄:
- When AbsDon[n] is greater than AbsDon[m], NAL unit n follows NAL unit m in NAL unit decoding order 🏃♀️.
- When AbsDon[n] is equal to AbsDon[m], the NAL unit decoding order of the two can be in either order 🤷♀️.
- When AbsDon[n] is less than AbsDon[m], NAL unit n precedes NAL unit m in decoding order ⏪.
Informative note: When two consecutive NAL units in the NAL unit decoding order have different values of AbsDon, the absolute difference between the two AbsDon values may be greater than or equal to 1.
Informative note: There are multiple reasons to allow the absolute difference of the values of AbsDon for two consecutive NAL units in the NAL unit decoding order to be greater than one. An increment by one is not required, because when AbsDon values are assigned to NAL units, it may not be known whether all NAL units are going to be delivered to the receiver. For example, a gateway might not forward VCL NAL units of higher sub-layers or some SEI NAL units when there is congestion in the network. In another example, the first intra-coded picture of a pre-encoded clip is transmitted in advance so it's readily available in the receiver. When transmitting the first intra-coded picture, the originator still determines how many NAL units will be encoded before the first intra-coded picture of the pre-encoded clip follows in decoding order. So the values of AbsDon for the NAL units of the first intra-coded picture of the pre-encoded clip have to be estimated when they are transmitted, and gaps in the values of AbsDon may occur.
5. Packetization Rules 🎀
okay the packetization rules are the ones that keep everything tidy 🧹💅. Here they are:
- If sprop-max-don-diff is greater than 0, the transmission order of NAL units carried in the RTP stream MAY be different from the NAL unit decoding order. Otherwise (where sprop-max-don-diff equals 0), the transmission order of NAL units carried in the RTP stream MUST be the same as the NAL unit decoding order 🔀.
- A NAL unit of small size SHOULD be encapsulated in an AP together with one or more other NAL units, so you avoid wasting packetization overhead on tiny NAL units 🐣. For example, non-VCL NAL units like access unit delimiters, parameter sets, or SEI NAL units are typically small and can often be aggregated with VCL NAL units without breaking MTU size constraints 📦✨.
- Each non-VCL NAL unit SHOULD, when possible from an MTU size match viewpoint, be encapsulated in an AP with its associated VCL NAL unit 💜. That's because a non-VCL NAL unit is typically meaningless without its associated VCL NAL unit being available 🥺.
- A single NAL unit packet MUST be used for carrying precisely one NAL unit in an RTP packet 🦄.
6. De-packetization Process ✨
okay so de-packetization is basically the art of pulling the NAL units out of the RTP packets in a stream and handing them to the decoder in NAL unit decoding order 🫶
The de-packetization process is implementation dependent 💅 so the walkthrough below is just an example of a solid implementation. Other schemes are totally valid too, as long as they give the same output for the same input. "Same output" means the set of output NAL units AND their order are both identical. Optimizations on top of these algorithms are possible ✨
All the normal RTP buffer management stuff applies 🧚 Duplicated or outdated RTP packets (spotted using the RTP sequence number and RTP timestamp) get removed 🗑️ And to figure out the exact time for decoding, you have to think about things like an intentional delay for inter-stream synchronization 🎀
NAL units with NAL unit type values in the range of 0 to 55, inclusive, may be passed to the decoder. NAL-unit-like structures with NAL unit type values in the range of 56 to 62, inclusive, MUST NOT be passed to the decoder 💀 hard boundary, bestie.
Okay, so the receiver has a receiver buffer 🌸 It's there to soak up transmission delay jitter within individual RTP streams and to reorder NAL units from transmission order into NAL unit decoding order. For this section, we're pretending there's zero transmission delay jitter within an RTP stream, just to keep things clean. To avoid mixing it up with a real-life receiver buffer (which also handles jitter), the buffer in this section is called the "de-packetization" buffer 🦋 Receivers should also prepare for jitter, either by reserving separate buffers for jitter and de-packetization, or by using one receiver buffer for both. Receivers should also take jitter into account while buffering, e.g., by adding extra initial buffering before decoding and playback starts ✨
Now the actual extraction 👑 When an RTP packet carries a single NAL unit packet, the payload of that RTP packet gets extracted as a single NAL unit, but the DONL field (that's the third and fourth bytes) gets excluded when sprop-max-don-diff is greater than 0. When an RTP packet carries an AP, several NAL units get pulled out of its payload. Each NAL unit is the part of an aggregation unit's payload that comes right after the NALU size field, as described in Section 4.3.2 🍓
When an RTP packet carries a Fragmentation Unit (FU), you collect every RTP packet from the first FU (with the S field equal to 1) of the fragmented NAL unit all the way up to the last FU (with the E field equal to 1) of that same NAL unit 🧩 Then you extract the NAL unit by concatenating all the FU payloads in the same order as their RTP packets, and appending a NAL unit header. That header's F and TID fields are set to the values of F and TID in the payload header of the FUs, and its NAL unit type is set to the value of the FuType field in the FU header. All of this is covered in Section 4.3.3 💖
When sprop-max-don-diff is equal to 0, the de-packetization buffer size is zero bytes, and the NAL units from the single RTP stream go straight to the decoder in their transmission order. That order is the same as their decoding order, so no reordering drama 🙌
When sprop-max-don-diff is greater than 0, the rest of this section applies 👀
The receiver has two buffering states: initial buffering and buffering while playing 🎬 Initial buffering starts when reception gets initialized. Once initial buffering is done, decoding and playback kick off, and the receiver switches to buffering-while-playing mode.
No matter which buffering state it's in, the receiver stores incoming NAL units in reception order into the de-packetization buffer 📥 Each NAL unit carried in an RTP packet is stored individually, and an AbsDon value gets calculated and stored for every single NAL unit 🔢
Initial buffering keeps going until the difference between the greatest and smallest AbsDon values of the NAL units in the de-packetization buffer is greater than or equal to the value of sprop-max-don-diff ⏳
After initial buffering, whenever that same difference between the greatest and smallest AbsDon values in the buffer is greater than or equal to sprop-max-don-diff, the following operation gets repeated until the difference is smaller than sprop-max-don-diff 🔁
The NAL unit in the de-packetization buffer with the smallest value of AbsDon is removed from the de-packetization buffer and passed to the decoder.
Finally, when no more NAL units are flowing into the de-packetization buffer, every NAL unit still sitting in there gets removed and passed to the decoder in order of increasing AbsDon values 🏁 Bye-bye buffer, you did great 🥹
7. Payload Format Parameters 🎀
this section is all about the optional parameters 🫶 It also includes a mapping of these parameters to the Session Description Protocol (SDP) [RFC8866], for any apps that use SDP.
Parameters that start with the string "sprop" (short for stream properties) can be used by a sender to tell a receiver about the properties of the stream that is, or will be, sent 📡 Here's the twist: the media sender (not the receiver) is the one who decides whether "sprop" parameters get sent at all, and what values they carry 👑 This is uncommon, and it may not feel intuitive if you're coming from some signaling protocol concepts, especially Offer/Answer. For sprop guidance specifically in the Offer/Answer case, check out Section 7.3.2 💅
7.1. Media Type Registration 📝
The receiver MUST ignore any parameter unspecified in this document 🙅♀️ That's a big one, so don't skip it!
Here's the full registration info, exactly as written:
Type name: video
Subtype name: evc
Required parameters: N/A
Optional parameters: profile-id, level-id, toolset-id, max-recv-
level-id, sprop-sps, sprop-pps, sprop-sei, sprop-max-don-diff,
sprop-depack-buf-bytes, depack-buf-cap (refer to Section 7.2 for
definitions)
Encoding considerations: This type is only defined for transfer via
RTP [RFC3550].
Security considerations: See Section 9 of RFC 9584.
Interoperability considerations: N/A
Published specification: Please refer to RFC 9584 and EVC standard
[EVC].
Applications that use this media type: Any application that relies
on EVC-based video services over RTP
Fragment identifier considerations: N/A
Additional information: N/A
Person & email address to contact for further information:
Stephan Wenger (stewe@stewe.org)
Intended usage: COMMON
Restrictions on usage: N/A
Author: See Authors' Addresses section of RFC 9584.
Change controller: IETF <avtcore@ietf.org>7.2. Optional Parameters Definition 💖
okay so this is where we go through the optional parameters, bestie 🎀 they say which profile, which level, and which constraints the bitstream carried by the RTP stream uses, or which profile, level, and constraints the receiver can handle ✨
More specifications of these parameters, including how they relate to syntax elements specified in [EVC] are provided below. 💅
profile-id
When profile-id is not present, a value of 0 (i.e., the Baseline profile) MUST be inferred. 🌸
When used to indicate properties of a bitstream, profile-id MUST be derived from the profile_idc in the SPS. 💜
EVC bitstreams transported over RTP using the technologies of this document SHOULD refer only to SPSs that have the same value in profile_idc, unless the sender has a priori knowledge that a receiver can correctly decode the EVC bitstream with different profile_idc values (for example, in walled garden scenarios). As exceptions to this rule, if the receiver is known to support a Baseline profile, a bitstream could safely end with CVS referring to an SPS wherein profile_idc indicates the Baseline Still picture profile. A similar exception can be made for Main profile and Main Still picture profile. 🦋
level-id
When level-id is not present, a value of 90 (corresponding to level 3, which allows for approximately standard-definition television (SD TV) resolution and frame rates; see Annex A of [EVC]) MUST be inferred. 📺
When used to indicate properties of a bitstream, level-id MUST be derived from the level_idc in the SPS. ✨
If the level-id parameter is used for capability exchange, the following applies. If max-recv-level-id is not present, the default level defined by level-id indicates the highest level the codec wishes to support. Otherwise, max-recv-level-id indicates the highest level the codec supports for receiving. For either receiving or sending, all levels that are lower than the highest level supported MUST also be supported. 🌈
toolset-id
This parameter is a base64-encoding representation (Section 4 of [RFC4648]) of a 64-bit unsigned integer bit mask derived from the concatenation, in network byte order, of the syntax elements toolset_idc_h and toolset_idc_l. When used to indicate properties of a bitstream, its value MUST be derived from toolset_idh_h and toolset_idc_l in the sequence parameter set. 🧚
max-recv-level-id
This parameter MAY be used to indicate the highest level a receiver supports. 🍓
The value of max-recv-level-id MUST be in the range of 0 to 255, inclusive. 🔢
When max-recv-level-id is not present, the value is inferred to be equal to level-id. 🫶
max-recv-level-id MUST NOT be present when the highest level the receiver supports is not higher than the default level. 🙈
sprop-sps
This parameter MAY be used to convey sequence parameter set NAL units of the bitstream for out-of-band transmission of sequence parameter sets. The value of the parameter is a comma-separated (',') list of base64-encoding representations (Section 4 of [RFC4648]) of the sequence parameter set NAL units as specified in Section 7.3.2.1 of [EVC]. 📦
sprop-pps
This parameter MAY be used to convey picture parameter set NAL units of the bitstream for out-of-band transmission of picture parameter sets. The value of the parameter is a comma-separated (',') list of base64-encoding representations (Section 4 of [RFC4648]) of the picture parameter set NAL units as specified in Section 7.3.2.2 of [EVC]. 📦
sprop-sei
This parameter MAY be used to convey one or more SEI messages that describe bitstream characteristics. When present, a decoder can rely on the bitstream characteristics that are described in the SEI messages for the entire duration of the session, independently from the persistence scopes of the SEI messages as specified in [VSEI]. 🎞️
The value of the parameter is a comma-separated (',') list of base64-encoding representations (Section 4 of [RFC4648]) of SEI NAL units as specified in [VSEI]. 💫
Informative note: Intentionally, no list of applicable or inapplicable SEI messages is specified here. Conveying certain SEI messages in sprop-sei may be sensible in some application scenarios and meaningless in others. However, a couple of examples are described below. 1. In an environment where the bitstream was created from film-based source material, and no splicing is going to occur during the lifetime of the session, the film grain characteristics SEI message is likely meaningful; and sending it in sprop-sei rather than in the bitstream at each entry point may help with saving bits and allow one to configure the renderer only once, avoiding unwanted artifacts. 2. Examples for SEI messages that would be meaningless to be conveyed in sprop-sei include the decoded picture hash SEI message (it is close to impossible that all decoded pictures have the same hashtag) or the filler payload SEI message (as there is no point in just having more bits in SDP).
sprop-max-don-diff
If there is no NAL unit naluA that is followed in transmission order by any NAL unit preceding naluA in decoding order (i.e., the transmission order of the NAL units is the same as the decoding order), the value of this parameter MUST be equal to 0. 🔁
Otherwise, this parameter specifies the maximum absolute difference between the decoding order number (i.e., AbsDon) values of any two NAL units naluA and naluB, where naluA follows naluB in decoding order and precedes naluB in transmission order. 🧠
The value of sprop-max-don-diff MUST be an integer in the range of 0 to 32767, inclusive. 🔢
When not present, the value of sprop-max-don-diff is inferred to be equal to 0. 🌷
sprop-depack-buf-bytes
This parameter signals the required size of the de-packetization buffer in units of bytes. The value of the parameter MUST be greater than or equal to the maximum buffer occupancy (in units of bytes) of the de-packetization buffer as specified in Section 6. 🪣
The value of sprop-depack-buf-bytes MUST be an integer in the range of 0 to 4294967295, inclusive. 🔢
When sprop-max-don-diff is present and greater than 0, this parameter MUST be present and the value MUST be greater than 0. When not present, the value of sprop-depack-buf-bytes is inferred to be equal to 0. 💅
Informative note: The value of sprop-depack-buf-bytes indicates the required size of the de-packetization buffer only. When network jitter can occur, an appropriately sized jitter buffer has to be available as well. 🫧
depack-buf-cap
This parameter signals the capabilities of a receiver implementation and indicates the amount of de-packetization buffer space in units of bytes that the receiver has available for reconstructing the NAL unit decoding order from NAL units carried in the RTP stream. A receiver is able to handle any RTP stream for which the value of the sprop-depack-buf-bytes parameter is smaller than or equal to this parameter. 🏠
When not present, the value of depack-buf-cap is inferred to be equal to 4294967295. The value of depack-buf-cap MUST be an integer in the range of 1 to 4294967295, inclusive. 👑
Informative note: The value of depack-buf-cap indicates the maximum possible size of the de-packetization buffer of the receiver only, without allowing for network jitter. When network jitter occurs, an appropriately sized jitter buffer has to be available as well. 🌸
7.3. SDP Parameters 📝
The receiver MUST ignore any parameter unspecified in this document. 🙅♀️
7.3.1. Mapping of Payload Type Parameters to SDP
okay so this part is where we map the video/evc media type onto SDP fields 🎀 The media type video/evc string maps to fields in the Session Description Protocol (SDP) [RFC8866] like this:
- The media name in the
m=line of SDP MUST bevideo. 💖 - The encoding name in the
a=rtpmapline of SDP MUST beevc(the media subtype). - The clock rate in the
a=rtpmapline MUST be90000. ⏱️ - The OPTIONAL parameters
profile-id,level-id,toolset-id,max-recv-level-id,sprop-max-don-diff,sprop-depack-buf-bytes, anddepack-buf-cap, when present, MUST be included in thea=fmtpline of SDP. Thea=fmtpline is written as a media type string, a semicolon-separated list ofparameter=valuepairs. ✨ - The OPTIONAL parameters
sprop-sps,sprop-pps, andsprop-sei, when present, MUST be included in thea=fmtpline of SDP, or conveyed using thefmtpsource attribute as specified in Section 6.3 of [RFC5576]. For a particular media format (i.e., RTP payload type),sprop-sps,sprop-pps, orsprop-seiMUST NOT be both included in thea=fmtpline of SDP and conveyed using thefmtpsource attribute. 🙅♀️ When included in thea=fmtpline of SDP, those parameters are written as a media type string, a semicolon-separated list ofparameter=valuepairs. When conveyed in thea=fmtpline of SDP for a particular payload type, the parameterssprop-sps,sprop-pps, andsprop-seiMUST be applied to each SSRC with the payload type. When conveyed using thefmtpsource attribute, these parameters are only associated with the given source and payload type as parts of thefmtpsource attribute. 🌸
Informative note: Conveyingsprop-spsandsprop-ppsusing thefmtpsource attribute allows for out-of-band transport of parameter sets in topologies like Topo-Video-switch-MCU, as specified in [RFC7667]. 🧚
A general usage of media representation in SDP looks like this:
m=video 49170 RTP/AVP 98
a=rtpmap:98 evc/90000
a=fmtp:98 profile-id=1;
sprop-sps=<sequence parameter set data>;
sprop-pps=<picture parameter set data>;A SIP Offer/Answer exchange where both parties are expected to both send and receive could look like this. Only the media codec-specific parts of the SDP are shown. 📞
Offerer->Answerer:
m=video 49170 RTP/AVP 98
a=rtpmap:98 evc/90000
a=fmtp:98 profile-id=1; level_id=90;So this offer is basically saying: symmetric video calls using [EVC] and its payload specification, at the main profile and level 3. 🎬 Informally, it tells the receiver of the offer that the sender is willing to receive up to xKpxx resolution at the maximum bitrates specified in [EVC]. At the same time, if this offer were accepted "as is", the offerer can expect the Answerer to be able to receive and properly decode EVC media up to and including level 3. 👑
Answerer->Offerer:
m=video 49170 RTP/AVP 98
a=rtpmap:98 evc/90000
a=fmtp:98 profile-id=1; level_id=60Informative note: level_id shall be set equal to a value of 30 times the level number specified in Table A.1 of [EVC]. 🔢
With this answer, the system receiving the offer tells the Offerer that it can't handle evc at level 3, but it can decode level 2. 🙈 EVC video codecs must support decoding at all levels below the maximum level they implement, so the resulting experience would likely be that both systems send video at level 2. Nothing stops an encoder from downgrading its sending even further, to level 1 for example, if it's short on cycles or bandwidth or has other reasons. 💅
7.3.2. Usage with SDP Offer/Answer Model
this section is all about negotiating unicast messages with the Offer/Answer model described in [RFC3264] and its updates. 🤝
This section applies to all profiles defined in [EVC], specifically Baseline, Main, and the associated still image profiles. 📸
The following limitations and rules about the media configuration apply:
The parameters identifying a media format configuration for EVC are profile-id and level-id. Profile_id MUST be used symmetrically. 🪞
The Answerer MUST structure its answer according to one of the following two options:
- maintain all configuration parameters with the values remaining the same as in the offer for the media format (payload type), with the exception that the value of
level-idis changeable, as long as the highest level indicated by the answer is not higher than the one indicated by the offer; or - remove the media format (payload type) completely (when one or more of the parameter values are not supported). 🗑️
Informative note: The above requirement for symmetric use does not apply for level-id, and it does not apply for the other bitstream or RTP stream properties and capability parameters, as described in Section 7.3.2.1 ("Payload Format Configuration"). 🦋
To keep handling and matching of these configurations simple, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [RFC3264]. 🌈
The answer MUST NOT contain a payload type number used in the offer for the media subtype, unless the configuration is the same as in the offer, or the configuration in the answer only differs from the offer in having a different value of level-id. 💜
7.3.2.1. Payload Format Configuration ✨
okay so this part is all about how the payload format buffer management gets set up, and there are some rules you really need to know 🎀
- The parameters
sprop-max-don-diffandsprop-depack-buf-bytesdescribe the properties of an RTP stream that the Offerer or the Answerer is sending for the media format configuration. This is different from the usual Offer/Answer vibe, where these parameters tell you what bitstream or RTP stream you're able to receive 👀 When it comes to EVC, the Offerer assumes the Answerer will be able to receive media encoded with the configuration being offered 💅
Informative note: The above parameters apply to any RTP stream, when present, sent by a declaring entity with the same configuration. So whether they apply to an RTP stream depends on the source endpoint 🧚 Instead of being tied to the payload type, the values may have to be applied to another payload type when sent, since they apply to the configuration.
- When an Offerer offers an interleaved stream, which you can spot by
sprop-max-don-diffshowing up with a value larger than zero, the Offerer MUST include the size of the de-packetization buffer insprop-depack-buf-bytes🌸 - So they can tell each other what they're capable of for de-packetization buffering when receiving RTP streams, both parties are RECOMMENDED to include
depack-buf-cap🫶 - The parameters
sprop-spsorsprop-pps, when present (included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute, as specified in Section 6.3 of [RFC5576]), are used for out-of-band transport of the parameter sets (SPS or PPS, respectively) 🦋 The Answerer MAY use either out-of-band or in-band transport of parameter sets for the bitstream it's sending, no matter whether out-of-band parameter set transport was used in the Offerer-to-Answerer direction. Parameter sets in an answer are totally independent of the ones in the offer, because they decode two different bitstreams: one from the Answerer to the Offerer, and one going the other way 💜 If some RTP packets get sent before the SDP Offer/Answer settles down, in-band parameter sets MUST be used for those RTP stream parts sent before the SDP Offer/Answer 😭 - The following rules apply to transport of parameter sets in the Offerer-to-Answerer direction 👑
- An offer MAY include
sprop-spsand/orsprop-pps. If none of these parameters are present in the offer, then only in-band transport of parameter sets is used. - If the level to use in the Offerer-to-Answerer direction is equal to the default level in the offer, the Answerer MUST be prepared to use the parameter sets included in
sprop-spsandsprop-pps(either included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute) for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams. Otherwise, the Answerer MUST ignoresprop-vps,sprop-sps, andsprop-pps(either included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute), and the Offerer MUST transmit parameter sets in-band.
- An offer MAY include
- The following rules apply to transport of parameter sets in the Answerer-to-Offerer direction 🌈
- An answer MAY include
sprop-spsand/orsprop-pps. If none of these parameters are present in the answer, then only in-band transport of parameter sets is used. - The Offerer MUST be prepared to use the parameter sets included in
sprop-spsandsprop-pps(either included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute) for decoding the incoming bitstream, e.g., by passing these parameter set NAL units to the video decoder before passing any NAL units carried in the RTP streams.
- An answer MAY include
- When
sprop-spsand/orsprop-ppsare conveyed using thefmtpsource attribute, as specified in Section 6.3 of [RFC5576], the receiver of the parameters MUST store the parameter sets included insprop-spsand/orsprop-ppsand associate them with the source given as part of thefmtpsource attribute. Parameter sets associated with one source (given as part of thefmtpsource attribute) MUST only be used to decode NAL units conveyed in RTP packets from that same source. When this mechanism is in use, SSRC collision detection and resolution MUST be performed as specified in [RFC5576] 🔥
Figure 11 lists the interpretation of all the parameters that MAY be used for the various combinations of offer, answer, and direction attributes 📋
sendonly --+
recvonly --+ |
sendrecv --+ | |
| | |
profile-id C C P
level-id D D P
toolset-id C C P
max-recv-level-id R R -
sprop-max-don-diff P - P
sprop-depack-buf-bytes P - P
depack-buf-cap R R -
sprop-sei P - P
sprop-sps P - P
sprop-pps P - PLegend:
C: configuration for sending and receiving bitstreams
D: changeable configuration; same as C, except possible to
answer with a different but consistent value (see the semantics
of the level-id parameter on these parameters being
consistent -- basically, level down-grading is allowed)
P: properties of the bitstream to be sent
R: receiver capabilities
-: not usable; when present MUST be ignoredFigure 11: Interpretation of Parameters for Various Combinations of Offers, Answers, and Direction Attributes
Parameters used for declaring receiver capabilities are, in general, downgradable, which means they express the upper limit for a sender's possible behavior 📉 So a sender MAY choose to set its encoder using only lower, lesser, or equal values of these parameters 💖
When a sender's capabilities are declared with the configuration parameters, these parameters express a configuration that is acceptable for the sender to receive bitstreams. To get high interoperability, it's often a good idea to offer multiple alternative configurations 🍓 You can't offer multiple configurations in a single payload type, so when you make multiple configuration offers, each offer needs its own RTP payload type associated with it 🎀
An implementation SHOULD be able to understand all media type parameters (including all optional media type parameters), even if it doesn't support the functionality related to the parameter. This, together with proper application logic, lets the implementation, after receiving an offer, create an answer by potentially downgrading one or more of the optional parameters to the point where it can cope. This leads to higher chances of interoperability beyond the most basic interop points (for which, as described above, no optional parameters are necessary) ✨
Informative note: In implementations of various H.26x video coding payload formats, including those for [AVC] and [HEVC], it was occasionally observed that implementations were incapable of parsing most (or all) of the optional parameters, and so they rejected offers other than the most basic ones. As a result, the Offer/Answer exchange ended up at baseline performance (using the default values for the optional parameters), which gave a suboptimal user experience 😬 Still, there are valid reasons to skip the implementation complexity of parsing some or all of the optional parameters, for example when there's predetermined knowledge, not negotiated by an SDP-based Offer/Answer process, of the capabilities of the systems involved (walled gardens, baseline requirements defined in application standards higher up in the stack, and similar).
An Answerer MAY extend the offer with additional media format configurations. However, to enable their usage, in most cases a second offer is required from the Offerer to provide the bitstream property parameters the media sender will use 🔁 This also means the Offerer has to be able to receive this media format configuration, not just send it 🫶
7.3.3. Multicast 📡
For bitstreams being delivered over multicast, the following rules apply:
- The media format configuration is identified by
profile-idandlevel-id. These media format configuration parameters, includinglevel-id, MUST be used symmetrically; that is, the Answerer MUST either maintain all configuration parameters or remove the media format (payload type) completely. Note that this implies thelevel-idfor Offer/Answer in multicast is not changeable 🙈 - To make these configurations easier to handle and match, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [RFC3264]. An answer MUST NOT contain a payload type number used in the offer unless the configuration is the same as in the offer 💅
- Parameter sets received MUST be associated with the originating source and MUST only be used in decoding the incoming bitstream from that same source 🔥
- The rules for other parameters are the same as above for unicast, as long as the three rules above are obeyed 🦋
7.3.4. Usage in Declarative Session Descriptions 📜
When EVC over RTP is offered with SDP in a declarative style, as in the Real-Time Streaming Protocol (RTSP) [RFC7826] or Session Announcement Protocol (SAP) [RFC2974], the following considerations apply 🌸
- All parameters that can indicate both bitstream properties and receiver capabilities are used to indicate only bitstream properties. For example, in this case,
profile-idandlevel-iddeclare the values used by the bitstream, not the capabilities for receiving bitstreams. As a result, the following interpretation of the parameters MUST be used:- Declaring actual configuration or bitstream properties:
profile-idlevel-idsprop-spssprop-ppssprop-max-don-diffsprop-depack-buf-bytessprop-sei
- Not usable (when present, they MUST be ignored):
depack-buf-caprecv-sublayer-id
- A receiver of the SDP is required to support all parameters and values of the parameters provided; otherwise, the receiver MUST reject (RTSP) or not participate in (SAP) the session. It falls on the creator of the session to use values that are expected to be supported by the receiving application 🧚
- Declaring actual configuration or bitstream properties:
7.3.5. Considerations for Parameter Sets ✨
okay so when parameter sets go out of band, they can still be sent in-band too 🫶 unless an application explicitly says no. parameter sets MAY still be additionally transported in-band, and some of those extra sets might update the out-of-band ones 🔄. an "update" just means sending a parameter set of the same type with the same parameter set ID but different values for at least one other parameter 💅
8. Use with Feedback Messages 💌
this section is all about how the Picture Loss Indication (PLI) [RFC4585] and Full Intra Request (FIR) [RFC5104] feedback messages work with [EVC] 🦋. the subsections below break it down.
bestie rule number one: a sender MUST NOT send Slice Loss Indication (SLI) or Reference Picture Selection Indication (RPSI) 🙅♀️. and a receiver MUST ignore RPSI and MUST treat a received SLI as a received PLI, ignoring the "First", "Number", and "PictureID" fields of the PLI ✨
8.1. Picture Loss Indication (PLI)
as Section 6.3.1 of [RFC4585] explains, when a media sender gets a PLI it means "the loss of an undefined amount of coded video data belonging to one or more pictures" 😭. so if the sender doesn't know anything about how the bitstream is set up (like where the in-band parameter sets are, where the IDR pictures live, the picture structures, all that), an EVC sender SHOULD respond by sending an IDR picture and relevant parameter sets 🌸, maybe with extra redundancy so they actually arrive 🛡️.
BUT sometimes you do know stuff about the bitstream 👀. like, parameter sets that were conveyed out of band through mechanisms not defined in this document and that stay static for the whole session. in that case there's no point resending them in-band because of a PLI 🙈. other examples could be made from a priori knowledge of different aspects of the bitstream structure 🧠. in all cases, the timing and congestion-control mechanisms of [RFC4585] MUST be observed 🔥
8.2. Full Intra Request (FIR)
the whole point of FIR is to force an encoder to send an independent decoder refresh point as soon as possible 🏃♀️, while still respecting applicable congestion-control constraints, like the ones in [RFC8082] 🚦
when a FIR comes in, a sender MUST send an IDR picture 💖. parameter sets MUST also be sent too, except when there's a priori knowledge that they've already been correctly established 🧚. a typical example is an understanding between sender and receiver, set up outside this document, that parameter sets are exclusively sent out of band 📦
9. Security Considerations 🔐
okay this one is serious, so buckle up 😤. the scope here is limited to the payload format itself and to one feature of [EVC] that could be a seriously big security risk if it's implemented naively 💀. the payload format on its own isn't a complete system. implementers are advised to read and understand the relevant security docs, especially the ones about RTP (see the Security Considerations in Section 14 of [RFC3550]) and the security of whatever call-control stack you pick (which might use the media type registration from this doc). implementers should also think about known security vulnerabilities of video coding and decoding implementations in general and avoid those 🙅♀️
within this RTP payload format, and except for the user data SEI message (more on that in a sec), no security threats other than the ones common to RTP payload formats are known 🫠. in other words, neither the media-plane mechanisms nor the signaling part of this document seem to pose a security risk beyond what's common to all RTP-based systems ✅
RTP packets using this payload format are subject to the security considerations in the RTP specification [RFC3550] and in any applicable RTP profile, such as RTP/AVP [RFC3551], RTP/AVPF [RFC4585], RTP/SAVP [RFC3711], or RTP/SAVPF [RFC5124] 📜. but, as "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution" [RFC7202] discusses, it is not an RTP payload format's job to discuss or mandate which solutions get used for the basic security goals (confidentiality, integrity, and source authenticity) for RTP in general 🤷♀️. that responsibility lies with whoever uses RTP in an application 💁♀️. they can find guidance on available security mechanisms and important considerations in "Options for Securing RTP Sessions" [RFC7201] 📚. applications SHOULD use one or more appropriate strong security mechanisms 💪. the rest of this section discusses the security-impacting properties of the payload format itself 🔍
because the data compression used with this payload format is applied end to end, any encryption needs to be performed after compression 🔒. there's a potential denial-of-service threat for data encodings that use compression techniques with non-uniform receiver-end computational load 😱. the attacker can inject pathological datagrams into the bitstream that are complex to decode and that overload the receiver 🚨
EVC is particularly vulnerable to these attacks, because it is extremely simple to generate datagrams containing NAL units that affect the decoding process of many future NAL units 💀. therefore, using data origin authentication and data integrity protection of at least the RTP packet is RECOMMENDED based on [RFC7202] 🫶
like HEVC [RFC7798] and VVC [VVC], EVC [EVC] includes a user data Supplemental Enhancement Information (SEI) message 🧩. this SEI message allows an arbitrary bitstring to be included in the video bitstream, and that bitstring could include JavaScript, machine code, and other active content 👾
EVC [EVC] leaves the handling of this SEI message up to the receiving system 🙃. so to avoid harmful side effects, decoder implementations cannot naively trust its content 🛑. for example, forwarding all received JavaScript code detected by a decoder to a web browser unchecked would be a bad and insecure implementation practice 😬. the safest way to deal with user data SEI messages is to simply discard them, but that can have negative side effects on the user's quality of experience 😭
end-to-end security with authentication, integrity, or confidentiality protection will prevent a MANE from performing media-aware operations other than discarding complete packets 🚫. with confidentiality protection, it won't even be able to discard packets in a media-aware way 🙈. to be allowed to perform those operations, a MANE is required to be a trusted entity that is included in the security context establishment 🛡️👑
10. Congestion Control 🌊
congestion control for RTP SHALL be used in accordance with RTP [RFC3550] and any applicable RTP profile, e.g., AVP [RFC3551] or AVPF [RFC4585] 🚦. if best-effort service is being used, there's an extra requirement: users of this payload format MUST monitor packet loss to make sure the loss rate stays within an acceptable range 👀. packet loss is considered acceptable if a TCP flow across the same network path and experiencing the same network conditions would achieve an average throughput, measured on a reasonable timescale, that is not less than all RTP streams combined 📊. this can be satisfied by implementing congestion-control mechanisms that adapt the transmission rate by changing the number of layers subscribed to for a layered multicast session, or by arranging for a receiver to leave the session if the loss rate is unacceptably high 🚪
bitrate adaptation for obeying the congestion control principle is totally achievable when real-time encoding is used, for example by tuning the quantization parameter 🎛️. but when pre-encoded content is being transmitted, bandwidth adaptation requires the pre-coded bitstream to be tailored for that adaptivity 🧵
the key mechanism available in [EVC] is temporal scalability 🦋. a media sender can remove NAL units belonging to higher temporal sub-layers (i.e., those NAL units with a large value of TID) until the sending bitrate drops to an acceptable range 📉
the mechanisms mentioned above generally work within a defined profile and level, so no renegotiation of the channel is required 🙌. only when non-downgradable parameters (such as the profile) need to be changed does it become necessary to terminate and restart the RTP streams 🔁. this may be accomplished by using different RTP payload types 🎀
MANEs MAY remove certain unusable packets from the RTP stream when that RTP stream was damaged due to previous packet losses 🧹. this can help reduce network load in certain special cases 🌸. for example, MANEs can remove those FUs where the leading FUs belonging to the same NAL unit have been lost, because the trailing FUs are meaningless to most decoders 🗑️. a MANE can also remove higher temporal scalable layers if the outbound transmission (from the MANE's viewpoint) experiences congestion 💨
11. IANA Considerations 📋
the media type specified in Section 7.1 has been registered with IANA 🎉💖
12. References 📚✨
12.1. Normative References 📚
okay so these are the docs the spec leans on and, like, can't function without 🫶
- [EVC] "Information technology -- General video coding -- Part 1: Essential video coding", ISO/IEC 23094-1:2020, October 2020,
https://www.iso.org/standard/57797.html. - [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997,
https://www.rfc-editor.org/info/rfc2119. - [RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, DOI 10.17487/RFC3264, June 2002,
https://www.rfc-editor.org/info/rfc3264. - [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, July 2003,
https://www.rfc-editor.org/info/rfc3550. - [RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, DOI 10.17487/RFC3551, July 2003,
https://www.rfc-editor.org/info/rfc3551. - [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, DOI 10.17487/RFC3711, March 2004,
https://www.rfc-editor.org/info/rfc3711. - [RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 10.17487/RFC4585, July 2006,
https://www.rfc-editor.org/info/rfc4585. - [RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006,
https://www.rfc-editor.org/info/rfc4648. - [RFC5104] Wenger, S., Chandra, U., Westerlund, M., and B. Burman, "Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, February 2008,
https://www.rfc-editor.org/info/rfc5104. - [RFC5124] Ott, J. and E. Carrara, "Extended Secure RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, February 2008,
https://www.rfc-editor.org/info/rfc5124. - [RFC5576] Lennox, J., Ott, J., and T. Schierl, "Source-Specific Media Attributes in the Session Description Protocol (SDP)", RFC 5576, DOI 10.17487/RFC5576, June 2009,
https://www.rfc-editor.org/info/rfc5576. - [RFC7826] Schulzrinne, H., Rao, A., Lanphier, R., Westerlund, M., and M. Stiemerling, Ed., "Real-Time Streaming Protocol Version 2.0", RFC 7826, DOI 10.17487/RFC7826, December 2016,
https://www.rfc-editor.org/info/rfc7826. - [RFC8082] Wenger, S., Lennox, J., Burman, B., and M. Westerlund, "Using Codec Control Messages in the RTP Audio-Visual Profile with Feedback with Layered Codecs", RFC 8082, DOI 10.17487/RFC8082, March 2017,
https://www.rfc-editor.org/info/rfc8082. - [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017,
https://www.rfc-editor.org/info/rfc8174. - [RFC8866] Begen, A., Kyzivat, P., Perkins, C., and M. Handley, "SDP: Session Description Protocol", RFC 8866, DOI 10.17487/RFC8866, January 2021,
https://www.rfc-editor.org/info/rfc8866. - [RFC9328] Zhao, S., Wenger, S., Sanchez, Y., Wang, Y.-K., and M. M. Hannuksela, "RTP Payload Format for Versatile Video Coding (VVC)", RFC 9328, DOI 10.17487/RFC9328, December 2022,
https://www.rfc-editor.org/info/rfc9328. - [VSEI] ITU-T, "Versatile supplemental enhancement information messages for coded video bitstreams", ITU-T Recommendation H.274, March 2024,
https://www.itu.int/rec/T-REC-H.274.
12.2. Informative References 🦋
these ones are the extra reading. good to know, not strictly needed to build the thing 💅
- [AVC] ITU-T, "Part 10: Advanced video coding", ITU-T Recommendation H.264, October 2014,
https://www.iso.org/standard/66069.html. - [HEVC] ITU-T, "High efficiency video coding", ITU-T Recommendation H.265, November 2019,
https://www.itu.int/rec/T-REC-H.265. - [MPEG2S] IS0/IEC, "Information technology - Generic coding of moving pictures and associated audio information - Part 1: Systems", ISO/IEC 13818-1:2013, June 2013.
- [RFC2974] Handley, M., Perkins, C., and E. Whelan, "Session Announcement Protocol", RFC 2974, DOI 10.17487/RFC2974, October 2000,
https://www.rfc-editor.org/info/rfc2974. - [RFC6184] Wang, Y.-K., Even, R., Kristensen, T., and R. Jesup, "RTP Payload Format for H.264 Video", RFC 6184, DOI 10.17487/RFC6184, May 2011,
https://www.rfc-editor.org/info/rfc6184. - [RFC6190] Wenger, S., Wang, Y.-K., Schierl, T., and A. Eleftheriadis, "RTP Payload Format for Scalable Video Coding", RFC 6190, DOI 10.17487/RFC6190, May 2011,
https://www.rfc-editor.org/info/rfc6190. - [RFC7201] Westerlund, M. and C. Perkins, "Options for Securing RTP Sessions", RFC 7201, DOI 10.17487/RFC7201, April 2014,
https://www.rfc-editor.org/info/rfc7201. - [RFC7202] Perkins, C. and M. Westerlund, "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution", RFC 7202, DOI 10.17487/RFC7202, April 2014,
https://www.rfc-editor.org/info/rfc7202. - [RFC7656] Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms for Real-Time Transport Protocol (RTP) Sources", RFC 7656, DOI 10.17487/RFC7656, November 2015,
https://www.rfc-editor.org/info/rfc7656. - [RFC7667] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 7667, DOI 10.17487/RFC7667, November 2015,
https://www.rfc-editor.org/info/rfc7667. - [RFC7798] Wang, Y.-K., Sanchez, Y., Schierl, T., Wenger, S., and M. M. Hannuksela, "RTP Payload Format for High Efficiency Video Coding (HEVC)", RFC 7798, DOI 10.17487/RFC7798, March 2016,
https://www.rfc-editor.org/info/rfc7798. - [VIDEO-CODING] ITU-T, "Video coding for low bit rate communication", ITU-T Recommendation H.263, January 2005,
https://www.itu.int/rec/T-REC-H.263. - [VVC] ITU-T, "Versatile video coding", ITU-T Recommendation H.266, August 2020,
http://www.itu.int/rec/T-REC-H.266.
Acknowledgements 🎀
big thank-you energy here 🫶 a lot of this spec literally shares text with the RTP payload format for VVC [RFC9328], so shoutout to those authors for their amazing work 👑. Roman Chernyak also gets a huge thanks for his super valuable review comments 💜
Authors' Addresses 🧚
Meet the authors ✨
Shuai Zhao
Shuai Zhao
Intel
2200 Mission College Blvd
Santa Clara, California 95054
United States of America
Email: shuai.zhao@ieee.orgStephan Wenger
Stephan Wenger
Tencent
2747 Park Blvd
Palo Alto, California 94588
United States of America
Email: stewe@stewe.orgYoungkwon Lim
Youngkwon Lim
Samsung Electronics
6625 Excellence Way
Plano, Texas 75013
United States of America
Email: yklwhite@gmail.comThat's the whole thing, bestie! 🌈✨
📦 where this page comes from
- ietf: rfc9584 2024-06
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms