RFC 6190
RTP Payload Format for Scalable Video Coding
RFC 6190: RTP Payload Format for Scalable Video Coding 🎀
okay bestie, this is RFC 6190, published in May 2011 as a Standards Track document 💖 It's an IETF doc, so the whole IETF community agreed on it. The authors are S. Wenger (Independent), Y.-K. Wang (Huawei Technologies), T. Schierl (Fraunhofer HHI), and A. Eleftheriadis (Vidyo) 🧚 The ISSN is 2070-1721. The header doesn't list any RFCs that it obsoletes or updates, so nothing gets replaced or patched by this one 🌸
Abstract ✨
so this memo is about an RTP payload format for Scalable Video Coding (SVC) 🎬 SVC is defined in Annex G of ITU-T Recommendation H.264, and it's technically identical to Amendment 3 of ISO/IEC International Standard 14496-10 💅
the format lets you pack one or more Network Abstraction Layer (NAL) units into a single RTP packet payload, and it can also split one NAL unit across several RTP packets when it's too big 📦 It works for an SVC stream sent over a single RTP session or over multiple RTP sessions, so you get options 🦋
it defines a new media subtype name, "H264-SVC" 🌈 It's still backward compatible with RFC 6184, because the base layer, when it's in its own RTP stream, has to use the H.264 media subtype name ("H264") and the packetization method from RFC 6184. Literally the best kind of compatibility 🫶
and the applications are everywhere: videoconferencing, Internet video streaming, high-bitrate entertainment-quality video, and more 🍓
Status of This Memo 🌸
this is an Internet Standards Track document 👑
it's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check Section 2 of RFC 5741 📚
for the current status, errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6190 🔗
Copyright Notice 🎀
Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. Please read them carefully, because they explain your rights and restrictions for this document 🧐 Code components taken from this document have to include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License 💅
heads up: this document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The people who control the copyright in some of that material may not have given the IETF Trust permission to allow modifications outside the IETF Standards Process. Unless you get an adequate license from them, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside that process. The only exceptions are formatting it for publication as an RFC or translating it into languages other than English 🙈
Table of Contents 🧚
1. Introduction ....................................................5
1.1. The SVC Codec ..............................................6
1.1.1. Overview ............................................6
1.1.2. Parameter Sets ......................................8
1.1.3. NAL Unit Header .....................................9
1.2. Overview of the Payload Format ............................12
1.2.1. Design Principles ..................................12
1.2.2. Transmission Modes and Packetization Modes .........13
1.2.3. New Payload Structures .............................15
2. Conventions ....................................................16
3. Definitions and Abbreviations ..................................16
3.1. Definitions ...............................................16
3.1.1. Definitions from the SVC Specification .............16
3.1.2. Definitions Specific to This Memo ..................18
3.2. Abbreviations .............................................22
4. RTP Payload Format .............................................23
4.1. RTP Header Usage ..........................................23
4.2. NAL Unit Extension and Header Usage .......................23
4.2.1. NAL Unit Extension .................................23
4.2.2. NAL Unit Header Usage ..............................24
4.3. Payload Structures ........................................25
4.4. Transmission Modes ........................................28
4.5. Packetization Modes .......................................28
4.5.1. Packetization Modes for Single-Session
Transmission .......................................28
4.5.2. Packetization Modes for Multi-Session
Transmission .......................................29
4.6. Single NAL Unit Packets ...................................32
4.7. Aggregation Packets .......................................33
4.7.1. Non-Interleaved Multi-Time Aggregation
Packets (NI-MTAPs) .................................33
4.8. Fragmentation Units (FUs) .................................35
4.9. Payload Content Scalability Information (PACSI) NAL Unit ..35
4.10. Empty NAL unit ...........................................43
4.11. Decoding Order Number (DON) ..............................43
4.11.1. Cross-Session DON (CS-DON) for
Multi-Session Transmission ........................43
5. Packetization Rules ............................................45
5.1. Packetization Rules for Single-Session Transmission .......45
5.2. Packetization Rules for Multi-Session Transmission ........46
5.2.1. NI-T/NI-TC Packetization Rules .....................47
5.2.2. NI-C/NI-TC Packetization Rules .....................49
5.2.3. I-C Packetization Rules ............................50
5.2.4. Packetization Rules for Non-VCL NAL Units ..........50
5.2.5. Packetization Rules for Prefix NAL Units ...........51
6. De-Packetization Process .......................................51
6.1. De-Packetization Process for Single-Session Transmission ..51
6.2. De-Packetization Process for Multi-Session Transmission ...51
6.2.1. Decoding Order Recovery for the NI-T and
NI-TC Modes ........................................52
6.2.1.1. Informative Algorithm for NI-T
Decoding Order Recovery within
an Access Unit ............................55
6.2.2. Decoding Order Recovery for the NI-C,
NI-TC, and I-C Modes ...............................57
7. Payload Format Parameters ......................................59
7.1. Media Type Registration ...................................60
7.2. SDP Parameters ............................................75
7.2.1. Mapping of Payload Type Parameters to SDP ..........75
7.2.2. Usage with the SDP Offer/Answer Model ..............76
7.2.3. Dependency Signaling in Multi-Session
Transmission .......................................84
7.2.4. Usage in Declarative Session Descriptions ..........85
7.3. Examples ..................................................86
7.3.1. Example for Offering a Single SVC Session ..........86
7.3.2. Example for Offering a Single SVC Session Using
scalable-layer-id ..................................87
7.3.3. Example for Offering Multiple Sessions in MST ......87
7.3.4. Example for Offering Multiple Sessions in
MST Including Operation with Answerer Using
scalable-layer-id ..................................89
7.3.5. Example for Negotiating an SVC Stream with
a Constrained Base Layer in SST ....................90
7.4. Parameter Set Considerations ..............................91
8. Security Considerations ........................................91
9. Congestion Control .............................................92
10. IANA Considerations ...........................................93
11. Informative Appendix: Application Examples ....................93
11.1. Introduction .............................................93
11.2. Layered Multicast ........................................93
11.3. Streaming ................................................94
11.4. Videoconferencing (Unicast to MANE, Unicast to
Endpoints) ...............................................95
11.5. Mobile TV (Multicast to MANE, Unicast to Endpoint) .......96
12. Acknowledgements ..............................................97
13. References ....................................................97
13.1. Normative References .....................................97
13.2. Informative References ...................................981. Introduction ✨
Okay bestie, this memo is the rulebook for getting Scalable Video Coding (SVC), the extension of H.264/AVC, into RTP packets 🎀 RTP itself is defined in [RFC3550]. SVC lives in Amendment 3 to ISO/IEC 14496 Part 10 [ISO/IEC14496-10], and it's also in Annex G of ITU-T Rec. H.264 [H.264]. Quick rule: unless it says otherwise, "H.264/AVC" in this memo means the [H.264] spec minus Annex G 💅
SVC covers the whole range of H.264/AVC, from low-bitrate mobile apps all the way up to High-Definition Television (HDTV) broadcasting, and even Digital Cinema, which needs nearly lossless coding and hundreds of megabits per second 🎬🍓 The scalability features SVC adds let a system adapt the signal to different conditions with no or minimal processing ✨ That adaptation is about the capabilities of potentially heterogeneous receivers (differing in screen resolution, processing speed, etc.) and about differing or time-varying network conditions. It can happen at the source, the destination, or in intermediate media-aware network elements (MANEs) 🧚 This payload format exposes all of these system-level functionalities so system designers can take direct advantage of them 💖
Informative note: Since SVC streams contain, by design, a sub-stream that is compliant with H.264/AVC, it is trivial for a MANE to filter the stream so that all SVC-specific information is removed. This memo, in fact, defines a media type parameter (sprop-avc-ready, Section 7.2) that indicates whether or not the stream can be converted to one compliant with [RFC6184] by eliminating RTP packets, and rewriting RTP Control Protocol (RTCP) to match the changes to the RTP packet stream as specified in Section 7 of [RFC3550].
The memo defines two basic modes for sending SVC data: single-session transmission (SST) and multi-session transmission (MST) 🦋 In SST, one RTP session carries all the scalability layers of an SVC bitstream. In MST, the scalability layers ride on different RTP sessions. SST packetization is a straightforward extension of [RFC6184]. For MST there are four different modes, and they differ on whether or not they allow interleaving (transmitting Network Abstraction Layer (NAL) units in an order different from the decoding order), and on the technique used to recover the inter-session NAL unit decoding order. That recovery uses either inter-session timestamp alignment [RFC3550] or cross-session decoding order numbers (CS-DONs). One of the MST modes supports both techniques, so receivers can pick their favourite 👑 More details are in Section 1.2.2.
The memo also defines three new NAL unit types, and they're giving main character energy 🌈 The first is the payload content scalability information (PACSI) NAL unit, which gives an informative summary of the scalability information in an RTP packet, plus ancillary data (e.g., CS-DON values). The second and third are the empty NAL unit and the non-interleaved multi-time aggregation packet (NI-MTAP) NAL unit. The empty NAL unit is there to ensure the inter-session timestamp alignment that decoding order recovery needs in MST. The NI-MTAP is a new payload structure that groups NAL units from different time instances in decoding order 🫶 More details on these packet structures are in Section 1.2.3.
The memo also defines signaling support for SVC transport over RTP, including a brand new media subtype name, H264-SVC 🔥
Everything in the rest of this section is a non-normative overview of the SVC codec and the payload, so it's just here to help you vibe with the details 🌸
1.1. The SVC Codec
1.1.1. Overview
Okay so SVC defines a coded video representation where one bitstream offers the source material at different levels of fidelity (hence "scalable") ✨ Scalable video coding bitstreams, or scalable bitstreams, are built like a pyramid: the coding process creates bitstream components that improve the fidelity of hierarchically lower components 🔺
The fidelity dimensions SVC offers are spatial (picture size), quality (or Signal-to-Noise Ratio (SNR)), and temporal (pictures per second) 💜 Bitstream components at a given level of spatial, quality, and temporal fidelity are identified with corresponding parameters in the bitstream: dependency_id, quality_id, and temporal_id (see also Section 1.1.3). These fidelity identifiers are integers, and higher values mean components that sit higher in the hierarchy. SVC is seriously flexible about how an encoder structures the dependencies between components 💅 Decoding a particular component needs all the components it depends on, either directly or indirectly. An operation point of an SVC bitstream is the set of bitstream components required to decode a particular dependency_id, quality_id, and temporal_id combination 🧚
"Layer" shows up in a bunch of contexts in this memo, so keep up 🙈 In terms like "Video Coding Layer" and "Network Abstraction Layer", it means conceptual organization levels. When talking about bitstream syntax elements like block layer or macroblock layer, it means hierarchical bitstream structure levels. When used for bitstream scalability, like "AVC base layer", it means a level of representation fidelity of the source signal with a specific set of NAL units included. The right meaning is always clear from context 💖
SVC keeps the bitstream organization from H.264/AVC. All bitstream components are wrapped in NAL units, and those are organized into Access Units (AUs). An AU is tied to a single sampling instance in time ⏰ Some NAL unit types make up the Video Coding Layer (VCL) and carry the coded picture data for the source content. Non-VCL NAL units carry ancillary data that may be needed for decoding (like parameter sets, explained below) or that helps with certain system operations but isn't needed by the decoding process itself. Coded picture data at the various fidelity dimensions is organized in slices 🍓 Within one AU, a coded picture of an operation point consists of all the coded slices required for decoding up to the particular combination of dependency_id and quality_id values at the time instance that AU corresponds to.
Here's a fun one: temporal scalability is already in H.264/AVC, because the profiles in Annex A of [H.264] already support it 🦋 H.264/AVC introduced the concept of sub-sequences, which allow optional use of temporal layers through Supplemental Enhancement Information (SEI) messages. SVC builds on that by exposing the temporal scalability information through temporal_id, right next to (and unified with) dependency_id and quality_id, which handle spatial and quality scalability respectively ✨ For coded picture data defined in Annex G of [H.264], this uses a new NAL unit type, the coded slice in scalable extension NAL unit (type 20), where the fidelity parameters live in its header. For coded picture data that follows H.264/AVC, and to stay compatible with existing H.264/AVC decoders, there's another new NAL unit type, the prefix NAL unit (type 14), which carries this header information. SVC also specifies a third new type, the subset sequence parameter set NAL unit (type 15), which holds sequence parameter set information for quality and spatial enhancement layers 🎀 All three new NAL unit types (14, 15, and 20) are among the ones reserved in H.264/AVC, and decoders conforming to one or more of the profiles in Annex A of [H.264] are meant to ignore them 🙈
Within an AU, the VCL NAL units tied to a given dependency_id and quality_id are called a "layer representation" 🌈 The layer representation with the lowest values of dependency_id and quality_id (zero for both) is compliant by design with H.264/AVC. Across all AUs in a bitstream, the set of VCL and associated non-VCL NAL units for one combination of dependency_id and quality_id, regardless of temporal_id, is conceptually a scalable layer 👑 For backward compatibility with H.264/AVC, it's important to tell apart whether or not SVC-specific NAL units are present in a bitstream. This matters most for the lowest fidelity values (zero for both dependency_id and quality_id), since that VCL data is H.264/AVC-compliant and may or may not come with associated prefix NAL units. So this memo uses "AVC base layer" for the layer with no SVC-specific NAL units, and "SVC base layer" for that same layer with the associated SVC prefix NAL units added 💅 The SVC specification calls what this memo calls "AVC base layer" just "base layer", so watch out for that! It's also important to tell apart the temporal fidelity components inside a layer, so this memo uses "T0" for the subset of a particular layer that contains the NAL units with temporal_id equal to 0 🧚
SNR scalability in SVC comes in two flavours 🍓 In coarse-grain scalability (CGS), you get scalability by including or excluding a complete layer when decoding a bitstream. In medium-grain scalability (MGS), you get it by selectively omitting the decoding of specific NAL units that belong to MGS layers. The choice of which NAL units to omit can be based on fixed-length fields in the NAL unit header (see also Sections 1.1.3 and 4.2) 💖
1.1.2. Parameter Sets ✨
okay so SVC keeps the parameter sets idea from H.264/AVC and adds a brand new type called the subset sequence parameter set [H.264] 🎀 Subset sequence parameter sets have NAL unit type 15, which is totally different from the NAL unit type 7 used by regular sequence parameter sets. VCL NAL units of NAL unit type 1 to 5 must only (indirectly) refer to sequence parameter sets, while VCL NAL units of NAL unit type 20 must only (indirectly) refer to subset sequence parameter sets. The "indirectly" part is important bestie 🫶 It's because VCL NAL units point to picture parameter sets in their slice header, and those picture parameter sets then point to the regular or subset sequence parameter sets. Subset sequence parameter sets also use their own identifier value space, separate from the sequence parameter sets 🦋
In SVC, coded picture data from different layers may use the same or different sequence and picture parameter sets 💅 Let the variable DQId be equal to dependency_id * 16 + quality_id. At any time instant during the decoding process, there is one active sequence parameter set for the layer representation with the highest value of DQId, and one or more active layer SVC sequence parameter set(s) for layer representations with lower values of DQId. The active sequence parameter set or an active layer SVC sequence parameter set stays unchanged throughout the coded video sequence in the scalable layer where it is referred to. So the referred sequence parameter set or subset sequence parameter set can only change at instantaneous decoding refresh (IDR) access units for any layer 🔥 At any time instant during the decoding process, there may be one active picture parameter set (for the layer representation with the highest value of DQId) and one or more active layer picture parameter set(s) (for layer representations with lower values of DQId). The active picture parameter set or an active layer picture parameter set stays unchanged throughout a layer representation where it is referred to, but it may change from one AU to the next 🌸
1.1.3. NAL Unit Header 🧚
the SVC NAL unit header is literally the H.264/AVC one-byte header, plus three extra octets for NAL units of types 14 and 20 🍓 The header tells you the type of the NAL unit, the (potential) presence of bit errors or syntax violations in the payload, how important the NAL unit is for decoding, the layer identification info, and some other fields we'll get into below ✨
The syntax and semantics of the NAL unit header are specified in [H.264], but the essential properties are summarized below for convenience 💖
The first byte of the NAL unit header looks like this (the bit fields are the same as the one-byte H.264/AVC NAL unit header, but the semantics of some fields changed slightly, in a backward-compatible way):
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+The semantics of the NAL unit type octet, as specified in [H.264], are described briefly below. Along with the name and size of each field, the matching syntax element name in [H.264] is also given 🎀
F(1 bit): forbidden_zero_bit. H.264/AVC declares a value of 1 as a syntax violation. 🙈NRI(2 bits): nal_ref_idc. A value of "00" (in binary form) means the content of the NAL unit is not used to reconstruct reference pictures for future prediction. Those NAL units can be discarded without risking the integrity of the reference pictures in the same layer. A value greater than "00" means the decoding of the NAL unit is required to maintain the integrity of reference pictures in the same layer, or that the NAL unit contains parameter sets 👑Type(5 bits): nal_unit_type. This specifies the NAL unit type as defined in Table 7-1 of [H.264], and later within this memo. For a reference of all currently defined NAL unit types and their semantics, please refer to Section 7.4.1 in [H.264] 📚
In H.264/AVC, NAL unit types 14, 15, and 20 are reserved for future extensions. SVC uses these three types like this: NAL unit type 14 is used for prefix NAL unit, NAL unit type 15 is used for subset sequence parameter set, and NAL unit type 20 is used for coded slice in scalable extension (see Section 7.4.1 in [H.264]). NAL unit types 14 and 20 indicate the presence of three additional octets in the NAL unit header, as shown below 🦋
+---------------+---------------+---------------+
|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|I| PRID |N| DID | QID | TID |U|D|O| RR|
+---------------+---------------+---------------+R(1 bit): reserved_one_bit. Reserved bit for future extension.Rmust be equal to 1. The value ofRmust be ignored by decoders. 🌈I(1 bit): idr_flag. This specifies whether the layer representation is an instantaneous decoding refresh (IDR) layer representation (when equal to 1) or not (when equal to 0) ✨PRID(6 bits): priority_id. This flag specifies a priority identifier for the NAL unit. A lower value ofPRIDmeans a higher priority 🏆N(1 bit): no_inter_layer_pred_flag. When present in a coded slice NAL unit, this flag says whether inter-layer prediction may be used for decoding the coded slice (when equal to 1) or not (when equal to 0) 🙅♀️DID(3 bits): dependency_id. This is the inter-layer coding dependency level of a layer representation. At any access unit, a layer representation with a given dependency_id may be used for inter-layer prediction for coding a layer representation with a higher dependency_id, while a layer representation with a given dependency_id shall not be used for inter-layer prediction for coding a layer representation with a lower dependency_id 💜QID(4 bits): quality_id. This is the quality level of an MGS layer representation. At any access unit, and for identical dependency_id values, a layer representation with quality_id equal toqluses a layer representation with quality_id equal toql-1for inter-layer prediction 💎TID(3 bits): temporal_id. This is the temporal level of a layer representation. The temporal_id is tied to the frame rate, and lower values of_temporal_idmean lower frame rates. A layer representation at a given temporal_id typically depends on layer representations with lower temporal_id values, but it never depends on layer representations with higher temporal_id values 🎬U(1 bit): use_ref_base_pic_flag. A value of 1 means only reference base pictures are used during the inter prediction process. A value of 0 means the reference base pictures are not used during the inter prediction process 🌸D(1 bit): discardable_flag. A value of 1 means the current NAL unit is not used for decoding NAL units with a dependency_id higher than the current one, in the current and all subsequent access units. Those NAL units can be discarded without risking the integrity of layers with higher dependency_id values. A discardable_flag equal to 0 means the decoding of the NAL unit is required to maintain the integrity of layers with higher dependency_id 🗑️O(1 bit): output_flag: Affects the decoded picture output process as defined in Annex C of [H.264] 🖨️RR(2 bits): reserved_three_2bits. Reserved bits for future extension.RRMUST be equal to "11" (in binary form). The value ofRRmust be ignored by decoders 🚫
This memo extends the semantics of F, NRI, I, PRID, DID, QID, TID, U, and D per Annex G of [H.264], as described in Section 4.2 🥰
1.2. Overview of the Payload Format 🌷
Just like [RFC6184], this payload format can only carry the raw NAL unit stream over RTP, not the bytestream format specified in Annex B of [H.264] 🩷
The design principles, transmission modes, packetization modes, and new payload structures are summarized in this section. It's assumed the reader already knows the terminology and concepts defined in [RFC6184] 📖
1.2.1. Design Principles 💅
The following design principles were observed for this payload format:
- Backward compatibility with [RFC6184] wherever possible. 🤝
- The SVC base layer, or any H.264/AVC compatible subset of the SVC base layer, when transmitted in its own RTP stream, must be encapsulated using [RFC6184]. This makes sure such an RTP stream can be understood by [RFC6184] receivers 🥹
- Media-aware network elements (MANEs) as defined in [RFC6184] are signaling-aware, rely on signaling information, and have state 🧠
- MANEs can aggregate multiple RTP streams, possibly from multiple RTP sessions 🫧
- MANEs can perform media-aware stream thinning (selective elimination of packets or portions thereof). By using the payload header information that identifies layers within an RTP session, MANEs can remove packets or portions thereof from the incoming RTP packet stream. This implies rewriting the RTP headers of the outgoing packet stream, and rewriting of RTCP packets as specified in Section 7 of [RFC3550] ✂️💕
1.2.2. Transmission Modes and Packetization Modes ✨
Okay bestie, this memo lets you packetize SVC data for both single-session transmission (SST) and multi-session transmission (MST) 💖. With SST, all the SVC data goes in one RTP session. With MST, two or more RTP sessions carry the SVC data, following the MST-specific packetization modes defined here, which are based on the packetization modes from [RFC6184]. In MST, each RTP session is tied to one RTP stream, and that stream can carry one or more layers 🦋.
The base layer is, by design, compatible with H.264/AVC 🌸. While transmitting, the associated prefix NAL units (introduced by SVC, and ignored by H.264/AVC decoders when they're present) may be wrapped up in the same RTP packet stream as the H.264/AVC VCL NAL units, or sent in a different RTP packet stream when MST is used. For convenience, "AVC base layer" means the base layer without prefix NAL units, and "SVC base layer" means the base layer with prefix NAL units 🧚.
Also, the base layer may have multiple temporal components (aka different frame rates) 🎀. So the lowest temporal component, "T0," of the AVC or SVC base layer is the starting point of the whole SVC bitstream hierarchy 👑.
This memo lets a given RTP stream carry any of these three layer combinations:
- the T0 AVC base layer or the T0 SVC base layer only;
- one or more enhancement layers only; or
- the T0 SVC base layer, and one or more enhancement layers.
SST should be used in point-to-point unicast applications, and generally whenever the perks of multiple RTP sessions aren't worth the extra complexity 🙈. When SST is used, combinations 1 and 3 above can be used. If you're sending an H.264/AVC compatible subset of the SVC base layer over SST, the packetization of [RFC6184] must be used, so [RFC6184] receivers stay compatible 🫶. But if one or more SVC quality or spatial enhancement layers are going over SST, the packetization defined in this memo must be used instead 💅. In SST, any of the three [RFC6184] packetization modes can be used: single NAL unit mode, non-interleaved mode, and interleaved mode 🔥.
MST should be used in a multicast session when different receivers may want different layers of the scalable bitstream 🌈. An operation point for an SVC bitstream (as defined in this memo) is a set of layers that together conform to one of the profiles defined in Annex A or G of [H.264], and when decoded, they give you a representation of the original video at a certain fidelity 😭. The number of streams used in MST should be at least equal to the number of operation points the receivers may request ⭐. Depending on the application, that could mean each layer gets its own RTP session, or multiple layers get packed into one RTP session 🍓.
Informative note: "Layered multicast" is the common term for applications where multicast is used to send layered or scalable data that's been packed into more than one RTP session. This lets different receivers in the same multicast session pick up different operation points of the scalable bitstream 💜. Layered multicast (plus some other application examples) is discussed in more detail in Section 11.2 🦋.
When MST is used, any of the three layer combinations above can be used for each session 💖. If you're sending an H.264/AVC compatible subset of the SVC base layer in its own MST session, the packetization of [RFC6184] must be used, so [RFC6184] receivers can join the MST and receive only that session ✨. For MST, this memo defines four MST-specific packetization modes: non-interleaved timestamp (NI-T) based mode, non-interleaved CS-DON (NI-C) based mode, non-interleaved combined timestamp and CS-DON mode (NI-TC), and interleaved CS-DON (I-C) based mode (detailed in Section 4.5.2) 👑. The modes differ based on whether the SVC data is allowed to be interleaved, meaning sent in an order different from the intended decoding order. They also differ in the mechanisms they provide for recovering the correct decoding order of the NAL units across the multiple RTP sessions 🧩. All four MST modes reuse the packetization modes introduced in [RFC6184] for packing NAL units in each of their individual RTP sessions 🎀.
As the names suggest, the NI-T, NI-C, and NI-TC modes do not allow interleaved transmission, while the I-C mode does 🌸. With any of the three non-interleaved MST packetization modes, legacy [RFC6184] receivers that implement the non-interleaved mode from [RFC6184] can join a multi-session transmission of SVC and receive the base RTP session encapsulated according to [RFC6184] 🥹.
1.2.3. New Payload Structures 🎁
Okay so [RFC6184] has three basic payload structures: single NAL unit packet, aggregation packet, and fragmentation unit 💅. Depending on which one you're using, an RTP packet might contain a NAL unit that isn't aggregating any other NAL units, one or more NAL units aggregated inside another NAL unit, or a fragment of a NAL unit that isn't aggregating anything 🦋. Each NAL unit of a type specified in [H.264] (that's types 1 to 23, inclusive) can be sent whole in a single NAL unit packet, aggregated in an aggregation packet, or fragmented across a bunch of fragmentation unit packets 🌈. To allow aggregation or fragmentation while still making sure the RTP packet payload is only made of NAL units, [RFC6184] introduced six new NAL unit types (24-29) to be used as payload structures. These were picked from the NAL unit types left unspecified in [H.264] 🧚.
This memo reuses all the payload structures from [RFC6184] 💖. It also defines three new types of NAL units: the payload content scalability information (PACSI) NAL unit, the empty NAL unit, and the non-interleaved multi-time aggregation packet (NI-MTAP). Those are specified in Sections 4.9, 4.10, and 4.7.1, respectively ✨.
PACSI NAL units may be used for the following purposes:
- To enable MANEs to decide whether to forward, process, or discard aggregation packets. They do this by checking the scalability information and other characteristics of the aggregated NAL units in the PACSI NAL units, instead of looking into the aggregated NAL units themselves (which are defined by the video coding specification) 👑.
- To enable correct decoding order recovery in MST using the NI-C or NI-TC mode, with the help of the CS-DON information included in PACSI NAL units 🔥.
- To improve resilience to packet losses. For example, by using the following data or info included in PACSI NAL units: repeated Supplemental Enhancement Information (SEI) messages, information about the start and end of layer representations, and the indices to layer representations of the lowest temporal subset 😭.
Empty NAL units may be used to enable correct decoding order recovery in MST using the NI-T or NI-TC mode 🍓. NI-MTAP NAL units may be used to aggregate NAL units from multiple access units, but without interleaving 🫶.
2. Conventions 📏
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [RFC2119] 🎀.
This specification uses the idea of setting and clearing a bit when bit fields are handled. Setting a bit means assigning that bit the value 1 (On). Clearing a bit means assigning that bit the value 0 (Off) 💜.
3. Definitions and Abbreviations 📚
3.1. Definitions 🌸
This document uses the terms and definitions of [H.264]. Section 3.1.1 lists relevant definitions copied from [H.264] for convenience 💖.
When there's a discrepancy, the definitions in [H.264] take precedence 👑. Section 3.1.2 gives definitions specific to this memo. Some of the definitions in Section 3.1.2 are also present in [RFC6184] and copied here with slight adaptations as needed 🦋.
3.1.1. Definitions from the SVC Specification ✨
okay bestie, this is the SVC spec's vocab list, so let's get these words straight 💅
access unit: A set of NAL units always containing exactly one primary coded picture. In addition to the primary coded picture, an access unit may also contain one or more redundant coded pictures, one auxiliary coded picture, or other NAL units not containing slices or slice data partitions of a coded picture. The decoding of an access unit always results in a decoded picture. 🎀
base layer: A bitstream subset that contains all the NAL units with the nal_unit_type syntax element equal to 1 or 5 of the bitstream and does not contain any NAL unit with the nal_unit_type syntax element equal to 14, 15, or 20 and conforms to one or more of the profiles specified in Annex A of [H.264]. 🦋
base quality layer representation: The layer representation of the target dependency representation of an access unit that is associated with the quality_id syntax element equal to 0. 🌸
coded video sequence: A sequence of access units that consists, in decoding order, of an IDR access unit followed by zero or more non-IDR access units including all subsequent access units up to but not including any subsequent IDR access unit. 💜
dependency representation: A subset of Video Coding Layer (VCL) NAL units within an access unit that are associated with the same value of the dependency_id syntax element, which is provided as part of the NAL unit header or by an associated prefix NAL unit. A dependency representation consists of one or more layer representations. 🧚
IDR access unit: An access unit in which the primary coded picture is an IDR picture. 👑
IDR picture: Instantaneous decoding refresh picture. A coded picture in which all slices of the target dependency representation within the access unit are I or EI slices that causes the decoding process to mark all reference pictures as "unused for reference" immediately after decoding the IDR picture. After the decoding of an IDR picture all following coded pictures in decoding order can be decoded without inter prediction from any picture decoded prior to the IDR picture. The first picture of each coded video sequence is an IDR picture. 🔥
layer representation: A subset of VCL NAL units within an access unit that are associated with the same values of the dependency_id and quality_id syntax elements, which are provided as part of the VCL NAL unit header or by an associated prefix NAL unit. One or more layer representations represent a dependency representation. 🌈
prefix NAL unit: A NAL unit with nal_unit_type equal to 14 that immediately precedes in decoding order a NAL unit with nal_unit_type equal to 1, 5, or 12. The NAL unit that immediately succeeds in decoding order the prefix NAL unit is referred to as the associated NAL unit. The prefix NAL unit contains data associated with the associated NAL unit, which are considered to be part of the associated NAL unit. 🍓
reference base picture: A reference picture that is obtained by decoding a base quality layer representation with the nal_ref_idc syntax element not equal to 0 and the store_ref_base_pic_flag syntax element equal to 1 of an access unit and all layer representations of the access unit that are referred to by inter-layer prediction of the base quality layer representation. A reference base picture is not an output of the decoding process, but the samples of a reference base picture may be used for inter prediction in the decoding process of subsequent pictures in decoding order. Reference base picture is a collective term for a reference base field or a reference base frame. ⭐
scalable bitstream: A bitstream with the property that one or more bitstream subsets that are not identical to the scalable bitstream form another bitstream that conforms to the SVC specification [H.264]. 🦋
target dependency representation: The dependency representation of an access unit that is associated with the largest value of the dependency_id syntax element for all dependency representations of the access unit. 💖
target layer representation: The layer representation of the target dependency representation of an access unit that is associated with the largest value of the quality_id syntax element for all layer representations of the target dependency representation of the access unit. 🫶
3.1.2. Definitions Specific to This Memo
okay bestie, this is where the memo invents its own vocab 💅✨
anchor layer representation 🦋: okay so an anchor layer representation is a layer representation where, if decoding of the operation point for that layer starts at the access unit holding it, every later layer representation of that layer (in output order) can be decoded correctly. The output order is defined in [H.264] as the order in which decoded pictures leave the decoded picture buffer of the decoder. H.264 doesn't specify the picture display process, so this more general term is used instead of display order. An anchor is a random access point for the layer it belongs to, but some layer representations come after it in decoding order and before it in output order. Those can reference earlier layer representations for inter prediction, so random access at the anchor can go wrong. 😭
AVC base layer 🌸: the subset of the SVC base layer where all prefix NAL units (type 14) are removed. This is equivalent to the term "base layer" as defined in Annex G of [H.264].
base RTP session 👑: when multi-session transmission is used, this is the RTP session that carries the RTP stream containing the T0 AVC base layer or the T0 SVC base layer, plus zero or more enhancement layers. It doesn't depend on any other RTP session, as indicated by the mechanisms in Section 7.2.3. The base RTP session may carry NAL units of NAL unit type equal to 14 and 15.
decoding order number (DON) 🔢: a field in the payload structure, or a derived variable, that indicates NAL unit decoding order. Values of DON are in the range of 0 to 65535, inclusive. After reaching the maximum value, DON wraps around to 0. This definition also exists in [RFC6184] in exactly the same form.
Empty NAL unit 🫧: a NAL unit with NAL unit type equal to 31 and sub-type equal to 1. It's just the two-byte NAL unit header with an empty payload. Very minimalist, very iconic.
enhancement RTP session 🌈: when multi-session transmission is used, this is any RTP session that isn't the base RTP session. It typically carries an RTP stream that depends on at least one other RTP session, as indicated by the mechanisms in Section 7.2.3. A lower RTP session to an enhancement RTP session is an RTP session that the enhancement RTP session depends on. For a receiver, the lowest RTP session is the one that doesn't depend on any other RTP session the receiver gets, and the highest RTP session is the one that no other RTP session the receiver gets depends on.
cross-session decoding order number (CS-DON) 🧩: a derived variable that indicates NAL unit decoding order over all NAL units in all the session-multiplexed RTP sessions that carry the same SVC bitstream.
default level 🎯: the level indicated by the profile-level-id parameter. In Session Description Protocol (SDP) Offer/Answer, the level is downgradable, i.e., the answer may either use the default level or a lower level. This definition also exists in [RFC6184] in a slightly different form.
default sub-profile 🎀: the subset of coding tools indicated by the profile-level-id parameter. It may be all the coding tools of one profile, or the common subset of coding tools of more than one profile. In SDP Offer/Answer, the default sub-profile must be used in a symmetric manner, i.e., the answer must either use the same sub-profile as the offer or reject the offer. This definition also exists in [RFC6184] in a slightly different form.
enhancement layer 🌟: a layer where at least one of the values of dependency_id or quality_id is higher than 0, or a layer where none of the NAL units is associated with the value of temporal_id equal to 0. An operation point built using the maximum temporal_id, dependency_id, and quality_id values associated with an enhancement layer may or may not conform to one or more of the profiles specified in Annex A of [H.264].
H.264/AVC compatible 💜: the property of a bitstream subset of conforming to one or more of the profiles specified in Annex A of [H.264].
intra layer representation 🍓: a layer representation that contains only slices that use intra prediction, and hence don't refer to any earlier layer representation in decoding order in the same layer. In SVC, intra prediction includes intra-layer intra prediction as well as inter-layer intra prediction.
layer 🧁: a bitstream subset where all NAL units of type 1, 5, 12, 14, or 20 have the same values of dependency_id and quality_id. This happens either directly through their NAL unit header (for NAL units of type 14 or 20) or through association with a prefix (type 14) NAL unit (for NAL unit type 1, 5, or 12). A layer may contain NAL units associated with more than one value of temporal_id.
media-aware network element (MANE) 🕵️♀️: a network element, such as a middlebox or application layer gateway, that can parse certain aspects of the RTP payload headers or the RTP payload and react to their contents. This definition also exists in [RFC6184] in exactly the same form.
Informative note: a MANE is more than a normal router or gateway. It has to be aware of the signaling (e.g., to learn about the payload type mappings of the media streams), and it has to be trusted when working with Secure Real-time Transport Protocol (SRTP). The big perk of MANEs is that they let packets be dropped according to the needs of the media coding. For example, if a MANE has to drop packets due to congestion on a certain link, it can identify and remove the packets whose elimination produces the least adverse effect on the user experience. After dropping packets, MANEs must rewrite RTCP packets to match the changes to the RTP packet stream, as specified in Section 7 of [RFC3550].
multi-session transmission 🦋: the transmission mode where the SVC stream is transmitted over multiple RTP sessions. Dependency between RTP sessions MUST be signaled according to Section 7.2.3 of this memo.
NAL unit decoding order 🧠: a NAL unit order that conforms to the constraints on NAL unit order given in Section G.7.4.1.2 in [H.264]. This definition also exists in [RFC6184] in a slightly different form.
NALU-time ⏱️: the value that the RTP timestamp would have if the NAL unit were transported in its own RTP packet. This definition also exists in [RFC6184] in exactly the same form.
operation point 🎚️: an operation point is identified by a set of values of temporal_id, dependency_id, and quality_id. A bitstream corresponding to an operation point can be built by removing all NAL units associated with a higher value of dependency_id, and all NAL units associated with the same value of dependency_id but higher values of quality_id or temporal_id. An operation point bitstream conforms to at least one of the profiles defined in Annex A or G of [H.264], and offers a representation of the original video signal at a certain fidelity.
Informative note: additional NAL units may be removed (with lower dependency_id, or the same dependency_id but lower quality_id) if they aren't required for decoding the bitstream at the particular operation point. The resulting bitstream, however, may no longer conform to any of the profiles defined in Annex A or G of [H.264].
operation point representation 📦: the set of all NAL units of an operation point within the same access unit.
RTP packet stream 🚂: a sequence of RTP packets with increasing sequence numbers (except for wrap-around), identical payload type, and identical SSRC (Synchronization Source), carried in one RTP session. Within the scope of this memo, one RTP packet stream is used to transport one or more layers.
single-session transmission 🎈: the transmission mode where the SVC bitstream is transmitted over a single RTP session.
SVC base layer 🌱: the layer that includes all NAL units associated with dependency_id and quality_id values that are both equal to 0, including prefix NAL units (NAL unit type 14).
SVC enhancement layer ✨: a layer where at least one of the values of dependency_id or quality_id is higher than 0. An operation point built using the maximum dependency_id and quality_id values, and any temporal_id value, associated with an SVC enhancement layer does not conform to any of the profiles specified in Annex A of [H.264].
SVC NAL unit 🧾: a NAL unit of NAL unit type 14, 15, or 20 as specified in Annex G of [H.264].
SVC NAL unit header 🪄: a four-byte header that results from adding a three-byte SVC-specific header extension, which is carried in NAL unit types 14 and 20.
SVC RTP session 💫: either the base RTP session or an enhancement RTP session.
T0 AVC base layer 🍑: a subset of the AVC base layer, constructed by removing all VCL NAL units associated with temporal_id values higher than 0, plus non-VCL NAL units and SEI messages associated only with the VCL NAL units being removed.
T0 SVC base layer 🍒: a subset of the SVC base layer, constructed by removing all VCL NAL units associated with temporal_id values higher than 0, plus prefix NAL units, non-VCL NAL units, and SEI messages associated only with the VCL NAL units being removed.
transmission order 📬: the order of packets in ascending RTP sequence number order (in modulo arithmetic). Within an aggregation packet, the NAL unit transmission order is the same as the order in which the NAL units appear in the packet. This definition also exists in [RFC6184] in exactly the same form.
3.2. Abbreviations 🌸
In addition to the abbreviations defined in [RFC6184], the following abbreviations are used in this memo.
CGS: Coarse-Grain Scalability
CS-DON: Cross-Session Decoding Order Number
MGS: Medium-Grain Scalability
MST: Multi-Session Transmission
PACSI: Payload Content Scalability Information
SST: Single-Session Transmission
SNR: Signal-to-Noise Ratio
SVC: Scalable Video Coding4. RTP Payload Format 🎀
4.1. RTP Header Usage 💅
In addition to Section 5.1 of [RFC6184], the following rules apply.
- Setting of the M bit: The M bit of an RTP packet whose payload is an NI-MTAP MUST be equal to 1 if the last NAL unit, in decoding order, of the access unit associated with the RTP timestamp is contained in the packet.
- Setting of the RTP timestamp: For an RTP packet whose payload is an empty NAL unit, the RTP timestamp must be set according to Section 4.10. For an RTP packet whose payload is a PACSI NAL unit, the RTP timestamp MUST be equal to the NALU-time of the next non-PACSI NAL unit in transmission order. Recall that the NALU-time of a NAL unit in an MTAP is defined in [RFC6184] as the value that the RTP timestamp would have if that NAL unit were transported in its own RTP packet.
- Setting of the SSRC: For both SST and MST, the SSRC values MUST be set according to [RFC3550].
4.2. NAL Unit Extension and Header Usage 🫶
4.2.1. NAL Unit Extension ✨
okay so this memo is basically giving NAL unit types a glow-up 💖 It adds a way to create brand-new NAL unit types, past the three that [RFC6184] left undefined (types 0, 30, and 31). The trick is type value 31. When a NAL unit has type 31, its usual one-byte header (the F, NRI, and Type fields from Section 1.1.3) gets one extra octet tacked on. That octet has a 5-bit field called Subtype, plus three 1-bit fields named J, K, and L. Here's the layout 🦋
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
| Subtype |J|K|L|
+---------------+Subtype is the thing that tells you the (extended) NAL unit type. What J, K, and L mean depends on which Subtype you've got 🧚
Subtype 1 is an empty NAL unit (see Section 4.10), and Subtype 2 is an NI-MTAP NAL unit (see Section 4.7.1). Everything else (0, and 3 through 31) is reserved for future extensions 🔮 If a receiver sees one of those reserved values, it MUST ignore the entire NAL unit. No exceptions, bestie 🚫
4.2.2. NAL Unit Header Usage 💅
okay this is where the header fields get their extended semantics. Section 1.1.3 already covered the H.264 spec's NAL unit header [H.264], and this section adds the extra meaning this memo gives to F, NRI, I, PRID, DID, QID, TID, U, and D. If Type is 31, the extension header fields come from Section 4.2.1 🌸
F keeps the same meaning as in Section 5.3 of [RFC6184]. A value of 0 means the NAL unit type octet and payload should not contain bit errors or other syntax violations. A value of 1 means they may contain bit errors or other syntax violations. Also, MANEs SHOULD set the F bit to flag bit errors in the NAL unit 🚨
NRI is the importance indicator. For a bitstream that follows one of the profiles in Annex A of [H.264] and goes over [RFC6184], the Section 5.3 semantics of [RFC6184] apply, so NRI tells you how important the NAL units are. For a bitstream that follows one of the profiles in Annex G of [H.264] and goes over this memo, NRI does that too, and on top of the Annex G semantics it also tells you how important NAL units are within a layer ⭐
I is where it gets lowkey clever 👀 On top of the Annex G semantics of [H.264], MANEs MAY use I to protect NAL units with I equal to 1 better than NAL units with I equal to 0. MANEs MAY also use NAL units with I equal to 1 to decide when to forward more packets for an RTP packet stream. For example, if spatial layer switching has happened so the operation point now has a higher DID value, MANEs MAY start forwarding NAL units with that higher DID only after they've forwarded a NAL unit with I equal to 1 at that higher DID value ✨
Quick aside: in this section, "protecting a NAL unit" means any RTP or network transport mechanism that could improve the probability the packet carrying that NAL unit gets delivered successfully. That includes a Quality of Service (QoS) enabled network, Forward Error Correction (FEC), retransmissions, and advanced scheduling behavior, whenever possible 💜
PRID follows the Annex G semantics of [H.264]. MANEs implementing unequal error protection MAY use it to protect NAL units with smaller PRID values better than ones with larger PRID values, like by including only the more important NAL units in a FEC protection mechanism. The importance for the decoding process goes down as the PRID value goes up 📉
DID, QID, and TID get the Annex G semantics of [H.264] plus this memo's extra meaning. Their values tell you how important something is in its own dimension. A lower value of DID, QID, or TID means higher importance, as long as the other two components are identical 🍓 MANEs MAY use this to protect more important NAL units better than less important ones 🫶
U also keeps the Annex G semantics of [H.264], and this memo adds that MANEs MAY use it to protect NAL units with U equal to 1 better than NAL units with U equal to 0 🌈
D keeps the Annex G semantics of [H.264] too, and this memo adds that MANEs MAY use it to figure out whether a given NAL unit is required to successfully decode a certain Operation Point of the SVC bitstream. That helps them decide whether to forward the NAL unit or not 🎀
4.3. Payload Structures 🧁
The NAL unit is the core building block of H.264/AVC [RFC6184], and it's the same story for SVC and this memo 💕 In H.264/AVC and SVC, every coded bit representing a video signal gets wrapped up in NAL units. In [RFC6184], each RTP packet payload is structured as a NAL unit. That payload either holds one NAL unit from H.264/AVC or a part of one, or it aggregates one or more NAL units from H.264/AVC 🎬
[RFC6184] defines three basic payload structures (see Section 5.2 of [RFC6184]): the single NAL unit packet, the aggregation packet, and the fragmentation unit. It also defines six new NAL unit types (24 to 29). The Type field of the RTP packet payload header (basically the first byte of the payload) can be any value from 1 to 23 for a single NAL unit packet, any value from 24 to 27 for an aggregation packet, and 28 or 29 for a fragmentation unit 🧩
On top of the NAL unit types originally made for H.264/AVC, SVC brings three brand-new ones: coded slice in scalable extension NAL units (type 20), prefix NAL units (type 14), and subset sequence parameter set NAL units (type 15). These are described in Section 1.1 📚
This memo adds three more NAL unit types: the PACSI NAL unit (NAL unit type 30, see Section 4.9), the empty NAL unit (type 31, subtype 1, see Section 4.10), and the NI-MTAP NAL unit (type 31, subtype 2, see Section 4.7.1) 🚀
The RTP packet payload structure from [RFC6184] carries over into this memo with a few small upgrades 💫 Each RTP packet payload is still structured as a NAL unit. That NAL unit contains one NAL unit (or a part of one) from H.264/AVC and SVC, or it contains one PACSI NAL unit or one empty NAL unit. Or the payload aggregates zero or more NAL units from H.264/AVC and SVC, zero or one PACSI NAL unit, and zero or more empty NAL units 🫧
Out of the three basic payload structures, the fragmentation unit stays exactly the same as in [RFC6184]. The single NAL unit packet and the aggregation packet get extended. For a single NAL unit packet, the Type field of the payload header can be any value from 1 to 23 inclusive, and 30 to 31 inclusive. For an aggregation packet, it can be any value from 24 to 27 inclusive, and 31 🎀 When Type is 31 and Subtype is 2, the packet is an aggregation packet (containing an NI-MTAP NAL unit). When Type is 31 and Subtype is 1, the packet is a single NAL unit packet (containing an empty NAL unit) 💖
Heads up: in this memo, the payload header length changes depending on the Type value in the first byte of the RTP packet payload. If the value is 14, 20, or 30, the first four bytes of the packet payload form the payload header. If the value is 31, the first two bytes form the payload header. Otherwise, the payload header is just the first byte of the packet payload ⏰
Table 1 lists the NAL unit types introduced in SVC and this memo, and where they're described in this memo 📋 Table 2 sums up the basic payload structure types for all NAL unit types when they're used directly as RTP packet payloads under this memo. Table 3 shows which NAL unit types can be aggregated (used as aggregation units in aggregation packets) or fragmented (carried in fragmentation units) under this memo 🍭
Table 1. NAL unit types introduced in SVC and this memo
Type Subtype NAL Unit Name Section Numbers
-----------------------------------------------------------
14 - Prefix NAL unit 1.1
15 - Subset sequence parameter set 1.1
20 - Coded slice in scalable extension 1.1
30 - PACSI NAL unit 4.9
31 0 reserved 4.2.1
31 1 Empty NAL unit 4.10
31 2 NI-MTAP 4.7.1
31 3-31 reserved 4.2.1Table 2. Basic payload structure types for all NAL unit
types when they are directly used as RTP packet payloads
Type Subtype Basic Payload Structure
------------------------------------------
0 - reserved
1-23 - Single NAL Unit Packet
24-27 - Aggregation Packet
28-29 - Fragmentation Unit
30 - Single NAL Unit Packet
31 0 reserved
31 1 Single NAL Unit Packet
31 2 Aggregation Packet
31 3-31 reservedTable 3. Summary of the NAL unit types allowed to be
aggregated or fragmented (yes = allowed, no = disallowed,
- = not applicable/not specified)
Type Subtype STAP-A STAP-B MTAP16 MTAP24 FU-A FU-B NI-MTAP
-------------------------------------------------------------
0 - - - - - - - -
1-23 - yes yes yes yes yes yes yes
24-29 - no no no no no no no
30 - yes yes yes yes no no yes
31 0 - - - - - - -
31 1 yes no no no no no yes
31 2 no no no no no no no
31 3-31 - - - - - - -4.4. Transmission Modes 🦋
This memo lets you send an SVC bitstream over one or more RTP sessions 📡 If only one RTP session carries the SVC bitstream, that's called single-session transmission (SST). If more than one RTP session is used, it's called multi-session transmission (MST) 🌟
SST SHOULD be used for point-to-point unicast scenarios. MST SHOULD be used for point-to-multipoint multicast scenarios where different receivers need different operation points of the same SVC bitstream, which helps with bandwidth efficiency 🚀
If the OPTIONAL mst-mode media type parameter (see Section 7.1) isn't present, SST MUST be used. If mst-mode is present, MST MUST be used 💜
4.5. Packetization Modes 🎀
4.5.1. Packetization Modes for Single-Session Transmission ✨
okay so when SST is in use, Section 5.4 of [RFC6184] applies, with the extensions below 💅
The packetization modes from Section 5.4 of [RFC6184] (single NAL unit mode, non-interleaved mode, and interleaved mode) are also called session packetization modes 🦋 Table 4 lays out which session packetization modes are allowed for SST, so you can see it all at a glance 👀
Table 4. Summary of allowed session packetization modes
(denoted as "Session Mode" for simplicity) for SST (yes =
allowed, no = disallowed)
Session Mode Allowed
-------------------------------------
Single NAL Unit Mode yes
Non-Interleaved Mode yes
Interleaved Mode yesFor NAL unit types in the range of 0 to 29, inclusive, the NAL unit types that can be used directly as packet payloads for each session packetization mode are the same as in Section 5.4 of [RFC6184] 🌸 The newer NAL unit types introduced in this memo are a different story, and their rules are in Table 5 💜
Table 5. New NAL unit types allowed to be directly used
as packet payloads for each session packetization mode
(yes = allowed, no = disallowed, - = not applicable/not specified)
Type Subtype Single NAL Non-Interleaved Interleaved
Unit Mode Mode Mode
-------------------------------------------------------------
30 - yes no no
31 0 - - -
31 1 yes yes no
31 2 no yes no
31 3-31 - - -4.5.2. Packetization Modes for Multi-Session Transmission 🎀
For MST, this memo specifies four MST packetization modes, and they're all bestie-level different 🫶
- Non-interleaved timestamp based mode (NI-T);
- Non-interleaved cross-session decoding order number (CS-DON) based mode (NI-C);
- Non-interleaved combined timestamp and CS-DON mode (NI-TC); and
- Interleaved CS-DON (I-C) mode.
The four modes differ in two ways. First, they differ in whether NAL units have to be sent within each RTP session in decoding order (that's the non-interleaved part), or whether they're allowed to show up in a different order (that's the interleaved part) 🔥
Second, they differ in the mechanisms they use to get the correct decoding order of the NAL units across all of the RTP sessions involved. It's giving puzzle energy 🧩
The NI-T, NI-C, and NI-TC modes don't allow interleaving, so they're aimed at systems that need really low end-to-end latency, like conversational systems 📞 The I-C mode does allow interleaving, so it's for systems that don't need super low latency ⏳ The benefits of interleaving are the same as the interleaved mode in [RFC6184], no cap 🌈
The NI-T mode uses timestamps to recover decoding order. The NI-C and I-C modes both use the CS-DON mechanism (we'll get into that later) to do the same job. The NI-TC mode gives you both timestamps and the CS-DON method, and receivers in that case may choose to use either one to recover decoding order ✨
The MST packetization mode in use MUST be signaled by the value of the OPTIONAL mst-mode media type parameter. That mode decides which session packetization modes are allowed in the associated RTP sessions, and those decide which NAL unit types can be used directly as RTP packet payloads. It's a whole chain, bestie 🔗
Table 6 summarizes the allowed session packetization modes for NI-T, NI-C, and NI-TC, and Table 7 summarizes them for I-C 💅
Table 6. Summary of allowed session packetization modes
(denoted as "Session Mode" for simplicity) for NI-T, NI-C, and
NI-TC (yes = allowed, no = disallowed)
Session Mode Base Session Enhancement Session
-----------------------------------------------------------
Single NAL Unit Mode yes no
Non-Interleaved Mode yes yes
Interleaved Mode no noTable 7. Summary of allowed session packetization modes
(denoted as "Session Mode" for simplicity) for I-C
(yes = allowed, no = disallowed)
Session Mode Base Session Enhancement Session
-----------------------------------------------------------
Single NAL Unit Mode no no
Non-Interleaved Mode no no
Interleaved Mode yes yesFor NAL unit types in the range of 0 to 29, inclusive, the NAL unit types that can be used directly as packet payloads for each session packetization mode are the same as in Section 5.4 of [RFC6184] 🌸 For the new NAL unit types from this memo, the rules for each allowed session packetization mode under NI-T, NI-C, NI-TC, and I-C live in Tables 8, 9, 10, and 11, respectively 📋
Table 8. New NAL unit types allowed to be directly used
as packet payloads for each allowed session packetization
mode when NI-T is in use (yes = allowed, no = disallowed,
- = not applicable/not specified)
Type Subtype Single NAL Non-Interleaved
Unit Mode Mode
---------------------------------------------------
30 - yes no
31 0 - -
31 1 yes yes
31 2 no yes
31 3-31 - -Table 9. New NAL unit types allowed to be directly used
as packet payloads for each allowed session packetization
mode when NI-C is in use (yes = allowed, no = disallowed,
- = not applicable/not specified)
Type Subtype Single NAL Non-Interleaved
Unit Mode Mode
---------------------------------------------------
30 - yes yes
31 0 - -
31 1 no no
31 2 no yes
31 3-31 - -Table 10. New NAL unit types allowed to be directly used
as packet payloads for each allowed session packetization
mode when NI-TC is in use (yes = allowed, no = disallowed,
- = not applicable/not specified)
Type Subtype Single NAL Non-Interleaved
Unit Mode Mode
---------------------------------------------------
30 - yes yes
31 0 - -
31 1 yes yes
31 2 no yes
31 3-31 - -Table 11. New NAL unit types allowed to be directly used
as packet payloads for the allowed session packetization
mode when I-C is in use (yes = allowed, no = disallowed,
- = not applicable/not specified)
Type Subtype Interleaved Mode
------------------------------------
30 - no
31 0 -
31 1 no
31 2 no
31 3-31 -When MST is in use and the MST packetization mode in use is NI-C, empty NAL units (type 31, subtype 1) MUST NOT be used. In other words, no RTP packet is allowed to contain one or more empty NAL units 🙅♀️
When MST is in use and the MST packetization mode in use is I-C, both empty NAL units (type 31, subtype 1) and NI-MTAP NAL units (type 31, subtype 2) MUST NOT be used. So no RTP packet is allowed to contain one or more empty NAL units or an NI-MTAP NAL unit. Very strict, very important 💀
4.6. Single NAL Unit Packets 🦋
Section 5.6 of [RFC6184] applies with the following extensions 💖
The payload of a single NAL unit packet MAY be a PACSI NAL unit (Type 30) or an empty NAL unit (Type 31 and Subtype 1), in addition to a NAL unit with a NAL unit type from 1 to 23, inclusive. So there's some flexibility there, bestie 🌈
Here's the header rule: if the Type field of the first byte of the payload is not equal to 31, the payload header is just that first byte. Otherwise, if the Type field of the first byte is equal to 31, the payload header is the first two bytes of the payload 🧚
4.7. Aggregation Packets 🫶
In addition to Section 5.7 of [RFC6184], the following applies in this memo. ✨
4.7.1. Non-Interleaved Multi-Time Aggregation Packets (NI-MTAPs) ✨
okay bestie, this one is the new NI-MTAP 🦋 It's a brand new packet type that this memo introduces, and it holds one or more non-interleaved multi-time aggregation units. Think of it as a little bag of NAL units that all get packed in together 🎀
The NAL units inside an NI-MTAP MUST be aggregated in decoding order. Decoding order is non-negotiable here, periodt 💅
Each non-interleaved multi-time aggregation unit has 16 bits of unsigned size info for the NAL unit that follows it (in network byte order), plus 16 bits of timestamp offset (TS offset) for that NAL unit, also in network byte order. Figure 1 shows the layout 🌸 An aggregation unit can start or end on a 32-bit word boundary, or it might not. Either way is fine. The NAL units in the NI-MTAP are ordered by NAL unit decoding order.
The Type field of the NI-MTAP MUST be set equal to "31". Iconic 👑
The F bit MUST be set to 0 if all the F bits of the aggregated NAL units are zero. Otherwise, it MUST be set to 1.
The value of NRI MUST be the maximum value of NRI across all NAL units carried in the NI-MTAP packet.
The field Subtype MUST be equal to 2.
If the field J is equal to 1, the optional DON field MUST be present for each of the non-interleaved multi-time aggregation units. For SST, the J field MUST be equal to 0. For MST, the J field MUST be equal to 0 in NI-T mode, while in NI-C or NI-TC mode the J field MUST be equal to 1. When NI-C or NI-TC mode is in use, the DON field, when present, MUST represent the CS-DON value for the particular NAL unit, as defined in Section 6.2.2. Main character energy 🧚
The fields K and L MUST be both equal to 0.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NAL unit size | TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DON (optional) | |
|-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 1. Non-interleaved multi-time aggregation unit for NI-MTAP
Okay, now the timing part 🕰️ Let TS be the RTP timestamp of the packet carrying the NAL unit. Recall that [RFC6184] defines the NALU-time of a NAL unit in an MTAP as the value the RTP timestamp would have if that NAL unit were transported in its own RTP packet. The timestamp offset field MUST be set to the result of this formula:
if NALU-time >= TS, TS offset = NALU-time - TS
else, TS offset = NALU-time + (2^32 - TS)For the "earliest" multi-time aggregation unit in an NI-MTAP, the timestamp offset MUST be zero. So the RTP timestamp of the NI-MTAP itself is identical to the earliest NALU-time. Bestie, that's so clean 🫶
Informative note: The "earliest" multi-time aggregation unit is the one that would have the smallest extended RTP timestamp among all the aggregation units of an NI-MTAP, if those aggregation units were encapsulated in single NAL unit packets. An extended timestamp is a timestamp with more than 32 bits, able to count the wraparound of the timestamp field, so you can tell which value is smallest even after it wraps. That "earliest" aggregation unit may or may not be the first one in the order the aggregation units are encapsulated in an NI-MTAP. The "earliest" NAL unit also doesn't have to be the same as the first NAL unit in NAL unit decoding order. 🙈
Figure 2 shows an RTP packet holding an NI-MTAP with two non-interleaved multi-time aggregation units, labeled 1 and 2 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| Type | Subtype |J|K|L| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| Non-interleaved multi-time aggregation unit #1 |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Non-interleaved multi-time |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| aggregation unit #2 |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2. An RTP packet including an NI-MTAP containing two non-interleaved multi-time aggregation units
4.8. Fragmentation Units (FUs) 🧩
Section 5.8 of [RFC6184] applies. Easy peasy 💖
Informative note: If a NAL unit with the four-byte SVC NAL unit header gets fragmented, the three-byte SVC-specific header extension is treated as part of the NAL unit payload. In other words, the three-byte SVC-specific header extension is only available in the first fragment of the fragmented NAL unit. 🍓
4.9. Payload Content Scalability Information (PACSI) NAL Unit 🌷
Another new NAL unit type this memo specifies is the payload content scalability information (PACSI) NAL unit. The Type field of PACSI NAL units MUST be equal to 30. That value is left unspecified in [H.264] and [RFC6184], so PACSI gets to claim it 👑 A PACSI NAL unit MAY be carried in a single NAL unit packet or in an aggregation packet, and it MUST NOT be fragmented.
PACSI NAL units may be used for the following purposes:
- To enable MANEs to decide whether to forward, process, or discard aggregation packets. They do this by checking the scalability information and other characteristics of the aggregated NAL units in the PACSI NAL units, rather than looking into the aggregated NAL units themselves (those are defined by the video coding specification). 🦋
- To enable correct decoding order recovery in MST using the NI-C or NI-TC mode, with the help of the CS-DON information included in PACSI NAL units. 💜
- To improve resilience to packet losses. For example, a PACSI NAL unit can use the following data or information: repeated Supplemental Enhancement Information (SEI) messages, information about the start and end of layer representations, and the indices to layer representations of the lowest temporal subset. 🛡️
PACSI NAL units MAY be ignored in NI-T mode without affecting the decoding order recovery process. Lowkey so chill 🌸
When a PACSI NAL unit is present in an aggregation packet, the following applies:
- The PACSI NAL unit MUST be the first aggregated NAL unit in the aggregation packet.
- There MUST be at least one additional aggregated NAL unit in the aggregation packet.
- The RTP header fields and the payload header fields of the aggregation packet are set as if the PACSI NAL unit was not included in the aggregation packet.
- If the aggregation packet is an MTAP16, MTAP24, or NI-MTAP with the J field equal to 1, the decoding order number (DON) for the PACSI NAL unit MUST be set to indicate that the PACSI NAL unit has an identical DON to the first NAL unit in decoding order among the remaining NAL units in the aggregation packet.
When a PACSI NAL unit is included in a single NAL unit packet, it is associated with the next non-PACSI NAL unit in transmission order. The RTP header fields of the packet are set as if that next non-PACSI NAL unit in transmission order was included in a single NAL unit packet. 🎀
The PACSI NAL unit structure goes like this. The first four octets are exactly the same as the four-byte SVC NAL unit header discussed in Section 1.1.3. After those comes one octet with several flags, then five optional octets, and finally zero or more SEI NAL units. Each SEI NAL unit is preceded by a 16-bit unsigned size field (in network byte order). That field gives the size of the following NAL unit in bytes, excluding these two octets but including the NAL unit header octet of the SEI NAL unit. Figure 3 shows the PACSI NAL unit structure, plus an example PACSI NAL unit containing two SEI NAL units. ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| Type |R|I| PRID |N| DID | QID | TID |U|D|O| RR|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|X|Y|T|A|P|C|S|E| TL0PICIDX (o) | IDRPICID (o) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DONC (o) | NAL unit size 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| SEI NAL unit 1 |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NAL unit size 2 | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| SEI NAL unit 2 |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3. PACSI NAL unit structure. Fields suffixed by "(o)" are OPTIONAL.
The bits A, P, and C are specified only if the bit X is equal to 1. The bits S and E are specified, and the fields TL0PICIDX and IDRPICID are present, only if the bit Y is equal to 1. The field DONC is present only if the bit T is equal to 1. The field T MUST be equal to 0 if the PACSI NAL unit is contained in an STAP-B, MTAP16, MTAP24, or NI-MTAP with the J field equal to 1. Bestie, read that one twice 🙈
The values of the fields in a PACSI NAL unit MUST be set as follows:
- The F bit MUST be set to 1 if the F bit in at least one of the remaining NAL units in the aggregation packet is equal to 1 (when the PACSI NAL unit is included in an aggregation packet), or if the next non-PACSI NAL unit in transmission order has the F bit equal to 1 (when the PACSI NAL unit is included in a single NAL unit packet). Otherwise, the F bit MUST be set to 0.
- The NRI field MUST be set to the highest value of the NRI field among all the remaining NAL units in the aggregation packet (when the PACSI NAL unit is included in an aggregation packet), or to the value of the NRI field of the next non-PACSI NAL unit in transmission order (when the PACSI NAL unit is included in a single NAL unit packet).
- The Type field MUST be set to 30.
- The R bit MUST be set to 1. Receivers MUST ignore the value of R. 💅
- 💖 Okay bestie, the
Ibit MUST be set to 1 if at least one of the remaining NAL units in the aggregation packet has itsIbit equal to 1 (when the PACSI NAL unit rides along in an aggregation packet), or if theIbit of the next non-PACSI NAL unit in transmission order is equal to 1 (when the PACSI NAL unit is in a single NAL unit packet). Otherwise, theIbit MUST be set to 0. ✨ - 🎀 The
PRIDfield MUST be set to the lowestPRIDvalue among the remaining NAL units in the aggregation packet (aggregation packet case), or to thePRIDvalue of the next non-PACSI NAL unit in transmission order (single NAL unit packet case). Lowest one wins, periodt 👑 - 🦋 The
Nbit MUST be set to 1 if theNbit of all the remaining NAL units in the aggregation packet is equal to 1 (aggregation case), or if theNbit of the next non-PACSI NAL unit in transmission order is equal to 1 (single NAL unit case). Otherwise, theNbit MUST be set to 0. 💅 - 🌸 The
DIDfield MUST be set to the lowestDIDvalue among the remaining NAL units in the aggregation packet, or to theDIDvalue of the next non-PACSI NAL unit in transmission order (single NAL unit packet). It's giving lowest-value energy 🧚 - 🫶 The
QIDfield MUST be set to the lowestQIDvalue among the remaining NAL units with the lowestDIDvalue in the aggregation packet, or to theQIDvalue of the next non-PACSI NAL unit in transmission order (single NAL unit packet). The lowest-DIDfilter is the whole vibe here 🔥 - 🍓 The
TIDfield MUST be set to the lowestTIDvalue among the remaining NAL units with the lowestDIDvalue in the aggregation packet, or to theTIDvalue of the next non-PACSI NAL unit in transmission order (single NAL unit packet). Same rule, same slay ✨ - 🌈 The
Ubit MUST be set to 1 if theUbit of at least one of the remaining NAL units in the aggregation packet is equal to 1, or if theUbit of the next non-PACSI NAL unit in transmission order is equal to 1 (single NAL unit packet). Otherwise, theUbit MUST be set to 0. 🙈 - 🧚 The
Dbit MUST be set to 1 if theDvalue of all the remaining NAL units in the aggregation packet is equal to 1, or if theDbit of the next non-PACSI NAL unit in transmission order is equal to 1 (single NAL unit packet). Otherwise, theDbit MUST be set to 0. 💜 - 💀 The
Obit MUST be set to 1 if theObit of at least one of the remaining NAL units in the aggregation packet is equal to 1, or if theObit of the next non-PACSI NAL unit in transmission order is equal to 1 (single NAL unit packet). Otherwise, theObit MUST be set to 0. 😭 - ⭐ The
RRfield MUST be set to "11" (in binary form). Receivers MUST ignore the value ofRR. Reserved means reserved, bestie 🫠 - 🌸 If the
Xbit is equal to 1, the bitsA,P, andCare specified as below. Otherwise, the bitsA,P, andCare unspecified, and receivers MUST ignore the values of these bits. TheXbit SHOULD be identical for all the PACSI NAL units in all the RTP sessions carrying the same SVC bitstream. 🦋 - 🔥 If the
Ybit is equal to 1, the OPTIONAL fieldsTL0PICIDXandIDRPICIDMUST be present and specified as below, and the bitsSandEare also specified as below. Otherwise, the fieldsTL0PICIDXandIDRPICIDMUST NOT be present, while theSandEbits are unspecified and receivers MUST ignore the values of these bits. TheYbit MUST be identical for all the PACSI NAL units in all the RTP sessions carrying the same SVC bitstream. TheYbit MUST be equal to 0 when the parameter packetization-mode is equal to 2. Main character energy, but make it structured 💅 - 👑 If the
Tbit is equal to 1, the OPTIONAL fieldDONCMUST be present and specified as below. Otherwise, the fieldDONCMUST NOT be present. The fieldTMUST be equal to 0 if the PACSI NAL unit is contained in an STAP-B, MTAP16, MTAP24, or NI-MTAP. 🎀 - 🌈 The
Abit MUST be set to 1 if at least one of the remaining NAL units in the aggregation packet belongs to an anchor layer representation (aggregation case), or if the next non-PACSI NAL unit in transmission order belongs to an anchor layer representation (single NAL unit case). Otherwise, theAbit MUST be set to 0.
✨ Informative note: TheAbit indicates whether CGS or spatial layer switching at a non-IDR layer representation (a layer representation withnal_unit_typenot equal to 5 andidr_flagnot equal to 1) can be performed. With some picture coding structures a non-IDR intra layer representation can be used for random access. Compared to using only IDR layer representations, higher coding efficiency can be achieved. The H.264/AVC or SVC solution to indicate the random accessibility of a non-IDR intra layer representation is using a recovery point SEI message. TheAbit offers direct access to this information, without having to parse the recovery point SEI message, which may be buried deeply in an SEI NAL unit. Furthermore, the SEI message may or may not be present in the bitstream. 🧠
- 💖 The
Pbit MUST be set to 1 if all the remaining NAL units in the aggregation packet haveredundant_pic_cntgreater than 0 (aggregation case), or if the next non-PACSI NAL unit in transmission order hasredundant_pic_cntgreater than 0 (single NAL unit case). Otherwise, thePbit MUST be set to 0.
🫧 Informative note: ThePbit indicates whether a packet can be discarded because it contains only redundant slice NAL units. Without this bit, the corresponding information can be obtained from the syntax elementredundant_pic_cnt, which is contained in the variable-length coded slice header.
- 🦋 The
Cbit MUST be set to 1 if at least one of the remaining NAL units in the aggregation packet belongs to an intra layer representation (aggregation case), or if the next non-PACSI NAL unit in transmission order belongs to an intra layer representation (single NAL unit case). Otherwise, theCbit MUST be set to 0.
🫧 Informative note: The C bit indicates whether a packet contains intra slices, which may be the only packets to be forwarded, e.g., when the network conditions are particularly adverse.
- 🍓 The
Sbit MUST be set to 1, if the first NAL unit following the PACSI NAL unit in an aggregation packet is the first VCL NAL unit, in decoding order, of a layer representation (aggregation case), or if the next non-PACSI NAL unit in transmission order is the first VCL NAL unit, in decoding order, of a layer representation (single NAL unit case). Otherwise, theSbit MUST be set to 0. - 🌸 The
Ebit MUST be set to 1, if the last NAL unit following the PACSI NAL unit in an aggregation packet is the last VCL NAL unit, in decoding order, of a layer representation (aggregation case), or if the next non-PACSI NAL unit in transmission order is the last VCL NAL unit, in decoding order, of a layer representation (single NAL unit case). Otherwise, theEbit MUST be set to 0.
🧚 Informative note: In an aggregation packet it is always possible to detect the beginning or end of a layer representation by detecting changes in the values ofdependency_id,quality_id, andtemporal_idin NAL unit headers, except from the first and last NAL units of a packet. TheSorEbits are used to provide this information, for both single NAL unit and aggregation packets, so that previous or following packets do not have to be examined. This enables MANEs to detect slice loss and take proper action such as requesting a retransmission as soon as possible, as well as to allow efficient playout buffer handling similarly to theMbit present in the RTP header. TheMbit in the RTP header still indicates the end of an access unit, not the end of a layer representation.
- 🔥 When present, the
TL0PICIDXfield MUST be set to equaltl0_dep_rep_idxas specified in Annex G of [H.264] for the layer representation containing the first NAL unit following the PACSI NAL unit in the aggregation packet (aggregation case), or containing the next non-PACSI NAL unit in transmission order (single NAL unit case). - 💅 When present, the
IDRPICIDfield MUST be set to equaleffective_idr_pic_idas specified in Annex G of [H.264] for the layer representation containing the first NAL unit following the PACSI NAL unit in the aggregation packet (aggregation case), or containing the next non-PACSI NAL unit in transmission order (single NAL unit case).
🫧 Informative note: TheTL0PICIDXandIDRPICIDfields enable the detection of the loss of layer representations in the most important temporal layer (withtemporal_idequal to 0) by receivers as well as MANEs. SVC provides a solution that uses SEI messages, which are harder to parse and may or may not be present in the bitstream. When the PACSI NAL unit is part of an NI-MTAP packet, it is possible to infer the correct values oftl0_dep_rep_idxandidr_pic_idfor all layer representations contained in the NI-MTAP by following the rules that specify how these parameters are set as given in Annex G of [H.264] and by detecting the different layer representations contained in the NI-MTAP packet by detecting changes in the values ofdependency_id,quality_id, andtemporal_idin the NAL unit headers as well as using theSandEflags. The only exception is if NAL units of an IDR picture are present in the NI-MTAP in a position other than the first NAL unit following the PACSI NAL unit, in which case the value ofidr_pic_idcannot be inferred. In this case the NAL unit has to be partially parsed to obtain theidr_pic_id. Note that, due to the large size of IDR pictures, their inclusion in an NI-MTAP, and especially in a position other than the first NAL unit following the PACSI NAL unit, may be neither practical nor useful. 🌈
- When present, the field
DONCgives the cross-session decoding order number (CS-DON) for the first of the remaining NAL units in the aggregation packet (when the PACSI NAL unit is in an aggregation packet), or the CS-DON of the next non-PACSI NAL unit in transmission order (when the PACSI NAL unit is in a single NAL unit packet). CS-DON gets the full tour in Section 4.11. 💖
The PACSI NAL unit MAY include a subset of the SEI NAL units tied to the access unit that the first non-PACSI NAL unit in the aggregation packet belongs to. It MUST NOT contain SEI NAL units tied to any other access unit. 🫶
Informative note: In H.264/AVC and SVC, every SEI NAL unit in an access unit must come before any VCL NAL unit in decoding order. So without PACSI NAL units, SEI messages usually only ride along in the first packet of an access unit. Senders may repeat SEI NAL units inside PACSI NAL units, so they show up in more than one packet, which boosts robustness against packet loss. Receivers may use those repeated SEI messages to stand in for missing ones. 🌸
For a PACSI NAL unit in an aggregation packet, an SEI message SHOULD NOT be in the PACSI NAL unit AND also in one of the other NAL units in that same aggregation packet. Don't double-book your SEI messages, bestie 🙈
4.10. Empty NAL unit 🍓
An empty NAL unit MAY be included in a single NAL unit packet, an STAP-A or an NI-MTAP packet. Empty NAL units MUST have an RTP timestamp (when sent in a single NAL unit packet) or a NALU-time (when sent in an aggregation packet) that's tied to an access unit with at least one NAL unit of type 1, 5, or 20. When MST is used, that type 1, 5, or 20 NAL unit may be in a different RTP session. Empty NAL units may be used in the decoding order recovery process of the NI-T mode, as described in Section 5.2.1. ✨
Here's the packet structure:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| Type | Subtype |J|K|L|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4. Empty NAL unit structure. 🦋
The fields MUST be set like this:
FMUST be equal to0NRIMUST be equal to3TypeMUST be equal to31SubtypeMUST be equal to1JMUST be equal to0KMUST be equal to0LMUST be equal to0
4.11. Decoding Order Number (DON) 💜
The DON concept comes from [RFC6184], and it's used to recover decoding order when interleaving happens within a single session. Section 5.5 of [RFC6184] applies when you're using SST. 👑
When you're using MST, decoding order has to be recovered across the various RTP sessions, whether or not interleaving is used. On top of the timestamp mechanism covered later, the CS-DON mechanism is an extension of the DON facility that can do this job. It's defined in the next section. 🌈
4.11.1. Cross-Session DON (CS-DON) for Multi-Session Transmission ⭐
The cross-session decoding order number (CS-DON) is a number that shows the decoding order of NAL units across all RTP sessions involved in MST. It's similar to the DON concept in [RFC6184], but where [RFC6184] only used DON for interleaved packetization, this memo uses CS-DON in the interleaved MST mode (I-C) AND in two of the non-interleaved MST modes (NI-C and NI-TC). It's giving main character energy 💅
When the NI-C or NI-TC MST modes are in use, the packetization of each session MUST be as specified in Section 5.2.2. PACSI NAL units explicitly code the CS-DON value in the field DONC. For non-PACSI NAL units, the CS-DON value is derived as follows. Let SN be the RTP sequence number of a packet. 🎀
- For each non-PACSI NAL unit carried in a session using the single NAL unit session packetization mode, the CS-DON value of the NAL unit is equal to the formula below, where "%" is the modulo operation,
DONC_prev_PACSIis theDONCvalue of the previous PACSI NAL unit with the same NALU-time as the current NAL unit, and SN_diff is calculated as shown after it.
(DONC_prev_PACSI + SN_diff - 1) % 65536if SN1 > SN2, SN_diff = SN1 - SN2
else SN_diff = SN2 + 65536 - SN1Here, SN1 and SN2 are the SNs of the current NAL unit and the previous PACSI NAL unit with the same NALU-time, respectively. 🧚
- For non-PACSI NAL units carried in a session using the non-interleaved session packetization mode, the CS-DON value of each non-PACSI NAL unit is derived as follows:
- For a non-PACSI NAL unit in a single NAL unit packet, the following applies.
- If the previous PACSI NAL unit is in a single NAL unit packet, the CS-DON value of the NAL unit is calculated as above.
- Otherwise (the previous PACSI NAL unit is in an STAP-A packet), the CS-DON value of the NAL unit is calculated as above, except that
DONC_prev_PACSIis replaced by the CS-DON value of the previous non-PACSI NAL unit in decoding order (that is, the CS-DON value of the last NAL unit of the STAP-A packet).
- For a non-PACSI NAL unit in an STAP-A packet, the following applies.
- If the non-PACSI NAL unit is the first non-PACSI NAL unit in the STAP-A packet, its CS-DON value is equal to the
DONCof the PACSI NAL unit in the STAP-A packet. - Otherwise (the non-PACSI NAL unit is not the first non-PACSI NAL unit in the STAP-A packet), its CS-DON value is equal to the formula below, where "%" is the modulo operation.
- If the non-PACSI NAL unit is the first non-PACSI NAL unit in the STAP-A packet, its CS-DON value is equal to the
- For a non-PACSI NAL unit in a single NAL unit packet, the following applies.
(the CS-DON value of the previous non-PACSI NAL unit in decoding order + 1) % 65536- For a non-PACSI NAL unit in a number of FU-A packets, the CS-DON value of the NAL unit is calculated the same way as when the single NAL unit session packetization mode is in use, with SN1 being the SN value of the first FU-A packet.
- For a non-PACSI NAL unit in an NI-MTAP packet, the CS-DON value is equal to the value of the DON field of the non-interleaved multi-time aggregation unit.
When the I-C MST packetization mode is in use, the DON values derived according to [RFC6184] for all the NAL units in each of the RTP sessions MUST indicate CS-DON values. 🌟
5. Packetization Rules 🌷
Section 6 of [RFC6184] applies in this memo, with the following additions. 💕
5.1. Packetization Rules for Single-Session Transmission 🎈
All receivers MUST support the single NAL unit packetization mode, so they stay backward compatible with endpoints that only support the single NAL unit mode of [RFC6184]. However, the single NAL unit packetization mode (packetization-mode equal to 0) SHOULD be avoided whenever possible. Putting small NAL units (like small parameter sets, prefix NAL units, or SEI messages) in their own packets is less efficient because of the packet header overhead. 😭
All receivers MUST support the non-interleaved mode. 🫶
Informative note: The non-interleaved mode of [RFC6184] does let an application put a single NAL unit in a single RTP packet. Historically, the single NAL unit mode was only included in [RFC6184] for compatibility with ITU-T Rec. H.241 Annex A [H.241]. There's no point carrying that historic ballast into a new application space like the one SVC opens up. Supporting the extra mechanisms of the non-interleaved mode (namely STAP-A and FU-A) adds only minor implementation complexity, and the benefits are significant. So support for STAP-A and FU-A is required. Support for two of the three NAL unit types defined in this memo, empty NAL units and NI-MTAP, is also needed, as specified in Section 4.5.1. 🍓
A NAL unit of small size SHOULD be encapsulated in an aggregation packet together with one or more other NAL units. Non-VCL NAL units like access unit delimiters, parameter sets, or SEI NAL units are typically small, so they're prime candidates. 🌸
A prefix NAL unit, and the NAL unit it's associated with (which follows it in decoding order), SHOULD be in the same aggregation packet whenever an aggregation packet is used for the associated NAL unit. The exception is when that would break session MTU constraints, or when fragmentation units are used for the associated NAL unit. 🦋
Informative note: Although an H.264/AVC decoder ignores the prefix NAL unit, it's necessary in the SVC decoding process. Because the prefix NAL unit is so small, it's best to send it in the same RTP packet as its associated NAL unit. 💜
When only an H.264/AVC compatible subset of the SVC base layer is sent in an RTP session, the subset MUST be encapsulated according to [RFC6184]. That way, an [RFC6184] receiver can still receive the H.264/AVC compatible bitstream subset. 👑
When a set of layers including one or more SVC enhancement layers is sent in an RTP session, the set SHOULD be carried in one RTP stream, and that stream SHOULD be encapsulated according to this memo. 🌈
5.2. Packetization Rules for Multi-Session Transmission ⭐
When MST is used, the packetization rules in Section 5.1 still apply. On top of those, the following packetization rules MUST be followed, so the decoding order of NAL units carried in the sessions can be correctly recovered for each MST packetization mode using the de-packetization process in Section 6.2. It's a whole plan, bestie 💅
The NI-T and NI-TC modes both use timestamps to recover decoding order. To do that, the RTP packet stream has to contain data for all sampling instances of a given RTP session in every enhancement RTP session that depends on it. The NI-C and I-C modes don't have this limitation. They use CS-DON values to explicitly indicate decoding order, either coded directly in PACSI NAL units or inferred from them using the packetization rules. NI-TC offers both options, and it's up to the receiver to pick which one to use. 🎀
5.2.1. NI-T/NI-TC Packetization Rules ✨
Okay bestie, first up: when you're using NI-T mode and a PACSI NAL unit is in the mix, the T bit MUST be equal to 0, which means the DONC field MUST NOT be present. 🙈
When you're using NI-T mode, the optional parameters sprop-mst-remux-buf-size, sprop-remux-buf-req, remux-buf-cap, sprop-remux-init-buf-time, and sprop-mst-max-don-diff MUST NOT be present. Not them, not today. 💅
When NI-T or NI-TC MST mode is in use, here's the main vibe: if one or more NAL units from an access unit at sampling time instance t show up in RTP session A, then one or more NAL units from that same access unit MUST be present in any enhancement RTP session that depends on RTP session A. It's all about keeping the squad together. 🫶
Informative note: The mapping between RTP and NTP format timestamps is conveyed in RTCP SR packets. Also, the mechanisms for faster media timestamp synchronization discussed in [RFC6051] may be used to speed up the acquisition of the RTP-to-wall-clock mapping. ⏰
Informative note: The rule above may require the insertion of NAL units, usually when temporal scalability is used. That happens when an enhancement RTP session has no NAL units for an access unit with a particular NTP timestamp (media timestamp), even though that access unit does show up in a lower enhancement RTP session or the base RTP session. There are two ways to slip in those extra NAL units: - One option is to use empty NAL units (defined in Section 4.10), which can be used by the process described in Section 6.2.1 for the access unit reordering process. 🧚 - Additional NAL units may also be added by the encoder itself, for example, by transmitting coded data that simply instructs the decoder to repeat the previous picture. This option, however, may be difficult to use with pre-encoded content. 🙃
If a packet has to be inserted to satisfy the rule above (like when a MANE is making multiple RTP streams out of a single RTP stream), the inserted packet must have an RTP timestamp that maps to the same wall-clock time (in NTP format) as the RTP timestamp of any packet of the access unit in any lower enhancement RTP session or the base RTP session. 🕰️ This is super easy if the NAL unit or packet can be inserted when the RTP stream is generated, because the media timestamp (NTP timestamp) has to match between the inserted packet and the packet of the corresponding access unit. If there's no knowledge of the media time at RTP stream generation, or if the RTP streams aren't generated at the same instance, you can still do it later in the transmission process. In that case, the NTP timestamp of the inserted packet can be calculated like this.
Say a packet A2 of an access unit with RTP timestamp TS_A2 is present in base RTP session A, and no packet of that access unit is in enhancement RTP session B, as shown in Figure 5. So a packet B2 has to be inserted into session B following the rule above. The most recent RTCP sender report in session A carries NTP timestamp NTP_A and RTP timestamp TS_A. The sender report in session B with a lower NTP timestamp than NTP_A is NTP_B, and it carries RTP timestamp TS_B.
RTP session B:..B0........B1........(B2)......................
RTCP session B:.....SR(NTP_B,TS_B).............................
RTP session A:..A0........A1........A2........................
RTCP session A:..................SR(NTP_A,TS_A)................
-----------------|--x------|-----x---|------------------------>
NTP time
--------------------+<---------->+<->+------------------------>
t1 t2 RTP TS(B) timeFigure 5. Example calculation of RTP timestamp for packet insertion in an enhancement layer RTP session
The vertical bars ("|") on the NTP time line in the figure mean access unit data is present in at least one of the sessions. The "x" marks show when the sender reports happen. Right under that, the RTP timestamp time line for session B has two time segments, t1 and t2. t1 is the time difference between the two sessions' sender reports, expressed in RTP timestamp clock ticks. t2 is the time difference from the session A sender report to the A2 packet, also in RTP timestamp clock ticks. You add those differences to the RTP timestamp of the session B sender report to get the correct RTP timestamp for the inserted packet B2. In other words:
TS_B2 = TS_B + t1 + t2Let toRTP() be a function that calculates the RTP time difference (in clock ticks of the used clock) from an NTP timestamp difference. Let effRTPdiff() be a function that calculates the effective difference between two timestamps, wraparounds included:
effRTPdiff( ts1, ts2 ):
if( ts1 <= ts2 ) then
effRTPdiff := ts1-ts2
else
effRTPDiff := (4294967296 + ts2) - ts1We have:
t1 = toRTP(NTP_A - NTP_B) and t2 = effRTPdiff(TS_A2, TS_A)So, to generate the RTP timestamp TS_B2 for the inserted packet B2, you can calculate it like this:
TS_B2 = TS_B + toRTP(NTP_A - NTP_B) + effRTPdiff(TS_A2, TS_A)5.2.2. NI-C/NI-TC Packetization Rules 🌸
When NI-C or NI-TC MST mode is in use, the following applies to each of the RTP sessions.
- For each single NAL unit packet containing a non-PACSI NAL unit, the previous packet, if present, MUST have the same RTP timestamp as the single NAL unit packet, and the following applies.
- If the NALU-time of the non-PACSI NAL unit is not equal to the NALU-time of the previous non-PACSI NAL unit in decoding order, the previous packet MUST contain a PACSI NAL unit containing the DONC field.
- In an STAP-A packet, the first NAL unit in the STAP-A packet MUST be a PACSI NAL unit containing the DONC field.
- For an FU-A packet, the previous packet MUST have the same RTP timestamp as the FU-A packet, and the following applies.
- If the FU-A packet is the start of the fragmented NAL unit, the following applies.
- If the NALU-time of the fragmented NAL unit is not equal to the NALU-time of the previous non-PACSI NAL unit in decoding order, the previous packet MUST contain a PACSI NAL unit containing the DONC field;
- Otherwise, (the NALU-time of the fragmented NAL unit is equal to the NALU-time of the previous non-PACSI NAL unit in decoding order), the previous packet MAY contain a PACSI NAL unit containing the DONC field.
- Otherwise, if the FU-A packet is the end of the fragmented NAL unit, the following applies.
- If the next non-PACSI NAL unit in decoding order has a NALU-time equal to the NALU-time of the fragmented NAL unit, and is carried in a number of FU-A packets or a single NAL unit packet, the next packet MUST be a single NAL unit packet containing a PACSI NAL unit containing the DONC field.
- Otherwise (the FU-A packet is neither the start nor the end of the fragmented NAL unit), the previous packet MUST be a FU-A packet.
- If the FU-A packet is the start of the fragmented NAL unit, the following applies.
- For each single NAL unit packet containing a PACSI NAL unit, if present, the PACSI NAL unit MUST contain the DONC field.
- When the optional media type parameter
sprop-mst-csdon-always-presentis equal to 1, the session packetization mode in use MUST be the non-interleaved mode, and only STAP-A and NI-MTAP packets can be used. 🎀
5.2.3. I-C Packetization Rules 🦋
When the I-C MST packetization mode is in use, the following applies.
- When a PACSI NAL unit is present, the
Tbit MUST be equal to 0, i.e., theDONCfield is not present, and theYbit MUST be equal to 0, i.e., theTL0PICIDXandIDRPICIDare not present.
5.2.4. Packetization Rules for Non-VCL NAL Units 🧁
NAL units that don't directly encode video slices are known in H.264 as non-VCL NAL units. Non-VCL units that are only used by, or only relevant to, enhancement RTP sessions SHOULD be sent in the lowest session they're relevant to. ✨
Some senders, though, like the ones sending pre-encoded data, may not be able to easily tell which non-VCL units are relevant to which session. So non-VCL NAL units MAY, instead, be sent in a session that the session using them depends on (for example, the base RTP session). 🌈
If a non-VCL unit is relevant to more than one RTP session, and none of those sessions depends on the others, the NAL unit MAY be sent in another session that all of these sessions depend on. 💜
5.2.5. Packetization Rules for Prefix NAL Units 🍓
Section 5.1 of this memo applies, with the following addition. If the base layer is sent in a base RTP session using [RFC6184], prefix NAL units MAY be sent in the lowest enhancement RTP session rather than in the base RTP session. 🫧
6. De-Packetization Process 👑
6.1. De-Packetization Process for Single-Session Transmission 🎧
For single-session transmission, where a single RTP session is used, the de-packetization process specified in Section 7 of [RFC6184] applies. Easy peasy. 💖
6.2. De-Packetization Process for Multi-Session Transmission 🌟
For multi-session transmission, where more than one RTP session is used to receive data from the same SVC bitstream, the de-packetization process goes like this. 🧩
Just like with a single RTP session, the big idea behind de-packetization is to reorder NAL units from transmission order back to NAL unit decoding order. 🔁
The sessions to be received MUST be identified by mechanisms specified in Section 7.2.3. An enhancement RTP session typically contains an RTP stream that depends on at least one other RTP session, as indicated by mechanisms defined in Section 7.2.3. A lower RTP session to an enhancement RTP session is an RTP session that the enhancement RTP session depends on. The lowest RTP session for a receiver is the base RTP session, which doesn't depend on any other RTP session the receiver gets. The highest RTP session for a receiver is the RTP session that no other RTP session the receiver gets depends on. 🪜
For each of the RTP sessions, the RTP reception process specified in RFC 3550 is applied. Then the received packets are passed into the payload de-packetization process defined in this memo. 📦
The decoding order of the NAL units carried in all the associated RTP sessions is then recovered by applying one of the following subsections, depending on which of the MST packetization modes is in use. Pick your mode and let's go! 🚀
6.2.1. Decoding Order Recovery for the NI-T and NI-TC Modes ✨
okay bestie, this is where we put the NAL units back in decoding order 🧩 If you're using the NI-T packetization mode, this process MUST be applied, no exceptions 💅 If you're in NI-TC, it MAY be applied, so that one's optional 🦋 The whole thing runs on RTP session dependency signaling, RTP sequence numbers, and timestamps 💜
The decoding order of NAL units inside an RTP packet stream in an RTP session is set by the sequence numbers SN of the RTP packets carrying them, and then by the order each NAL unit appears inside its packet 🌸
Timing is the next piece ⏰ The media timestamp TS is the NTP timestamp derived from an RTP packet's RTP timestamp, and it's attached to every NAL unit inside that same RTP packet received in an RTP session. So they all share one timestamp, iykyk 🫶
For NI-MTAP packets it's a little different 🎀 Each contained NAL unit gets its own NALU-time, worked out from the "TS offset" value in the NI-MTAP packet (see Section 4.10). That NALU-time gets used instead of the RTP packet timestamp when deriving the media timestamp, for example using the NTP wall clock provided through RTCP sender reports. NAL units inside fragmentation packets are treated as fully defragmented NAL units, each with its own media timestamp ✨ All NAL units that share the same media timestamp TS belong to the same access unit AU(TS), which is so satisfying 😭
Now the empty NAL units. Any empty NAL units SHOULD be kept, because they effectively act as access unit indicators during the reordering process 🧁 Empty NAL units and PACSI NAL units SHOULD be removed before the access unit data is passed to the decoder 🧹
Informative note: These empty NAL units are used to associate NAL units present in other RTP sessions with RTP sessions not containing any data for an access unit of a particular time instance. They act as access unit indicators in sessions that would otherwise contain no data for the particular access unit. The presence of these NAL units is ensured by the packetization rules in Section 5.2.1. 🌈
It's assumed the receiver has already set up an operation point (DID, QID, and TID values) and has identified the highest enhancement RTP session for that operation point 👑 The decoding order of NAL units coming from multiple RTP streams across multiple RTP sessions MUST be recovered into one single sequence of NAL units, grouped into access units, using any process equivalent to the steps below. The general process is described in Section 4.2 of [RFC6051]. For convenience, the instructions of [RFC6051] are repeated and applied to NAL units instead of full RTP packets. On top of that, the SVC-specific extensions to the procedure in Section 4.2. of [RFC6051] look like this:
- The process should be started with the NAL units received in the highest RTP session with the first media timestamp TS (in NTP format) available in the session's (de-jittering) buffer. It is assumed that packets in the de-jittering buffer are already stored in RTP sequence number order. 🍓
- Collect all NAL units associated with the same value of media timestamp TS, starting from the highest RTP session, from all the (de-jittering) buffers of the received RTP sessions. The collected NAL units will be those associated with the access unit AU(TS). 🧺
- Place the collected NAL units in the order of session dependency as derived by the dependency indication as specified in Section 7.2.3, starting from the lowest RTP session. 🪜
- Place the session ordered NAL units in decoding order within the particular access unit by satisfying the NAL unit ordering rules for SVC access units, as described in the informative algorithm provided in Section 6.2.1.1. 🧠
- Remove NI-MTAP and any PACSI NAL units from the access unit AU(TS). 🗑️
- The access units can then be transferred to the decoder. Access units AU(TS) are transferred to the decoder in the order of appearance (given by the order of RTP sequence numbers) of media timestamp values TS in the highest RTP session associated with access unit AU(TS). 🚀
Informative note: Due to packet loss it is possible that not all sessions may have NAL units present for the media timestamp value TS present in the highest RTP session. In such a case, an algorithm may: a) proceed to the next complete access unit with NAL units present in all the received RTP sessions; or b) consider a new highest RTP session, the highest RTP session for which the access unit is complete, and apply the process above. The algorithm may return to the original highest RTP session when a complete and error-free access unit that contains NAL units in all the sessions is received. 😬
Here comes the informative example, and it's a good one 🦋
The example shown in Figure 6 refers to three RTP sessions A, B, and C containing an SVC bitstream transmitted as 3 sources. In the example, the dependency signaling (described in Section 7.2.3) indicates that session A is the base RTP session, B is the first enhancement RTP session and depends on A, and C is the second enhancement RTP session and depends on A and B. A hierarchical picture coding prediction structure is used, in which session A has the lowest frame rate and sessions B and C have the same but higher frame rate 💖
The figure shows NAL units contained in RTP packets that are stored in the de-jittering buffer at the receiver for session de-packetization. The NAL units are already reordered according to their RTP sequence number order and, if within an aggregation packet, according to the order of their appearance within the aggregation packet. The figure indicates for the received NAL units the decoding order within the sessions, as well as the associated media (NTP) timestamps ("TS[..]"). NAL units of the same access unit within a session are grouped by "(.,.)" and share the same media timestamp TS, which is shown at the bottom of the figure. Note that the timestamps are not in increasing order since, in this example, the decoding order is different from the output/display order 🙈
The process first goes to the NAL units associated with the first media timestamp TS[1] present in the highest session C and removes/ignores all preceding (in decoding order) NAL units to NAL units with TS[1] in each of the de-jittering buffers of RTP sessions A, B, and C. Then, starting from session C, the first media timestamp available in decoding order (TS[1]) is selected, and NAL units starting from RTP session A, and sessions B and C are placed in order of the RTP session dependency as required by Section 7.2.3 of this memo (in the example for TS[1]: first session B and then session C) into the access unit AU(TS[1]) associated with media timestamp TS[1]. Then the next media timestamp TS[3] in order of appearance in the highest RTP session C is processed, and the process described above is repeated. Note that there may be access units with no NAL units present, e.g., in the lowest RTP session A (see, e.g., TS[1]). With TS[8], the first access unit with NAL units present in all the RTP sessions appears in the buffers 🌟
C: ------------(1,2)-(3,4)--(5)---(6)---(7,8)(9,10)-(11)--(12)----
| | | | | | | | | |
B: -(1,2)-(3,4)-(5)---(6)--(7,8)-(9,10)-(11)-(12)--(13,14)(15,15)-
| | | | | |
A: -------(1)---------------(2)---(3)---------------(4)----(5)----
---------------------------------------------------decoding order-->
TS: [4] [2] [1] [3] [8] [6] [5] [7] [12] [10] Key:
A, B, C - RTP sessions
Integer values in "()" - NAL unit decoding order within RTP session
"( )" - groups the NAL units of an access unit
in an RTP session
"|" - indicates corresponding NAL units of the
same access unit AU(TS[..]) in the RTP
sessions
Integer values in "[]" - media timestamp TS, sampling time
as derived, e.g., from NTP timestamp
associated with the access unit AU(TS[..]),
consisting of NAL units in the sessions
above each TS value.Figure 6. Example of decoding order recovery in multi-source transmission.
6.2.1.1. Informative Algorithm for NI-T Decoding Order Recovery within an Access Unit 🎀
okay so within one access unit, the [H.264] spec (Sections 7.4.1.2.3 and G.7.4.1.2.3) puts limits on which decoding orders are valid for the NAL units. 💅 ngl this is the rule book
these limits mean you can rebuild a valid decoding order for an access unit using only the order of NAL units in each session, the NAL unit headers, and the Supplemental Enhancement Information message headers. 🧩 the whole thing is literally built on that
this section gives an informative algorithm for rebuilding that valid order. other NAL unit orderings can be valid too, but any compliant ordering describes the same video stream and ancillary data as what this algorithm spits out. ✨
an actual implementation only needs to behave "as if" this reordering happened. NAL units that the decoding process throws away don't need to be reordered. 🙈
here's how the algorithm works. NAL units inside an access unit get sorted by NAL unit type, following the order in Table 12 below, except for NAL unit type 14, which gets special handling as the table explains. NAL units of the same type are then ordered the way that type says, if it needs ordering at all. 🦋
for this algorithm, "session order" means the order of NAL units implied by their transmission order within an RTP session. for the non-interleaved and single NAL unit modes, that's the RTP sequence number order, combined with the order of NAL units inside an aggregation unit. 🌸
Type Description / Comments
-----------------------------------------------------------
9 Access unit delimiter
7 Sequence parameter set
13 Sequence parameter set extension
15 Subset sequence parameter set
8 Picture parameter set
16-18 Reserved
6 Supplemental enhancement information (SEI)
If an SEI message with a first payload of 0 (Buffering
Period) is present, it must be the first SEI message.
If SEI messages with a Scalable Nesting (30) payload and
a nested payload of 0 (Buffering Period) are present,
these then follow the first SEI message. Such an SEI
message with the all_layer_representations_in_au_flag
equal to 1 is placed first, followed by any others,
sorted in increasing order of DQId.
All other SEI messages follow in any order.
14 Prefix NAL unit in scalable extension
1 Coded slice of a non-IDR picture
5 Coded slice of an IDR picture
NAL units of type 1 or 5 will be sent within only a
single session for any given access unit. They are
placed in session order. (Note: Any given access unit
will contain only NAL units of type 1 or type 5, not
both.)
If NAL units of type 14 are present, every NAL unit of
type 1 or 5 is prefixed by a NAL unit of type 14. (Note:
Within an access unit, every NAL unit of type 14 is
identical, so correlation of type 14 NAL units with the
other NAL units is not necessary.)
12 Filler data
The only restriction of filler data NAL units within an
access unit is that they shall not precede the first VCL
NAL unit with the same access unit.
19 Coded slice of an auxiliary coded picture without
partitioning
These NAL units will be sent within only a single
session for any given access unit, and are placed in
session order.
20 Coded slice in scalable extension
21-23 Reserved
Type 20 NAL units are placed in increasing order of DQId.
Within each DQId value, they are placed in session order.
(Note: SVC slices with a given DQId value will be sent
within only a single session for any given access unit.)
Type 21-23 NAL units are placed immediately following
the non-reserved-type VCL NAL unit they follow in
session order.
10 End of sequence
11 End of streamTable 12. Ordering of NAL unit types within an Access Unit 🍓
6.2.2. Decoding Order Recovery for the NI-C, NI-TC, and I-C Modes 💜
okay this part is the big one! 😭 the following process MUST be used whenever the NI-C or I-C MST packetization mode is in use. it MAY be applied when the NI-TC MST packetization mode is in use. so NI-C and I-C are mandatory, and NI-TC is your optional glow-up. 💖
the RTP packets that come out of the RTP-level reception processing for each session get dropped into a re-multiplexing buffer. 🧺
it is RECOMMENDED to set the size of the re-multiplexing buffer (in bytes) to be equal to or greater than the value of the sprop-remux-buf-req media type parameter of the highest RTP session the receiver gets. 🌈
the CS-DON value gets calculated and stored for each NAL unit. ✨
Informative note: the CS-DON value of a NAL unit can depend on info carried in a different packet from the one that contains the NAL unit. this happens, e.g., when the CS-DON values need to be worked out for non-PACSI NAL units inside single NAL unit packets, because those single NAL unit packets don't carry any CS-DON info themselves. in that case, if no packet with the required CS-DON info shows up for a NAL unit, the receiver has to discard that NAL unit, because it can't be fed to the decoder in the correct order. when the optional media type parameter sprop-mst-csdon-always-present is equal to 1, there's no such dependency, so the CS-DON value of any NAL unit can be worked out just from the packet that contains it, and the receiver doesn't have to throw anything away. 🙌
the receiver operation is explained below using the following functions and constants: 🧠
- Function AbsDON is specified in Section 8.1 of [RFC6184].
- Function don_diff is specified in Section 5.5 of [RFC6184].
- Constant N is the value of the OPTIONAL sprop-mst-remux-buf-size media type parameter of the highest RTP session incremented by 1.
initial buffering lasts until one of the following conditions is met: ⏳
- There are N or more VCL NAL units in the re-multiplexing buffer.
- If sprop-mst-max-don-diff of the highest RTP session is present, don_diff(m,n) is greater than the value of sprop-mst-max-don-diff of the highest RTP session, where n corresponds to the NAL unit having the greatest value of AbsDON among the received NAL units and m corresponds to the NAL unit having the smallest value of AbsDON among the received NAL units.
- Initial buffering has lasted for the duration equal to or greater than the value of the OPTIONAL sprop-remux-init-buf-time media type parameter of the highest RTP session.
the NAL units that get removed from the re-multiplexing buffer are decided like this: 🫶
- If the re-multiplexing buffer contains at least N VCL NAL units, NAL units are removed from the re-multiplexing buffer and passed to the decoder in the order specified below until the buffer contains N-1 VCL NAL units.
- If sprop-mst-max-don-diff of the highest RTP session is present, all NAL units m for which don_diff(m,n) is greater than sprop-max-don-diff of the highest RTP session are removed from the re-multiplexing buffer and passed to the decoder in the order specified below. here, n corresponds to the NAL unit having the greatest value of AbsDON among the NAL units in the re-multiplexing buffer.
the order that NAL units get passed to the decoder is: 👑
- Let PDON be a variable that is initialized to 0 at the beginning of the RTP sessions.
- For each NAL unit associated with a value of CS-DON, a CS-DON distance is calculated as follows. If the value of CS-DON of the NAL unit is larger than the value of PDON, the CS-DON distance is equal to CS-DON - PDON. Otherwise, the CS-DON distance is equal to 65535 - PDON + CS-DON + 1.
- NAL units are delivered to the decoder in increasing order of CS-DON distance. If several NAL units share the same value of CS-DON distance, they can be passed to the decoder in any order.
- When a desired number of NAL units have been passed to the decoder, the value of PDON is set to the value of CS-DON for the last NAL unit passed to the decoder.
7. Payload Format Parameters 🎀
okay this section is all about the parameters that MAY be used to pick optional features of the payload format and certain features of the bitstream. ✨ the parameters are defined here as part of the media type registration for the SVC codec. there's also a mapping of them into the Session Description Protocol (SDP) [RFC4566] for apps that use SDP. other control protocols that don't use SDP can get equivalent parameters defined somewhere else. 🦋
some parameters tell the receiver about the properties of the stream that's about to be sent. their names all start with "sprop" for stream properties. some of these "sprop" parameters are limited by other payload or codec configuration parameters. for example, the sprop-parameter-sets parameter is constrained by the profile-level-id parameter. the media sender picks all the "sprop" parameters, not the receiver. this is a pretty uncommon thing for parameters to do, and it may clash with some signaling protocol concepts, in which case using these parameters SHOULD be avoided. 🙈
7.1. Media Type Registration 💅
the media subtype for the SVC codec has been allocated from the IETF tree. 🌸
the receiver MUST ignore any unspecified parameter. 🙅♀️
Informative note: making the receiver ignore unspecified parameters keeps things backward compatible with future extensions. for example, if a future spec that's backward compatible with this one defines some new parameters, a receiver following this spec can still receive data per the new payload and just ignore those newly specified parameters. this same provision is also in [RFC6184]. 🧚
Media Type name: video
Media subtype name: H264-SVC
Required parameters: none
OPTIONAL parameters:in the definitions of parameters below, "the stream" or "the NAL unit stream" means all NAL units carried in the current RTP session in SST, and, in MST, all NAL units carried in the current RTP session plus all NAL units carried in any other RTP sessions that the current RTP session depends on. 🌈
profile-level-id: this is a base16 [RFC4648] (hexadecimal) representation of three bytes from the sequence parameter set or subset sequence parameter set NAL unit specified in [H.264]. those bytes are: 1) profile_idc; 2) a byte called profile-iop, made of the values of constraint_set0_flag, constraint_set1_flag, constraint_set2_flag, constraint_set3_flag, constraint_set4_flag, constraint_set5_flag, and reserved_zero_2bits, in bit-significance order starting from the most-significant bit; and 3) level_idc. ✨ heads up on the last bit: reserved_zero_2bits is required to be equal to 0 in [H.264], but ITU-T or ISO/IEC may specify other values for it in the future. 🔥
The profile-level-id parameter is the default vibe check 💅 It tells everyone which default sub-profile is in play, meaning the subset of coding tools that may have been used to make the stream or that the receiver supports, plus the default level of the stream or the level the receiver can handle 🎀
The default sub-profile comes from the profile_idc byte together with some fields in the profile-iop byte. Depending on those profile-iop fields, the default sub-profile might match exactly the coding tools of one profile, or it might be a shared subset of tools across several profiles, as spelled out in Subsection G.7.4.2.1.1 of [H.264] ✨ The default level comes from the level_idc byte. When profile_idc is 66, 77, or 88 (Baseline, Main, or Extended) and level_idc is 11, bit 4 (constraint_set3_flag) of the profile-iop byte also gets a say. If profile_idc is 66, 77, or 88, level_idc is 11, and that bit is 1, the default level is Level 1b 🦋
Table 13 is the cheat sheet 📋 for every profile defined in Annexes A and G of [H.264]. It shows, for each profile, the combos of profile_idc and profile-iop that all point at the same sub-profile 🌈
Table 13. Combinations of profile_idc and profile-iop
representing the same sub-profile corresponding to the full set
of coding tools supported by one profile.Here's the decoder list for the abbreviations, where x can be 0 or 1:
- CB: Constrained Baseline profile
- B: Baseline profile
- M: Main profile
- E: Extended profile
- H: High profile
- H10: High 10 profile
- H42: High 4:2:2 profile
- H44: High 4:4:4 Predictive profile
- H10I: High 10 Intra profile
- H42I: High 4:2:2 Intra profile
- H44I: High 4:4:4 Intra profile
- C44I: CAVLC 4:4:4 Intra profile
- SB: Scalable Baseline profile
- SH: Scalable High profile
- SHI: Scalable High Intra profile
Profile profile_idc profile-iop
(hexadecimal) (binary)
CB 42 (B) x1xx0000
same as: 4D (M) 1xxx0000
same as: 58 (E) 11xx0000
B 42 (B) x0xx0000
same as: 58 (E) 10xx0000
M 4D (M) 0x0x0000
E 58 00xx0000
H 64 00000000
H10 6E 00000000
H42 7A 00000000
H44 F4 00000000
H10I 6E 00010000
H42I 7A 00010000
H44I F4 00010000
C44I 2C 00010000
SB 53 x0000000
SH 56 0x000000
SHI 56 0x010000Quick example from the table 👀 profile_idc 58 (Extended) with profile-iop 11xx0000 is the same sub-profile as profile_idc 42 (Baseline) with profile-iop x1xx0000. Also, other combos that aren't in Table 13 can still describe a sub-profile equal to the shared subset of tools for more than one profile 🧠 And a decoder that conforms to one profile might be able to decode bitstreams from other profiles, so it's not all or nothing 💜
If profile-level-id is used to describe stream properties, it says that to decode the stream, the minimum subset of coding tools a decoder has to support is the default sub-profile, and the lowest level the decoder has to support is the default level 🎯
If profile-level-id is used for capability exchange or session setup, it means the subset of coding tools (equal to the default sub-profile) that the codec supports for both receiving and sending 🫶 If max-recv-level is not present, the default level from profile-level-id is the highest level the codec wishes to support. If max-recv-level is present, it's the highest level the codec supports for receiving. For either receiving or sending, all levels lower than the highest supported level MUST also be supported 🔥
Informative note: Capability exchange and session setup procedures should provide means to list the capabilities for each supported sub-profile separately. For example, the one-of-N codec selection procedure of the SDP Offer/Answer model can be used (Section 10.2 of [RFC3264]). The one-of-N codec selection procedure may also be used to provide different combinations of profile_idc and profile-iop that represent the same sub-profile. When there are many different combinations of profile_idc and profile-iop that represent the same sub-profile, using the one-of-N codec selection procedure may result in a fairly large SDP message. Therefore, a receiver should understand the different equivalent combinations of profile_idc and profile-iop that represent the same sub-profile, and be ready to accept an offer using any of the equivalent combinations.
If no profile-level-id is present, the Baseline Profile without additional constraints at Level 1 MUST be implied 🍓
max-recv-level:
This parameter MAY be used to tell everyone the highest level a receiver supports when that level is higher than the default level (the one indicated by profile-level-id) 🌸 The value of max-recv-level is a base16 (hexadecimal) representation of the two bytes after the profile_idc syntax element in the sequence parameter set NAL unit specified in [H.264]: profile-iop (as defined above) and level_idc. If the level_idc byte of max-recv-level is 11 and bit 4 of its profile-iop byte is 1, or the level_idc byte is 9 and bit 4 of its profile-iop byte is 0, the highest level the receiver supports is Level 1b. Otherwise, the highest level is the level_idc byte of max-recv-level divided by 10 ✨
max-recv-level MUST NOT be present if the highest level the receiver supports is not higher than the default level 🙈
max-recv-base-level:
This parameter MAY be used to indicate the highest level a receiver supports for the base layer when negotiating an SVC stream 🧚 The value of max-recv-base-level is a base16 (hexadecimal) representation of the two bytes after the profile_idc syntax element in the sequence parameter set NAL unit specified in [H.264]: profile-iop (as defined above) and level_idc. If the level_idc byte of max-recv-level is 11 and bit 4 of its profile-iop byte is 1, or the level_idc byte of max-recv-level is 9 and bit 4 of its profile-iop byte is 0, the highest level the receiver supports for the base layer is Level 1b. Otherwise, it's the level_idc byte of max-recv-level divided by 10 💅
max-mbps, max-fs, max-cpb, max-dpb, and max-br:
These parameters share common properties that are specified in [RFC6184] 🎀
max-mbps: This one is as specified in [RFC6184] 💖
max-fs: This one is as specified in [RFC6184] 💖
max-cpb: The value of max-cpb is an integer for the maximum coded picture buffer size, in units of 1000 bits for the VCL HRD parameters and in units of 1200 bits for the NAL HRD parameters. Heads up, this parameter doesn't use units of cpbBrVclFactor and cpbBrNALFactor (see Table A-1 of [H.264]) 👀 The max-cpb parameter signals that the receiver has more memory than the minimum amount of coded picture buffer memory required by the highest level signaled in profile-level-id or max-recv-level 🧠 When max-cpb is signaled, the receiver MUST be able to decode NAL unit streams that conform to the signaled highest level, with one exception: the MaxCPB value in Table A-1 of [H.264] for that level is replaced with the value of max-cpb (after taking cpbBrVclFactor and cpbBrNALFactor into consideration when needed). The value of max-cpb (after taking cpbBrVclFactor and cpbBrNALFactor into consideration when needed) MUST be greater than or equal to the value of MaxCPB given in Table A-1 of [H.264] for the highest level. Senders MAY use this knowledge to build coded video streams with more bitrate variation than the MaxCPB value in Table A-1 of [H.264] would allow 🚀
Informative note: The coded picture buffer is used in the Hypothetical Reference Decoder (HRD, Annex C) of [H.264]. The use of the HRD is recommended in SVC encoders to verify that the produced bitstream conforms to the standard and to control the output bitrate. Thus, the coded picture buffer is conceptually independent of any other potential buffers in the receiver, including de-interleaving, re-multiplexing, and de-jitter buffers. The coded picture buffer need not be implemented in decoders as specified in Annex C of [H.264]; standard-compliant decoders can have any buffering arrangements provided that they can decode standard-compliant bitstreams. Thus, in practice, the input buffer for video decoder can be integrated with the de-interleaving, re-multiplexing, and de-jitter buffers of the receiver.
max-dpb: This one is as specified in [RFC6184] 💖
max-br: The value of max-br is an integer for the maximum video bitrate, in units of 1000 bits per second for the VCL HRD parameters and in units of 1200 bits per second for the NAL HRD parameters. Again, this parameter doesn't use units of cpbBrVclFactor and cpbBrNALFactor (see Table A-1 of [H.264]) 🍓
The max-br parameter signals that the receiver's video decoder can decode video at a higher bitrate than the highest level signaled in profile-level-id or max-recv-level requires 🌈
When max-br is signaled, the video codec of the receiver MUST be able to decode NAL unit streams that conform to the signaled highest level, with the following exceptions within the limits specified by the highest level:
Okay so max-br is giving "two bitrates, one vibe" 💅 here's how it works:
- The value of max-br (after taking cpbBrVclFactor and cpbBrNALFactor into consideration when needed) replaces the MaxBR value in Table A-1 of [H.264] for the highest level. ✨
- When the max-cpb parameter is not present, this formula replaces the value of MaxCPB in Table A-1 of [H.264] 🧮
(MaxCPB of the signaled level) * max-br / (MaxBR of the signaled highest level)Let's do an example, bestie 🎀 If a receiver signals Main profile Level 1.2 with max-br equal to 1550, that means a maximum video bitrate of 1550 kbits/sec for VCL HRD parameters, a maximum video bitrate of 1860 kbits/sec for NAL HRD parameters, and a CPB size of 4036458 bits, calculated like this:
1550000 / 384000 * 1000 * 1000The value of max-br (after taking cpbBrVclFactor and cpbBrNALFactor into consideration when needed) MUST be greater than or equal to the value MaxBR given in Table A-1 of [H.264] for the signaled highest level. No cap, this is a hard floor 🫶
Senders MAY use this knowledge to send higher-bitrate video as allowed in the level definition of SVC, to get that glow-up in video quality 🦋
Informative note: This parameter was added primarily to complement a similar codepoint in the ITU-T Recommendation H.245, so as to facilitate signaling gateway designs. No assumption can be made from the value of this parameter that the network is capable of handling such bitrates at any given time. In particular, no conclusion can be drawn that the signaled bitrate is possible under congestion control constraints. 💀 Translation: just because you say you can take it doesn't mean the network can.
redundant-pic-cap: This parameter is as specified in [RFC6184]. Easy, no drama 🌸
sprop-parameter-sets: This parameter MAY be used to convey any sequence parameter set, subset sequence parameter set, and picture parameter set NAL units (we're calling these the initial parameter set NAL units 🧚) that can be placed in the NAL unit stream to precede any other NAL units in decoding order and that are associated with the default level of profile-level-id. The parameter MUST NOT be used to indicate codec capability in any capability exchange procedure. The value is a comma (,) separated list of base64 [RFC4648] representations of the parameter set NAL units as specified in Sections 7.3.2.1, 7.3.2.2, and G.7.3.2.1 of [H.264]. Also, a parameter set NAL unit is typically less than 10 bytes, but a picture parameter set NAL unit can be several hundreds of bytes, so it can get chunky 🍓
Informative note: When several payload types are offered in the SDP Offer/Answer model, each with its ownsprop-parameter-setsparameter, the receiver cannot assume those parameter sets don't use conflicting storage locations (i.e., identical values of parameter set identifiers). So a receiver should buffer allsprop-parameter-setsand make them available to the decoder instance that decodes a certain payload type. Storage drama, iykyk 🙈
sprop-level-parameter-sets: This parameter MAY be used to convey any sequence, subset sequence, and picture parameter set NAL units (the initial parameter set NAL units, again 🧚) that can be placed in the NAL unit stream to precede any other NAL units in decoding order and that are associated with one or more levels different than the default level of profile-level-id. The parameter MUST NOT be used to indicate codec capability in any capability exchange procedure.
This one is so organized it's giving librarian energy 📚 The sprop-level-parameter-sets parameter contains parameter sets for one or more levels that are different than the default level. All parameter sets targeted for use when one level of the default sub-profile is accepted by a receiver get clustered together and prefixed with a three-byte field that has the same syntax as profile-level-id. That lets the receiver install the parameter sets for the accepted level and discard the rest. The three-byte field is named PLId, and all parameter sets associated with one level are named PSL, which has the same syntax as sprop-parameter-sets. Each level is represented as PLId:PSL, meaning PLId followed by a colon (:) and the base64 [RFC4648] representation of the initial parameter set NAL units for that level. Each pair of PLId:PSL is also separated by a colon. A PSL can hold multiple parameter sets for that level, separated with commas (,) 🌈
The subset of coding tools indicated by each PLId field MUST be equal to the default sub-profile, and the level indicated by each PLId field MUST be different than the default level.
Informative note: This parameter allows for efficient level downgrade or upgrade in SDP Offer/Answer and out-of-band transport of parameter sets, simultaneously. Level glow-ups and downgrades in one parameter, slay 👑
in-band-parameter-sets: This parameter MAY be used to indicate a receiver capability. The value MAY be equal to either 0 or 1. The value 1 indicates that the receiver discards out-of-band parameter sets in sprop-parameter-sets and sprop-level-parameter-sets, therefore the sender MUST transmit all parameter sets in-band. The value 0 indicates that the receiver utilizes out-of-band parameter sets included in sprop-parameter-sets and/or sprop-level-parameter-sets. However, in this case, the sender MAY still choose to send parameter sets in-band. When the parameter is not present, this receiver capability is not specified, so the sender MAY send out-of-band parameter sets only, MAY send in-band-parameter-sets only, or MAY send both. Options, options, options 🦋
packetization-mode: This parameter is as specified in [RFC6184]. When the mst-mode parameter is present, the value of this parameter is additionally constrained as follows. If mst-mode is equal to "NI-T", "NI-C", or "NI-TC", packetization-mode MUST NOT be equal to 2. Otherwise (mst-mode is equal to "I-C"), packetization-mode MUST be equal to 2. 🔥
sprop-interleaving-depth: This parameter is as specified in [RFC6184]. 🌸
sprop-deint-buf-req: This parameter is as specified in [RFC6184]. 🌸
deint-buf-cap: This parameter is as specified in [RFC6184]. 🌸
sprop-init-buf-time: This parameter is as specified in [RFC6184]. 🌸
sprop-max-don-diff: This parameter is as specified in [RFC6184]. 🌸
max-rcmd-nalu-size: This parameter is as specified in [RFC6184]. 🌸
mst-mode: This parameter MAY be used to signal the properties of a NAL unit stream or the capabilities of a receiver implementation. If this parameter is present, multi-session transmission MUST be used. Otherwise (this parameter is not present), single-session transmission MUST be used. When this parameter is present, the following applies. When the value of mst-mode is equal to "NI-T", the NI-T mode MUST be used. When the value of mst-mode is equal to "NI-C", the NI-C mode MUST be used. When the value of mst-mode is equal to "NI-TC", the NI-TC mode MUST be used. When the value of mst-mode is equal to "I-C", the I-C mode MUST be used. The value of mst-mode MUST have one of the following tokens: "NI-T", "NI-C", "NI-TC", or "I-C". 🧩
All RTP sessions in an MST MUST have the same value of mst-mode. Consistency is everything, bestie 💜
sprop-mst-csdon-always-present: This parameter MUST NOT be present when mst-mode is not present or the value of mst-mode is equal to "NI-T" or "I-C". It signals the properties of the NAL unit stream. When sprop-mst-csdon-always-present is present and the value is equal to 1, packetization-mode MUST be equal to 1, and all the RTP packets carrying the NAL unit stream MUST be STAP-A packets containing a PACSI NAL unit that further contains the DONC field, or NI-MTAP packets with the J field equal to 1. Also, when it's present and equal to 1, the CS-DON value of any particular NAL unit can be derived solely according to information in the packet containing the NAL unit. Main character energy, the packet carries its own receipts 🎀
When sprop-mst-csdon-always-present is present in the current RTP session, it MUST be present also in all the RTP sessions the current RTP session depends on, and the value of sprop-mst-csdon-always-present MUST be identical for the current RTP session and all the RTP sessions on which it depends. Everyone in the friend group has to match 👯♀️
sprop-mst-remux-buf-size: This parameter MUST NOT be present when mst-mode is not present or the value of mst-mode is equal to "NI-T". This parameter MUST be present when mst-mode is present and the value of mst-mode is equal to "NI-C", "NI-TC", or "I-C".
It signals the properties of the NAL unit stream. It MUST be set to a value one less than the minimum re-multiplexing buffer size (in NAL units), so that it is guaranteed that receivers can reconstruct NAL unit decoding order as specified in Subsection 6.2.2. The value of sprop-mst-remux-buf-size MUST be an integer in the range of 0 to 32767, inclusive. 🧠
sprop-remux-buf-req: This parameter MUST NOT be present when mst-mode is not present or the value of mst-mode is equal to "NI-T". It MUST be present when mst-mode is present and the value of mst-mode is equal to "NI-C", "NI-TC", or "I-C".
sprop-remux-buf-req signals the required size of the re-multiplexing buffer for the NAL unit stream. It is guaranteed that receivers can recover the decoding order of the received NAL units from the current RTP session and the RTP sessions the current RTP session depends on as specified in Section 6.2.2, when the re-multiplexing buffer size is of at least the value of sprop-remux-buf-req in units of bytes. The value MUST be an integer in the range of 0 to 4294967295, inclusive. Big buffer, big energy 💅
remux-buf-cap: This parameter MUST NOT be present when mst-mode is not present or the value of mst-mode is equal to "NI-T". This parameter MAY be used to signal the capabilities of a receiver implementation and indicates the amount of re-multiplexing buffer space, in units of bytes, that the receiver has available for recovering the NAL unit decoding order as specified in Section 6.2.2. A receiver is able to handle any NAL unit stream for which the value of the sprop-remux-buf-req parameter is smaller than or equal to this parameter. 🦋
If the parameter is not present, then a value of 0 MUST be used for remux-buf-cap. The value of remux-buf-cap MUST be an integer in the range of 0 to 4294967295, inclusive. ✨
Okay so, we're picking up mid-parameter list, bestie 🎀
sprop-remux-init-buf-time: this parameter MAY be used to signal the properties of the NAL unit stream. It MUST NOT be present if mst-mode is not present, or if the value of mst-mode is equal to "NI-T" 🙈
The parameter signals the initial buffering time that a receiver MUST wait before it starts recovering the NAL unit decoding order, as specified in Section 6.2.2 of this memo ⏳
It's coded as a non-negative base10 integer representation in clock ticks of a 90-kHz clock. If the parameter isn't present, then no initial buffering time value is defined. Otherwise, the value of sprop-remux-init-buf-time MUST be an integer in the range of 0 to 4294967295, inclusive 🔥
sprop-mst-max-don-diff: this parameter MAY be used to signal the properties of the NAL unit stream. It MUST NOT be used to signal transmitter or receiver or codec capabilities 💅 It MUST NOT be present if mst-mode is not present, or if the value of mst-mode is equal to "NI-T". sprop-mst-max-don-diff is an integer in the range of 0 to 32767, inclusive. If sprop-mst-max-don-diff is not present, the value of the parameter is unspecified. It's calculated the same way as sprop-max-don-diff as specified in [RFC6184], except the decoding order number gets replaced by the cross-session decoding order number 🧚
sprop-scalability-info: this parameter MAY be used to convey the NAL unit containing the scalability information SEI message, as specified in Annex G of [H.264]. It MAY also be used to signal the contained layers of an SVC bitstream ✨ It MUST NOT be used to indicate codec capability in any capability exchange procedure. The value is the base64 [RFC4648] representation of the NAL unit containing the scalability information SEI message. If it's present, the NAL unit MUST contain only one SEI message, and that message MUST be a scalability information SEI message 🦋
This parameter MAY be used in an offering or declarative SDP message to indicate which layers (operation points) can be provided. A receiver MAY indicate its choice of one layer using the optional media type parameter scalable-layer-id 💜
scalable-layer-id: this parameter MAY be used to signal a receiver's choice of the offered or declared operation points or layers, using sprop-scalability-info or sprop-operation-point-info. Its value is a base16 representation of the layer_id[ i ] syntax element in the scalability information SEI message, as specified in Annex G of [H.264], or the layer-ID contained in sprop-operation-point-info 🌸
sprop-operation-point-info: this parameter MAY be used to describe the operation points of an RTP session. Its value is a comma-separated list of operation-point-description vectors. The values given by those vectors are the same as, or are derived from, the values that would be given for a scalable layer in the scalability information SEI message as specified in Annex G of [H.264]. In the scalability information SEI message, the term scalable layer refers to all NAL units associated with the same values of temporal_id, dependency_id, and quality_id. In this memo, that set of NAL units is called an operation point 👑
Each operation-point-description vector has ten elements, provided as a comma-separated list of values, defined below. The first value of the vector is preceded by a '<', and the last value is followed by a '>'. If sprop-operation-point-info is followed by exactly one operation-point-description vector, it describes the highest operation point contained in the RTP session. If there are two or more vectors, the first describes the lowest operation point and the last describes the highest operation point contained in the RTP session 🍓
The values given by the operation-point-description vector are as follows, in this order:
- layer-ID: This value specifies the layer identifier of the operation point, which is identical to the
layer_idthat would be indicated (for the same values ofdependency_id,quality_id, andtemporal_id) in the scalability information SEI message. This field MAY be empty, indicating that the value is unspecified. When there are multiple operation-point-description vectors with layer-ID, the values of layer-ID do not need to be consecutive. - temporal-ID: This value specifies the
temporal_idof the operation point. This field MUST NOT be empty. - dependency-ID: This value specifies the
dependency_idof the operation point. This field MUST NOT be empty. - quality-ID: This value specifies the
quality_idof the operation point. This field MUST NOT be empty. - profile-level-ID: This value specifies the profile-level-idc of the operation point in the base16 format. The default sub-profile or default level indicated by the parameter
profile-level-IDin thesprop-operation-point-infovector SHALL be equal to or lower than the default sub-profile or default level indicated byprofile-level-id, which may be either present or the default value is taken. This field MAY be empty, indicating that the value is unspecified. - avg-framerate: This value specifies the average frame rate of the operation point. This value is given as an integer in frames per 256 seconds. The field MAY be empty, indicating that the value is unspecified.
- width: This value specifies the width dimension in pixels of decoded frames for the operation point. This parameter is not directly given in the scalability information SEI message. This field MAY be empty, indicating that the value is unspecified.
- height: This value gives the height dimension in pixels of decoded frames for the operation point. This parameter is not directly given in the scalability information SEI. This field MAY be empty, indicating that the value is unspecified.
- avg-bitrate: This value specifies the average bitrate of the operation point. This parameter is given as an integer in kbits per second over the entire stream. Note that this parameter is provided in the scalability information SEI message in bits per second and calculated over a variable time window. This field MAY be empty, indicating that the value is unspecified.
- max-bitrate: This value specifies the maximum bitrate of the operation point. This parameter is given as an integer in kbits per second and describes the maximum bitrate per each one-second window. Note that this parameter is provided in the scalability information SEI message in bits per second and is calculated over a variable time window. This field MAY be empty, indicating that the value is unspecified.
Similarly to sprop-scalability-info, this parameter MAY be used in an offering or declarative SDP message to indicate which layers (operation points) can be provided. A receiver MAY indicate its choice of the highest layer it wants to send and/or receive using the optional media type parameter scalable-layer-id 🫶
sprop-no-NAL-reordering-required: this parameter MAY be used to signal the properties of the NAL unit stream. It MUST NOT be present when mst-mode is not present, or when the value of mst-mode is not equal to "NI-T". If it's present, that means no reordering of non-VCL or VCL NAL units is required for the decoding order recovery process 😭
sprop-avc-ready: this parameter MAY be used to indicate the properties of the NAL unit stream. If it's present, the RTP session can be processed by a [RFC6184] receiver, whether it's used in SST, or in MST combined with other RTP sessions that also have this parameter present 💖 This parameter MAY be used with RTP sessions with media subtype H264-SVC.
Encoding considerations: This media type is framed and binary; see Section 4.8 of RFC 4288 [RFC4288].
Security considerations: See Section 8 of RFC 6190.
Published specification: Please refer to RFC 6190 and its Section 13.
Additional information: none
File extensions: none
Macintosh file type code: none
Object identifier or OID: none
Person & email address to contact for further information: Ye-Kui Wang, yekui.wang@huawei.com
Intended usage: COMMON
Restrictions on usage: This media type depends on RTP framing, and hence is only defined for transfer via RTP [RFC3550]. Transport within other framing protocols is not defined at this time.
Interoperability considerations: The media subtype name contains "SVC" to avoid potential conflict with RFC 3984 and its potential future replacement RTP payload format for H.264 non-SVC profiles 🦋
Applications that use this media type: Real-time video applications like video streaming, video telephony, and video conferencing 📹
Author: Ye-Kui Wang, yekui.wang@huawei.com
Change controller: IETF Audio/Video Transport working group delegated from the IESG.
7.2. SDP Parameters ✨
7.2.1. Mapping of Payload Type Parameters to SDP ✨
Okay so here's how the video/H264-SVC media type gets translated into SDP fields 🎀
- The media name in the
m=line of SDP MUST bevideo. - The encoding name in the
a=rtpmapline MUST beH264-SVC(the media subtype). - The clock rate in the
a=rtpmapline MUST be90000. - The OPTIONAL parameters
profile-level-id,max-recv-level,max-recv-base-level,max-mbps,max-fs,max-cpb,max-dpb,max-br,redundant-pic-cap,in-band-parameter-sets,packetization-mode,sprop-interleaving-depth,deint-buf-cap,sprop-deint-buf-req,sprop-init-buf-time,sprop-max-don-diff,max-rcmd-nalu-size,mst-mode,sprop-mst-csdon-always-present,sprop-mst-remux-buf-size,sprop-remux-buf-req,remux-buf-cap,sprop-remux-init-buf-time,sprop-mst-max-don-diff, andscalable-layer-id, when present, MUST be included in thea=fmtpline of SDP. These are written as a media type string, which is a semicolon-separated list ofparameter=valuepairs. 💜 - The OPTIONAL parameters
sprop-parameter-sets,sprop-level-parameter-sets,sprop-scalability-info,sprop-operation-point-info,sprop-no-NAL-reordering-required, andsprop-avc-ready, when present, MUST be included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute as specified in Section 6.3 of [RFC5576]. For a particular media format (i.e., RTP payload type), asprop-parameter-setsorsprop-level-parameter-setsMUST NOT be both included in thea=fmtpline of SDP and conveyed using thefmtpsource attribute. When they're in thea=fmtpline, these parameters are a media type string, a semicolon-separated list ofparameter=valuepairs. When they're conveyed using thefmtpsource attribute, these parameters are only associated with the given source and payload type as parts of thefmtpsource attribute. 🌸
Informative note: Conveyingsprop-parameter-setsandsprop-level-parameter-setsusing thefmtpsource attribute allows for out-of-band transport of parameter sets in topologies like Topo-Video-switch-MCU [RFC5117]. 🦋
7.2.2. Usage with the SDP Offer/Answer Model
When an SVC stream (with media subtype H264-SVC) is offered over RTP using SDP in an Offer/Answer model [RFC3264] for unicast usage, the following limitations and rules apply. Let's gooo 🚀
- The parameters identifying a media format configuration for SVC are
profile-level-id,packetization-mode, andmst-mode. These media configuration parameters (except for the level part ofprofile-level-id) MUST be used symmetrically when the answerer does not includescalable-layer-idin the answer. In other words, the answerer MUST either maintain all configuration parameters or remove the media format (payload type) completely, if one or more of the parameter values are not supported. Note that the level part ofprofile-level-idincludeslevel_idc, and, for indication of level 1b whenprofile_idcis equal to 66, 77, or 88, bit 4 (constraint_set3_flag) ofprofile-iop. The level part ofprofile-level-idis changeable. 🎀
Informative note: The requirement for symmetric use does not apply for the level part of profile-level-id, and does not apply for the other stream properties and capability parameters. 💅
Informative note: In [H.264], all the levels except for Level 1b are equal to the value oflevel_idcdivided by 10. Level 1b is a level higher than Level 1.0 but lower than Level 1.1, and is signaled in an ad hoc manner. For the Baseline, Main, and Extended profiles (withprofile_idcequal to 66, 77, and 88, respectively), Level 1b is indicated bylevel_idcequal to 11 (i.e., the same as level 1.1) andconstraint_set3_flagequal to 1. For other profiles, Level 1b is indicated bylevel_idcequal to 9 (but note that Level 1b for these profiles is still higher than Level 1, which haslevel_idcequal to 10, and lower than Level 1.1). In SDP Offer/Answer, an answer may indicate a level equal to or lower than the level indicated in the offer. Due to the ad hoc indication of Level 1b, offerers and answerers must check the value of bit 4 (constraint_set3_flag) of the middle octet of the parameterprofile-level-id, whenprofile_idcis equal to 66, 77, or 88 andlevel_idcis equal to 11. 🧚
To keep these configurations easy to handle and match, the same RTP payload type number used in the offer should also be used in the answer, as specified in [RFC3264]. The same RTP payload type number used in the offer MUST also be used in the answer when the answer includes scalable-layer-id. When the answer does not include scalable-layer-id, the answer MUST NOT contain a payload type number used in the offer unless the configuration is exactly the same as in the offer, or the configuration in the answer only differs from that in the offer with a level lower than the default level offered. 🌈
Informative note: When an offerer receives an answer that does not includescalable-layer-id, it has to compare payload types not declared in the offer based on the media type (i.e.,video/H264-SVC) and the above media configuration parameters with any payload types it has already declared. This lets it figure out whether the configuration in question is new or equivalent to a configuration already offered, since a different payload type number may be used in the answer. 🔥
Since an SVC stream may contain multiple operation points, there's a facility so that an answerer can select a different operation point than the entire SVC stream. Specifically, different operation points MAY be described using the sprop-scalability-info or sprop-operation-point-info parameters. The first one carries the entire scalability information SEI message defined in Annex G of [H.264], whereas the second one may be derived, e.g., as a subset of this SEI message that only contains key information about an operation point. For both cases, operation points are associated with a layer identifier. ✨
If such information (sprop-operation-point-info or sprop-scalability-info) is provided in an offer, an answerer MAY select from the various operation points offered in the sprop-scalability-information or sprop-operation-point-info parameters by including scalable-layer-id in the answer. By doing this, the answerer indicates its selection of a particular operation point in the received and/or in the sent stream. When such operation point selection takes place (i.e., the answerer includes scalable-layer-id in the answer), the media configuration parameters MUST NOT be present in the answer. Instead, the media configuration that the answerer will use for receiving and/or sending is the one used for the selected operation point as indicated in the offer. 👑
Informative note: Operation point selection lets a receiver make the most of the scalable nature of an SVC stream. 🍓
- The parameter
max-recv-level, when present, declares the highest level supported for receiving. Ifmax-recv-levelis not present, the highest level supported for receiving is equal to the default level indicated by the level part ofprofile-level-id.max-recv-level, when present, MUST be higher than the default level. 🙈 - The parameter
max-recv-base-level, when present, declares the highest level of the base layer supported for receiving. Whenmax-recv-base-levelis not present, the highest level supported for the base layer is not constrained separately from the SVC stream containing the base layer. The endpoint at the other side MUST NOT send a scalable stream for which the base layer is of a level higher thanmax-recv-base-level. Parameters declaring receiver capabilities above the default level (max-mbps,max-smbps,max-fs,max-cpb,max-dpb,max-br, andmax-recv-level) do not apply to the base layer whenmax-recv-base-levelis present. 🫶 - The parameters
sprop-deint-buf-req,sprop-interleaving-depth,sprop-max-don-diff,sprop-init-buf-time,sprop-mst-csdon-always-present,sprop-remux-buf-req,sprop-mst-remux-buf-size,sprop-remux-init-buf-time,sprop-mst-max-don-diff,sprop-scalability-information,sprop-operation-point-info,sprop-no-NAL-reordering-required, andsprop-avc-readydescribe the properties of the NAL unit stream that the offerer or answerer is sending for the media format configuration. This is different from the normal usage of the Offer/Answer parameters: normally, such parameters declare the properties of the stream that the offerer or the answerer is able to receive. When dealing with SVC, the offerer assumes that the answerer will be able to receive media encoded using the configuration being offered. 🌷
Informative note: The above parameters apply to any stream sent by the declaring entity with the same configuration. In other words, they depend on their source. Rather than being bound to the payload type, the values may have to be applied to another payload type when being sent, as they apply for the configuration. 🦋
- The capability parameters
max-mbps,max-fs,max-cpb,max-dpb,max-br,redundant-pic-cap, andmax-rcmd-nalu-sizeMAY be used to declare further capabilities of the offerer or answerer for receiving. These parameters MUST NOT be present when the direction attribute issendonly, and the parameters describe the limitations of what the offerer or answerer accepts for receiving streams. 💖 - When
mst-modeis not present andpacketization-modeis equal to 2, the following applies.- An offerer has to include the size of the de-interleaving buffer,
sprop-deint-buf-req, in the offer. To let the offerer and answerer tell each other about their capabilities for de-interleaving buffering, both parties are RECOMMENDED to includedeint-buf-cap. It is also RECOMMENDED to consider offering multiple payload types with different buffering requirements when the capabilities of the receiver are unknown. 🧃
- An offerer has to include the size of the de-interleaving buffer,
- When
mst-modeis present and equal to "NI-C", "NI-TC", or "I-C", the following applies.- An offerer has to include
sprop-remux-buf-reqin the offer. To let the offerer and answerer tell each other about their capabilities for re-multiplexing buffering, both parties are RECOMMENDED to includeremux-buf-cap. It is also RECOMMENDED to consider offering multiple payload types with different buffering requirements when the capabilities of the receiver are unknown. 🍭
- An offerer has to include
- The
sprop-parameter-setsorsprop-level-parameter-setsparameter, when present (included in thea=fmtpline of SDP or conveyed using thefmtpsource attribute as specified in Section 6.3 of [RFC5576]), is used for out-of-band transport of parameter sets. However, when out-of-band transport of parameter sets is used, parameter sets MAY still be additionally transported in-band. 🌟
Okay bestie, the answerer MAY use either out-of-band or in-band transport of parameter sets for the stream it's sending 💖, no matter whether out-of-band parameter sets were used in the offerer-to-answerer direction. Parameter sets in an answer are totally independent of the ones in the offer, because they decode two different video streams: one going from the answerer to the offerer, and one going the other way 🎀.
Here's how it works for the offerer-to-answerer direction 🦋
- An offer MAY include either or both of sprop-parameter-sets and sprop-level-parameter-sets. If neither sprop-parameter-sets nor sprop-level-parameter-sets is present in the offer, then only in-band transport of parameter sets is used.
- If the answer includes in-band-parameter-sets equal to 1, then the offerer MUST transmit parameter sets in-band. Otherwise, the following applies.
- If the level to use in the offerer-to-answerer direction is equal to the default level in the offer, the following applies.
- The answerer MUST be prepared to use the parameter sets included in sprop-parameter-sets, when present, for decoding the incoming NAL unit stream, and ignore sprop-level-parameter-sets, when present.
- When sprop-parameter-sets is not present in the offer, in-band transport of parameter sets MUST be used.
- Otherwise (the level to use in the offerer-to-answerer direction is not equal to the default level in the offer), the following applies.
- The answerer MUST be prepared to use the parameter sets that are included in sprop-level-parameter-sets for the accepted level (i.e., the default level in the answer, which is also the level to use in the offerer-to-answerer direction), when present, for decoding the incoming NAL unit stream, and ignore all other parameter sets included in sprop-level-parameter-sets and sprop-parameter-sets, when present.
- When no parameter sets for the accepted level are present in the sprop-level-parameter-sets, in-band transport of parameter sets MUST be used.
- If the level to use in the offerer-to-answerer direction is equal to the default level in the offer, the following applies.
Now the answerer-to-offerer direction, which gets its own set of rules ✨
- An answer MAY include either sprop-parameter-sets or sprop-level-parameter-sets, but MUST NOT include both of the two. If neither sprop-parameter-sets nor sprop-level-parameter-sets is present in the answer, then only in-band transport of parameter sets is used.
- If the offer includes in-band-parameter-sets equal to 1, then the answerer MUST NOT include sprop-parameter-sets or sprop-level-parameter-sets in the answer and MUST transmit parameter sets in-band. Otherwise, the following applies.
- If the level to use in the answerer-to-offerer direction is equal to the default level in the answer, the following applies.
- The offerer MUST be prepared to use the parameter sets included in sprop-parameter-sets, when present, for decoding the incoming NAL unit stream, and ignore sprop-level-parameter-sets, when present.
- When sprop-parameter-sets is not present in the answer, the answerer MUST transmit parameter sets in-band.
- Otherwise (the level to use in the answerer-to-offerer direction is not equal to the default level in the answer), the following applies.
- The offerer MUST be prepared to use the parameter sets that are included in sprop-level-parameter-sets for the level to use in the answerer-to-offerer direction, when present in the answer, for decoding the incoming NAL unit stream, and ignore all other parameter sets included in sprop-level-parameter-sets and sprop-parameter-sets, when present in the answer.
- When no parameter sets for the level to use in the answerer-to-offerer direction are present in sprop-level-parameter-sets in the answer, the answerer MUST transmit parameter sets in-band.
- If the level to use in the answerer-to-offerer direction is equal to the default level in the answer, the following applies.
When sprop-parameter-sets or sprop-level-parameter-sets is conveyed using the "fmtp" source attribute as specified in Section 6.3 of [RFC5576], things get a little more serious 🧐. The receiver of the parameters MUST store the parameter sets included in the sprop-parameter-sets or sprop-level-parameter-sets for the accepted level and associate them to the source given as a part of the "fmtp" source attribute. Parameter sets associated with one source MUST only be used to decode NAL units conveyed in RTP packets from the same source. When this mechanism is in use, SSRC collision detection and resolution MUST be performed as specified in [RFC5576].
Informative note: Conveyance of sprop-parameter-sets and sprop-level-parameter-sets using the "fmtp" source attribute may be used in topologies like Topo-Video-switch-MCU [RFC5117] to enable out-of-band transport of parameter sets.
For streams being delivered over multicast, the following rules apply 🌸
- The media format configuration is identified by profile-level-id, including the level part, packetization-mode, and mst-mode. These media format configuration parameters (including the level part of profile-level-id) MUST be used symmetrically, i.e., the answerer MUST either maintain all configuration parameters or remove the media format (payload type) completely. Note that this implies that the level part of profile-level-id for Offer/Answer in multicast is not changeable.
- To simplify handling and matching of these configurations, the same RTP payload type number used in the offer should also be used in the answer, as specified in [RFC3264]. An answer MUST NOT contain a payload type number used in the offer unless the configuration is the same as in the offer.
- Parameter sets received MUST be associated with the originating source, and MUST be only used in decoding the incoming NAL unit stream from the same source.
- The rules for other parameters are the same as above for unicast as long as the above rules are obeyed.
Table 14. Interpretation of parameters for various combinations of offers, answers, direction attributes, with and without scalable-layer-id. Columns that do not indicate offer or answer apply to both.
Table 14 is the ultimate cheat sheet 📋 It lists the interpretation of all the parameters that MUST be used for the various combinations of offer, answer, and direction attributes. Heads up: the two columns where scalable-layer-id is used only apply to answers, while the other columns apply to both offers and answers 💅
sendonly --+
answer: recvonly,scalable-layer-id --+ |
recvonly w/o scalable-layer-id --+ | |
answer: sendrecv, scalable-layer-id --+ | | |
sendrecv w/o scalable-layer-id --+ | | | |
| | | | |
profile-level-id C X C X P
max-recv-level R R R R -
max-recv-base-level R R R R -
packetization-mode C X C X P
mst-mode C X C X P
sprop-avc-ready P P - - P
sprop-deint-buf-req P P - - P
sprop-init-buf-time P P - - P
sprop-interleaving-depth P P - - P
sprop-max-don-diff P P - - P
sprop-mst-csdon-always-present P P - - P
sprop-mst-max-don-diff P P - - P
sprop-mst-remux-buf-size P P - - P
sprop-no-NAL-reordering-required P P - - P
sprop-operation-point-info P P - - P
sprop-remux-buf-req P P - - P
sprop-remux-init-buf-time P P - - P
sprop-scalability-info P P - - P
deint-buf-cap R R R R -
max-br R R R R -
max-cpb R R R R -
max-dpb R R R R -
max-fs R R R R -
max-mbps R R R R -
max-rcmd-nalu-size R R R R -
redundant-pic-cap R R R R -
remux-buf-cap R R R R -
in-band-parameter-sets R R R R -
sprop-parameter-sets S S - - S
sprop-level-parameter-sets S S - - S
scalable-layer-id X O X O -Legend:
C: configuration for sending and receiving streams
P: properties of the stream to be sent
R: receiver capabilities
S: out-of-band parameter sets
O: operation point selection
X: MUST NOT be present
-: not usable, when present SHOULD be ignoredParameters used for declaring receiver capabilities are in general downgradable, i.e., they express the upper limit for a sender's possible behavior. Thus, a sender MAY select to set its encoder using only lower/lesser or equal values of these parameters.
Parameters declaring a configuration point are not changeable, with the exception of the level part of the profile-level-id parameter for unicast usage. This expresses values a receiver expects to be used and must be used verbatim on the sender side. If level downgrading (for profile-level-id) is used, an answerer MUST NOT include the scalable-layer-id parameter.
When a sender's capabilities are declared, and non-downgradable parameters are used in this declaration, then these parameters express a configuration that is acceptable for the sender to receive streams. In order to achieve high interoperability levels, it is often advisable to offer multiple alternative configurations, e.g., for the packetization mode. It is impossible to offer multiple configurations in a single payload type. Thus, when multiple configuration offers are made, each offer requires its own RTP payload type associated with the offer.
A receiver SHOULD understand all media type parameters, even if it only supports a subset of the payload format's functionality. This ensures that a receiver is capable of understanding when an offer to receive media can be downgraded to what is supported by the receiver of the offer. 🫶
An answerer MAY extend the offer with additional media format configurations. However, to enable their usage, in most cases a second offer is required from the offerer to provide the stream property parameters that the media sender will use. This also has the effect that the offerer has to be able to receive this media format configuration, not only to send it.
If an offerer wishes to have non-symmetric capabilities between sending and receiving, the offerer can allow asymmetric levels via level-asymmetry-allowed equal to 1. Alternatively, the offerer can offer different RTP sessions, i.e., different media lines declared as "recvonly" and "sendonly", respectively. This may have further implications on the system, and may require additional external semantics to associate the two media lines. 🌈
7.2.3. Dependency Signaling in Multi-Session Transmission ✨
okay so if you're using MST, the rules for signaling media decoding dependency in SDP from [RFC5583] apply 💅. The "hierarchical or layered encoding" rules for multicast in Section 5.7 of [RFC4566] do NOT apply here, so the Connection Data c= notation SHALL NOT be used with more than one address 🙅♀️. Also, the order of dependencies between the RTP sessions, shown with the a=depend attribute from [RFC5583], MUST match the decoding order of the VC) NAL units in an access unit 🎀. Basically, session dependency goes from the base (or lowest enhancement) RTP session, which is the most important one, up to the highest enhancement RTP session, which is the least important 🦋.
7.2.4. Usage in Declarative Session Descriptions 📜
so when SVC over RTP is offered with SDP in a declarative style, like in Real Time Streaming Protocol (RTSP) [RFC2326] or Session Announcement Protocol (SAP) [RFC2974], there are a few things you need to keep in mind 🧚
- All parameters that could indicate both stream properties and receiver capabilities only get used to indicate stream properties 💖. For example,
profile-level-idtells you what values the stream is actually using, not what the receiver is able to take in 🌸. So this is how the parameters MUST be read:- Declaring actual configuration or stream properties:
profile-level-idpacketization-modemst-modesprop-deint-buf-reqsprop-interleaving-depthsprop-max-don-diffsprop-init-buf-timesprop-mst-csdon-always-presentsprop-mst-remux-buf-sizesprop-remux-buf-reqsprop-remux-init-buf-timesprop-mst-max-don-diffsprop-scalability-infosprop-operation-point-infosprop-no-NAL-reordering-requiredsprop-avc-ready
- Out-of-band transporting of parameter sets:
sprop-parameter-setssprop-level-parameter-sets
- Not usable (when present, they SHOULD be ignored) 🙈:
max-mbpsmax-fsmax-cpbmax-dpbmax-brmax-recv-levelmax-recv-base-levelredundant-pic-capmax-rcmd-nalu-sizedeint-buf-capremux-buf-capscalable-layer-id
- Declaring actual configuration or stream properties:
- A receiver of the SDP is required to support all the parameters and values that get provided 🫶. If it can't, it MUST reject the session (RTSP) or not participate in it (SAP) 💀. It's totally on the creator of the session to pick values the receiving app is expected to support 👑.
7.3. Examples 🎀
in the following examples, "{data}" is used to show a data string encoded as base64 🍓.
7.3.1. Example for Offering a Single SVC Session 🦋
Example 1 is giving two-payload-type energy 💅. The offerer offers one video media description with two RTP payload types. The first one offers H264, and the second one offers H264-SVC. Both payload types have different fmtp parameters, like profile-level-id, packetization-mode, and sprop-parameter-sets ✨.
Offerer -> Answerer SDP message:
m=video 20000 RTP/AVP 97 96
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=4de00a; packetization-mode=0;
sprop-parameter-sets={sps0},{pps0};
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 profile-level-id=53000c; packetization-mode=1;
sprop-parameter-sets={sps0},{pps0},{sps1},{pps1};if the answerer doesn't support the H264-SVC media subtype, it can send back an answer that only accepts the base layer offer (payload type 96) 🥺. In the next example, the receiver DOES support H264-SVC, so it lists payload type 97 first as its fave 🌈.
Answerer -> Offerer SDP message:
m=video 40000 RTP/AVP 97 96
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=4de00a; packetization-mode=0;
sprop-parameter-sets={sps2},{pps2};
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 profile-level-id=53000c; packetization-mode=1;
sprop-parameter-sets={sps2},{pps2},{sps3},{pps3};7.3.2. Example for Offering a Single SVC Session Using scalable-layer-id 🧚♀️
Example 2 is the offerer offering the same media configurations as the example above, for both receiving and sending the stream 🫶. The difference is that it uses a single RTP payload type and includes sprop-operation-point-info ✨.
Offerer -> Answerer SDP message:
m=video 20000 RTP/AVP 97
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 profile-level-id=53000c; packetization-mode=1;
sprop-parameter-sets={sps0},{sps1},{pps0},{pps1};
sprop-operation-point-info=<1,0,0,0,4de00a,3200,176,144,128,
256>,<2,1,1,0,53000c,6400,352,288,256,512>;in this one, the receiver supports H264-SVC and goes with the lower operation point offered in the RTP payload type, for both sending and receiving the stream 💜.
Answerer -> Offerer SDP message:
m=video 40000 RTP/AVP 97
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 sprop-parameter-sets={sps2},{sps3},{pps2},{pps3};
scalable-layer-id=1;and in an equivalent example that uses sprop-scalability-info instead of sprop-operation-point-info, the sprop-operation-point-info gets swapped out for sprop-scalability-info, followed by the binary (base16) representation of the Scalability Information SEI message 🦋.
7.3.3. Example for Offering Multiple Sessions in MST 🌸
Example 3 is the offerer offering a multi-session transmission with up to three sessions 🎀. The base session media description includes payload types that are backward compatible with [RFC6184], and three different payload types are offered 💅. The other two media use payload types with media subtype H264-SVC. Each media description has different values for profile-level-id, packetization-mode, mst-mode, and sprop-parameter-sets 🌈.
Offerer -> Answerer SDP message:
a=group:DDP L1 L2 L3
m=video 20000 RTP/AVP 96 97 98
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=4de00a; packetization-mode=0;
mst-mode=NI-T; sprop-parameter-sets={sps0},{pps0};
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=4de00a; packetization-mode=1;
mst-mode=NI-TC; sprop-parameter-sets={sps0},{pps0};
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=4de00a; packetization-mode=2;
mst-mode=I-C; init-buf-time=156320;
sprop-parameter-sets={sps0},{pps0};
a=mid:L1
m=video 20002 RTP/AVP 99 100
a=rtpmap:99 H264-SVC/90000
a=fmtp:99 profile-level-id=53000c; packetization-mode=1;
mst-mode=NI-T; sprop-parameter-sets={sps1},{pps1};
a=rtpmap:100 H264-SVC/90000
a=fmtp:100 profile-level-id=53000c; packetization-mode=2;
mst-mode=I-C; sprop-parameter-sets={sps1},{pps1};
a=mid:L2
a=depend:99 lay L1:96,97; 100 lay L1:98
m=video 20004 RTP/AVP 101
a=rtpmap:101 H264-SVC/90000
a=fmtp:101 profile-level-id=53001F; packetization-mode=1;
mst-mode=NI-T; sprop-parameter-sets={sps2},{pps2};
a=mid:L3
a=depend:101 lay L1:96,97 L2:99in this example, the answerer only supports the NI-T mode for multi-session transmission 🥹. So it picks the matching payload type (96) for the base RTP session, and for the two enhancement RTP sessions it picks the payload types that use NI-T mode (99 and 101) 🫶.
Answerer -> Offerer SDP message:
a=group:DDP L1 L2 L3
m=video 40000 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=4de00a; packetization-mode=0;
mst-mode=NI-T; sprop-parameter-sets={sps3},{pps3};
a=mid:L1
m=video 40002 RTP/AVP 99
a=rtpmap:99 H264-SVC/90000
a=fmtp:99 profile-level-id=53000c; packetization-mode=1;
mst-mode=NI-T; sprop-parameter-sets={sps4},{pps4};
a=mid:L2
a=depend:99 lay L1:96
m=video 40004 RTP/AVP 101
a=rtpmap:101 H264-SVC/90000
a=fmtp:101 profile-level-id=53001F; packetization-mode=1;
mst-mode=NI-T; sprop-parameter-sets={sps5},{pps5};
a=mid:L3
a=depend:101 lay L1:96 L2:997.3.4. Example for Offering Multiple Sessions in MST Including Operation with Answerer Using scalable-layer-id 💖
Example 4 is the offerer offering a multi-session transmission of three layers, with up to two sessions 🧁. The base session media description has a payload type that's backward compatible with [RFC6184]. Heads up: no parameter sets are provided here, so in-band transport must be used 👀. The other media description has two enhancement layers and uses the H264-SVC media subtype. It includes two operation point definitions ✨.
Offerer -> Answerer SDP message:
a=group:DDP L1 L2
m=video 20000 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=4de00a; packetization-mode=0;
mst-mode=NI-T;
a=mid:L1
m=video 20002 RTP/AVP 97
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 profile-level-id=53001F; packetization-mode=1;
mst-mode=NI-TC; sprop-operation-point-info=<2,0,1,0,53000c,
3200,352,288,384,512>,<3,1,2,0,53001F,6400,704,576,768,1024>;
a=mid:L2
a=depend:97 lay L1:96so the answerer wants to send and receive the base layer (payload type 96), but it only wants to send and receive the lower enhancement layer, which is the one with layer id 2 🎯. That's why the response selects the layer it wants by setting scalable-layer-id to 2. Notice that the answer only includes the scalable-layer-id information 💅. The answer could include sprop-parameter-sets in the response too 🌷.
Answerer -> Offerer SDP message:
a=group:DDP L1 L2
m=video 40000 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=4de00a; packetization-mode=0;
mst-mode=NI-T;
a=mid:L1
m=video 40002 RTP/AVP 97
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 scalable-layer-id=2;
a=mid:L2
a=depend:97 lay L1:967.3.5. Example for Negotiating an SVC Stream with a Constrained Base Layer in SST 🧸
Example 5 is Alice (the offerer) offering one video description with two RTP payload types, which have different levels and packetization modes 🌟.
Offerer -> Answerer SDP message:
m=video 20000 RTP/AVP 97 96
a=rtpmap:96 H264-SVC/90000
a=fmtp:96 profile-level-id=53001e; packetization-mode=0;
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 profile-level-id=53001f; packetization-mode=1;the answerer (Bridge) picks packetization mode 1, and says it would receive an SVC stream where the base layer is constrained 🦋.
Answerer -> Offerer SDP message:
m=video 40000 RTP/AVP 97
a=rtpmap:97 H264-SVC/90000
a=fmtp:97 profile-level-id=53001f; packetization-mode=1;
max-recv-base-level=000dthe answering endpoint must send an SVC stream at Level 3.1 💅. Since the offering endpoint didn't declare max-recv-base-level, the base layer of the SVC stream that the answering endpoint sends isn't specifically constrained 🫧. The offering endpoint (Alice) must send an SVC stream at Level 3.1 too, and for that one, the base layer has to be at a level no higher than Level 1.3 💖.
7.4. Parameter Set Considerations ✨
okay so this bit is about parameter sets, and it stacks on top of what [RFC6184] already says. Section 8.4 of [RFC6184] applies in this memo, and the following applies additionally for multi-session transmission (MST). 🧚
In MST, it doesn't matter whether parameter sets are being sent out-of-band or in-band. Parameter sets needed for decoding NAL units carried in one particular RTP session SHOULD be carried in that same session, MAY be carried in a session that the particular RTP session depends on, and MUST NOT be carried in a session that the particular RTP session does not depend on. The dependency rules are non-negotiable, bestie 💅
8. Security Considerations 🔐
The security considerations of the RTP Payload Format for H.264 Video specification [RFC6184] apply. Additionally, the following applies. 🫶
Decoders MUST exercise caution with respect to the handling of reserved NAL unit types and reserved SEI messages, particularly if they contain active elements, and MUST restrict their domain of applicability to the presentation containing the stream. The safest way is to simply discard these NAL units and SEI messages. 😭 Seriously, when in doubt, just toss them.
When integrity protection is applied to a stream, care MUST be taken that the stream being transported may be scalable. That means a receiver may only be able to access part of the entire stream. Iykyk 🙈
End-to-end security with either authentication, integrity, or confidentiality protection will prevent a MANE from performing media-aware operations other than discarding complete packets. And in the case of confidentiality protection, it will even be prevented from discarding packets in a media-aware way. 💀 So to let any MANE do its job, it will be required to be a trusted entity that is included in the security context establishment. This applies both for the media path and for the RTCP path, if RTCP packets need to be rewritten.
9. Congestion Control 🌊
Within any given RTP session carrying payload according to this specification, the provisions of Section 10 of [RFC6184] apply. Reducing the session bitrate is possible by one or more of the following means. Main character energy 👑
a) Within the highest layer identified by the DID field, remove any NAL units with QID higher than a certain value.
b) Remove all NAL units with TID higher than a certain value.
c) Remove all NAL units associated with a DID higher than a certain value.
Informative note: Removal of all coded slice NAL units associated with DIDs higher than a certain value in the entire stream is required in order to preserve conformance of the resulting SVC stream. 🍓
d) Utilize the PRID field to indicate the relative importance of NAL units, and remove all NAL units associated with a PRID higher than a certain value. Note that the use of the PRID is application-specific. 🦋
e) Remove NAL units or entire packets according to application-specific rules. The result will depend on the particular coding structure used, as well as any additional application-specific functionality (e.g., concealment performed at the receiving decoder). In general, this will result in the reception of a non-conforming bitstream, and hence the decoder behavior is not specified by [H.264]. Significant artifacts may therefore appear in the decoded output if the particular decoder implementation does not take appropriate action in response to congestion control. 😬
Informative note: The discussion above is centered on NAL units rather than packets, primarily because that is the level where senders can meaningfully manipulate the scalable bitstream. The mapping of NAL units to RTP packets is fairly flexible when using aggregation packets. Depending on the nature of the congestion control algorithm, the "dimension" of congestion measurement (packet count or bitrate) and reaction to it (reducing packet count or bitrate or both) can be adjusted accordingly. ✨
All aforementioned means are available to the RTP sender, regardless of whether that sender is located in the sending endpoint or in a mixer-based MANE. 💖
When a translator-based MANE is employed, then the MANE MAY manipulate the session only on the MANE's outgoing path, so that the sensed end-to-end congestion falls within the permissible envelope. As with all translators, in this case, the MANE needs to rewrite RTCP RRs to reflect the manipulations it has performed on the session. 🔁
Informative note: Applications MAY also implement, in addition or separately, other congestion control mechanisms, e.g., as described in [RFC5775] and [Yan]. 🌸
10. IANA Considerations 📚
A new media type, as specified in Section 7.1 of this memo, has been registered with IANA. Done and dusted, periodt 💅
11. Informative Appendix: Application Examples 🎀
11.1. Introduction
Scalable video coding is a concept that has been around since at least MPEG-2 [MPEG2], which goes back as early as 1993. Nevertheless, it has never gained wide acceptance, perhaps partly because applications didn't materialize in the form envisioned during standardization. It's giving "great idea, wrong timing" 🥲
ISO/IEC MPEG and ITU-T VCEG, respectively, performed a requirement analysis for the SVC project. The MPEG and VCEG requirement documents are available in [JVT-N026] and [JVT-N027], respectively. 🧠
The following introduces four main application scenarios that the authors consider relevant and that are implementable with this specification. Four scenarios, all doable, let's gooo 🚀
11.2. Layered Multicast
This well-understood form of the use of layered coding [McCanne] implies that all layers are individually conveyed in their own RTP packet streams, each carried in its own RTP session using the IP (multicast) address and port number as the single demultiplexing point. Receivers "tune" into the layers by subscribing to the IP multicast, normally by using IGMP [IGMP]. Depending on the application scenario, it is also possible to convey a number of layers in one RTP session, when finer operation points within the subset of layers are not needed. 📻
Layered multicast has the great advantage of simplicity and easy implementation. However, it has also the great disadvantage of utilizing many different transport addresses. While the authors consider this not to be a major problem for a professionally maintained content server, receiving client endpoints need to open many ports to IP multicast addresses in their firewalls. This is a practical problem from a firewall and network address translation (NAT) viewpoint. Furthermore, even today IP multicast is not as widely deployed as many wish. 😩
The authors consider layered multicast an important application scenario for the following reasons. First, it is well understood and the implementation constraints are well known. Second, there may well be large-scale IP networks outside the immediate Internet context that may wish to employ layered multicast in the future. One possible example could be a combination of content creation and core-network distribution for the various mobile TV services, e.g., those being developed by 3GPP (MBMS) [MBMS] and DVB (DVB-H) [DVB-H]. 📺✨
11.3. Streaming
In this scenario, a streaming server has a repository of stored SVC coded layers for a given content. At the time of streaming, and according to the capabilities, connectivity, and congestion situation of the client(s), the streaming server generates and serves a scalable stream. Both unicast and multicast serving is possible. At the same time, the streaming server may use the same repository of stored layers to compose different streams (with a different set of layers) intended for other audiences. Reuse the content, slay 🍰
As every endpoint receives only a single SVC RTP session, the number of firewall pinholes can be optimized to one. 🔥
The main difference between this scenario and straightforward simulcasting lies in the architecture and the requirements of the streaming server, and is therefore out of the scope of IETF standardization. However, compelling arguments can be made why such a streaming server design makes sense. One possible argument is related to storage space and channel bandwidth. Another is bandwidth adaptability without transcoding -- a considerable advantage in a congestion controlled network. When the streaming server learns about congestion, it can reduce the sending bitrate by choosing fewer layers when composing the layered stream; see Section 9. 📉
SVC is designed to gracefully support both bandwidth ramp-down and bandwidth ramp-up with a considerable dynamic range. This payload format is designed to allow for bandwidth flexibility in the mentioned sense. While, in theory, a transcoding step could achieve a similar dynamic range, the computational demands are impractically high and video quality is typically lowered -- therefore, few (if any) streaming servers implement full transcoding. Transcoding is the villain era of this story 🦹♀️💜
11.4. Videoconferencing (Unicast to MANE, Unicast to Endpoints) 📹✨
okay so videoconferencing has traditionally been run by these things called Multipoint Control Units (MCUs) 💅 and bestie, they are a lot. Everybody connects to the MCU in a star shape. Each endpoint sends its coded video to the MCU, the MCU decodes it, scales it, and composites it into output frames, then re-encodes everything and sends it back out to the endpoint(s) 🌸. If the system supports personalized layout (where each user picks how their own screen looks), the MCU has to do the whole compositing and encoding process again for every single receiving endpoint 😭. Even without personalized layout, rate matching still means the MCU has to encode separately for each endpoint. So MCUs are super complicated, they add a ton of delay 🙈, and the cascaded encodings make the video quality worse. Multipoint calls especially get clunky because the end-to-end delay is very high [G.114].
There's a simpler version called the switching MCU 🦋, where one of the incoming video streams just gets redirected to the receiving endpoints. The problem? Only one person can be seen at a time, and rate matching can't happen at all. That means every transmitting endpoint is forced to send at the lowest bit rate available on any MCU-to-endpoint connection. Not a vibe.
With scalable video coding, the MCU can be swapped out for an application-level router (ALR) 🧚. The ALR just picks which incoming packets get sent to which receiving endpoint [Eleft]. Each endpoint then does its own composition of the incoming video streams, which is so iconic. Say you have a system using spatial scalability with two layers. Personalized layout is basically the same as telling the ALR to only send the packets for the resolution that particular endpoint needs ✨. Rate matching at the ALR works the same way, by picking the right subset of incoming video packets for that endpoint. So personalized layout and rate matching both become routing decisions, and they need zero signal processing 💜. Also, scalability means everyone can enjoy the best video quality their own links can handle. Nobody has to be dragged down to the quality of the weakest endpoint 👑.
The biggest deal is that the ALR adds an insignificant amount to the end-to-end delay, typically an order of magnitude less than an MCU 😭🔥. That makes fully interactive multipoint conferences with even a very large number of participants actually possible. The error resilience is also way better, and error tolerance can go up by nearly an order of magnitude too (for example, by using unequal error protection) 🫶. And because an ALR's delay is so tiny, these systems can be cascaded, which is a huge win for system design and deployment 🌈. Traditional MCUs can't be cascaded at all, since even one MCU introduces such high delay that it's a no-go 💀.
Scalable video coding is a major paradigm shift for videoconferencing 🎀. It brings the complexity of video communication systems (especially the servers sitting inside the network) in line with other types of network applications. Iconic. 💅
11.5. Mobile TV (Multicast to MANE, Unicast to Endpoint) 📱🎬
this one is a bit more complex, ngl 🙈. It's designed to optimize the network traffic in a core network, while still only needing a single pinhole in the endpoint's firewall. One of its biggest uses is the mobile TV market 🍓.
Picture a large private IP network, like the core network of the Third Generation Partnership Project (3GPP) 🌐. Streaming servers inside that core network can be assumed to be professionally maintained 🧑💻. It's also assumed that these servers can have lots of ports open to the network, and that layered multicast is a real option for them 🔥. So the streaming server multicasts SVC scalable layers, instead of simulcasting different representations of the same content at different bitrates. Main character energy for sure ✨.
Next up, there are lots of different classes of endpoints 💖. Some of them may not have the processing power or the display size to decode all the layers, and others totally do. Some users don't want to pay for high quality and are happy with a base service, which might be cheaper or even free 🌸. Other users are willing to pay for the good stuff 💅. And then there are connected users who have a bandwidth problem, meaning they can't receive as much bandwidth as they'd like, whether that's from congestion, connectivity, a change of service quality, or whatever other reasons 😭. The thing all of these users have in common is that they don't want to be exposed too much, so the number of firewall pinholes needs to stay small 🦋.
So the best way to handle this is to put middleboxes close to the edge of the core network 🧚. They receive the layered multicast streams and compose a single SVC scalable bitstream based on the needs of each connected endpoint. These middleboxes are called MANEs throughout this specification 💜. In practice, the authors picture the MANE as part of (or at least physically and topologically close to) the base station of a mobile network, where all the signaling and media traffic is multiplexed on the same physical link 📶.
MANEs have to be fairly complex devices 🤓. They definitely need to understand the signaling. For example, they need to associate the payload type octet in the RTP header with the SVC payload type 🫶.
A MANE can aggregate multiple RTP streams, possibly from multiple RTP sessions. This reduces the number of firewall pinholes needed at the endpoints 🔒. It can also optimize the outgoing RTP stream to the MTU size of the outgoing path, by using the aggregation and fragmentation mechanisms of this memo ✨. This type of MANE is conceptually easy to implement and still offers powerful features 🎀. The reason is that it can totally "see" the payload (including the RTP payload headers), use all the layering information in there, and manipulate it 👀.
A MANE can also do stream thinning, which helps it follow the congestion control principles in Section 9 🌈. The forward (media) channel of this kind of MANE is pretty simple to build, but having to rewrite RTCP RRs makes even this MANE a complex device 🙈.
Okay, so while the implementation complexity of either kind of MANE (as described above) is pretty high, the computational demands are comparatively low 💅✨.
12. Acknowledgements 🫶🌸
Miska Hannuksela contributed a lot to the designs of the PACSI NAL unit and the NI-C mode for decoding order recovery 👑. Roni Even organized and coordinated the design team for this memo, and gave valuable comments 💖. Jonathan Lennox contributed to the NAL unit reordering algorithm for MST and gave input on several parts of this memo 🦋. Peter Amon, Sam Ganesan, Mike Nilsson, Colin Perkins, and Thomas Wiegand were design team members and made valuable contributions 🧚. Magnus Westerlund also left valuable comments 💜. Charles Eckel and Stuart Taylor gave valuable comments after the first WGLC for this document 🌈. Xiaohui (Joanne) Wei helped improve Table 13 and the SDP examples 🎀.
The work of Thomas Schierl has been supported by the European Commission under contract number FP7-ICT-248036, project COAST 🇪🇺✨.
13. References 📚
13.1. Normative References 💅
[H.264] ITU-T Recommendation H.264, "Advanced video coding for generic audiovisual services", March 2010.
[RFC6184] Wang, Y.-K., Even, R., Kristensen, T., and R. Jesup, "RTP Payload Format for H.264 Video", RFC 6184, May 2011.
[ISO/IEC14496-10] ISO/IEC International Standard 14496-10:2005.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
[RFC4288] Freed, N. and J. Klensin, "Media Type Specifications and Registration Procedures", BCP 13, RFC 4288, December 2005.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006.
[RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 4648, October 2006.
[RFC5576] Lennox, J., Ott, J., and T. Schierl, "Source-Specific Media Attributes in the Session Description Protocol (SDP)", RFC 5576, June 2009.
[RFC5583] Schierl, T. and S. Wenger, "Signaling Media Decoding Dependency in the Session Description Protocol (SDP)", RFC 5583, July 2009.
[RFC6051] Perkins, C. and T. Schierl, "Rapid Synchronisation of RTP Flows", RFC 6051, November 2010.
13.2. Informative References 📚✨
Okay bestie, these are the informative refs, the extra reading that backs up the spec 🤓💜
[DVB-H] DVB - Digital Video Broadcasting (DVB); DVB-H Implementation Guidelines, ETSI TR 102 377, 2005. 📡
[Eleft] Eleftheriadis, A., R. Civanlar, and O. Shapiro, "Multipoint Videoconferencing with Scalable Video Coding", Journal of Zhejiang University SCIENCE A, Vol. 7, Nr. 5, April 2006, pp. 696-705. (Proceedings of the Packet Video 2006 Workshop.) 🎥✨
[G.114] ITU-T Rec. G.114, "One-way transmission time", May 2003. ⏱️
[H.241] ITU-T Rec. H.241, "Extended video procedures and control signals for H.300-series terminals", May 2006. 🎞️
[IGMP] Cain, B., Deering, S., Kouvelas, I., Fenner, B., and A. Thyagarajan, "Internet Group Management Protocol, Version 3", RFC 3376, October 2002. 👑
[JVT-N026] Ohm J.-R., Koenen, R., and Chiariglione, L. (ed.), "SVC requirements specified by MPEG (ISO/IEC JTC1 SC29 WG11)", JVT-N026, available from http://ftp3.itu.ch/av-arch/jvt-site/2005_01_HongKong/JVT-N026.doc, Hong Kong, China, January 2005. 🌸
[JVT-N027] Sullivan, G. and Wiegand, T. (ed.), "SVC requirements specified by VCEG (ITU-T SG16 Q.6)", JVT-N027, available from http://ftp3.itu.int/av-arch/jvt-site/2005_01_HongKong/JVT-N027.doc, Hong Kong, China, January 2005. 🌸
[McCanne] McCanne, S., Jacobson, V., and Vetterli, M., "Receiver-driven layered multicast", in Proc. of ACM SIGCOMM'96, pages 117-130, Stanford, CA, August 1996. 🦋 this one is a classic, no cap
[MBMS] 3GPP - Technical Specification Group Services and System Aspects; Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs (Release 6), December 2005. 📱
[MPEG2] ISO/IEC International Standard 13818-2:1993. 🎬
[RFC2326] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, April 1998. ▶️
[RFC2974] Handley, M., Perkins, C., and E. Whelan, "Session Announcement Protocol", RFC 2974, October 2000. 📣
[RFC5117] Westerlund, M. and S. Wenger, "RTP Topologies", RFC 5117, January 2008. 🧚
[RFC5775] Luby, M., Watson, M., and L. Vicisano, "Asynchronous Layered Coding (ALC) Protocol Instantiation", RFC 5775, April 2010. 💖
[Yan] Yan, J., Katrinis, K., May, M., and Plattner, R., "Media- and TCP-friendly congestion control for scalable video streams", in IEEE Trans. Multimedia, pages 196-206, April 2006. 🌈
Authors' Addresses 🫶
Here's where to find the authors, bestie, the real MVPs who made this whole thing happen 👑
Stephan Wenger
2400 Skyfarm Dr.
Hillsborough, CA 94010
USA
Phone: +1-415-713-5473
EMail: stewe@stewe.org
Ye-Kui Wang
Huawei Technologies
400 Crossing Blvd, 2nd Floor
Bridgewater, NJ 08807
USA
Phone: +1-908-541-3518
EMail: yekui.wang@huawei.com
Thomas Schierl
Fraunhofer HHI
Einsteinufer 37
D-10587 Berlin
Germany
Phone: +49-30-31002-227
EMail: ts@thomas-schierl.de
Alex Eleftheriadis
Vidyo, Inc.
433 Hackensack Ave.
Hackensack, NJ 07601
USA
Phone: +1-201-467-5135
EMail: alex@vidyo.comThat's the whole doc, bestie! 🎀💅 Every author is literally iconic.
📦 where this page comes from
- ietf: rfc6190 2011-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms