RFC 3984
RTP Payload Format for H.264 Video
Status of This Memo ✨
Okay so this doc is a standards track protocol spec for the whole Internet community 🌐 It's literally asking people to chat about it and drop suggestions for making it better 💬 For the official standardization state of this protocol, check the current "Internet Official Protocol Standards" (STD 1) 📋 And distribution of this memo is unlimited, so share it with everyone 💖
Copyright Notice
Copyright (C) The Internet Society (2005). 🔒
Abstract 🦋
Okay, the big picture: this memo describes an RTP payload format for the ITU-T Recommendation H.264 video codec, and it's technically identical to the ISO/IEC International Standard 14496-10 codec 🎬 The format lets one or more Network Abstraction Layer Units (NALUs), made by an H.264 video encoder, get packed into each RTP payload 📦 It's giving versatility, because it works for simple low bit-rate conversations, Internet video streaming with interleaved transmission, and high bit-rate video-on-demand 🔥
Table of Contents 🌸
Table of Contents
1. Introduction.................................................. 3
1.1. The H.264 Codec......................................... 3
1.2. Parameter Set Concept................................... 4
1.3. Network Abstraction Layer Unit Types.................... 5
2. Conventions................................................... 6
3. Scope......................................................... 6
4. Definitions and Abbreviations................................. 6
4.1. Definitions............................................. 6
5. RTP Payload Format............................................ 8
5.1. RTP Header Usage........................................ 8
5.2. Common Structure of the RTP Payload Format.............. 11
5.3. NAL Unit Octet Usage.................................... 12
5.4. Packetization Modes..................................... 14
5.5. Decoding Order Number (DON)............................. 15
5.6. Single NAL Unit Packet.................................. 18
5.7. Aggregation Packets..................................... 18
5.8. Fragmentation Units (FUs)............................... 27
6. Packetization Rules........................................... 31
6.1. Common Packetization Rules.............................. 31
6.2. Single NAL Unit Mode.................................... 32
6.3. Non-Interleaved Mode.................................... 32
6.4. Interleaved Mode........................................ 33
7. De-Packetization Process (Informative)........................ 33
7.1. Single NAL Unit and Non-Interleaved Mode................ 33
7.2. Interleaved Mode........................................ 34
7.3. Additional De-Packetization Guidelines.................. 36
8. Payload Format Parameters..................................... 37
8.1. MIME Registration....................................... 37
8.2. SDP Parameters.......................................... 52
8.3. Examples................................................ 58
8.4. Parameter Set Considerations............................ 60
9. Security Considerations....................................... 62
10. Congestion Control............................................ 63
11. IANA Considerations........................................... 64
12. Informative Appendix: Application Examples.................... 65
12.1. Video Telephony according to ITU-T Recommendation H.241
Annex A................................................. 65
12.2. Video Telephony, No Slice Data Partitioning, No NAL
Unit Aggregation........................................ 65
12.3. Video Telephony, Interleaved Packetization Using NAL
Unit Aggregation........................................ 66
12.4. Video Telephony with Data Partitioning.................. 66
12.5. Video Telephony or Streaming with FUs and Forward
Error Correction........................................ 67
12.6. Low Bit-Rate Streaming.................................. 69
12.7. Robust Packet Scheduling in Video Streaming............. 70
13. Informative Appendix: Rationale for Decoding Order Number..... 71
13.1. Introduction............................................ 71
13.2. Example of Multi-Picture Slice Interleaving............. 71
13.3. Example of Robust Packet Scheduling..................... 73
13.4. Robust Transmission Scheduling of Redundant Coded
Slices.................................................. 77
13.5. Remarks on Other Design Possibilities................... 77
14. Acknowledgements.............................................. 78
15. References.................................................... 78
15.1. Normative References.................................... 78
15.2. Informative References.................................. 79
Authors' Addresses................................................ 81
Full Copyright Statement.......................................... 831. Introduction ✨
Okay, so this is where the party starts 🎉 The doc is the title block's whole vibe: an RFC by S. Wenger, M.M. Hannuksela, T. Stockhammer, M. Westerlund, and D. Singer 👑 It's a Standards Track doc dated February 2005, and the title block doesn't list anything it obsoletes or updates 🎀
1.1. The H.264 Codec 🎬
This memo specifies an RTP payload specification for the video coding standard known as ITU-T Recommendation H.264 [1] and ISO/IEC International Standard 14496 Part 10 [2] (both also known as Advanced Video Coding, or AVC) 💜 Recommendation H.264 was approved by ITU-T on May 2003, and the approved draft specification is available for public review [8] 📖 In this memo, the H.264 acronym is used for the codec and the standard, but the memo is equally applicable to the ISO/IEC counterpart of the coding standard 🤝
The H.264 video codec has a seriously huge application range 😭 It covers all forms of digital compressed video, from low bit-rate Internet streaming to HDTV broadcast and Digital Cinema applications with nearly lossless coding 🌈 Compared to the current state of technology, H.264 performance is such that bit rate savings of 50% or more are reported 🔥 Digital Satellite TV quality, for example, was reported to be achievable at 1.5 Mbit/s, compared to the current operation point of MPEG 2 video at around 3.5 Mbit/s [9] 🍓 Not me being impressed by that gap 💅
The codec specification [1] itself distinguishes conceptually between a video coding layer (VCL) and a network abstraction layer (NAL) 🧚 The VCL holds the signal processing functionality of the codec: mechanisms such as transform, quantization, and motion compensated prediction, plus a loop filter ⚙️ It follows the general concept of most of today's video codecs, which is a macroblock-based coder that uses inter picture prediction with motion compensation and transform coding of the residual signal 🦋 The VCL encoder outputs slices, which are a bit string containing the macroblock data of an integer number of macroblocks, plus the information of the slice header (like the spatial address of the first macroblock in the slice, the initial quantization parameter, and similar info) ✨ Macroblocks in slices are arranged in scan order unless a different macroblock allocation is specified using the so-called Flexible Macroblock Ordering syntax 🎀 In-picture prediction is used only within a slice 🙈 More information is provided in [9] 📚
The Network Abstraction Layer (NAL) encoder wraps the slice output of the VCL encoder into Network Abstraction Layer Units (NAL units) 📦 These are suitable for transmission over packet networks or use in packet oriented multiplex environments 🌐 Annex B of H.264 defines an encapsulation process to transmit such NAL units over byte-stream oriented networks, but in the scope of this memo, Annex B is not relevant 🙅♀️
Internally, the NAL uses NAL units 🤯 A NAL unit is a one-byte header plus the payload byte string 🧩 The header tells you the type of the NAL unit, the (potential) presence of bit errors or syntax violations in the NAL unit payload, and info about how important the NAL unit is for the decoding process 💀 This RTP payload specification is designed to be unaware of the bit string in the NAL unit payload 🫶
One of the main properties of H.264 is the complete decoupling of the transmission time, the decoding time, and the sampling or presentation time of slices and pictures ⏰ The decoding process in H.264 is unaware of time, and the H.264 syntax doesn't carry info like the number of skipped frames (which is common in the form of the Temporal Reference in earlier video compression standards) 🕰️ Also, there are NAL units that affect many pictures and are therefore inherently timeless 🌌 Because of this, handling the RTP timestamp needs some special considerations for NAL units whose sampling or presentation time is not defined or, at transmission time, unknown 💜
1.2. Parameter Set Concept 🎀
One super fundamental design concept of H.264 is generating self-contained packets 💖 That makes mechanisms like the header duplication of RFC 2429 [10] or MPEG-4's Header Extension Code (HEC) [11] unnecessary ✨ This was achieved by decoupling information relevant to more than one slice from the media stream 🧠 This higher layer meta information should be sent reliably, asynchronously, and in advance from the RTP packet stream that contains the slice packets 🚀 (Provisions for sending this information in-band are also available for applications that don't have an out-of-band transport channel appropriate for the purpose 🙌) The combination of the higher-level parameters is called a parameter set 🦄 H.264 includes two types of parameter sets: the sequence parameter set and the picture parameter set 🌸 An active sequence parameter set stays unchanged throughout a coded video sequence, and an active picture parameter set stays unchanged within a coded picture 🔒 These structures hold info like picture size, optional coding modes employed, and the macroblock to slice group map 🗺️
To be able to change picture parameters (like the picture size) without transmitting parameter set updates synchronously to the slice packet stream, the encoder and decoder can maintain a list of more than one sequence and picture parameter set 🍰 Each slice header contains a codeword that indicates which sequence and picture parameter set to use 🎯
This mechanism decouples the transmission of parameter sets from the packet stream 🌊 They can be sent by external means (for example, as a side effect of the capability exchange) or through a (reliable or unreliable) control protocol 📡 It may even be possible that they're never transmitted at all, but are fixed by an application design specification 💅✨
1.3. Network Abstraction Layer Unit Types 🧚
Okay bestie, the tutorial info on the NAL design is in [12], [13], and [14] if you want to go deeper 📚✨
Every NAL unit starts with a single NAL unit type octet, and that same octet doubles as the payload header for this RTP payload format 🎀. The NAL unit's payload comes right after it, no gaps.
The full syntax and semantics of this octet are in [1], but here's the lowkey essential stuff 💅. The format looks like this:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+Here's what each piece means, based on the H.264 spec:
- F (1 bit):
forbidden_zero_bit. H.264 treats a value of 1 as a syntax violation 🙅♀️. - NRI (2 bits):
nal_ref_idc. A value of00means the NAL unit's content isn't used to rebuild reference pictures for inter picture prediction. That means it can be discarded without risking the integrity of the reference pictures 🗑️. Any value greater than00means decoding that NAL unit is required to keep the reference pictures intact 🔒. - Type (5 bits):
nal_unit_type. This tells you the NAL unit payload type, as defined in table 7-1 of [1] and later in this memo. For every currently defined NAL unit type and its semantics, check section 7.4.1 in [1] 🔍.
This memo introduces new NAL unit types, which you'll find in section 5.2 ✨. These new types are marked as unspecified in [1]. This spec also extends the semantics of F and NRI, as described in section 5.3 🌈.
2. Conventions 📝
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [3]. No cap, these words are doing serious work 🫶.
This spec talks about setting and clearing a bit whenever bit fields come up 🎀. Setting a bit just means assigning it the value 1 (On) ✅. Clearing a bit means assigning it the value 0 (Off) ❌.
3. Scope 🌸
This payload spec can only carry the "naked" H.264 NAL unit stream over RTP. It can't carry the bitstream format discussed in Annex B of H.264 🙅♀️. The first applications of this spec will likely be in conversational multimedia, like video telephony or video conferencing 📞. But the payload format also covers other stuff, like Internet streaming and TV over IP 📺🦋.
4. Definitions and Abbreviations 💜
4.1. Definitions
This document uses the definitions from [1]. For convenience, here's a quick recap of the terms defined there 📖:
access unit: A set of NAL units that always contains a primary coded picture. Along with that primary coded picture, an access unit may also contain one or more redundant coded pictures, or other NAL units that don't contain slices or slice data partitions of a coded picture 🧩. Decoding an access unit always results in a decoded picture 🖼️.
coded video sequence: A sequence of access units. In decoding order, it starts with an instantaneous decoding refresh (IDR) access unit, followed by zero or more non-IDR access units. It includes every access unit after that up to, but not including, the next IDR access unit 🔁.
IDR access unit: An access unit whose primary coded picture is an IDR picture 🔄.
IDR picture: A coded picture that contains only slices with I or SI slice types, and that causes a "reset" in the decoding process 🧹. After an IDR picture is decoded, every following coded picture in decoding order can be decoded without inter prediction from any picture decoded before the IDR picture 💖.
primary coded picture: The coded representation of a picture that the decoding process uses for a bitstream conforming to H.264. It contains all the macroblocks of the picture 🧱.
redundant coded picture: A coded representation of a picture, or of a part of a picture. The decoding process for a bitstream conforming to H.264 shall not use its content. The decoding process for a bitstream that contains errors or losses may use its content 🛟.
VCL NAL unit: A catch-all term for coded slice and coded data partition NAL units 🎬.
In addition, the following definitions apply 👑:
decoding order number (DON): A field in the payload structure, or a derived variable, that indicates NAL unit decoding order. DON values are in the range of 0 to 65535, inclusive. Once it hits the maximum value, DON wraps around to 0 🔄.
NAL unit decoding order: A NAL unit order that follows the constraints on NAL unit order given in section 7.4.1.2 in [1] 📏.
transmission order: The order of packets in ascending RTP sequence number order, using modulo arithmetic 🔢. Inside an aggregation packet, the NAL unit transmission order is the same as the order in which the NAL units appear in the packet 📦.
media aware network element (MANE): A network element, like a middlebox or application layer gateway, that can parse certain aspects of the RTP payload headers or the RTP payload, and react to what it finds 🕵️♀️.
Informative note: A MANE is more than a normal router or gateway 🧐. It has to be aware of the signaling, for example to learn the payload type mappings of the media streams. It also has to be trusted when working with SRTP 🔐. The big upside is that MANEs can drop packets according to what the media coding needs. Say a MANE has to drop packets because of congestion on a certain link. It can pick the packets whose dropping does the least harm to the user experience and remove those, which clears the congestion and/or keeps the delay low ⚡.
Abbreviations
DON: Decoding Order Number
DONB: Decoding Order Number Base
DOND: Decoding Order Number Difference
FEC: Forward Error Correction
FU: Fragmentation Unit
IDR: Instantaneous Decoding Refresh
IEC: International Electrotechnical Commission
ISO: International Organization for Standardization
ITU-T: International Telecommunication Union,
Telecommunication Standardization Sector
MANE: Media Aware Network Element
MTAP: Multi-Time Aggregation Packet
MTAP16: MTAP with 16-bit timestamp offset
MTAP24: MTAP with 24-bit timestamp offset
NAL: Network Abstraction Layer
NALU: NAL Unit
SEI: Supplemental Enhancement Information
STAP: Single-Time Aggregation Packet
STAP-A: STAP type A
STAP-B: STAP type B
TS: Timestamp
VCL: Video Coding Layer5. RTP Payload Format 🦋
5.1. RTP Header Usage ✨
okay so the RTP header format is defined in RFC 3550 [4], and I've reprinted it in Figure 1 so you don't have to go hunting 🔍 this payload format uses the header fields in a way that matches that spec, no cap 💅
when one NAL unit goes in each RTP packet, the RECOMMENDED RTP payload format is the one in section 5.6 🎀 the RTP payload (plus the settings for some RTP header bits) for aggregation packets and fragmentation units lives in sections 5.7 and 5.8, respectively 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X| CC |M| PT | sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| synchronization source (SSRC) identifier |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
| contributing source (CSRC) identifiers |
| .... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 1. RTP header according to RFC 3550
the RTP header info you set for this payload format goes like this, bestie 🫶
Marker bit (M): 1 bit 🚩 Set this on the very last packet of the access unit that the RTP timestamp points to. It matches how the M bit is normally used in video formats, and it helps the playout buffer handle things efficiently 🎬 for aggregation packets (STAP and MTAP), the marker bit in the RTP header MUST be set to whatever the marker bit of the last NAL unit in the aggregation packet would have been if that NAL unit had its own RTP packet 😭 decoders MAY use this bit as an early hint that they've hit the last packet of an access unit, but they MUST NOT rely on that property 🙈
Informative note: Only one M bit is attached to an aggregation packet carrying multiple NAL units. So if a gateway re-packetizes an aggregation packet into several packets, it can't reliably set the M bit on those packets. 💀
Payload type (PT): 7 bits 🍓 The RTP payload type for this new packet format is out of scope for this document and won't be specified here. Assigning a payload type has to happen either through the profile being used or dynamically 🌈
Sequence number (SN): 16 bits 🦋 Set and used according to RFC 3550. For the single NALU and non-interleaved packetization modes, the sequence number is what decides the decoding order of the NALU 🧠
Timestamp: 32 bits ⏰ The RTP timestamp is set to the sampling timestamp of the content. A 90 kHz clock rate MUST be used, no exceptions 👑
If the NAL unit has no timing properties of its own (think parameter set and SEI NAL units), the RTP timestamp gets set to the RTP timestamp of the primary coded picture of the access unit the NAL unit is part of, following section 7.4.1.2 of [1] 🫧 the RTP Timestamp for MTAPs is defined in section 5.7.2 ✨
receivers SHOULD ignore any picture timing SEI messages in access units that only have one display timestamp. Instead, receivers SHOULD use the RTP timestamp to sync the display process 🎀 and RTP senders SHOULD NOT transmit picture timing SEI messages for pictures that aren't supposed to be displayed as multiple fields 🙅♀️
now the tricky bit, lowkey my favorite 🤓 if one access unit has more than one display timestamp in a picture timing SEI message, the SEI info SHOULD be treated as relative to the RTP timestamp. The earliest event happens at the RTP timestamp, and the later ones come after, based on the difference in SEI picture timing values. Say the display timestamps in an access unit's SEI message are tSEI1, tSEI2, ..., tSEIn, where tSEI1 is the earliest of them all. Then tmadjst() is a function that converts the SEI message's time scale to a 90-kHz time scale, and TS is the RTP timestamp. The display time for the event tied to tSEI1 is TS. The display time for the event with tSEIx, where x is [2..n], is TS + tmadjst (tSEIx - tSEI1) 🧮✨
Informative note: Displaying coded frames as fields is commonly needed for an operation called 3:2 pulldown, where film content made of coded frames gets shown on a display that uses interlaced scanning. The picture timing SEI message lets you carry multiple timestamps for the same coded picture, so 3:2 pulldown can be perfectly controlled 🎞️ this SEI mechanism is necessary because only one timestamp per coded frame can be conveyed in the RTP timestamp 💜
Informative note: H.264 lets the decoding order differ from the display order, so RTP timestamps may not be monotonically non-decreasing as a function of RTP sequence numbers 😵💫 also, the interarrival jitter value reported in RTCP reports may not be a trustworthy sign of network performance, because the interarrival jitter calculation rules (section 6.4.1 of RFC 3550) assume an RTP timestamp is directly proportional to its transmission time 📉
5.2. Common Structure of the RTP Payload Format 🧩
okay the payload format defines three basic payload structures. A receiver can tell which structure it's looking at from the first byte of the RTP payload. That byte does double duty as the RTP payload header and, in some cases, as the first byte of the payload too 🫶 it's always structured as a NAL unit header, and the NAL unit type field tells you which structure is present. The possible structures are:
Single NAL Unit Packet: 🎯 contains just one NAL unit in the payload. The NAL header type field will equal the original NAL unit type, meaning it's in the range of 1 to 23, inclusive. Specified in section 5.6.
Aggregation packet: 📦 the packet type used to bundle multiple NAL units into one RTP payload. It comes in four versions: the Single-Time Aggregation Packet type A (STAP-A), the Single-Time Aggregation Packet type B (STAP-B), the Multi-Time Aggregation Packet (MTAP) with 16-bit offset (MTAP16), and the Multi-Time Aggregation Packet (MTAP) with 24-bit offset (MTAP24). The NAL unit type numbers assigned to STAP-A, STAP-B, MTAP16, and MTAP24 are 24, 25, 26, and 27, respectively. Specified in section 5.7.
Fragmentation unit: ✂️ used to split a single NAL unit across multiple RTP packets. It exists in two versions, FU-A and FU-B, identified by the NAL unit type numbers 28 and 29, respectively. Specified in section 5.8.
Table 1 is the cheat sheet for all of this 🌸
Table 1. Summary of NAL unit types and their payload structures
Type Packet Type name Section
---------------------------------------------------------
0 undefined -
1-23 NAL unit Single NAL unit packet per H.264 5.6
24 STAP-A Single-time aggregation packet 5.7.1
25 STAP-B Single-time aggregation packet 5.7.1
26 MTAP16 Multi-time aggregation packet 5.7.2
27 MTAP24 Multi-time aggregation packet 5.7.2
28 FU-A Fragmentation unit 5.8
29 FU-B Fragmentation unit 5.8
30-31 undefined -Informative note: This specification doesn't limit the size of NAL units encapsulated in single NAL unit packets and fragmentation units. The maximum size of a NAL unit encapsulated in any aggregation packet is 65535 bytes. 🙌
5.3. NAL Unit Octet Usage ✨
Okay so we already met the NAL unit octet back in section 1.3, but it's worth another look 🎀 Here's the layout reprinted for convenience:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+This section explains what F and NRI mean according to this specification 💅
F: 1 bit (forbidden_zero_bit) 🙈 A value of 0 means the NAL unit type octet and payload should not contain bit errors or other syntax violations. A value of 1 means the NAL unit type octet and payload may contain bit errors or other syntax violations.
MANEs SHOULD set the F bit to flag detected bit errors in the NAL unit 🚨 The H.264 specification requires that the F bit is equal to 0. When the F bit is set, the decoder is warned that bit errors or other syntax violations may be present in the payload or in the NAL unit type octet. The simplest decoder reaction to a NAL unit with the F bit equal to 1 is to discard it and conceal the lost data in the discarded NAL unit. Honestly, a very chill plan 😌
NRI: 2 bits (nal_ref_idc) ✨ The meaning of the value 00 and of any non-zero value stays the same as in the H.264 specification. Basically, a value of 00 means the content of the NAL unit is not used to reconstruct reference pictures for inter picture prediction. Those NAL units can be discarded without risking the integrity of the reference pictures. Values greater than 00 mean the NAL unit needs to be decoded to keep the reference pictures intact.
On top of that, this RTP payload spec says that NRI values greater than 00 show the relative transport priority, as set by the encoder 👑 MANEs can use this to protect the more important NAL units better than the less important ones. The highest transport priority is 11, then 10, then 01, and finally 00 is the lowest 🧚
Informative note: Any non-zero value of NRI is handled identically in H.264 decoders. Therefore, receivers need not manipulate the value of NRI when passing NAL units to the decoder.
An H.264 encoder MUST set the value of NRI according to the H.264 specification (subclause 7.4.1) when the value of nal_unit_type is in the range of 1 to 12, inclusive. In particular, the H.264 specification requires that the value of NRI SHALL be equal to 0 for all NAL units having nal_unit_type equal to 6, 9, 10, 11, or 12. The rules are strict here, no cap 🫶
For NAL units with nal_unit_type equal to 7 or 8 (sequence parameter set or picture parameter set, respectively), an H.264 encoder SHOULD set the value of NRI to 11 (in binary format). For coded slice NAL units of a primary coded picture with nal_unit_type equal to 5 (a coded slice belonging to an IDR picture), an H.264 encoder SHOULD set the value of NRI to 11 (in binary format).
For mapping the remaining nal_unit_types to NRI values, the following example MAY be used. It has been shown to be efficient in a certain environment [13]. Other mappings MAY also be desirable, depending on the application and the H.264/AVC Annex A profile in use 🌸
Informative note: Data Partitioning is not available in certain profiles, e.g., in the Main or Baseline profiles. Consequently, the nal unit types 2, 3, and 4 can occur only if the video bitstream conforms to a profile in which data partitioning is allowed, and not in streams that conform to the Main or Baseline profiles.
Table 2. Example of NRI values for coded slices and coded slice data partitions of primary coded reference pictures
NAL Unit Type Content of NAL unit NRI (binary)
----------------------------------------------------------------
1 non-IDR coded slice 10
2 Coded slice data partition A 10
3 Coded slice data partition B 01
4 Coded slice data partition C 01Informative note: As mentioned before, the NRI value of non-reference pictures is 00 as mandated by H.264/AVC.
An H.264 encoder SHOULD set the value of NRI for coded slice and coded slice data partition NAL units of redundant coded reference pictures equal to 01 (in binary format).
Definitions of the NRI values for NAL unit types 24 to 29, inclusive, are given in sections 5.7 and 5.8 of this memo 📖
No recommendation for the NRI value is given for NAL units having nal_unit_type in the range of 13 to 23, inclusive, because those values are reserved for ITU-T and ISO/IEC 🔒 No recommendation for the NRI value is given for NAL units having nal_unit_type equal to 0 or in the range of 30 to 31, inclusive, since the semantics of those values are not specified in this memo.
5.4. Packetization Modes 🦋
This memo specifies three packetization modes, and they're giving three very different vibes:
- Single NAL unit mode
- Non-interleaved mode
- Interleaved mode
The single NAL unit mode is aimed at conversational systems that comply with ITU-T Recommendation H.241 [15] (see section 12.1) 📞 The non-interleaved mode is aimed at conversational systems that may not comply with ITU-T Recommendation H.241. In non-interleaved mode, NAL units are transmitted in NAL unit decoding order. The interleaved mode is aimed at systems that don't need very low end-to-end latency. It allows NAL units to be transmitted out of NAL unit decoding order 🌈
The packetization mode in use MAY be signaled by the value of the OPTIONAL packetization-mode MIME parameter, or by external means ✨ The packetization mode in use governs which NAL unit types are allowed in RTP payloads. Table 3 summarizes the allowed NAL unit types for each packetization mode. Some NAL unit type values (marked as undefined in Table 3) are reserved for future extensions. NAL units of those types SHOULD NOT be sent by a sender and MUST be ignored by a receiver 💀 For example, the Types 1-23, with the associated packet type "NAL unit", are allowed in "Single NAL Unit Mode" and in "Non-Interleaved Mode", but disallowed in "Interleaved Mode". Packetization modes are explained in more detail in section 6 🍓
Table 3. Summary of allowed NAL unit types for each packetization mode (yes = allowed, no = disallowed, ig = ignore)
Type Packet Single NAL Non-Interleaved Interleaved
Unit Mode Mode Mode
-------------------------------------------------------------
0 undefined ig ig ig
1-23 NAL unit yes yes no
24 STAP-A no yes no
25 STAP-B no no yes
26 MTAP16 no no yes
27 MTAP24 no no yes
28 FU-A no yes yes
29 FU-B no no yes
30-31 undefined ig ig ig5.5. Decoding Order Number (DON) 🔢
Okay bestie, in interleaved mode the NAL units are allowed to be sent in a different order than they get decoded 🌀. The decoding order number (DON) is either a field in the payload structure or a derived variable, and its whole job is to tell you the NAL unit decoding order ✨. The reasons you'd ever send things out of decoding order, plus examples of how DON helps, live in section 13 💅.
Whether send order has to match decode order is controlled by the OPTIONAL sprop-interleaving-depth MIME parameter, and it works like this 👑:
- If
sprop-interleaving-depthis equal to 0 (set explicitly or just by default), OR sending NAL units out of decoding order is disallowed by external means, then the transmission order of NAL units MUST conform to the NAL unit decoding order 🚫. - If
sprop-interleaving-depthis greater than 0, OR sending NAL units out of decoding order is allowed by external means, then:- the order of NAL units in an MTAP16 and an MTAP24 is NOT REQUIRED to be the NAL unit decoding order, and
- the order of NAL units generated by decapsulating STAP-Bs, MTAPs, and FUs in two consecutive packets is NOT REQUIRED to be the NAL unit decoding order 🦋.
The RTP payload structures for a single NAL unit packet, an STAP-A, and an FU-A do not include DON 🍓. STAP-B and FU-B structures do include DON, and the structure of MTAPs lets you derive DON the way section 5.7.2 spells out 🧚.
Informative note: When an FU-A occurs in interleaved mode, it always follows an FU-B, which sets its DON. 🌸
Informative note: If a transmitter wants to encapsulate a single NAL unit per packet and transmit packets out of their decoding order, STAP-B packet type can be used. 💖
In the single NAL unit packetization mode, the transmission order of NAL units (as determined by the RTP sequence number) MUST be the same as their NAL unit decoding order 📦. In the non-interleaved packetization mode, the transmission order of NAL units in single NAL unit packets, STAP-As, and FU-As MUST be the same as their NAL unit decoding order ✨. The NAL units within an STAP MUST appear in the NAL unit decoding order too. So the decoding order comes first from the implicit order inside a STAP, and second from the RTP sequence number for the order between STAPs, FUs, and single NAL unit packets 🎀.
Signaling the DON value for NAL units carried in STAP-B, MTAP, and a series of fragmentation units that starts with an FU-B is covered in sections 5.7.1, 5.7.2, and 5.8, respectively 🔥. The DON value of the first NAL unit in transmission order MAY be set to any value 🌈. DON values run from 0 to 65535, inclusive, and once you hit the max, it wraps back around to 0 🔄.
Now here's how you figure out the decoding order of any two NAL units inside an STAP-B, an MTAP, or a series of fragmentation units starting with an FU-B 🧮. Let DON(i) be the decoding order number of the NAL unit at index i in transmission order. The function don_diff(m,n) is defined like this:
If DON(m) == DON(n), don_diff(m,n) = 0
If (DON(m) < DON(n) and DON(n) - DON(m) < 32768),
don_diff(m,n) = DON(n) - DON(m)
If (DON(m) > DON(n) and DON(m) - DON(n) >= 32768),
don_diff(m,n) = 65536 - DON(m) + DON(n)
If (DON(m) < DON(n) and DON(n) - DON(m) >= 32768),
don_diff(m,n) = - (DON(m) + 65536 - DON(n))
If (DON(m) > DON(n) and DON(m) - DON(n) < 32768),
don_diff(m,n) = - (DON(m) - DON(n))A positive don_diff(m,n) means the NAL unit at transmission index n comes after the NAL unit at transmission index m in decoding order 📈. When don_diff(m,n) is equal to 0, the two NAL units can be in either order in decoding 🤷♀️. A negative don_diff(m,n) means the NAL unit at transmission index n comes before the NAL unit at transmission index m in decoding order 📉.
Values of the DON-related fields (DON, DONB, and DOND; see section 5.7) MUST be such that the decoding order determined by the DON values, as specified above, conforms to the NAL unit decoding order 💜. If two NAL units in NAL unit decoding order get switched and the new order does NOT conform, those NAL units MUST NOT have the same DON value 🙅♀️. If two consecutive NAL units get switched and the new order STILL conforms, they MAY have the same DON value 🌷. For example, when arbitrary slice order is allowed by the video coding profile in use, all the coded slice NAL units of a coded picture are allowed to share one DON value 🌺. So NAL units with the same DON can be decoded in any order, and two NAL units with different DON values should be passed to the decoder in the order specified above 🫶. When two consecutive NAL units in decoding order have different DON values, the DON for the second one in decoding order SHOULD be the first one's DON, incremented by one ⭐.
An example of the decapsulation process that recovers the NAL unit decoding order is given in section 7 🗺️.
Informative note: Receivers should not expect that the absolute difference of values of DON for two consecutive NAL units in the NAL unit decoding order will be equal to one, even in error-free transmission. An increment by one is not required, as at the time of associating values of DON to NAL units, it may not be known whether all NAL units are delivered to the receiver. For example, a gateway may not forward coded slice NAL units of non-reference pictures or SEI NAL units when there is a shortage of bit rate in the network to which the packets are forwarded. In another example, a live broadcast is interrupted by pre-encoded content, such as commercials, from time to time. The first intra picture of a pre-encoded clip is transmitted in advance to ensure that it is readily available in the receiver. When transmitting the first intra picture, the originator does not exactly know how many NAL units will be encoded before the first intra picture of the pre-encoded clip follows in decoding order. Thus, the values of DON for the NAL units of the first intra picture of the pre-encoded clip have to be estimated when they are transmitted, and gaps in values of DON may occur. 🙈
5.6. Single NAL Unit Packet 🦋
The single NAL unit packet defined here MUST contain only one NAL unit, of the types defined in [1] 🎯. That means neither an aggregation packet nor a fragmentation unit can be used inside a single NAL unit packet 🚫. A NAL unit stream made by decapsulating single NAL unit packets in RTP sequence number order MUST conform to the NAL unit decoding order 💅. The structure of the single NAL unit packet is shown in Figure 2 👇.
Informative note: The first byte of a NAL unit co-serves as the RTP payload header. 🧁
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| type | |
+-+-+-+-+-+-+-+-+ |
| |
| Bytes 2..n of a Single NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2. RTP payload format for single NAL unit packet ✨
5.7. Aggregation Packets ✨
okay so aggregation packets are the NAL unit aggregation scheme of this payload spec 🎀 and they exist because two worlds have wildly different MTU sizes. on wireline IP networks the MTU is often capped by Ethernet, so roughly 1500 bytes. then on IP or non-IP wireless systems (ITU-T H.324/M is one example) the preferred transmission unit is 254 bytes or less 😭. the goal is to avoid media transcoding between the two worlds and keep packetization overhead from getting out of hand, so we aggregate 💅
There are two types of aggregation packets in this spec 👑
- Single-time aggregation packet (STAP): aggregates NAL units with identical NALU-time. There are two flavors, one without DON (STAP-A) and one that includes DON (STAP-B).
- Multi-time aggregation packet (MTAP): aggregates NAL units with potentially differing NALU-time. There are two different MTAPs, and they differ in the length of the NAL unit timestamp offset.
quick reminder that NALU-time is the value the RTP timestamp would have if that NAL unit rode in its own RTP packet 🧚
Each NAL unit inside an aggregation packet gets wrapped in an aggregation unit. The four aggregation units and their characteristics are coming up below 🦋
The structure of an aggregation packet's RTP payload is shown in Figure 3 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F|NRI| type | |
+-+-+-+-+-+-+-+-+ |
| |
| one or more aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3. RTP payload format for aggregation packets
Now the packetization rules that MTAPs and STAPs both follow 🫶 The RTP timestamp MUST be set to the earliest of the NALU times of all the NAL units being aggregated. The type field of the NAL unit type octet MUST be set to the right value, as listed in Table 4. The F bit MUST be cleared if every F bit of the aggregated NAL units is zero, and otherwise it MUST be set. The value of NRI MUST be the maximum of all the NAL units carried in the aggregation packet 🔥
Table 4 has the type values, which you need to get exactly right 💀
Type Packet Timestamp offset DON related fields
field length (DON, DONB, DOND)
(in bits) present
--------------------------------------------------------
24 STAP-A 0 no
25 STAP-B 0 yes
26 MTAP16 16 yes
27 MTAP24 24 yesTable 4. Type field for STAPs and MTAPs
The marker bit in the RTP header gets set to whatever the marker bit of the last NAL unit in the aggregated packet would have been if that NAL unit had its own RTP packet 🍓
The payload of an aggregation packet is one or more aggregation units. Check sections 5.7.1 and 5.7.2 for the four different aggregation unit types 💜 An aggregation packet can hold as many aggregation units as it needs, but the total data in the packet MUST fit into one IP packet, and the size SHOULD be picked so the resulting IP packet ends up smaller than the MTU. An aggregation packet MUST NOT contain the fragmentation units from section 5.8. And aggregation packets MUST NOT be nested, meaning an aggregation packet MUST NOT contain another aggregation packet 🙈
5.7.1. Single-Time Aggregation Packet 🌸
The single-time aggregation packet (STAP) SHOULD be used whenever the NAL units being aggregated all share the same NALU-time ⭐ The payload of an STAP-A has no DON and is made of at least one single-time aggregation unit, shown in Figure 4. The payload of an STAP-B is a 16-bit unsigned decoding order number (DON), sent in network byte order, followed by at least one single-time aggregation unit, shown in Figure 5 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: |
+-+-+-+-+-+-+-+-+ |
| |
| single-time aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4. Payload format for STAP-A
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: decoding order number (DON) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| single-time aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5. Payload format for STAP-B
The DON field gives the DON value of the first NAL unit in an STAP-B, in transmission order 🧚 For each NAL unit after that, in appearance order inside the STAP-B, the DON value is (the DON value of the previous NAL unit in the STAP-B + 1) % 65536, where '%' is the modulo operation 🔥
A single-time aggregation unit is made of 16-bit unsigned size information, in network byte order, that gives the size in bytes of the NAL unit that follows. That size excludes these two octets but includes the NAL unit type octet of that NAL unit. After the size comes the NAL unit itself, type byte and all 💅 A single-time aggregation unit is byte aligned within the RTP payload, but it may not be aligned on a 32-bit word boundary. Figure 6 shows the structure 🦋
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NAL unit size | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6. Structure for single-time aggregation unit
Figure 7 is an example RTP packet holding an STAP-A. The STAP has two single-time aggregation units, labeled 1 and 2 in the figure 🍓
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|STAP-A NAL HDR | NALU 1 Size | NALU 1 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Data |
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 Data |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7. An example of an RTP packet including an STAP-A and two single-time aggregation units
Figure 8 is an example RTP packet holding an STAP-B. Again the STAP has two single-time aggregation units, labeled 1 and 2 in the figure 🫶
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|STAP-B NAL HDR | DON | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 HDR | NALU 1 Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 Size | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 Data |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 8. An example of an RTP packet including an STAP-B and two single-time aggregation units
5.7.2. Multi-Time Aggregation Packets (MTAPs) 🎀
okay bestie, so the NAL unit payload of an MTAP is two things: a 16-bit unsigned decoding order number base (DONB), sent in network byte order 🧚, plus one or more multi-time aggregation units 💖 (check Figure 9). The DONB MUST contain the DON value of the first NAL unit in NAL unit decoding order among the NAL units in the MTAP ✨
Informative note: The first NAL unit in the NAL unit decoding order is not necessarily the first NAL unit in the order in which the NAL units are encapsulated in an MTAP. 🙈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: decoding order number base | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| multi-time aggregation units |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 9. NAL unit payload format for MTAPs 📸
Two different multi-time aggregation units are defined in this spec 🦋. They both have 16 bits of unsigned size info for the following NAL unit (network byte order), an 8-bit unsigned decoding order number difference (DOND), and n bits of timestamp offset (TS offset) for that NAL unit, in network byte order, where n can be 16 or 24. Which MTAP type you pick (MTAP16 or MTAP24) is application dependent. The bigger the timestamp offset, the more flexible the MTAP is, but the overhead goes up too, so it's a trade-off 💅
The structure of the multi-time aggregation units for MTAP16 and MTAP24 is shown in Figures 10 and 11. The starting or ending position of an aggregation unit within a packet is NOT REQUIRED to sit on a 32-bit word boundary 🌸. The DON of the following NAL unit is equal to (DONB + DOND) % 65536, where % is the modulo operation 🧮. This memo does not specify how the NAL units within an MTAP are ordered, but in most cases NAL unit decoding order SHOULD be used 👑.
The timestamp offset field MUST be set to the result of this formula 💜: if the NALU-time is larger than or equal to the RTP timestamp of the packet, then the timestamp offset equals (the NALU-time of the NAL unit - the RTP timestamp of the packet). If the NALU-time is smaller than the RTP timestamp of the packet, then the timestamp offset is equal to the NALU-time + (2^32 - the RTP timestamp of the packet).
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NAL unit size | DOND | TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS offset | |
+-+-+-+-+-+-+-+-+ NAL unit |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 10. Multi-time aggregation unit for MTAP16 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: NALU unit size | DOND | TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS offset | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| NAL unit |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 11. Multi-time aggregation unit for MTAP24 🦋
For the "earliest" multi-time aggregation unit in an MTAP, the timestamp offset MUST be zero 🌈. So the RTP timestamp of the MTAP itself is identical to the earliest NALU-time, no cap ✨.
Informative note: The "earliest" multi-time aggregation unit is the one that would have the smallest extended RTP timestamp among all the aggregation units of an MTAP if the aggregation units were encapsulated in single NAL unit packets. An extended timestamp is a timestamp that has more than 32 bits and is capable of counting the wraparound of the timestamp field, thus enabling one to determine the smallest value if the timestamp wraps. Such an "earliest" aggregation unit may not be the first one in the order in which the aggregation units are encapsulated in an MTAP. The "earliest" NAL unit need not be the same as the first NAL unit in the NAL unit decoding order either. 🧠
Figure 12 gives an example of an RTP packet holding a multi-time aggregation packet of type MTAP16, with two multi-time aggregation units, labeled 1 and 2 in the figure 💖.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MTAP16 NAL HDR | decoding order number base | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 DOND | NALU 1 TS offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 HDR | NALU 1 DATA |
+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 SIZE | NALU 2 DOND |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 TS offset | NALU 2 HDR | NALU 2 DATA |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 12. An RTP packet including a multi-time aggregation packet of type MTAP16 and two multi-time aggregation units 🍓
Figure 13 gives an example of an RTP packet holding a multi-time aggregation packet of type MTAP24, with two multi-time aggregation units, labeled 1 and 2 in the figure 🧚♀️.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MTAP24 NAL HDR | decoding order number base | NALU 1 Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 1 Size | NALU 1 DOND | NALU 1 TS offs |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|NALU 1 TS offs | NALU 1 HDR | NALU 1 DATA |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
: :
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | NALU 2 SIZE | NALU 2 DOND |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 TS offset | NALU 2 HDR |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NALU 2 DATA |
: :
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 13. An RTP packet including a multi-time aggregation packet of type MTAP24 and two multi-time aggregation units ✨
5.8. Fragmentation Units (FUs) ✨
okay so this payload type lets you chop one NAL unit into a bunch of RTP packets 🔪🎀 and doing it at the application layer (instead of leaving it to lower layers like IP) is lowkey iconic for two reasons:
- The payload format can carry NAL units bigger than 64 kbytes over an IPv4 network 😭. This matters for pre-recorded video, especially High Definition stuff. There's a cap on slices per picture, which caps the NAL units per picture, which can make those NAL units huge 🐘.
- The fragmentation mechanism lets you fragment a single picture and apply generic forward error correction, as described in section 12.5 🛡️✨.
Fragmentation is only defined for a single NAL unit, never for aggregation packets 🙅♀️. A fragment is an integer number of consecutive octets from that NAL unit. Every octet of the NAL unit MUST be part of exactly one fragment of that NAL unit 💅. The fragments of the same NAL unit MUST be sent in consecutive order with ascending RTP sequence numbers, and no other RTP packets in the same RTP packet stream can sneak in between the first and last fragment 🚫🧚. Same deal on the receiving end: a NAL unit MUST be reassembled in RTP sequence number order 🔁.
When a NAL unit gets fragmented and carried inside fragmentation units (FUs), it's called a fragmented NAL unit 🦋. STAPs and MTAPs MUST NOT be fragmented 🙈, and FUs MUST NOT be nested, which means an FU MUST NOT contain another FU 💀.
The RTP timestamp of an RTP packet carrying an FU is set to the NALU time of the fragmented NAL unit ⏰🌸.
Figure 14 shows the RTP payload format for FU-As 💖. An FU-A is one octet of fragmentation unit indicator, one octet of fragmentation unit header, and then the fragmentation unit payload 🎀.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 14. RTP payload format for FU-A
Figure 15 shows the RTP payload format for FU-Bs 🧁. An FU-B is basically an FU-A with one extra thing: a decoding order number (DON), sent in network byte order, sitting between the fragmentation unit header and the payload. So the structure is the same as FU-A, plus that DON field 🫶.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | DON |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 15. RTP payload format for FU-B
NAL unit type FU-B MUST be used in the interleaved packetization mode for the first fragmentation unit of a fragmented NAL unit 🌈. NAL unit type FU-B MUST NOT be used in any other case. So in interleaved mode, every NALU that gets fragmented starts with an FU-B, followed by one or more FU-A fragments 🧚♀️.
The FU indicator octet looks like this 👀:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+Type values 28 and 29 in the FU indicator octet mean FU-A and FU-B, respectively 🔢. The F bit is used as described in section 5.3. The NRI field MUST be set according to the NRI value of the fragmented NAL unit 💜.
The FU header looks like this 💫:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|R| Type |
+---------------+- S: 1 bit. When set to one, the Start bit marks the start of a fragmented NAL unit 🏁. If the FU payload isn't the start of a fragmented NAL unit payload, the Start bit is set to zero.
- E: 1 bit. When set to one, the End bit marks the end of a fragmented NAL unit, meaning the last byte of the payload is also the last byte of the fragmented NAL unit 🏁✨. If the FU payload isn't the last fragment of a fragmented NAL unit, the End bit is set to zero.
- R: 1 bit. The Reserved bit MUST be equal to 0 and MUST be ignored by the receiver 🤫.
- Type: 5 bits. The NAL unit payload type, as defined in table 7-1 of [1] 📋.
The DON value in FU-Bs is selected as described in section 5.5 🔢.
Informative note: The DON field in FU-Bs lets gateways fragment NAL units into FU-Bs without having to put the incoming NAL units in NAL unit decoding order. Literally so helpful 🙌.
A fragmented NAL unit MUST NOT be sent in one FU. In other words, the Start bit and End bit MUST NOT both be set to one in the same FU header 🚫.
The FU payload is made of fragments of the fragmented NAL unit's payload, so if you concatenate the fragmentation unit payloads of consecutive FUs in order, you can rebuild the payload of the fragmented NAL unit 🧩. The NAL unit type octet of the fragmented NAL unit isn't included as-is in the fragmentation unit payload. Instead, its information lives in the F and NRI fields of the FU indicator octet and in the Type field of the FU header 🗂️. An FU payload MAY have any number of octets and MAY be empty 🫧.
Informative note: Empty FUs are allowed to reduce latency for a certain class of senders in nearly lossless environments. These senders packetize NALU fragments before the NALU is completely generated, so they don't know the NALU size yet. If zero-length NALU fragments weren't allowed, the sender would have to generate at least one bit of the following fragment before the current fragment could be sent. Because of how H.264 works, where sometimes several macroblocks take zero bits, that's undesirable and can add delay 😤. Still, the (potential) use of zero-length NALUs should be carefully weighed against the higher risk of losing the NALU, since more packets get used to send it 🥺.
If a fragmentation unit is lost, the receiver SHOULD discard all following fragmentation units in transmission order that belong to the same fragmented NAL unit 🗑️💅.
A receiver in an endpoint or in a MANE MAY aggregate the first n-1 fragments of a NAL unit into an (incomplete) NAL unit, even if fragment n of that NAL unit isn't received 🧃. In that case, the forbidden_zero_bit of the NAL unit MUST be set to one to flag a syntax violation 🚨.
6. Packetization Rules 🎀
The packetization modes are introduced in section 5.2 🌸. The packetization rules shared by more than one of the packetization modes are specified in section 6.1. The packetization rules for the single NAL unit mode, the non-interleaved mode, and the interleaved mode are specified in sections 6.2, 6.3, and 6.4, respectively 🗺️.
6.1. Common Packetization Rules
All senders MUST enforce the following packetization rules, no matter which packetization mode they're using 👑:
- Coded slice NAL units or coded slice data partition NAL units that belong to the same coded picture (and therefore share the same RTP timestamp value) MAY be sent in any order permitted by the applicable profile defined in [1]. However, for delay-critical systems, they SHOULD be sent in their original coding order to keep delay down ⏱️. Note that coding order isn't necessarily the scan order. It's the order the NAL packets become available to the RTP stack 🧠.
- Parameter sets are handled according to the rules and recommendations in section 8.4 📦.
- MANEs MUST NOT duplicate any NAL unit except for sequence or picture parameter set NAL units, since neither this memo nor the H.264 specification gives a way to identify duplicated NAL units 🙅♀️. Sequence and picture parameter set NAL units MAY be duplicated to make their correct reception more likely, but any such duplication MUST NOT affect the contents of any active sequence or picture parameter set 🛡️. Duplication SHOULD happen at the application layer, not by duplicating RTP packets (with identical sequence numbers) 💫.
Senders using the non-interleaved mode and the interleaved mode MUST enforce the following packetization rule too 🫶:
- MANEs MAY convert single NAL unit packets into one aggregation packet, convert an aggregation packet into several single NAL unit packets, or mix both concepts, in an RTP translator 🔄. The RTP translator SHOULD take into account at least the following parameters: path MTU size, unequal protection mechanisms (for example, packet-based FEC according to RFC 2733 [18], especially for sequence and picture parameter set NAL units and coded slice data partition A NAL units), bearable latency of the system, and buffering capabilities of the receiver 📊✨.
Informative note: An RTP translator is required to handle RTCP as per RFC 3550 📡.
6.2. Single NAL Unit Mode
This mode is in use when the OPTIONAL packetization-mode MIME parameter is equal to 0, when the packetization-mode isn't present, or when no other packetization mode is signaled by external means 🎯. All receivers MUST support this mode 🙌. It's mainly for low-delay applications that work with systems using ITU-T Recommendation H.241 [15] (see section 12.1) 🐢➡️⚡. Only single NAL unit packets MAY be used in this mode. STAPs, MTAPs, and FUs MUST NOT be used 🚫. The transmission order of single NAL unit packets MUST comply with the NAL unit decoding order 📜.
6.3. Non-Interleaved Mode
This mode is in use when the OPTIONAL packetization-mode MIME parameter is equal to 1, or when the mode is turned on by external means 🔛. This mode SHOULD be supported. It's mainly for low-delay applications 🏃♀️. Only single NAL unit packets, STAP-As, and FU-As MAY be used in this mode. STAP-Bs, MTAPs, and FU-Bs MUST NOT be used 🙅♀️. The transmission order of NAL units MUST comply with the NAL unit decoding order 💖.
6.4. Interleaved Mode ✨
Okay bestie, this mode kicks in when the OPTIONAL packetization-mode MIME parameter is equal to 2, or when external means turn it on 💅. Some receivers MAY support this mode 🫶. STAP-Bs, MTAPs, FU-As, and FU-Bs MAY be used 🦋. STAP-As and single NAL unit packets MUST NOT be used 🙅♀️. The transmission order of packets and NAL units is constrained as specified in section 5.5 🔥.
7. De-Packetization Process (Informative) 🧚
The de-packetization process is implementation dependent 🤷♀️. So the description here is just an example of a suitable implementation, and other schemes may be used as well 🌈. Optimizations relative to the described algorithms are likely possible ✨. Section 7.1 covers the single NAL unit and non-interleaved packetization modes, section 7.2 covers the interleaved mode, and section 7.3 has extra decapsulation guidelines for intelligent receivers 💜.
All normal RTP mechanisms related to buffer management apply 🧃. Specifically, duplicated or outdated RTP packets (as indicated by the RTP sequence number and the RTP timestamp) are removed 🗑️. To figure out the exact time for decoding, you have to factor in things like a possible intentional delay for proper inter-stream synchronization ⏰.
7.1. Single NAL Unit and Non-Interleaved Mode 🎀
The receiver has a receiver buffer to soak up transmission delay jitter 🧽. It stores incoming packets in reception order into that buffer 📥. Packets get decapsulated in RTP sequence number order 🔢. If a decapsulated packet is a single NAL unit packet, the NAL unit inside is passed straight to the decoder 🚀. If it's an STAP-A, the NAL units inside are passed to the decoder in the order they were encapsulated in the packet 📦. If it's an FU-A, all the fragments of the fragmented NAL unit get concatenated and passed to the decoder 🧩.
Informative note: If the decoder supports Arbitrary Slice Order, coded slices of a picture can be passed to the decoder in any order regardless of their reception and transmission order. 🌀
7.2. Interleaved Mode 🦋
The big idea behind these de-packetization rules is to reorder NAL units from transmission order back into NAL unit decoding order 🔄.
The receiver has a receiver buffer, which handles transmission delay jitter and reorders packets from transmission order into NAL unit decoding order 🧠. In this section, we assume there is no transmission delay jitter 🙈. To tell it apart from a practical receiver buffer that also deals with jitter, this section calls it the deinterleaving buffer 🏷️. Receivers SHOULD also prepare for transmission delay jitter 🫧. That means either reserving separate buffers for jitter buffering and deinterleaving buffering, or using one receiver buffer for both. Moreover, receivers SHOULD take transmission delay jitter into account in the buffering operation, e.g., by doing some extra initial buffering before decoding and playback start 🎬.
This section is organized as follows: subsection 7.2.1 shows how to calculate the size of the deinterleaving buffer 📏. Subsection 7.2.2 specifies the receiver process for organizing received NAL units into NAL unit decoding order 🗂️.
7.2.1. Size of the Deinterleaving Buffer 📏
When the SDP Offer/Answer model or any other capability exchange procedure is used in session setup, the properties of the received stream SHOULD be such that the receiver capabilities are not exceeded 🙏. In the SDP Offer/Answer model, the receiver can indicate its capabilities for allocating a deinterleaving buffer with the deint-buf-cap MIME parameter 📣. The sender indicates its requirement for the deinterleaving buffer size with the sprop-deint-buf-req MIME parameter 📨. It is therefore RECOMMENDED to set the deinterleaving buffer size, in terms of number of bytes, equal to or greater than the value of the sprop-deint-buf-req MIME parameter 💖. See section 8.1 for more on deint-buf-cap and sprop-deint-buf-req, and section 8.2.2 for how they're used in the SDP Offer/Answer model 📚.
When a declarative session description is used in session setup, the sprop-deint-buf-req MIME parameter signals the requirement for the deinterleaving buffer size 📜. It is therefore RECOMMENDED to set the deinterleaving buffer size, in terms of number of bytes, equal to or greater than the value of the sprop-deint-buf-req MIME parameter 💅.
7.2.2. Deinterleaving Process 🌸
There are two buffering states in the receiver: initial buffering and buffering while playing 🎭. Initial buffering happens when the RTP session is initialized 🌱. Once it's done, decoding and playback start, and the buffering-while-playing mode takes over 🎶.
No matter which buffering state it's in, the receiver stores incoming NAL units, in reception order, in the deinterleaving buffer 📥. NAL units of aggregation packets get stored individually 📦. The value of DON is calculated and stored for all NAL units 🔢.
The receiver operation is described below using the following functions and constants 🧮:
- Function AbsDON is specified in section 8.1.
- Function don_diff is specified in section 5.5.
- Constant N is the value of the OPTIONAL
sprop-interleaving-depthMIME type parameter (see section 8.1), incremented by 1.
Initial buffering lasts until one of the following conditions is fulfilled 🕰️:
- There are N VCL NAL units in the deinterleaving buffer.
- If
sprop-max-don-diffis present,don_diff(m,n)is greater than the value ofsprop-max-don-diff, where n corresponds to the NAL unit having the greatest value of AbsDON among the received NAL units and m corresponds to the NAL unit having the smallest value of AbsDON among the received NAL units. - Initial buffering has lasted for a duration equal to or greater than the value of the OPTIONAL
sprop-init-buf-timeMIME parameter.
The NAL units to be removed from the deinterleaving buffer are determined as follows 🗑️:
- If the deinterleaving buffer contains at least N VCL NAL units, NAL units are removed from it and passed to the decoder in the order specified below, until the buffer contains N-1 VCL NAL units.
- If
sprop-max-don-diffis present, all NAL units m for whichdon_diff(m,n)is greater thansprop-max-don-diffare removed from the deinterleaving buffer and passed to the decoder in the order specified below. Here, n corresponds to the NAL unit having the greatest value of AbsDON among the received NAL units.
The order in which NAL units are passed to the decoder works like this 🚦:
- Let PDON be a variable that is initialized to
0at the beginning of an RTP session. - For each NAL unit associated with a value of DON, a DON distance is calculated. If the DON of the NAL unit is larger than PDON, the DON distance is
DON - PDON. Otherwise, the DON distance is65535 - PDON + DON + 1🧮. - NAL units are delivered to the decoder in ascending order of DON distance. If several NAL units share the same DON distance, they can be passed to the decoder in any order 🤷♀️.
- When a desired number of NAL units have been passed to the decoder, PDON is set to the DON of the last NAL unit passed 🏁.
7.3. Additional De-Packetization Guidelines 💎
The following additional de-packetization rules may be used to implement an operational H.264 de-packetizer 🛠️:
- Intelligent RTP receivers (e.g., in gateways) may identify lost coded slice data partitions A (DPAs). If a lost DPA is found, a gateway may decide not to send the corresponding coded slice data partitions B and C, since their information is meaningless for H.264 decoders. This way, a MANE can cut network load by discarding useless packets without parsing a complex bitstream 🙌.
- Intelligent RTP receivers (e.g., in gateways) may identify lost FUs. If a lost FU is found, a gateway may decide not to send the following FUs of the same fragmented NAL unit, since their information is meaningless for H.264 decoders. Again, a MANE can reduce network load by discarding useless packets without parsing a complex bitstream 🧹.
- Intelligent receivers having to discard packets or NALUs should first discard all packets/NALUs in which the value of the NRI field of the NAL unit type octet is equal to 0. This minimizes the impact on user experience and keeps the reference pictures intact 🖼️. If more packets have to be discarded, then packets with a numerically lower NRI value should be discarded before packets with a numerically higher NRI value. However, discarding any packets with an NRI bigger than 0 very likely leads to decoder drift and SHOULD be avoided 😭.
8. Payload Format Parameters 🎀
This section specifies the parameters that MAY be used to select optional features of the payload format and certain features of the bitstream 🎛️. The parameters are specified here as part of the MIME subtype registration for the ITU-T H.264 | ISO/IEC 14496-10 codec. A mapping of the parameters into the Session Description Protocol (SDP) [5] is also provided for apps that use SDP 📡. Equivalent parameters could be defined elsewhere for use with control protocols that don't use MIME or SDP 🧩.
Some parameters give a receiver the properties of the stream that will be sent 📬. The names of all these parameters start with "sprop" for stream properties ✨. Some of the "sprop" parameters are limited by other payload or codec configuration parameters. For example, the sprop-parameter-sets parameter is constrained by the profile-level-id parameter 🔗. The media sender selects all "sprop" parameters, not the receiver 📤. This uncommon characteristic may not be compatible with some signaling protocol concepts, in which case the use of these parameters SHOULD be avoided 🙃.
8.1. MIME Registration 📝
The MIME subtype for the ITU-T H.264 | ISO/IEC 14496-10 codec is allocated from the IETF tree 🌳.
The receiver MUST ignore any unspecified parameter 🙅♀️.
Media Type name: video
Media subtype name: H264
Required parameters: none
OPTIONAL parameters:
profile-level-id:
A base16 [6] (hexadecimal) representation of
the following three bytes in the sequence
parameter set NAL unit specified in [1]: 1)
profile_idc, 2) a byte herein referred to as
profile-iop, composed of the values of
constraint_set0_flag, constraint_set1_flag,
constraint_set2_flag, and reserved_zero_5bits
in bit-significance order, starting from the
most significant bit, and 3) level_idc. Note
that reserved_zero_5bits is required to be
equal to 0 in [1], but other values for it may
be specified in the future by ITU-T or ISO/IEC.If the profile-level-id parameter is used to describe a NAL unit stream, it tells you the profile and level a decoder has to support in order to comply with [1] when it decodes that stream 🎀. The profile-iop byte then tells you whether the stream also obeys all the constraints of the profiles. If bit 7 (the most significant bit), bit 6, or bit 5 of profile-iop is equal to 1, then all constraints of the Baseline profile, the Main profile, or the Extended profile, respectively, are obeyed in that NAL unit stream 💖.
If profile-level-id is used for capability exchange or session setup, it's a different vibe ✨. It indicates the profile the codec supports and the highest level supported for that signaled profile. The profile-iop byte shows whether the codec has extra limitations, so that only the common subset of the algorithmic features and limitations of the profiles signaled with profile-iop and of the profile indicated by profile_idc is supported 🦋. For example, if a codec only supports the common subset of the coding tools of the Baseline profile and the Main profile at level 2.1 and below, the profile-level-id becomes 42E015. In there, 42 stands for the Baseline profile, E0 indicates that only the common subset for all profiles is supported, and 15 indicates level 2.1 💅.
Informative note: Capability exchange and session setup procedures should provide means to list the capabilities for each supported codec profile separately. For example, the one-of-N codec selection procedure of the SDP Offer/Answer model can be used (section 10.2 of [7]) 🌸.
If no profile-level-id is present, the Baseline Profile without additional constraints at Level 1 MUST be implied 🧚. So no ID just means Baseline, Level 1, no extras, periodt.
max-mbps, max-fs, max-cpb, max-dpb, and max-br: These parameters MAY be used to signal the capabilities of a receiver implementation 👑. They MUST NOT be used for any other purpose 🙈. The profile-level-id parameter MUST be present in the same receiver capability description that contains any of these parameters. The level conveyed in the value of profile-level-id MUST be such that the receiver is fully capable of supporting. max-mbps, max-fs, max-cpb, max-dpb, and max-br MAY be used to indicate capabilities of the receiver that extend the required capabilities of the signaled level, as specified below ⭐.
When more than one parameter from the set (max-mbps, max-fs, max-cpb, max-dpb, max-br) is present, the receiver MUST support all signaled capabilities simultaneously 🔥. For example, if both max-mbps and max-br are present, the signaled level with the extension of both the frame rate and bit rate is supported. That means the receiver is able to decode NAL unit streams in which the macroblock processing rate is up to max-mbps (inclusive), the bit rate is up to max-br (inclusive), the coded picture buffer size is derived as specified in the semantics of the max-br parameter below, and other properties comply with the level specified in the value of the profile-level-id parameter 🌈.
A receiver MUST NOT signal values of max-mbps, max-fs, max-cpb, max-dpb, and max-br that meet the requirements of a higher level, referred to as level A herein, compared to the level specified in the value of the profile-level-id parameter, if the receiver can support all the properties of level A 💀.
Informative note: When the OPTIONAL MIME type parameters are used to signal the properties of a NAL unit stream, max-mbps, max-fs, max-cpb, max-dpb, and max-br are not present, and the value of profile-level-id must always be such that the NAL unit stream complies fully with the specified profile and level 🦋.
max-mbps: The value of max-mbps is an integer indicating the maximum macroblock processing rate in units of macroblocks per second. The max-mbps parameter signals that the receiver is capable of decoding video at a higher rate than is required by the signaled level conveyed in the value of the profile-level-id parameter 😭. When max-mbps is signaled, the receiver MUST be able to decode NAL unit streams that conform to the signaled level, with the exception that the MaxMBPS value in Table A-1 of [1] for the signaled level is replaced with the value of max-mbps. The value of max-mbps MUST be greater than or equal to the value of MaxMBPS for the level given in Table A-1 of [1]. Senders MAY use this knowledge to send pictures of a given size at a higher picture rate than is indicated in the signaled level 🍓.
max-fs: The value of max-fs is an integer indicating the maximum frame size in units of macroblocks. The max-fs parameter signals that the receiver is capable of decoding larger picture sizes than are required by the signaled level conveyed in the value of the profile-level-id parameter 🌸. When max-fs is signaled, the receiver MUST be able to decode NAL unit streams that conform to the signaled level, with the exception that the MaxFS value in Table A-1 of [1] for the signaled level is replaced with the value of max-fs. The value of max-fs MUST be greater than or equal to the value of MaxFS for the level given in Table A-1 of [1]. Senders MAY use this knowledge to send larger pictures at a proportionally lower frame rate than is indicated in the signaled level 💜.
max-cpb: The value of max-cpb is an integer indicating the maximum coded picture buffer size in units of 1000 bits for the VCL HRD parameters (see A.3.1 item i of [1]) and in units of 1200 bits for the NAL HRD parameters (see A.3.1 item j of [1]). The max-cpb parameter signals that the receiver has more memory than the minimum amount of coded picture buffer memory required by the signaled level conveyed in the value of the profile-level-id parameter 🧁. When max-cpb is signaled, the receiver MUST be able to decode NAL unit streams that conform to the signaled level, with the exception that the MaxCPB value in Table A-1 of [1] for the signaled level is replaced with the value of max-cpb. The value of max-cpb MUST be greater than or equal to the value of MaxCPB for the level given in Table A-1 of [1]. Senders MAY use this knowledge to construct coded video streams with greater variation of bit rate than can be achieved with the MaxCPB value in Table A-1 of [1] 🎀.
Informative note: The coded picture buffer is used in the hypothetical reference decoder (Annex C) of H.264. The use of the hypothetical reference decoder is recommended in H.264 encoders to verify that the produced bitstream conforms to the standard and to control the output bitrate. Thus, the coded picture buffer is conceptually independent of any other potential buffers in the receiver, including de-interleaving and de-jitter buffers. The coded picture buffer need not be implemented in decoders as specified in Annex C of H.264, but rather standard-compliant decoders can have any buffering arrangements provided that they can decode standard-compliant bitstreams. Thus, in practice, the input buffer for video decoder can be integrated with de-interleaving and de-jitter buffers of the receiver 🫶.
bestie, okay so max-dpb is an integer for the max decoded picture buffer size, counted in units of 1024 bytes 💾✨ It basically says "hey, I've got more memory than the minimum the level in profile-level-id needs" 🫶 When you signal max-dpb, the receiver MUST still decode NAL unit streams that match the signaled level, except the MaxDPB value in Table A-1 of [1] for that level gets swapped out for your max-dpb number 💅 So a receiver that signals max-dpb MUST be able to store this many decoded frames, complementary field pairs, and non-paired fields in its DPB 👑
Min(1024 * max-dpb / ( PicWidthInMbs *
FrameHeightInMbs * 256 * ChromaFormatFactor ),
16)PicWidthInMbs, FrameHeightInMbs, and ChromaFormatFactor are all defined in [1] 🧚
Also, the value of max-dpb MUST be greater than or equal to the value of MaxDPB for the level given in Table A-1 of [1]. Senders MAY use this knowledge to build coded video streams with improved compression 🦋
Informative note: ngl this parameter was added mainly to match a similar codepoint in ITU-T Recommendation H.245, so gateway designs are easier to set up 🌸 The decoded picture buffer holds reconstructed samples and only belongs to the video decoder. There's no relationship between the DPB size and the buffers used in RTP, especially de-interleaving and de-jitter buffers 🙈
max-br is an integer for the maximum video bit rate, in units of 1000 bits per second for the VCL HRD parameters (see A.3.1 item i of [1]) and in units of 1200 bits per second for the NAL HRD parameters (see A.3.1 item j of [1]) 🍓
This parameter says the receiver's video decoder can handle a higher bit rate than the signaled level in profile-level-id requires 💜 The value of max-br MUST be greater than or equal to the value of MaxBR for the level given in Table A-1 of [1].
When max-br is signaled, the video codec of the receiver MUST be able to decode NAL unit streams that conform to the signaled level (from profile-level-id), with these exceptions to the limits of that level:
- The value of max-br replaces the MaxBR value of the signaled level (in Table A-1 of [1]).
- When the max-cpb parameter is not present, the result of the following formula replaces the value of MaxCPB in Table A-1 of [1]: (MaxCPB of the signaled level) * max-br / (MaxBR of the signaled level).
Here's an example, bestie 🎀 If a receiver signals capability for Level 1.2 with max-br equal to 1550, that means a max video bitrate of 1550 kbits/sec for VCL HRD parameters, a max video bitrate of 1860 kbits/sec for NAL HRD parameters, and a CPB size of 4036458 bits (1550000 / 384000 * 1000 * 1000) 😭
The value of max-br MUST be greater than or equal to the value MaxBR for the signaled level given in Table A-1 of [1].
Senders MAY use this knowledge to send higher bitrate video, as allowed in the level definitions of Annex A of H.264, to get better video quality ✨
Informative note: This one was also added mainly to match a codepoint in ITU-T Recommendation H.245, to make gateway signaling easier 🌈 Don't assume from this value that the network can actually handle those bit rates at any given time. You also can't conclude that the signaled bit rate is possible under congestion control constraints 💀
redundant-pic-cap signals what the receiver implementation can do 🧁 When it's equal to 0, the receiver makes no attempt to use redundant coded pictures to fix primary coded pictures that decoded incorrectly. That same 0 also means the receiver can't use redundant slices, so a sender SHOULD avoid sending them to save bandwidth 🙅♀️ When it's equal to 1, the receiver can decode any redundant slice that covers a corrupted area in a primary decoded picture (at least partly), so a sender MAY send redundant slices 🫧 When the parameter isn't present, the value MUST be 0 for redundant-pic-cap. When it is present, the value of redundant-pic-cap MUST be either 0 or 1.
When profile-level-id is in the same capability signaling as redundant-pic-cap, and the profile it indicates disallows redundant coded pictures (e.g., Main Profile), the value of redundant-pic-cap MUST be equal to 0. When a receiver indicates redundant-pic-cap equal to 0, the received stream SHOULD NOT contain redundant coded pictures 🚫
Informative note: Even when redundant-pic-cap is 0, the decoder can ignore redundant coded pictures as long as it supports a profile (Baseline, Extended) where redundant coded pictures are allowed 💅
Informative note: Even when redundant-pic-cap is 1, the receiver may also pick other error concealment strategies to replace or complement decoding of redundant slices 🦋
sprop-parameter-sets MAY be used to convey any sequence and picture parameter set NAL units (called the initial parameter set NAL units here) that MUST come before any other NAL units in decoding order 📦 This parameter MUST NOT be used to indicate codec capability in any capability exchange procedure. The value is the base64 [6] representation of the initial parameter set NAL units, as specified in sections 7.3.2.1 and 7.3.2.2 of [1] 🍬 The parameter sets are conveyed in decoding order, and there's no framing of the parameter set NAL units. A comma separates any pair of parameter sets in the list. Heads up: the number of bytes in a parameter set NAL unit is usually less than 10, but a picture parameter set NAL unit can contain several hundreds of bytes 🌷
Informative note: When several payload types are offered in the SDP Offer/Answer model, each with its own sprop-parameter-sets parameter, the receiver can't assume those parameter sets don't use conflicting storage locations (i.e., identical values of parameter set identifiers). So a receiver SHOULD double-buffer all sprop-parameter-sets and make them available to the decoder instance that decodes a certain payload type 🫶
parameter-add MAY be used to signal whether the receiver of this parameter is allowed to add parameter sets to its signaling response using the sprop-parameter-sets MIME parameter 🎟️ The value is either 0 or 1. 0 means false, so it's not allowed to add parameter sets. 1 means true, so it is allowed to add them. If the parameter isn't present, its value MUST be 1 ✨
packetization-mode signals the properties of an RTP payload type or the capabilities of a receiver implementation 🧚 Only a single configuration point can be indicated, so if capabilities to support more than one packetization-mode are declared, multiple configuration points (RTP payload types) MUST be used 💖
When packetization-mode is 0 or isn't there at all, the single NAL mode from section 6.2 of RFC 3984 MUST be used 💅. This mode shows up in standards that use ITU-T Recommendation H.241 [15] (see section 12.1) 🎀. When packetization-mode is 1, the non-interleaved mode from section 6.3 of RFC 3984 MUST be used ✨. When it's 2, the interleaved mode from section 6.4 of RFC 3984 MUST be used 🦋. The packetization mode value MUST be an integer from 0 to 2, inclusive 🫶.
sprop-interleaving-depth 🌈: This parameter MUST NOT be present when packetization-mode is missing or is 0 or 1 🙈. It MUST be present when packetization-mode is 2 🔥.
This parameter describes the properties of a NAL unit stream 💜. It gives the maximum number of VCL NAL units that come before any VCL NAL unit in the stream in transmission order and come after that VCL NAL unit in decoding order 🧚. So receivers can rebuild the NAL unit decoding order as long as their buffer for that job is at least the value of sprop-interleaving-depth + 1, counted in VCL NAL units 👑.
The value of sprop-interleaving-depth MUST be an integer from 0 to 32767, inclusive 💖.
sprop-deint-buf-req 🍓: This parameter MUST NOT be present when packetization-mode is missing or is 0 or 1. It MUST be present when packetization-mode is 2 ✨.
sprop-deint-buf-req tells you how big the deinterleaving buffer needs to be for the NAL unit stream 🦋. The value MUST be greater than or equal to the maximum buffer occupancy, in bytes, that such a deinterleaving buffer needs, as specified in section 7.2 of RFC 3984 💅. If the deinterleaving buffer is at least the sprop-deint-buf-req value in bytes, receivers can put interleaved NAL units back into NAL unit decoding order 🫶.
The value of sprop-deint-buf-req MUST be an integer from 0 to 4294967295, inclusive 🌸.
Informative note: sprop-deint-buf-req only covers the size of the deinterleaving buffer 🥺. When network jitter can happen, you also have to provision an appropriately sized jitter buffer 💀.
deint-buf-cap 🎀: This parameter tells you what a receiver can handle. It's the amount of deinterleaving buffer space, in bytes, that the receiver has free for rebuilding the NAL unit decoding order 🧁. A receiver can handle any stream whose sprop-deint-buf-req value is less than or equal to this parameter 💜.
If the parameter isn't there, a value of 0 MUST be used for deint-buf-cap 🙃. The value of deint-buf-cap MUST be an integer from 0 to 4294967295, inclusive 🍓.
Informative note: deint-buf-cap is only the largest possible deinterleaving buffer the receiver can have 🫠. When network jitter can happen, you still need an appropriately sized jitter buffer 😭.
sprop-init-buf-time ⭐: This parameter MAY be used to describe the properties of a NAL unit stream 🦋. It MUST NOT be present if packetization-mode is 0 or 1 🙅♀️.
This parameter gives the initial buffering time that a receiver MUST buffer before it starts decoding, so it can rebuild the NAL unit decoding order from transmission order 🌊. It's the maximum value of (transmission time of a NAL unit minus decoding time of that NAL unit). This assumes reliable, instantaneous transmission, the same timeline for transmission and decoding, and that decoding starts when the first packet arrives 🕰️.
Here's an example of how to figure out sprop-init-buf-time 🧠. A NAL unit stream is sent in this interleaved order, where each number is the decoding time, and transmission goes left to right:
0 2 1 3 5 4 6 8 7 ...If NAL units are sent at a steady rate, the transmission times are:
0 1 2 3 4 5 6 7 8 ...Now subtract the decoding time from the transmission time, column by column, and you get:
0 -1 1 0 -1 1 0 -1 1 ...So in terms of intervals of NAL unit transmission times, sprop-init-buf-time is 1 in this example 🎯.
This parameter is a non-negative base-10 integer, counted in clock ticks of a 90-kHz clock 🕐. If it isn't there, no initial buffering time value is defined 🫥. Otherwise, the value of sprop-init-buf-time MUST be an integer from 0 to 4294967295, inclusive 💅.
On top of the signaled sprop-init-buf-time, receivers SHOULD also account for transmission delay jitter buffering 🌸. That includes buffering for delay jitter caused by mixers, translators, gateways, proxies, traffic shapers, and other network elements 🛸.
sprop-max-don-diff 🔮: This parameter MAY be used to describe the properties of a NAL unit stream 🦋. It MUST NOT be used to signal transmitter, receiver, or codec capabilities 🚫. It MUST NOT be present if packetization-mode is 0 or 1 🙈. sprop-max-don-diff is an integer from 0 to 32767, inclusive 💖. If it isn't present, the value of the parameter is unspecified 🤷♀️.
sprop-max-don-diff is calculated like this:
sprop-max-don-diff = max{AbsDON(i) -
AbsDON(j)},
for any i and any j>i,Here, i and j are the indexes of NAL units in transmission order. AbsDON is the decoding order number of a NAL unit that doesn't wrap around to 0 after 65535 🌀. So for the very first NAL unit in transmission order (index 0), AbsDON(0) = DON(0). For every other NAL unit, take two consecutive NAL units in transmission order, m and n, and use these cases:
If DON(m) == DON(n), AbsDON(n) = AbsDON(m)
If (DON(m) < DON(n) and DON(n) - DON(m) <
32768),
AbsDON(n) = AbsDON(m) + DON(n) - DON(m)
If (DON(m) > DON(n) and DON(m) - DON(n) >=
32768),
AbsDON(n) = AbsDON(m) + 65536 - DON(m) + DON(n)
If (DON(m) < DON(n) and DON(n) - DON(m) >=
32768),
AbsDON(n) = AbsDON(m) - (DON(m) + 65536 -
DON(n))
If (DON(m) > DON(n) and DON(m) - DON(n) <
32768),
AbsDON(n) = AbsDON(m) - (DON(m) - DON(n))Here, DON(i) is the decoding order number of the NAL unit at index i in transmission order. The decoding order number is specified in section 5.5 of RFC 3984 🧾.
Informative note: receivers may use sprop-max-don-diff to decide which NAL units in the receiver buffer can be passed to the decoder 🎀.
max-rcmd-nalu-size 🌷: This parameter MAY be used to signal a receiver's capabilities 💜. It MUST NOT be used for any other purpose 🙅♀️. The value gives the largest NALU size, in bytes, that the receiver can handle efficiently 🫧. It's a recommendation, not a strict upper boundary 💅. The sender MAY create larger NALUs, but must be aware that handling them may cost more than handling NALUs that stay within the limit 💀.
The max-rcmd-nalu-size value MUST be an integer from 0 to 4294967295, inclusive 🎀. If this parameter isn't specified, there's no known limit on NALU size ✨. Senders still have to think about the MTU size between sender and receiver, and they SHOULD run MTU discovery for that 🦋.
This parameter exists for stuff like an IP to H.223 video telephony gateway, where NALUs smaller than the H.223 transport data unit are more efficient 🌸. A gateway may terminate IP, so MTU discovery normally won't work beyond the gateway 🙈.
Informative note: setting this parameter to a lower than necessary value may have a negative impact. 💀
Encoding considerations:
This type is only defined for transfer via RTP
(RFC 3550).
A file format of H.264/AVC video is defined in
[29]. This definition is utilized by other
file formats, such as the 3GPP multimedia file
format (MIME type video/3gpp) [30] or the MP4
file format (MIME type video/mp4).
Security considerations:
See section 9 of RFC 3984.
Public specification:
Please refer to RFC 3984 and its section 15.
Additional information:
None
File extensions: none
Macintosh file type code: none
Object identifier or OID: none
Person & email address to contact for further information:
stewe@stewe.org
Intended usage: COMMON
Author:
stewe@stewe.org
Change controller:
IETF Audio/Video Transport working group
delegated from the IESG.8.2. SDP Parameters
8.2.1. Mapping of MIME Parameters to SDP
The MIME media type video/H264 string gets mapped to fields in the Session Description Protocol (SDP) [5] like this 💜:
- The media name in the "m=" line of SDP MUST be video.
- The encoding name in the "a=rtpmap" line of SDP MUST be H264 (the MIME subtype).
- The clock rate in the "a=rtpmap" line MUST be 90000.
- The OPTIONAL parameters "profile-level-id", "max-mbps", "max-fs", "max-cpb", "max-dpb", "max-br", "redundant-pic-cap", "sprop-parameter-sets", "parameter-add", "packetization-mode", "sprop-interleaving-depth", "deint-buf-cap", "sprop-deint-buf-req", "sprop-init-buf-time", "sprop-max-don-diff", and "max-rcmd-nalu-size", when present, MUST be included in the "a=fmtp" line of SDP. These parameters are expressed as a MIME media type string, in the form of a semicolon separated list of parameter=value pairs ✨.
Here's an example of media representation in SDP 🧚 (Baseline Profile, Level 3.0, some of the constraints of the Main profile may not be obeyed):
m=video 49170 RTP/AVP 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A01E;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==8.2.2. Usage with the SDP Offer/Answer Model ✨
okay so, when H.264 gets offered over RTP using SDP in an Offer/Answer model [7] for unicast usage, there are some limits and rules you need to know 🫶
- The three must-match parameters 💖 are
profile-level-id,packetization-mode, and, ifpacketization-modecalls for it,sprop-deint-buf-req. These three MUST be used symmetrically, i.e., the answerer MUST either maintain all configuration parameters or remove the media format (payload type) completely, if one or more of the parameter values are not supported.
Informative note: The requirement for symmetric use applies only to the above three parameters, not to the other stream properties and capability parameters.
To keep things simple, the same RTP payload type number used in the offer SHOULD also be used in the answer, as specified in [7]. An answer MUST NOT contain a payload type number used in the offer unless the configuration (profile-level-id, packetization-mode, and, if present, sprop-deint-buf-req) is the same as in the offer.
Informative note: When the offerer gets the answer, it has to compare any payload types it didn't declare in the offer, based on media type (i.e., video/h264) and the above three parameters, against the payload types it has already declared. That's how it figures out whether a configuration is new or equivalent to one already offered 🔍
- The stream description parameters 🌸 are
sprop-parameter-sets,sprop-deint-buf-req,sprop-interleaving-depth,sprop-max-don-diff, andsprop-init-buf-time. They describe the NAL unit stream that the offerer or answerer is sending for this media format configuration. This is different from the usual Offer/Answer parameters, which normally describe what the offerer or answerer is able to receive. With H.264, the offerer assumes the answerer will be able to receive media encoded using the configuration being offered.
Informative note: The above parameters apply to any stream sent by the declaring entity with the same configuration. In other words, they depend on their source. Rather than being bound to the payload type, the values may have to be applied to another payload type when being sent, since they apply to the configuration.
- The capability parameters 🦋 are
max-mbps,max-fs,max-cpb,max-dpb,max-br,"redundant-pic-cap","max-rcmd-nalu-size", and they MAY be used to declare further capabilities. How they're read depends on the direction attribute. When the direction attribute issendonly, the parameters describe the limits of the RTP packets and the NAL unit stream that the sender is capable of producing. When the direction attribute issendrecvorrecvonly, the parameters describe the limitations of what the receiver accepts. - Deinterleaving buffer info 🧺 As specified above, an offerer has to include the size of the deinterleaving buffer in the offer for an interleaved H.264 stream. So that the offerer and answerer can tell each other about their deinterleaving buffering capabilities, both parties are RECOMMENDED to include
deint-buf-cap. This information MAY be used when the value forsprop-deint-buf-reqis selected in a second round of offer and answer. For interleaved streams, it is also RECOMMENDED to consider offering multiple payload types with different buffering requirements when the capabilities of the receiver are unknown. - The parameter sets rules 🍓 The
sprop-parameter-setsparameter works as described above. On top of that, an answerer MUST maintain all parameter sets received in the offer in its answer. What happens next depends on the value ofparameter-add. Ifparameter-addis false (0), the answer MUST NOT add any additional parameter sets. Ifparameter-addis true (1), the answerer, in its answer, MAY add additional parameter sets to thesprop-parameter-setsparameter. No matter whatparameter-addsays, the answerer MUST also accept receiving a video stream using thesprop-parameter-setsit declared in the answer.
Informative note: Be careful when adding parameter sets. Using conflicting parameter set identifiers could overwrite parameter sets that were already transmitted 😬
For streams delivered over multicast, these extra rules apply too 📡
- The stream properties parameters (
sprop-parameter-sets,sprop-deint-buf-req,sprop-interleaving-depth,sprop-max-don-diff, andsprop-init-buf-time) MUST NOT be changed by the answerer. So a payload type can either be accepted unaltered or removed. - The receiver capability parameters
max-mbps,max-fs,max-cpb,max-dpb,max-br, andmax-rcmd-nalu-sizeMUST be supported by the answerer for all streams declared assendrecvorrecvonly. Otherwise, one of the following actions MUST be performed: the media format is removed, or the session rejected. - The receiver capability parameter
redundant-pic-capSHOULD be supported by the answerer for all streams declared assendrecvorrecvonly, as follows. The answerer SHOULD NOT include redundant coded pictures in the transmitted stream if the offerer indicatedredundant-pic-capequal to 0. Otherwise (whenredundant_pic_capis equal to 1), it is beyond the scope of this memo to recommend how the answerer should use redundant coded pictures.
Here are the complete lists of how each parameter should be interpreted for each combo of offer or answer and direction attribute 💅
- Offers and answers with
a=sendrecv, no direction attribute, ora=recvonly💜 The following interpretation of the parameters MUST be used.
Declaring the actual configuration or properties for receiving:
profile-level-idpacketization-mode
Declaring the actual properties of the stream to be sent (only applicable when a=sendrecv or no direction attribute is used):
sprop-deint-buf-reqsprop-interleaving-depthsprop-parameter-setssprop-max-don-diffsprop-init-buf-time
Declaring receiver implementation capabilities:
max-mbpsmax-fsmax-cpbmax-dpbmax-brredundant-pic-capdeint-buf-capmax-rcmd-nalu-size
Declaring how Offer/Answer negotiation shall be performed:
parameter-add
- Offers and answers with
a=sendonly🌈 For a media stream with the direction attributea=sendonly, the following interpretation of the parameters MUST be used.
Declaring the actual configuration and properties of the stream proposed to be sent:
profile-level-idpacketization-modesprop-deint-buf-reqsprop-max-don-diffsprop-init-buf-timesprop-parameter-setssprop-interleaving-depth
Declaring the capabilities of the sender when it receives a stream:
max-mbpsmax-fsmax-cpbmax-dpbmax-brredundant-pic-capdeint-buf-capmax-rcmd-nalu-size
Declaring how Offer/Answer negotiation shall be performed:
parameter-add
And a few more considerations, bestie 🎀
- Parameters used for declaring receiver capabilities are generally downgradable. They express the upper limit for a sender's possible behavior. So a sender MAY choose to set its encoder using only lower or equal values of these parameters.
sprop-parameter-setsMUST NOT be used in a sender's declaration of its capabilities, because the limits of the values carried inside the parameter sets are implicit in the profile and level used. - Parameters declaring a configuration point are not downgradable, except for the level part of the
profile-level-idparameter. They express values a receiver expects to be used, and they must be used verbatim on the sender side. - When a sender's capabilities are declared and non-downgradable parameters are used in that declaration, those parameters express a configuration that is acceptable. To get high interoperability, it is often advisable to offer multiple alternative configurations, e.g., for the packetization mode. It is impossible to offer multiple configurations in a single payload type. So when multiple configuration offers are made, each offer requires its own RTP payload type associated with it.
- A receiver SHOULD understand all MIME parameters, even if it only supports a subset of the payload format's functionality. This way, the receiver can tell when an offer to receive media can be downgraded to what it supports ✨
- An answerer MAY extend the offer with additional media format configurations. To actually use them, though, in most cases a second offer is needed from the offerer to provide the stream properties parameters the media sender will use. This also means the offerer has to be able to receive this media format configuration, not just send it.
- If an offerer wants non-symmetric capabilities between sending and receiving, the offerer has to offer different RTP sessions, i.e., different media lines declared as
recvonlyandsendonly, respectively. This may have further implications for the system 🧚
8.2.3. Usage in Declarative Session Descriptions
When H.264 over RTP is offered with SDP in a declarative style, as in RTSP [27] or SAP [28], the following considerations are necessary 🌟
- All parameters that can indicate the properties of both a NAL unit stream and a receiver are used to indicate the properties of a NAL unit stream. For example, in this case the parameter
profile-level-iddeclares the values used by the stream, instead of the capabilities of the sender. As a result, the following interpretation of the parameters MUST be used.
Declaring actual configuration or properties:
profile-level-idsprop-parameter-setspacketization-modesprop-interleaving-depthsprop-deint-buf-reqsprop-max-don-diffsprop-init-buf-time
Not usable:
max-mbpsmax-fsmax-cpbmax-dpbmax-brredundant-pic-capmax-rcmd-nalu-sizeparameter-adddeint-buf-cap
- A receiver of the SDP is required to support all parameters and values of the parameters provided. Otherwise, the receiver MUST reject (RTSP) or not participate in (SAP) the session. It falls on the creator of the session to use values that are expected to be supported by the receiving application 💖
8.3. Examples ✨
okay bestie, this one is a SIP Offer/Answer exchange where both parties are expected to both send AND receive 💖 Only the media codec specific parts of the SDP are shown, and some lines are wrapped because of text constraints, so don't panic 🙈
Offerer -> Answer SDP message:
m=video 49170 RTP/AVP 100 99 98
a=rtpmap:98 H264/90000
a=fmtp:98 profile-level-id=42A01E; packetization-mode=0;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==
a=rtpmap:99 H264/90000
a=fmtp:99 profile-level-id=42A01E; packetization-mode=1;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==
a=rtpmap:100 H264/90000
a=fmtp:100 profile-level-id=42A01E; packetization-mode=2;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==;
sprop-interleaving-depth=45; sprop-deint-buf-req=64000;
sprop-init-buf-time=102478; deint-buf-cap=128000So the offer is basically serving the same codec config in three different packetization formats 🎀 PT 98 is single NALU mode, PT 99 is non-interleaved mode, and PT 100 is the interleaved mode. For PT 100, the offerer also includes the interleaving parameters it would use if the answer says yes to PT 100 🦋 In all three cases, sprop-parameter-sets carries the initial parameter sets the answerer needs when it's receiving a stream from the offerer, as long as that config (profile-level-id and packetization mode) gets accepted. The value is identical in this example, but it could be different for each payload type, so don't assume it's the same 👀
Answerer -> Offerer SDP message:
m=video 49170 RTP/AVP 100 99 97
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42A01E; packetization-mode=0;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==,As0DEWlsIOp==,
KyzFGleR
a=rtpmap:99 H264/90000
a=fmtp:99 profile-level-id=42A01E; packetization-mode=1;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==,As0DEWlsIOp==,
KyzFGleR; max-rcmd-nalu-size=3980
a=rtpmap:100 H264/90000
a=fmtp:100 profile-level-id=42A01E; packetization-mode=2;
sprop-parameter-sets=Z0IACpZTBYmI,aMljiA==,As0DEWlsIOp==,
KyzFGleR; sprop-interleaving-depth=60;
sprop-deint-buf-req=86000; sprop-init-buf-time=156320;
deint-buf-cap=128000; max-rcmd-nalu-size=3980Here's the tea ☕ Offer/Answer negotiation covers both the sending and receiving streams, so an offer shows the exact parameters the offerer is willing to receive, and the answer shows the same thing for what the answerer accepts to receive 💅 In this case the offerer said it's willing to receive payload type 98, and the answerer accepts that by declaring an equivalent payload type 97. That's because the values for "profile-level-id", packetization-mode, and "sprop-deint-buf-req" are identical 🫶
This has implications for both sides about the parameters that declare properties. The offerer first declared a certain value of "sprop-parameter-sets" in the payload definition for PT=98. But since the answerer accepted it as PT=97, the values of "sprop-parameter-sets" in PT=98 are the ones that now have to be used when the offerer sends PT=97 😭 Same vibe in reverse: when the answerer sends PT=98 to the offerer, it has to use the properties it declared in PT=97 🔄
The answerer also accepts reception of the two configurations that payload types 99 and 100 represent. It provides the initial parameter sets for the answerer-to-offerer direction, plus the buffering related parameters it'll use to send those payload types. It also tells the offerer its memory limit for deinterleaving operations through the "deint-buf-cap" parameter 🧠 That's only really useful if the offerer decides to make a second offer, where it can take the new value into account. The "max-rcmd-nalu-size" value says the answerer can efficiently process NALUs up to 3980 bytes. But there's no guarantee the network supports that size, so don't treat it as a promise 🙃
Please note that the parameter sets in the above example do not represent a legal operation point of an H.264 codec. The base64 strings are only used for illustration. 🚨
8.4. Parameter Set Considerations 🌈
The H.264 parameter sets are a fundamental part of the video codec and vital to its operation, see section 1.2 💜 Because of how they work and how important they are for decoding, lost or erroneously transmitted parameter sets can hardly be concealed locally at the receiver 💀 A reference to a corrupt parameter set normally has fatal results for the decoding process. Corruption could happen from erroneous transmission or loss of a parameter set data structure, but also from the untimely transmission of a parameter set update. So the following recommendations are given as a guideline for the implementer of the RTP sender 🫶
Parameter set NALUs can be transported using three different principles:
A. Using a session control protocol (out-of-band) prior to the actual RTP session.
B. Using a session control protocol (out-of-band) during an ongoing RTP session.
C. Within the RTP stream in the payload (in-band) during an ongoing RTP session.
Principles A and B have to be implemented within a session control protocol. SIP and SDP can be used as described in the SDP Offer/Answer model and in the previous sections of this memo. This section has guidelines on how principles A and B must be implemented within session control protocols, and it doesn't depend on which particular protocol you use ✨ Principle C is supported by the RTP payload format defined in this specification 🦋
The picture and sequence parameter set NALUs SHOULD NOT be transmitted in the RTP payload unless reliable transport is provided for RTP, because losing a parameter set of either type will likely stop a big chunk of the corresponding RTP stream from decoding 😱 So sending parameter sets through a reliable session control protocol (basically principle A or B above) is RECOMMENDED 💅
For the rest of the section, it's assumed that out-of-band signaling gives reliable transport of parameter set NALUs and that in-band transport doesn't. If you do use in-band signaling for parameter sets, the sender SHOULD take the error characteristics into account and use mechanisms that give a high probability of delivering the parameter sets correctly 🔥 Ways to boost the chance of correct reception include packet repetition, FEC, and retransmission. An unreliable, out-of-band control protocol has similar disadvantages to in-band signaling (possible loss) and, on top of that, may also cause synchronization difficulties (see below). Therefore, it is NOT RECOMMENDED 🙅♀️
Parameter sets MAY be added or updated during the lifetime of a session using principles B and C. It is required that parameter sets are present at the decoder prior to the NAL units that refer to them. Updating or adding parameter sets can cause further problems, so the following recommendations should be considered 🧐
- When parameter sets are added or updated, principle C is vulnerable to transmission errors as described above, so principle B is RECOMMENDED 💖
- When parameter sets are added or updated, care SHOULD be taken to make sure any parameter set is delivered before it's used. It's common for there to be no synchronization between out-of-band signaling and in-band traffic. If out-of-band signaling is used, it is RECOMMENDED that a sender does not start sending NALUs that require the updated parameter sets until it gets acknowledgement of delivery from the signaling protocol 🎀
- When parameter sets are updated, the following synchronization issue should be taken into account. When overwriting a parameter set at the receiver, the sender has to make sure that parameter set isn't needed by any NALU still in the network or in receiver buffers. Otherwise, decoding with a wrong parameter set may happen 😬 To lessen this problem, it is RECOMMENDED either to overwrite only those parameter sets that haven't been used for a sufficiently long time (so you know all the related NALUs have been consumed), or to add a new parameter set instead (which may have negative consequences for the efficiency of the video coding) 🧠
- When new parameter sets are added, previously unused parameter set identifiers are used. This avoids the problem from the previous point 🙌 However, in a multiparty session, unless a synchronized control protocol is used, there's a risk that multiple entities try to add different parameter sets for the same identifier, and that has to be avoided 🚫
- Adding or modifying parameter sets by using both principles B and C in the same RTP session may lead to inconsistencies in the parameter sets, because the control and RTP channels aren't synchronized. Therefore, principles B and C MUST NOT both be used in the same session unless sufficient synchronization can be provided 🙅♀️
In some scenarios (for example, when only the subset of this payload format specification that corresponds to H.241 is used), it isn't possible to use out-of-band parameter set transmission. In that case, parameter sets have to be sent in-band 🍓 Here the synchronization with the non-parameter-set data in the bitstream is implicit, but the possibility of loss still has to be taken into account. The loss probability should be reduced using the mechanisms discussed above 🛡️
- When parameter sets are initially provided using principle A and then later added or updated in-band (principle C), there's a risk tied to updating the parameter sets that were delivered out-of-band. If receivers miss some in-band updates (for example, because of a loss or a late tune-in), they'll try to decode the bitstream with out-dated parameters 😵 It is RECOMMENDED that parameter set IDs be partitioned between the out-of-band and in-band parameter sets 🧩
To allow for maximum flexibility and the best performance from the H.264 coder, it is recommended, if possible, to let any sender add its own parameter sets to be used in a session 👑 Setting the "parameter-add" parameter to false should only be done in cases where the session topology stops a participant from adding its own parameter sets 🥺
9. Security Considerations ✨
okay bestie, this section is SERIOUS but also iconic 💅 RTP packets using this payload format are subject to the security considerations in the RTP specification [4], and in any appropriate RTP profile (for example, [16]) 🎀. That basically means confidentiality of the media streams is achieved by encryption, for example through SRTP [26] 🔐. And because the data compression is applied end-to-end, any encryption needs to be performed after compression 🫶.
Here's the scary-but-real part 😭 A potential denial-of-service threat exists for data encodings using compression techniques that have non-uniform receiver-end computational load. The attacker can inject pathological datagrams into the stream that are complex to decode and overload the receiver 💀. H.264 is particularly vulnerable, because it's extremely simple to generate datagrams containing NAL units that affect the decoding process of many future NAL units 🙈. So the usage of data origin authentication and data integrity protection of at least the RTP packet is RECOMMENDED, for example with SRTP [26] 👑.
Note that the right mechanism for confidentiality and integrity of RTP packets and their payloads is very dependent on the application and on the transport and signaling protocols employed. So although SRTP is the example above, other choices exist 🌸.
Decoders MUST exercise caution with how they handle user data SEI messages, especially if those contain active elements, and they MUST restrict their domain of applicability to the presentation containing the stream 🚨.
Last but not least, end-to-end security with authentication, integrity or confidentiality protection will stop a MANE from doing media-aware operations other than discarding complete packets. With confidentiality protection, it even stops media-aware packet discarding 😭. So for any MANE to do its job, it has to be a trusted entity included in the security context establishment 🧚.
10. Congestion Control 🦋
Congestion control for RTP SHALL be used in accordance with RFC 3550 [4], and with any applicable RTP profile, e.g., RFC 3551 [16] 🌈. There's one extra requirement if best-effort service is being used: users of this payload format MUST monitor packet loss to make sure the loss rate stays within acceptable parameters 👀. Packet loss counts as acceptable if a TCP flow across the same network path, under the same network conditions, would achieve an average throughput (measured on a reasonable timescale) that is not less than what the RTP flow is achieving 💜.
This condition can be satisfied by implementing congestion control mechanisms that adapt the transmission rate (or the number of layers subscribed to in a layered multicast session), or by having a receiver leave the session if the loss rate gets unacceptably high 🫶.
Bit rate adaptation for congestion control is easy when real-time encoding is used ✨. With pre-encoded content, though, adapting bandwidth needs more than one coded representation of the same content at different bit rates, or non-reference pictures or sub-sequences [22] in the bitstream 🍓. Switching between representations can normally happen in the same RTP session, e.g. with SI/SP slices of the Extended Profile, or by switching streams at IDR picture boundaries 🌸. Only when non-downgradable parameters (such as the profile part of the profile/level ID) need to change does the media stream have to be terminated and restarted, which may be done by using a different RTP payload type 🔥.
MANEs MAY follow the suggestions in Section 7.3 and remove certain unusable packets from the packet stream when that stream was damaged by earlier packet losses 🧚. This can cut network load in certain special cases 💅.
11. IANA Consideration 📚
IANA has registered one new MIME type; see Section 8.1 🎀.
12. Informative Appendix: Application Examples 🌈
This payload specification is super flexible, so it can cover the massive application space anticipated for H.264 🦋. But that flexibility also makes it hard for an implementer to pick a reasonable packetization scheme 😭. Some info on applying this spec to real-world scenarios is likely to show up in academic publications and a test model software and description in the near future, but a few preliminary usage scenarios are described here too ✨.
12.1. Video Telephony according to ITU-T Recommendation H.241 Annex A 📞
H.323-based video telephony systems that use H.264 as an optional video compression scheme are REQUIRED to support H.241 Annex A [15] as a packetization scheme 💜. The packetization mechanism in this Annex is technically identical with a small subset of this specification 🫶.
When a system operates according to H.241 Annex A, parameter set NAL units are sent in-band 🔥. Only Single NAL unit packets are used 🍓. Lots of these systems don't send IDR pictures regularly; they only send them when user interaction or control protocol means call for it, e.g. when switching video channels in a Multipoint Control Unit, or for error recovery requested by feedback 🙈.
12.2. Video Telephony, No Slice Data Partitioning, No NAL Unit Aggregation 🎀
The RTP part of this scheme is implemented and tested (though not the control-protocol part; see below) ✅.
In most real-world video telephony apps, picture parameters like picture size or optional modes never change during the life of a connection 🌸. So all necessary parameter sets (usually just one) are sent as a side effect of the capability exchange/announcement process, e.g. according to the SDP syntax in Section 8.2 of this document 👑. Since all the necessary parameter set info is established before the RTP session starts, there's no need to send any parameter set NAL units 🧚. Slice data partitioning isn't used either, so the RTP packet stream basically consists of NAL units that carry single coded slices 🦋.
The encoder picks the size of coded slice NAL units so they offer the best performance 💅. Often this means adapting the coded slice size to the MTU size of the IP network 🌈. For small picture sizes, this may end up as a one-picture-per-one-packet strategy 🍓. Intra refresh algorithms clean up after packet loss and the drift-related artifacts it causes ✨.
12.3. Video Telephony, Interleaved Packetization Using NAL Unit Aggregation 🧚
This scheme allows better error concealment and is used in H.263 based designs using RFC 2429 packetization [10]. It has been implemented, and good results were reported [12] 💖.
The VCL encoder codes the source picture so that all macroblocks (MBs) of one MB line are assigned to one slice 🌸. All slices with even MB row addresses are combined into one STAP, and all slices with odd MB row addresses go into another. Those STAPs are transmitted as RTP packets 🫶. The parameter sets are established as discussed above 🔥.
Note that STAPs are essential here, because the high number of individual slices (18 for a CIF picture) would lead to unacceptably high IP/UDP/RTP header overhead 😭 (unless the source coding tool FMO is used, which isn't assumed in this scenario). Also, some wireless video transmission systems, such as H.324M and the IP-based video telephony specified in 3GPP, are likely to use a relatively small transport packet size. For example, a typical MTU size of H.223 AL3 SDU is around 100 bytes [17] 📱. Coding individual slices this way gives further advantage in communication between wired and wireless networks, since individual slices are likely to be smaller than the preferred maximum packet size of wireless systems 🦋. As a result, a gateway can convert the STAPs used in a wired network into several RTP packets with only one NAL unit (which are preferred in a wireless network), and vice versa 👑.
12.4. Video Telephony with Data Partitioning 💜
This scheme has been implemented and has been shown to offer good performance, especially at higher packet loss rates [12] 🎀.
Data Partitioning is only known to be useful when some form of unequal error protection is available 🛡️. Normally, in single-session RTP environments, error characteristics are assumed even, i.e. the packet loss probability of all packets in the session is statistically the same 😴. However, there are means to reduce the packet loss probability of individual packets in an RTP session. A FEC packet according to RFC 2733 [18], for example, specifies which media packets are associated with the FEC packet 🌈.
In all cases the incurred overhead is substantial, but it's in the same order of magnitude as the number of bits otherwise spent on intra information. However, this mechanism doesn't add any delay to the system ⭐.
Again, the complete parameter set establishment is performed through control protocol means 🫶.
12.5. Video Telephony or Streaming with FUs and Forward Error Correction 🦋
okay bestie, this scheme has actually been implemented, and it showed really good performance, especially when packet loss rates get higher 💖 [19]
when retransmissions aren't on the table, the most efficient way to fight packet losses is forward error correction (FEC) 🛡️ ngl it's giving main character energy. Application layer, end-to-end FEC is often less efficient than FEC-based protection of individual links (especially when links with different characteristics are in the transmission path). But it's unavoidable in some scenarios, so it's still a whole thing 💅 RFC 2733 [18] provides ways to use generic, application layer, end-to-end FEC in packet-loss environments. The code is made by applying the XOR operation to the bits at the same bit position in different packets ✨ It's specified by the parameters (n,k), where k is the number of information packets used in the connection and n is the total number of packets generated for those k information packets. So i.e., n-k parity packets get generated for k information packets 🧮
once you're using a code with parameters (n,k) inside the RFC 2733 framework, these properties are well known (iykyk 🙈):
- a) If it's applied over one RTP packet, RFC 2733 only provides packet repetition.
- b) RFC 2733 is most bit rate efficient if XOR-connected packets have equal length.
- c) At the same packet loss probability p and for a fixed k, the greater n is, the smaller the residual error probability becomes. For example, for a packet loss probability of 10%, k=1, and n=2, the residual error probability is about 1%, whereas for n=3, the residual error probability is about 0.1% 🤯
- d) At the same packet loss probability p and for a fixed code rate k/n, the greater n is, the smaller the residual error probability becomes. For example, at a packet loss probability of p=10%, k=1 and n=2, the residual error rate is about 1%, whereas for an extended Golay code with k=12 and n=24, the residual error rate is about 0.01% 😭🔥
for applying RFC 2733 with H.264 baseline coded video without using FUs, there are several options to think about 🌸
- The video encoder produces NAL units where each video frame is coded in a single slice. Applying FEC, you could use a simple code, e.g., (n=2, k=1). So each NAL unit would basically just be repeated. The downside is the bad code performance according to d) above, plus the low flexibility, since only (n, k=1) codes can be used 💀
- The video encoder produces NAL units where each video frame is encoded in one or more consecutive slices. Applying FEC, you could use a better code, e.g., (n=24, k=12), over a sequence of NAL units. Depending on the number of RTP packets per frame, a loss may introduce a significant delay, and that delay gets reduced when more RTP packets are used per frame. Packets of completely different length might also get connected, which decreases bit rate efficiency according to b) above. However, with some care and for slices of 1kb or larger, similar lengths (100-200 bytes difference) can be produced, and that won't lower the bit efficiency catastrophically 🧁
- The video encoder produces NAL units where a certain frame contains k slices of possibly almost equal length. Then, applying FEC, a better code, e.g., (n=24, k=12), can be used over the sequence of NAL units for each frame. The delay compared to option 2) above may be reduced, but there are several obvious disadvantages 🙃 First, the coding efficiency of the encoded video is lowered significantly, since slice-structured coding reduces intra-frame prediction and adds extra slice overhead. Second, pre-encoded content (or, when operating over a gateway, the video) is usually not appropriately coded with k slices so FEC can be applied. Finally, encoding video that produces k slices of equal length is not straightforward and might need more than one encoding pass 😤
Many of these disadvantages can be avoided by applying FUs in combination with FEC 👑 Each NAL unit can be split into any number of FUs of basically equal length. That means FEC with a reasonable k and n can be applied even if the encoder made no effort to produce slices of equal length. For example, a coded slice NAL unit containing an entire frame can be split into k FUs, and a parity check code (n=k+1, k) can be applied. But here's the tea ☕ unless all the created fragments can be recovered, the whole slice is lost. So a larger section gets lost than would be lost if the frame had been split into several slices.
The presented technique makes it possible to achieve good transmission error tolerance, even if no additional source coding layer redundancy (such as periodic intra frames) is present. So the same coded video sequence can be used to get the maximum compression efficiency and quality over error-free transmission and over error-prone networks 🦋 It also lets FEC get applied to pre-encoded sequences without adding delay. Pre-encoded sequences that weren't encoded for error-prone networks can still be transmitted almost reliably without adding extensive delays. And FUs of equal length result in a bit rate efficient use of RFC 2733 ✨
if the error probability depends on the length of the transmitted packet (e.g., in case of mobile transmission [14]), the benefits of applying FUs with FEC are even more obvious 📱 Basically, the flexibility in the size of FUs lets appropriate FEC be applied for each NAL unit, which gives unequal error protection of NAL units. Slay 💅
When FUs and FEC are used, the incurred overhead is substantial, but it's in the same order of magnitude as the number of bits that have to be spent on intra-coded macroblocks if no FEC is applied. In [19], it was shown that the FEC-based approach enhanced the overall performance (quality) at the same error rate and same overall bit rate, including the overhead 🏆
12.6. Low Bit-Rate Streaming 🍓
this scheme has been implemented with H.263 and non-standard RTP packetization and it gave good results [20]. There's no technical reason why similarly good results couldn't be achievable with H.264 🧚
in today's Internet streaming, some of the offered bit rates are pretty low so terminals with dial-up modems can still access the content 📞 In wired IP networks, relatively large packets (say 500 - 1500 bytes) are preferred over smaller, more frequently occurring packets, in order to reduce network congestion. Plus, using large packets decreases the amount of RTP/UDP/IP header overhead 💜 For low bit-rate video, though, using large packets means that sometimes up to a few pictures should be encapsulated in one packet.
but here's the catch 😬 losing a packet that includes many coded pictures would have drastic consequences for visual quality, since there's practically no other way to conceal the loss of an entire picture than to repeat the previous one. One way to construct relatively large packets and keep the possibilities for successful loss concealment is to construct MTAPs that contain interleaved slices from several pictures ✨ An MTAP should not contain spatially adjacent slices from the same picture or spatially overlapping slices from any picture. If a packet is lost, a lost slice is likely to be surrounded by spatially adjacent slices of the same picture, plus spatially corresponding slices of the temporally previous and succeeding pictures. So concealment of the lost slice is likely to be relatively successful 🫶
12.7. Robust Packet Scheduling in Video Streaming 🌈
robust packet scheduling has been implemented with MPEG-4 Part 2 and simulated in a wireless streaming environment [21]. There's no technical reason why similar or better results couldn't be achievable with H.264 🎀
streaming clients typically have a receiver buffer that can store a relatively large amount of data 📦 When a streaming session is first established, a client doesn't start playing the stream back immediately. Instead, it typically buffers the incoming data for a few seconds. This buffering helps keep playback continuous, because if there are occasional increases in transmission delays or drops in network throughput, the client can decode and play the buffered data. Without initial buffering, the client would have to freeze the display, stop decoding, and wait for incoming data 🥺 The buffering is also necessary for either automatic or selective retransmission at any protocol level. If any part of a picture is lost, a retransmission mechanism may be used to resend the lost data. If the retransmitted data arrives before its scheduled decoding or playback time, the loss is recovered perfectly 🎉
coded pictures can be ranked by how important they are to the subjective quality of the decoded sequence. For example, non-reference pictures, such as conventional B pictures, are subjectively the least important, because their absence doesn't affect decoding of any other pictures 🤷♀️ On top of non-reference pictures, the ITU-T H.264 | ISO/IEC 14496-10 standard includes a temporal scalability method called sub-sequences [22]. Subjective ranking can also be done on a coded slice data partition or slice group basis. Coded slices and coded slice data partitions that are subjectively the most important can be sent earlier than their decoding order indicates, whereas coded slices and coded slice data partitions that are subjectively the least important can be sent later than their natural coding order indicates. Consequently, any retransmitted parts of the most important slices and coded slice data partitions are more likely to arrive before their scheduled decoding or playback time, compared to the least important slices and slice data partitions 👑
13. Informative Appendix: Rationale for Decoding Order Number 🧠
13.1. Introduction 🌟
the Decoding Order Number (DON) concept was introduced mainly to enable efficient multi-picture slice interleaving (see section 12.6) and robust packet scheduling (see section 12.7). In both of these applications, NAL units are transmitted out of decoding order. DON indicates the decoding order of NAL units and should be used in the receiver to recover the decoding order 🔥 Example use cases for efficient multi-picture slice interleaving and for robust packet scheduling are given in sections 13.2 and 13.3, respectively. Section 13.4 describes the benefits of the DON concept in error resiliency achieved by redundant coded pictures. Section 13.5 summarizes the alternatives considered to DON and explains why DON was chosen for this RTP payload specification 💖
13.2. Example of Multi-Picture Slice Interleaving ✨
okay so this one is my fave 🦋 we're looking at a little subset of a coded video sequence in output order. R means reference picture, N means non-reference picture, and the number is the relative output time 💅
... R1 N2 R3 N4 R5 ...now the decoding order from left to right is different, bestie 👀
... R1 R3 N2 R5 N4 ...the NAL units of R1, R3, N2, R5, and N4 get marked with a DON equal to 1, 2, 3, 4, and 5, respectively 🌸
each reference picture has three slice groups scattered around (the number is the slice group for each macroblock in a QCIF frame) 🧚
0 1 2 0 1 2 0 1 2 0 1
2 0 1 2 0 1 2 0 1 2 0
1 2 0 1 2 0 1 2 0 1 2
0 1 2 0 1 2 0 1 2 0 1
2 0 1 2 0 1 2 0 1 2 0
1 2 0 1 2 0 1 2 0 1 2
0 1 2 0 1 2 0 1 2 0 1
2 0 1 2 0 1 2 0 1 2 0
1 2 0 1 2 0 1 2 0 1 2to keep it simple, we assume all the macroblocks of a slice group are in one slice 🎀 then we build three MTAPs from three consecutive reference pictures, so each MTAP holds three aggregation units, and each unit has all the macroblocks from one slice group 💖
- the first MTAP has slice group 0 of R1, slice group 1 of R3, and slice group 2 of R5
- the second MTAP has slice group 1 of R1, slice group 2 of R3, and slice group 0 of R5
- the third MTAP has slice group 2 of R1, slice group 0 of R3, and slice group 1 of R5
and each non-reference picture gets encapsulated into an STAP-B 🍓
so the transmission order of NAL units ends up looking like this, iykyk 🫶
R1, slice group 0, DON 1, carried in MTAP, RTP SN: N
R3, slice group 1, DON 2, carried in MTAP, RTP SN: N
R5, slice group 2, DON 4, carried in MTAP, RTP SN: N
R1, slice group 1, DON 1, carried in MTAP, RTP SN: N+1
R3, slice group 2, DON 2, carried in MTAP, RTP SN: N+1
R5, slice group 0, DON 4, carried in MTAP, RTP SN: N+1
R1, slice group 2, DON 1, carried in MTAP, RTP SN: N+2
R3, slice group 1, DON 2, carried in MTAP, RTP SN: N+2
R5, slice group 0, DON 4, carried in MTAP, RTP SN: N+2
N2, DON 3, carried in STAP-B, RTP SN: N+3
N4, DON 5, carried in STAP-B, RTP SN: N+4the receiver can put the NAL units back in decoding order using the DON value attached to each one 👑 it's giving organized, periodt 💅
and the glow-up part: if one of the MTAPs gets lost, the spatially adjacent and temporally co-located macroblocks still arrive, so they can be used to conceal the loss really efficiently 🔥 and if one of the STAPs is lost, the damage doesn't spread across time 🦋 not me being impressed 😭
13.3. Example of Robust Packet Scheduling ✨
okay this one is a whole system tour 🎀 an example of robust packet scheduling, and the communication system has these components, in the order the video gets processed from source to sink:
- camera and capturing
- pre-encoding buffer
- encoder
- encoded picture buffer
- transmitter
- transmission channel
- receiver
- receiver buffer
- decoder
- decoded picture buffer
- display
so here's the vibe 🌈 processing of the video stream happens gradually, and at the same time, in all components of the system. the source video sequence gets shot and captured into a pre-encoding buffer. that buffer can reorder pictures from sampling order into encoding order, or analyze multiple uncompressed frames for bit rate control, for example. in some cases the pre-encoding buffer may not exist at all, and the sampled pictures just get encoded right away 💅 the encoder encodes pictures from the pre-encoding buffer and stores the output (aka coded pictures) into the encoded picture buffer. the transmitter encapsulates coded pictures from the encoded picture buffer into transmission packets and sends them through a transmission channel to a receiver ✨
the receiver stores the received packets in the receiver buffer. that buffering typically includes buffering for transmission delay jitter, and it can also be used to recover the correct decoding order of coded data 🧚 the decoder reads coded data from the receiver buffer and produces decoded pictures into the decoded picture buffer, which is used to recover the output (or display) order of pictures. finally, the pictures get displayed 🌸
in the following figures, I is an IDR picture, R is a reference picture, N is a non-reference picture, and the number after I, R, or N is the sampling time relative to the previous IDR picture in decoding order. the values under the sequence are scaled system clock timestamps. the system clock is initialized arbitrarily here, and time runs left to right. each I, R, and N picture maps into the same timeline compared to the previous processing step (assuming encoding, transmission, and decoding take no time), so events happening at the same time sit in the same column across all the figures 🍓
a subset of a sequence of coded pictures in sampling order looks like this:
... N58 N59 I00 N01 N02 R03 N04 N05 R06 ... N58 N59 I00 N01 ...
... --|---|---|---|---|---|---|---|---|- ... -|---|---|---|- ...
... 58 59 60 61 62 63 64 65 66 ... 128 129 130 131 ...Figure 16. Sequence of pictures in sampling order
the sampled pictures get buffered in the pre-encoding buffer so they can be arranged into encoding order. in this example, the non-reference pictures are predicted from both the previous and the next reference picture in output order, except for the non-reference pictures right before an IDR picture, which are predicted only from the previous reference picture in output order 💖 so the pre-encoding buffer has to hold at least two pictures, and the buffering causes a delay of two picture intervals. the output of that buffering process, which is also the encoding (and decoding) order, looks like this:
... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...
... -|---|---|---|---|---|---|---|---|- ...
... 60 61 62 63 64 65 66 67 68 ...Figure 17. Re-ordered pictures in the pre-encoding buffer
the encoder or the transmitter can set the DON for each picture to the DON of the previous picture in decoding order plus one 🔥
for the sake of simplicity, let's assume that:
- the frame rate of the sequence is constant,
- each picture consists of only one slice,
- each slice is encapsulated in a single NAL unit packet,
- there is no transmission delay, and
- pictures are transmitted at constant intervals (that is, 1 / frame rate).
when pictures are transmitted in decoding order, they're received like this 🦋
... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...
... -|---|---|---|---|---|---|---|---|- ...
... 60 61 62 63 64 65 66 67 68 ...Figure 18. Received pictures in decoding order
the OPTIONAL sprop-interleaving-depth MIME type parameter is set to 0, since the transmission (or reception) order is identical to the decoding order 🌸
the decoder has to buffer for one picture interval initially in its decoded picture buffer, so it can organize pictures from decoding order into output order, as shown below 🎀
... N58 N59 I00 N01 N02 R03 N04 N05 R06 ...
... -|---|---|---|---|---|---|---|---|- ...
... 61 62 63 64 65 66 67 68 69 ...Figure 19. Output order
the amount of required initial buffering in the decoded picture buffer can be signaled in the buffering period SEI message, or with the num_reorder_frames syntax element of H.264 video usability information. num_reorder_frames is the maximum number of frames, complementary field pairs, or non-paired fields that precede any frame, complementary field pair, or non-paired field in the sequence in decoding order and follow it in output order 💅 to keep things simple, we assume num_reorder_frames is used to indicate the initial buffer in the decoded picture buffer. in this example, num_reorder_frames is equal to 1 ✨
now here's the spicy bit 😭 if the IDR picture I00 gets lost in transmission and a retransmission request is issued when the system clock reads 62, there's one picture interval (until the clock hits timestamp 63) to receive the retransmitted IDR picture I00. yikes, but fine 💀
so let's say IDR pictures are transmitted two frame intervals earlier than their decoding position, i.e., the pictures get sent like this:
... I00 N58 N59 R03 N01 N02 R06 N04 N05 ...
... --|---|---|---|---|---|---|---|---|- ...
... 62 63 64 65 66 67 68 69 70 ...Figure 20. Interleaving: Early IDR pictures in sending order
the OPTIONAL sprop-interleaving-depth MIME type parameter is set equal to 1 according to its definition 🧚 (quick derivation for this example: picture I00 is the only picture that comes before N58 or N59 in transmission order and after it in decoding order. apart from I00, N58, and N59, the transmission order matches the decoding order. since each coded picture is encapsulated in exactly one NAL unit, sprop-interleaving-depth equals the maximum number of pictures that precede any picture in transmission order and follow that picture in decoding order 🫶)
the receiver buffering process holds two pictures at a time, based on the sprop-interleaving-depth value, and it orders pictures from reception order into the correct decoding order using the DON value of each picture. the output of that process looks like this 🌈
... N58 N59 I00 R03 N01 N02 R06 N04 N05 ...
... -|---|---|---|---|---|---|---|---|- ...
... 63 64 65 66 67 68 69 70 71 ...Figure 21. Interleaving: Receiver buffer
and once again, an initial buffering delay of one picture interval is needed to go from decoding order to output order, as shown below 💖
... N58 N59 I00 N01 N02 R03 N04 N05 ...
... -|---|---|---|---|---|---|---|- ...
... 64 65 66 67 68 69 70 71 ...Figure 22. Interleaving: Receiver buffer after reordering
last but not least 👑 the maximum delay that IDR pictures can undergo during transmission, including any possible application, transport, or link layer retransmission, is equal to three picture intervals. so the loss resiliency of IDR pictures is improved in systems that support retransmission, compared to the case where pictures are transmitted in their decoding order. main character energy, no cap 🦋✨
13.4. Robust Transmission Scheduling of Redundant Coded Slices 🦋
okay so a redundant coded picture is a coded version of a picture (or a piece of one) that doesn't get used in decoding as long as the matching primary coded picture decodes correctly 💖 There should be no noticeable difference between any area of the decoded primary picture and the matching area you'd get from running the H.264 decoding process on any redundant picture in the same access unit. And a redundant coded slice is just a coded slice that lives inside a redundant coded picture ✨
Redundant coded pictures are giving unequal error protection in error-prone video transmission 🛡️ If a primary coded representation of a picture gets decoded incorrectly, the corresponding redundant coded picture can be decoded instead. Examples of apps and coding techniques that use the redundant codec picture feature include the video redundancy coding [23] and the protection of "key pictures" in multicast streaming [24] 🌸
Here's the thing about error-prone video systems: their transmission errors are often bursty 😭 So they can knock out more than one consecutive transmission packet in transmission order. In low bit-rate video communication, it's pretty common for an entire coded picture to fit into one transmission packet. That means a primary coded picture and its redundant coded pictures may end up in consecutive packets in transmission order 💀 To make the scheme more tolerant of bursty errors, it's beneficial to send the primary coded picture and the redundant coded picture separated by more than a single packet. The DON concept is what makes this possible 🍓
13.5. Remarks on Other Design Possibilities 🧚
The slice header syntax structure of the H.264 coding standard has a frame_num syntax element that can indicate the decoding order of coded frames. BUT, bestie, using frame_num to recover the decoding order is not feasible or desirable, and here's why 🙈
- The receiver is required to parse at least one slice header per coded picture (before passing the coded data to the decoder).
- Coded slices from multiple coded video sequences cannot be interleaved, as the frame number syntax element is reset to 0 in each IDR picture.
- The coded fields of a complementary field pair share the same value of the frame_num syntax element. Thus, the decoding order of the coded fields of a complementary field pair cannot be recovered based on the frame_num syntax element or any other syntax element of the H.264 coding syntax.
The RTP payload format for transport of MPEG-4 elementary streams [25] lets you interleave access units and send multiple access units in the same RTP packet ✨ In H.264, an access unit is defined to comprise all NAL units associated with a primary coded picture, according to subclause 7.4.1.2 of [1]. So slices of different pictures can't be interleaved, and the multi-picture slice interleaving technique (see section 12.6) for improved error resilience can't be used with it 😤 It's a whole mood, tbh 👑
14. Acknowledgements 🫶
The authors thank Roni Even, Dave Lindbergh, Philippe Gentric, Gonzalo Camarillo, Gary Sullivan, Joerg Ott, and Colin Perkins for careful review 💜
15. References 📚
15.1. Normative References
[1] ITU-T Recommendation H.264, "Advanced video coding for generic audiovisual services", May 2003.
[2] ISO/IEC International Standard 14496-10:2003.
[3] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[4] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
[5] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998.
[6] Josefsson, S., "The Base16, Base32, and Base64 Data Encodings", RFC 3548, July 2003.
[7] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002.
15.2. Informative References
[8] "Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU-T Rec. H.264 | ISO/IEC 14496-10 AVC)", available from http://ftp3.itu.int/av-arch/jvt-site/2003_03_Pattaya/JVT-G050r1.zip, May 2003.
[9] Luthra, A., Sullivan, G.J., and T. Wiegand (eds.), Special Issue on H.264/AVC. IEEE Transactions on Circuits and Systems on Video Technology, July 2003.
[10] Bormann, C., Cline, L., Deisher, G., Gardos, T., Maciocco, C., Newell, D., Ott, J., Sullivan, G., Wenger, S., and C. Zhu, "RTP Payload Format for the 1998 Version of ITU-T Rec. H.263 Video (H.263+)", RFC 2429, October 1998.
[11] ISO/IEC IS 14496-2.
[12] Wenger, S., "H.26L over IP", IEEE Transaction on Circuits and Systems for Video technology, Vol. 13, No. 7, July 2003.
[13] Wenger, S., "H.26L over IP: The IP Network Adaptation Layer", Proceedings Packet Video Workshop 02, April 2002.
[14] Stockhammer, T., Hannuksela, M.M., and S. Wenger, "H.26L/JVT Coding Network Abstraction Layer and IP-based Transport" in Proc. ICIP 2002, Rochester, NY, September 2002.
[15] ITU-T Recommendation H.241, "Extended video procedures and control signals for H.300 series terminals", 2004.
[16] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, July 2003.
[17] ITU-T Recommendation H.223, "Multiplexing protocol for low bit rate multimedia communication", July 2001.
[18] Rosenberg, J. and H. Schulzrinne, "An RTP Payload Format for Generic Forward Error Correction", RFC 2733, December 1999.
[19] Stockhammer, T., Wiegand, T., Oelbaum, T., and F. Obermeier, "Video Coding and Transport Layer Techniques for H.264/AVC-Based Transmission over Packet-Lossy Networks", IEEE International Conference on Image Processing (ICIP 2003), Barcelona, Spain, September 2003.
[20] Varsa, V. and M. Karczewicz, "Slice interleaving in compressed video packetization", Packet Video Workshop 2000.
[21] Kang, S.H. and A. Zakhor, "Packet scheduling algorithm for wireless video streaming," International Packet Video Workshop 2002.
[22] Hannuksela, M.M., "Enhanced concept of GOP", JVT-B042, available http://ftp3.itu.int/av-arch/video-site/0201_Gen/JVT-B042.doc, January 2002.
[23] Wenger, S., "Video Redundancy Coding in H.263+", 1997 International Workshop on Audio-Visual Services over Packet Networks, September 1997.
[24] Wang, Y.-K., Hannuksela, M.M., and M. Gabbouj, "Error Resilient Video Coding Using Unequally Protected Key Pictures", in Proc. International Workshop VLBV03, September 2003.
[25] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., and P. Gentric, "RTP Payload Format for Transport of MPEG-4 Elementary Streams", RFC 3640, November 2003.
[26] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004.
[27] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, April 1998.
[28] Handley, M., Perkins, C., and E. Whelan, "Session Announcement Protocol", RFC 2974, October 2000.
[29] ISO/IEC 14496-15: "Information technology - Coding of audio-visual objects - Part 15: Advanced Video Coding (AVC) file format".
[30] Castagno, R. and D. Singer, "MIME Type Registrations for 3rd Generation Partnership Project (3GPP) Multimedia files", RFC 3839, July 2004.
Authors' Addresses 🎀
Stephan Wenger
TU Berlin / Teles AG
Franklinstr. 28-29
D-10587 Berlin
Germany
Phone: +49-172-300-0813
EMail: stewe@stewe.org
Miska M. Hannuksela
Nokia Corporation
P.O. Box 100
33721 Tampere
Finland
Phone: +358-7180-73151
EMail: miska.hannuksela@nokia.com
Thomas Stockhammer
Nomor Research
D-83346 Bergen
Germany
Phone: +49-8662-419407
EMail: stockhammer@nomor.de
Magnus Westerlund
Multimedia Technologies
Ericsson Research EAB/TVA/A
Ericsson AB
Torshamsgatan 23
SE-164 80 Stockholm
Sweden
Phone: +46-8-7190000
EMail: magnus.westerlund@ericsson.com
David Singer
QuickTime Engineering
Apple
1 Infinite Loop MS 302-3MT
Cupertino
CA 95014
USA
Phone +1 408 974-3162
EMail: singer@apple.comFull Copyright Statement ⚖️
Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🔐
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the IETF's procedures with respect to rights in IETF Documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
Acknowledgement 💌
Funding for the RFC Editor function is currently provided by the Internet Society. 🌈
📦 where this page comes from
- ietf: rfc3984 2005-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms