RFC 3640
RTP Payload Format for Transport of MPEG-4 Elementary Streams
RFC 3640: RTP Payload Format for Transport of MPEG-4 Elementary Streams 🎀
- Number: RFC 3640
- Title: RTP Payload Format for Transport of MPEG-4 Elementary Streams
- Authors: J. van der Meer (Philips Electronics), D. Mackie (Apple Computer), V. Swaminathan (Sun Microsystems Inc.), D. Singer (Apple Computer), P. Gentric (Philips Electronics)
- Working group: Network Working Group
- Category: Standards Track
- Date: November 2003
- Obsoletes and updates: the title block doesn't list anything this RFC obsoletes or updates 🦋
Status of this Memo
Okay so this doc is an Internet standards track protocol spec for the Internet community, and it's asking for discussion and suggestions for improvements 🫶 For the standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited ✨
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved. 💜
Abstract
The Motion Picture Experts Group (MPEG) Committee (ISO/IEC JTC1/SC29 WG11) is the working group inside ISO that made the MPEG-4 standard 👑 MPEG defines tools that compress things like audio-visual info into elementary streams. This spec defines a simple but generic RTP payload format for carrying any non-multiplexed MPEG-4 elementary stream 🌸
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Carriage of MPEG-4 Elementary Streams Over RTP . . . . . . . . 4
2.1. Signaling by MIME Format Parameters . . . . . . . . . . 4
2.2. MPEG Access Units . . . . . . . . . . . . . . . . . . . 5
2.3. Concatenation of Access Units . . . . . . . . . . . . . 5
2.4. Fragmentation of Access Units . . . . . . . . . . . . . 6
2.5. Interleaving . . . . . . . . . . . . . . . . . . . . . . 6
2.6. Time Stamp Information . . . . . . . . . . . . . . . . . 7
2.7. State Indication of MPEG-4 System Streams . . . . . . . 8
2.8. Random Access Indication . . . . . . . . . . . . . . . . 8
2.9. Carriage of Auxiliary Information . . . . . . . . . . . 8
2.10. MIME Format Parameters and Configuring Conditional Field 8
2.11. Global Structure of Payload Format . . . . . . . . . . . 9
2.12. Modes to Transport MPEG-4 Streams . . . . . . . . . . . 9
2.13. Alignment with RFC 3016 . . . . . . . . . . . . . . . . 10
3. Payload Format . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1. Usage of RTP Header Fields and RTCP . . . . . . . . . . 10
3.2. RTP Payload Structure . . . . . . . . . . . . . . . . . 11
3.2.1. The AU Header Section . . . . . . . . . . . . . 11
3.2.1.1. The AU-header . . . . . . . . . . . . 12
3.2.2. The Auxiliary Section . . . . . . . . . . . . . 14
3.2.3. The Access Unit Data Section . . . . . . . . . . 15
3.2.3.1. Fragmentation. . . . . . . . . . . . . 16
3.2.3.2. Interleaving . . . . . . . . . . . . . 16
3.2.3.3. Constraints for Interleaving . . . . . 17
3.2.3.4. Crucial and Non-Crucial AUs with
MPEG-4 System Data . . . . . . . . . . 20
3.3. Usage of this Specification. . . . . . . . . . . . . . . 21
3.3.1. General. . . . . . . . . . . . . . . . . . . . . 21
3.3.2. The Generic Mode . . . . . . . . . . . . . . . . 22
3.3.3. Constant Bit Rate CELP . . . . . . . . . . . . . 22
3.3.4. Variable Bit Rate CELP . . . . . . . . . . . . . 23
3.3.5. Low Bit Rate AAC . . . . . . . . . . . . . . . . 24
3.3.6. High Bit Rate AAC. . . . . . . . . . . . . . . . 25
3.3.7. Additional Modes . . . . . . . . . . . . . . . . 26
4. IANA Considerations. . . . . . . . . . . . . . . . . . . . . . 27
4.1. MIME Type Registration . . . . . . . . . . . . . . . . . 27
4.2. Registration of Mode Definitions with IANA . . . . . . . 33
4.3. Concatenation of Parameters. . . . . . . . . . . . . . . 33
4.4. Usage of SDP . . . . . . . . . . . . . . . . . . . . . . 34
4.4.1. The a=fmtp Keyword . . . . . . . . . . . . . . . 34
5. Security Considerations. . . . . . . . . . . . . . . . . . . . 34
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 35
APPENDIX: Usage of this Payload Format. . . . . . . . . . . . . . 36
Appendix A. Interleave Analysis . . . . . . . . . . . . . . . . . 36
A. Examples of Delay Analysis with Interleave. . . . . . . . . . 36
A.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 36
A.2. De-interleaving and Error Concealment . . . . . . . . . 36
A.3. Simple Group Interleave . . . . . . . . . . . . . . . . 36
A.3.1. Introduction . . . . . . . . . . . . . . . . . . 36
A.3.2. Determining the De-interleave Buffer Size . . . 37
A.3.3. Determining the Maximum Displacement . . . . . . 37
A.4. More Subtle Group Interleave . . . . . . . . . . . . . . 38
A.4.1. Introduction . . . . . . . . . . . . . . . . . . 38
A.4.2. Determining the De-interleave Buffer Size. . . . 38
A.4.3. Determining the Maximum Displacement . . . . . . 39
A.5. Continuous Interleave . . . . . . . . . . . . . . . . . 39
A.5.1. Introduction . . . . . . . . . . . . . . . . . . 39
A.5.2. Determining the De-interleave Buffer Size . . . 40
A.5.3. Determining the Maximum Displacement . . . . . . 40
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Normative References . . . . . . . . . . . . . . . . . . . . . . . 41
Informative References . . . . . . . . . . . . . . . . . . . . . . 41
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 42
Full Copyright Statement . . . . . . . . . . . . . . . . . . . . . 431. Introduction ✨
Okay so the MPEG Committee is Working Group 11 (WG11) in ISO/IEC JTC1 SC29, and they're the squad that specified MPEG-1, MPEG-2 and, more recently, MPEG-4 [1] 💅 MPEG-4 is all about compressing audio-visual data into things like an audio or video elementary stream. Its streams are made of audio-visual objects, which can be arranged into an audio-visual scene using a scene description. Each MPEG-4 elementary stream is a sequence of Access Units, and examples of an Access Unit (AU) are an audio frame and a video picture 🦋
This spec gives you a general, configurable payload structure for carrying MPEG-4 elementary streams. That includes MPEG-4 audio (speech too), MPEG-4 video, and MPEG-4 systems streams like BIFS (BInary Format for Scenes), OCI (Object Content Information), OD (Object Descriptor) and IPMP (Intellectual Property Management and Protection) streams 🌈 The RTP payload here is simple to implement and reasonably efficient. It also allows for optional interleaving of Access Units (such as audio frames) to boost error resiliency when packets get lost 🧚
Some MPEG-4 elementary streams carry "crucial" information that you really can't afford to lose. The catch is that RTP doesn't provide reliable transmission, so receiving that crucial info isn't guaranteed 😭 Section 3.2.3.4 explains how stream state gets conveyed, so the receiver can spot lost crucial info and stop decoding until the next random access point shows up 🫶 Apps sending crucial stuff, like OD commands, BIFS commands, or programmatic content such as MPEG-J (Java) and ECMAScript, should include random access points. They should space them out based on the probability of loss, so stream corruption drops to an acceptable level 💖 An example is the carousel mechanism MPEG defines in ISO/IEC 14496-1 [1].
Apps can also bring in extra protocols or services to cut down on loss 🌸 At the RTP layer, that means payload formats and profiles for retransmission or forward error correction (such as in RFC 2733 [10]), which must be used with due consideration to congestion control. Another option that may suit some apps is carrying RTP over TCP (such as in RFC 2326 [8], section 10.12). At the network layer, resource allocation or preferential service may be available to reduce loss. For a general overview of ways to repair streaming media, see RFC 2354 [9] ⭐
Though this payload format can carry any MPEG-4 stream, more specific formats may exist, like RFC 3016 [12] for MPEG-4 video transport (ISO/IEC 14496 [1] part 2) 👑
Configuration lets you transport any MPEG-4 stream at any possible bit rate. But for one specific elementary stream, you typically only need a few configurations. So, to make simplified but dedicated receivers possible, this specification requires that specific modes be defined for MPEG-4 transport. This document defines modes for MPEG-4 CELP and AAC streams, plus a generic mode that can carry any MPEG-4 stream. New RFCs are expected to add more modes in the future 🎀
This payload format carries system-related info that's often equivalent to what the MPEG-4 Sync Layer (SL) can contain, as defined in MPEG-4 Systems [1]. This document doesn't prescribe how to transcode or map information from the SL into the RTP payload format fields. That's left to the discretion of the application 🙈 Still, to be ready for any additional system-related info down the line, an auxiliary field can be configured to carry whatever data is needed 🍓
Okay, the BCP 14 power words in this doc, "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL", are to be read as described in BCP 14, RFC 2119 [4] 🫶
2. Carriage of MPEG-4 Elementary Streams over RTP 🌈
2.1. Signaling by MIME Format Parameters 🧚
With this payload format, a single MPEG-4 elementary stream can be transported ✨ Info about what type of MPEG-4 stream is in the payload gets conveyed by MIME format parameters, either in an SDP [5] message or by other means (see section 4) 💅 These MIME format parameters specify the payload's configuration. For simplified, dedicated receivers, there's a MIME format parameter that signals a specific mode of using this payload 🦋 A mode definition MAY include the type of MPEG-4 elementary stream along with the applied configuration. That way receivers don't have to parse every single MIME format parameter 💖 The applied mode MUST be signaled. 🔥
2.2. MPEG Access Units
Okay bestie, MPEG has this concept called Access Units 🎀. An MPEG Access Unit (AU) is the smallest chunk of data that gets timing info attached to it ⏱️. For audio, an AU can be one audio frame, and for video it can be a picture 📸. Access Units are octet-aligned by definition 💅. If an audio frame isn't octet-aligned, up to 7 zero-padding bits MUST be inserted at the end of the frame to get the octet-aligned Access Units, as the MPEG-4 spec requires 🫶. MPEG-4 decoders MUST be able to decode AUs where that padding is applied ✨.
Consistent with the MPEG-4 spec, this document requires that each MPEG-4 part 2 video Access Unit include all the coded data of a picture 🌸. That includes any video stream headers that come before the coded picture data, and any video stream stuffing that comes after it. It all goes up to, but doesn't include, the startcode that marks the start of a new video stream or the next Access Unit 🦋.
2.3. Concatenation of Access Units
Putting multiple Access Units in one RTP packet is so iconic 👑. It's especially useful for audio. For example, when AAC is used to encode a stereo signal at 64 kbits/sec, AAC frames average about 200 octets. On a LAN with a 1500 octet MTU, that works out to an average of 7 complete AAC frames per RTP packet 🔥.
Access Units can have a fixed size in octets, but a variable size is also possible 🌈. To make parsing easier when there are multiple concatenated AUs in one RTP packet, the receiver needs to know the size of each AU ✨. When the AUs are a constant size, that size gets communicated "out of band" through a MIME format parameter 💜. When the AUs are variable size, the RTP payload carries an AU size field for each contained AU "in band" 🧚.
Combined with the RTP payload length, the size info lets the receiver split the RTP payload back into the individual AUs 💖.
To keep RTP receivers simple to implement, it is required that when multiple AUs are carried in an RTP packet, each AU MUST be complete, i.e., the number of AUs in an RTP packet MUST be integral 🫶. Also, an AU MUST NOT be repeated in other RTP packets. That means repeating an AU is only possible when using a duplicate RTP packet 🙈.
2.4. Fragmentation of Access Units
MPEG allows for very large Access Units 😭. Most IP networks have much smaller MTU sizes, so this payload format allows an Access Unit to be fragmented over multiple RTP packets 🧩. That way, if an IP packet gets lost after IP-level fragmentation, only an AU fragment may get lost instead of the whole AU 💅.
To keep things simple for RTP receivers, an RTP packet SHALL either carry one or more complete Access Units or a single fragment of one AU. In other words, packets MUST NOT contain fragments of multiple Access Units 💀.
2.5. Interleaving
When an RTP packet carries a contiguous run of Access Units, losing that one packet can cause a "decoding gap" for the user 😩. One way to fix this is to interleave the Access Units across RTP packets 🦋. For a modest cost in latency and implementation complexity, you can get a lot of error resiliency against packet loss 🌸.
To support optional interleaving, this payload format lets index information get sent for each Access Unit 🎀. After receivers have been told which buffer resources to allocate for de-interleaving, the RTP sender is free to pick the interleaving pattern without sending that info to the receiver(s) ahead of time 🍓. The sender could even adjust the pattern on the fly based on Access Unit size, error rates, etc. 👑. The RTP receiver doesn't need to know the interleaving pattern. It just reads the index info of each Access Unit and slots the AU into the right spot in the decoding or rendering queue ✨.
Here's an example. Say an RTP packet contains 3 AUs, the AUs are numbered 0, 1, 2, 3, 4, and so forth, and we pick an interleaving group length of 9. Then RTP packet(i) contains these AU(n)s:
RTP packet(0): AU(0), AU(3), AU(6)
RTP packet(1): AU(1), AU(4), AU(7)
RTP packet(2): AU(2), AU(5), AU(8)
RTP packet(3): AU(9), AU(12), AU(15)
RTP packet(4): AU(10), AU(13), AU(16) Etc.2.6. Time Stamp Information
The RTP time stamp MUST carry the sampling instant of the first AU (fragment) in the RTP packet ⏰. When there are multiple AUs in one RTP packet, the time stamps of the later AUs can be calculated if the frame period of each AU is known 🧮. For audio and video, that's doable when the frame rate is constant. But sometimes you can't do that math, like with variable frame rate video or MPEG-4 BIFS streams carrying composition info 🙈. So this payload format can be configured to carry a time stamp in the RTP payload for each contained Access Unit 💜. A time stamp MAY be conveyed in the RTP payload only for non-first AUs in the RTP packet. It SHALL NOT be conveyed for the first AU (fragment), because the first AU's time stamp already lives in the RTP time stamp 🔥.
MPEG-4 has two kinds of time stamps: the composition time stamp (CTS) and the decoding time stamp (DTS) 🌈. The CTS is the sampling instant of an AU, so it's equivalent to the RTP time stamp ✨. The DTS may be used in MPEG-4 video streams that use bi-directional coding. That's when pictures are predicted in both the forward and backward direction, using either a reference picture from the past or one from the future 🦋. The DTS can't be carried in the RTP header 😤. Sometimes you can derive the DTS from the RTP time stamp using frame rate info, but that requires deep parsing of the video stream, which may be considered objectionable 🙅♀️. And if the video frame rate is variable, the needed info might not even be in the video stream 💀. For both reasons, there's an optional capability to carry the DTS in the RTP payload for each contained Access Unit 🎀.
To keep time stamp coding efficient, each time stamp in the RTP payload is coded as a difference. For the CTS, the offset from the RTP time stamp is provided. For the DTS, the offset from the CTS is provided 🧁.
2.7. State Indication of MPEG-4 System Streams
ISO/IEC 14496-1 defines states for MPEG-4 system streams 🧠. To carry state info when transporting those streams, this payload format optionally allows the stream state for each contained Access Unit to be carried in the RTP payload 💖. Stream states are used to signal "crucial" AUs, the ones carrying info that can't be lost 😱. They're also handy when repeating AUs according to the carousel mechanism defined in ISO/IEC 14496-1 🔁.
2.8. Random Access Indication
You can't always jump into MPEG-4 elementary streams at any point. Random access may only be possible at some Access Units 🚪. To signal which AUs allow random access, a random access point flag can optionally be carried in the RTP payload for each contained Access Unit 🌟. Random access points are especially useful for MPEG-4 system streams combined with the stream state 🍓.
2.9. Carriage of Auxiliary Information
This payload format defines a specific field for auxiliary data 🎁. The auxiliary data field is preceded by a field that gives its length, so you can skip the data without parsing it 🏃♀️. The coding of the auxiliary data is not defined in this document. Instead, the format, meaning and signaling of auxiliary info is expected to be specified in one or more future RFCs 🔮. Auxiliary information MUST NOT be transmitted until its format, meaning and signaling have been specified and its use has been signaled 🚫. Receivers that know about the auxiliary data MAY decode it. But receivers without that knowledge MUST skip the auxiliary data field 🙈.
2.10. MIME Format Parameters and Configuring Conditional Fields
To support the features from the previous sections, several fields are defined for carriage in the RTP payload 🎀. Their use really depends on the type of MPEG-4 elementary stream being carried 🌈. Sometimes a field needs a certain length, and other times it isn't needed at all. To stay efficient either way, these fields are configurable through MIME format parameters 💜. In general, a MIME format parameter defines whether the associated field is present and how long it is. A length of zero means the field is absent ✨. That also means parsing the payload requires knowing the MIME format parameters 🧩. The MIME format parameters get conveyed to the receiver via SDP [5] messages, as specified in section 4.4.1, or through other means 📨.
2.11. Global Structure of Payload Format
The RTP payload that follows the RTP header has three octet-aligned data sections. The first two MAY be empty. Check out Figure 1 🍰.
+---------+-----------+-----------+---------------+
| RTP | AU Header | Auxiliary | Access Unit |
| Header | Section | Section | Data Section |
+---------+-----------+-----------+---------------+
<----------RTP Packet Payload----------->
Figure 1: Data sections within an RTP packetThe first data section is the AU (Access Unit) Header Section, which holds one or more AU-headers. Each AU-header MAY be empty, and if they all are, the entire AU Header Section is empty 🫧. The second section is the Auxiliary Section, which holds auxiliary data. This section MAY also be configured empty 🌸. The third section is the Access Unit Data Section, which holds either a single fragment of one Access Unit or one or more complete Access Units. The Access Unit Data Section MUST NOT be empty 💅.
2.12. Modes to Transport MPEG-4 Streams
You can build fully configurable receivers that handle any MPEG-4 stream, but this spec also allows simpler, dedicated receivers 🛠️. Those might only receive one type of MPEG-4 stream, for example. This works because specific modes have to be defined before you can use this specification. Each mode may define constraints for transporting one or more types of MPEG-4 streams, such as on the payload configuration 🎯.
The applied mode MUST be signaled 📣. Signaling the mode matters most for receivers that can only decode one or more specific modes. Those receivers need to figure out whether the applied mode is supported, so they don't choke on payloads beyond what they can handle 😅.
This document defines several modes for transporting MPEG-4 CELP and AAC streams, plus a generic mode that works for any MPEG-4 stream 🌈. New RFCs may define other modes for using this specification in the future 🔭. However, each mode MUST be in full compliance with this specification (see section 3.3.7) 🫶.
2.13. Alignment with RFC 3016
okay so this payload can be set up to be nearly identical to the payload format in RFC 3016 [12] for the MPEG-4 video configurations that RFC 3016 recommends 💖 that means receivers that comply with RFC 3016 can decode this RTP payload, as long as the extra packets with the video decoder configuration (VO, VOL, VOSH) get inserted into the stream, as RFC 3016 [12] requires ✨ the other way around, receivers that comply with this spec SHOULD be able to decode the payloads, names and parameters defined for MPEG-4 video in RFC 3016 [12] 🫶 and honestly, it is strongly RECOMMENDED that the implementation lets you ignore "in band" video decoder configuration packets that might show up in streams following the RFC 3016 video payload 💅
also, the "out of band" availability of the video decoder configuration is optional in RFC 3016 [12] 🌸 so if you want maximum interoperability with this RTP payload format, applications that use RFC 3016 to carry MPEG-4 video (part 2) are recommended to make the video decoder configuration available as a MIME parameter 🦋
3. Payload Format ✨
3.1. Usage of RTP Header Fields and RTCP
Payload Type (PT): the assignment of an RTP payload type for this packet format is outside the scope of this document 🎀 it's specified by the RTP profile this payload format is used under, or signaled dynamically out-of-band (for example, using SDP) 🧚
Marker (M) bit: the M bit is set to 1 to indicate that the RTP packet payload contains either the final fragment of a fragmented Access Unit or one or more complete Access Units 🌈
Extension (X) bit: defined by the RTP profile used 💜
Sequence Number: the RTP sequence number SHOULD be generated by the sender in the usual manner, with a constant random offset 🍓
Timestamp: indicates the sampling instant of the first AU contained in the RTP payload 👑 this sampling instant is equivalent to the CTS in the MPEG-4 time domain ✨ when using SDP, the clock rate of the RTP time stamp MUST be expressed using the "rtpmap" attribute 🔥 if an MPEG-4 audio stream is transported, the rate SHOULD be set to the same value as the sampling rate of the audio stream 🎶 if an MPEG-4 video stream is transported, it is RECOMMENDED that the rate be set to 90 kHz 🦋
In all cases, the sender SHALL make sure that RTP time stamps are identical only if the RTP time stamp refers to fragments of the same Access Unit 💅
According to RFC 3550 [2] (section 5.1), it is RECOMMENDED that RTP time stamps start at a random value for security reasons 🔒 this is not an issue for synchronization of multiple RTP streams 🙌 however, when streams from multiple sources are to be synchronized (for example, one stream from local storage and another from an RTP streaming server), synchronization may become impossible if the receiver only knows the original time stamp relationships 😭 in such cases, the time stamp relationship required for obtaining synchronization may be provided by out-of-band means 🫶 the format of such information, as well as the methods to convey it, are beyond the scope of this specification 🌸
SSRC: set as described in RFC 3550 [2] 💖
CC and CSRC fields are used as described in RFC 3550 [2] ✨
RTCP SHOULD be used as defined in RFC 3550 [2] 🎀 note that time stamps in RTCP Sender Reports may be used to synchronize multiple MPEG-4 elementary streams, and also to synchronize MPEG-4 streams with non-MPEG-4 streams, in case the delivery of these streams uses RTP 🌈
3.2. RTP Payload Structure
3.2.1. The AU Header Section
When present, the AU Header Section consists of the AU-headers-length field, followed by a number of AU-headers, see Figure 2 🧚
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+
|AU-headers-length|AU-header|AU-header| |AU-header|padding|
| | (1) | (2) | | (n) | bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+-+Figure 2: The AU Header Section
The AU-headers are configured using MIME format parameters and MAY be empty 🍓 if the AU-header is configured empty, the AU-headers-length field SHALL NOT be present, and consequently the AU Header Section is empty 🙈 if the AU-header is not configured empty, then the AU-headers-length is a two octet field that specifies the length in bits of the immediately following AU-headers, excluding the padding bits 🫶
Each AU-header is associated with a single Access Unit (fragment) contained in the Access Unit Data Section in the same RTP packet 💜
For each contained Access Unit (fragment), there is exactly one AU-header 👑 within the AU Header Section, the AU-headers are bit-wise concatenated in the order in which the Access Units are contained in the Access Unit Data Section 🔥 hence, the n-th AU-header refers to the n-th AU (fragment) ✨ if the concatenated AU-headers consume a non-integer number of octets, up to 7 zero-padding bits MUST be inserted at the end in order to achieve octet-alignment of the AU Header Section 🎀
3.2.1.1. The AU-header
Each AU-header may contain the fields given in Figure 3 🌸 the length in bits of the fields, with the exception of the CTS-flag, the DTS-flag and the RAP-flag fields, is defined by MIME format parameters; see section 4.1 💅 if a MIME format parameter has the default value of zero, then the associated field is not present 🦋 the number of bits for fields that are present and that represent the value of a parameter MUST be chosen large enough to correctly encode the largest value of that parameter during the session 🌈
If present, the fields MUST occur in the mutual order given in Figure 3 💖 in the general case, a receiver can only discover the size of an AU-header by parsing it, since the presence of the CTS-delta and DTS-delta fields is signaled by the value of the CTS-flag and DTS-flag, respectively 🧚
+---------------------------------------+
| AU-size |
+---------------------------------------+
| AU-Index / AU-Index-delta |
+---------------------------------------+
| CTS-flag |
+---------------------------------------+
| CTS-delta |
+---------------------------------------+
| DTS-flag |
+---------------------------------------+
| DTS-delta |
+---------------------------------------+
| RAP-flag |
+---------------------------------------+
| Stream-state |
+---------------------------------------+Figure 3: The fields in the AU-header. If used, the AU-Index field only occurs in the first AU-header within an AU Header Section; in any other AU-header, the AU-Index-delta field occurs instead.
AU-size: indicates the size in octets of the associated Access Unit in the Access Unit Data Section in the same RTP packet 🍓 when the AU-size is associated with an AU fragment, the AU size indicates the size of the entire AU and not the size of the fragment 🙌 in this case, the size of the fragment is known from the size of the AU data section 🫶 this can be exploited to determine whether a packet contains an entire AU or a fragment, which is particularly useful after losing a packet carrying the last fragment of an AU 😭
AU-Index: indicates the serial number of the associated Access Unit (fragment) 🔢 for each (in decoding order) consecutive AU or AU fragment, the serial number is incremented by 1 ✨ when present, the AU-Index field occurs in the first AU-header in the AU Header Section, but MUST NOT occur in any subsequent (non-first) AU-header in that Section 🎀 to encode the serial number in any such non-first AU-header, the AU-Index-delta field is used 💜
AU-Index-delta: the AU-Index-delta field is an unsigned integer that specifies the serial number of the associated AU as the difference with respect to the serial number of the previous Access Unit 🦋 hence, for the n-th (n>1) AU, the serial number is found from:
AU-Index(n) = AU-Index(n-1) + AU-Index-delta(n) + 1If the AU-Index field is present in the first AU-header in the AU Header Section, then the AU-Index-delta field MUST be present in any subsequent (non-first) AU-header 🌈 when the AU-Index-delta is coded with the value 0, it indicates that the Access Units are consecutive in decoding order 💖 an AU-Index-delta value larger than 0 signals that interleaving is applied 🧚
CTS-flag: indicates whether the CTS-delta field is present 🌸 a value of 1 indicates that the field is present, and a value of 0 indicates that it is not present 💅
The CTS-flag field MUST be present in each AU-header if the length of the CTS-delta field is signaled to be larger than zero 🔥 in that case, the CTS-flag field MUST have the value 0 in the first AU-header and MAY have the value 1 in all non-first AU-headers 🎀 the CTS-flag field SHOULD be 0 for any non-first fragment of an Access Unit 🙈
CTS-delta: encodes the CTS by specifying the value of CTS as a 2's complement offset (delta) from the time stamp in the RTP header of this RTP packet 🧮 the CTS MUST use the same clock rate as the time stamp in the RTP header 🫶
DTS-flag: indicates whether the DTS-delta field is present 🌈 a value of 1 indicates that DTS-delta is present, and a value of 0 indicates that it is not present 💜
The DTS-flag field MUST be present in each AU-header if the length of the DTS-delta field is signaled to be larger than zero 👑 the DTS-flag field MUST have the same value for all fragments of an Access Unit ✨
DTS-delta: specifies the value of the DTS as a 2's complement offset (delta) from the CTS 🦋 the DTS MUST use the same clock rate as the time stamp in the RTP header 🍓 the DTS-delta field MUST have the same value for all fragments of an Access Unit 💖
RAP-flag: when set to 1, indicates that the associated Access Unit provides a random access point to the content of the stream 🧚 if an Access Unit is fragmented, the RAP flag, if present, MUST be set to 0 for each non-first fragment of the AU 🌸
Stream-state: specifies the state of the stream for an AU of an MPEG-4 system stream; each state is identified by a value of a modulo counter 🔄 in ISO/IEC 14496-1, MPEG-4 system streams use the AU_SequenceNumber to signal stream states 💅 when the stream state changes, the value of the stream-state MUST be incremented by one 🎀
Note: no relation is required between stream-states of different streams. 🙈
3.2.2. The Auxiliary Section 🎀
Okay so the Auxiliary Section is basically two fields stuck together: the auxiliary-data-size field, then the auxiliary-data field. Receivers MAY parse the auxiliary-data field, but they're not required to 🙈. The size field tells you how many bits of auxiliary data to expect, so you can skip right past it if you don't care. If the size field plus the data don't add up to a whole number of octets, up to 7 zero padding bits MUST be inserted right after the auxiliary data to get everything octet-aligned ✨. See Figure 4 for the full picture 🧚.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+
| auxiliary-data-size | auxiliary-data |padding bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- .. -+-+-+-+-+-+-+-+-+Figure 4: The fields in the Auxiliary Section
The length in bits of the auxiliary-data-size field is configurable with a MIME format parameter, so check out section 4.1 for that 💅. A default length of zero means the whole Auxiliary Section is absent, so no drama 🌸.
- auxiliary-data-size: specifies the length in bits of the immediately following auxiliary-data field.
- auxiliary-data: the auxiliary-data field contains data of a format not defined by this specification 👑.
3.2.3. The Access Unit Data Section 🦋
The Access Unit Data Section holds either a whole number of complete Access Units, or one single fragment of one AU. It is never empty, bestie 🫶. If there's more than one AU in there, they get mashed together into one contiguous string of octets. Peep Figure 5 🌈. The AUs inside this section MUST be in decoding order, though they don't necessarily sit next to each other when interleaving is happening 🔥.
The size and number of AUs SHOULD be adjusted so the resulting RTP packet doesn't end up bigger than the path MTU. For anything bigger, this payload format leans on the lower layers to do fragmentation, which may result in reduced performance 😭.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|AU(1) |
+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |AU(2) |
+-+-+-+-+-+-+-+-+ |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | AU(n) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|AU(n) continued|
|-+-+-+-+-+-+-+-+Figure 5: Access Unit Data Section; each AU is octet-aligned.
When you're carrying multiple AUs, the size of each one MUST be made available to the receiver 💖. If the AU size varies, then each AU's size MUST be indicated in the AU-size field of its matching AU-header. BUT, if the AU size is constant for the whole stream, this mechanism SHOULD NOT be used. Instead, the fixed size SHOULD be signaled with the MIME format parameter "constantSize", which you can read about in section 4.1 ✨.
No AU-size in the AU-header AND no constantSize parameter? That means you're getting a single AU (fragment) per RTP packet for that stream 🍓.
3.2.3.1. Fragmentation 💜
A packet SHALL carry either one or more complete Access Units, or a single fragment of an Access Unit 🎀. Fragments of the same Access Unit share the same time stamp but have different RTP sequence numbers 🧚. The marker bit in the RTP header is 1 on the last fragment of an Access Unit, and 0 on all the other fragments 💅.
3.2.3.2. Interleaving 🌈
Unless the signaled mode prohibits it, a sender MAY interleave Access Units 🦋. Receivers that can handle modes which support interleaving MUST be able to decode interleaved Access Units. No skipping this part, periodt 😤.
When a sender interleaves AUs, it has to provide enough info so the receiver can figure out the original order without any confusion, even if packets show up out of order, go missing, or get duplicated 🔥. What info is needed depends on whether the AUs have a constant time duration. AUs have a constant time duration if:
TS(i+1) - TS(i) = constant
for any i, where:
i indicates the index of the AU in the original order, and
TS(i) denotes the time stamp of AU(i)The MIME parameter "constantDuration" SHOULD be used to signal that the AUs have a constant time duration. See section 4.1 for the deets 🫶.
If the "constantDuration" parameter is present, the receiver can rebuild the original AU timing using only the RTP timestamp and the AU-Index-delta ✨. So when you send AUs of constant duration, the AU-Index, if it's there, MUST be set to the value 0 🌸. Receivers of constant duration AUs MUST use the RTP timestamp to work out the index of the first AU in the RTP packet 👑. The AU-Index-delta header and the signaled "constantDuration" are what get used to rebuild the AU timing, so that's the whole team working together 🙈.
If the "constantDuration" parameter is NOT present, senders MAY signal AUs of constant duration by coding the AU-Index with zero in each RTP packet. Without the constantDuration parameter, receivers MUST conclude that the AUs have constant duration if the AU-Index is zero in two consecutive RTP packets 💀.
Now when it's variable duration, the "constantDuration" parameter MUST NOT be present ❗. The transmitter MUST use the AU-Index to encode the index info needed for re-ordering, and the receiver MUST use that value to figure out the index of each AU in the RTP packet 🔥. The number of bits in the AU-Index field MUST be chosen so that valid index info is provided by the interleaving scheme in use, without causing problems when the AU-Index field rolls over 🦋. On top of that, the CTS-delta MUST be coded in the AU header for each non-first AU in the RTP packet, so receivers can place the AUs correctly in time 💖.
When interleaving is applied, receivers need a de-interleave buffer to put the AUs back into their correct logical, consecutive decoding order 🌈. That means computing the time stamp for each AU. For a constant time duration per AU, the time stamp of the i-th AU in an RTP packet with RTP time stamp T is calculated like this:
Timestamp[0] = T
Timestamp[i, i > 0] = T +(Sum(for k=1 to i of (AU-Index-delta[k]
+ 1))) * access-unit-durationWhen AU-Index-delta is always 0, this reduces to T + i * (access-unit-duration). That's the non-interleaved case, where the frames are consecutive in decoding order 🧚. Also worth noting: the AU-Index field (present for the first Access Unit) isn't actually needed in this calculation 🙈✨.
3.2.3.3. Constraints for Interleaving ✨
Okay bestie, packet size is a whole vibe 🎀. It has to work for both the path MTU and how much room the receiver's de-interleave buffer has, so it's gotta be picked wisely. The maximum packet size for a session SHOULD be chosen to not exceed the path MTU.
Receivers need to know how much space to set aside for de-interleaving, so senders MUST provide that information as described in this section. 💖
AUs enter the decoder in decoding order. The de-interleave buffer is what puts a shuffled stream of interleaved AUs back into decoding order 🧚. When interleaving is applied, the decoding of "early" AUs has to wait until every AU that comes before it in decoding order has shown up. So those "early" AUs get parked in the de-interleave buffer. Here's the example from Figure 6, using the interleaving pattern from section 2.5:
+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
+--+--+--+--+--+--+--+--+--+--+--+-
Storage of "early" AUs 3 3 3 3 3 3
6 6 6 6 6 6
4 4 4
7 7 7
12 12Figure 6: Storage of "early" AUs in the de-interleave buffer per interleaved AU.
AU(3) has to be handed to the decoder after AU(0), AU(1) and AU(2). Of those, AU(2) is the last to arrive from the network, so AU(3) has to hang out in the buffer until AU(2) shows up 🥹. Same deal for AU(6), which waits until AU(5) is here, while AU(4) and AU(7) wait for AU(2) and AU(5), respectively. Also, the buffer's fullness keeps changing over time. In Figure 6 it holds at most 4 AUs, but often fewer. ✨
To get a rough idea of how many resources a receiver needs for de-interleaving, we define the maximum displacement in time of an AU. For any AU(j) in the pattern, we can figure out each AU(i) with i<j that hasn't shown up yet. The maximum displacement in time of an AU is the biggest difference between the time stamp of an AU in the pattern and the time stamp of the earliest AU that is not yet present. Put another way, for a sequence of interleaved AUs:
Maximum displacement = max{TS(i) - TS(j)}
for any i and any j>i, where:
i and j indicate the index of the AU in the interleaving
pattern, and
TS denotes the time stamp of the AU.Here's Figure 7, using the same interleaving pattern from section 2.5. For each AU in the pattern, it shows the index of the earliest earlier AU that isn't present yet. So for each AU(n), you get the smallest index k (with k<n) of an AU that hasn't been delivered. A "-" means every previous AU is already present 🌸. If the AU period is constant, the maximum displacement is 5 AU periods, as you can see for AU(6) and AU(7).
+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 3| 6| 1| 4| 7| 2| 5| 8| 9|12|..
+--+--+--+--+--+--+--+--+--+--+--+-
Earliest not yet present AU - 1 1 - 2 2 - - - - 10Figure 7: For each AU in the interleaving pattern, the earliest of any earlier AUs not yet present.
When interleaving, senders MUST signal the maximum displacement in time during the session using the MIME format parameter "maxDisplacement". Details are in section 4.1. 📣
To estimate the size of the de-interleave buffer, multiply the maximum displacement by the maximum bit rate:
size(de-interleave buffer) = {(maxDisplacement) * Rate(max)} / (RTP
clock frequency),
where:
Rate(max) is the maximum bit-rate of the transported stream.Receivers can work out Rate(max) from the MIME format parameters streamType, profile-level-id, and config. Smart, right? 🧠
BUT, this calculation is only an estimate, so the actual size you need might be different 🙃. If the calculation under-estimates the buffer size, then senders, when interleaving, MUST signal the buffer size using the MIME format parameter "de-interleaveBufferSize" (see section 4.1). If the calculation over-estimates it, then senders, when interleaving, MAY signal the buffer size using that same parameter.
The signaled buffer size MUST be big enough to hold all "early" AUs at any point during the session. The minimum de-interleave buffer size is:
minimum de-interleave buffer size = max [sum {if TS(i) > TS(j) then
AU-size(i) else 0}]
for any j and any i<j, where:
i and j indicate the index of an AU in the interleaving
pattern,
TS(i) denotes the time stamp of AU(i), and
AU-size(i) denotes the size of AU(i) in number of octets.If the "de-interleaveBufferSize" parameter is present, the buffer a receiver actually uses for de-interleaving MUST be at least as large as the signaled size. If the parameter isn't there, it MUST be at least as large as the calculated size. 🫶
No matter which interleaving scheme you use, you have to analyze it to work out the right maxDisplacement value and the required de-interleave buffer size 🔍. Senders SHOULD signal values that are not larger than the strictly required values. If bigger values get signaled, the receiver ends up buffering way too much 😭.
Also, for low bit-rate material, interleaving might make packets shorter than the MTU size. Just so you know! 🦋
3.2.3.4. Crucial and Non-Crucial AUs with MPEG-4 System Data
Some Access Units with MPEG-4 system data are called "crucial" AUs. They carry information you really can't afford to lose, whether for the presentation or for the decoder 💀. At each crucial AU in an MPEG-4 system stream, the stream state changes. The stream-state MAY remain constant at non-crucial AUs. In ISO/IEC 14496-1, MPEG-4 system streams use the AU_SequenceNumber to signal stream states.
Example time! 💅 Say we have three AUs: AU1 = "Insertion of node X", AU2 = "Set position of node X", and AU3 = "Set position of node X". AU1 is crucial, because if it gets lost, AU2 can't be executed. AU2 is not crucial, though, since AU3 can still be executed even if AU2 goes missing. Iconic behaviour 👑.
When a crucial AU is (possibly) lost, the stream is corrupted. For example, if an AU gets lost and the stream state has changed by the time the next AU arrives, then the lost AU might have been crucial. Once corrupted, the stream stays corrupted until the next random access point. Losing non-crucial AUs doesn't corrupt the stream, though 🌈. When a decoder starts receiving a stream, it MUST consider the stream corrupted until it receives an AU that provides a random access point.
An AU that provides a random access point (as signaled by the RAP-flag) may or may not be crucial. Non-crucial RAP AUs are like a "repeated" random access point, there for decoders that just joined the stream or that need to restart decoding after corruption. Non-crucial RAP AUs MUST include all updates since the last crucial RAP AU.
When AUs arrive, decoders react like this:
a) If the RAP-flag is set to 1 and the stream-state changes, then the AU is a crucial RAP AU, and the AU MUST be decoded.
b) If the RAP-flag is set to 1 and the stream state does not change, then the AU is a non-crucial RAP AU. The receiver SHOULD decode it if the stream is corrupted. Otherwise, the decoder MUST ignore the AU.
c) If the RAP-flag is set to 0, then the AU MUST be decoded, unless the stream is corrupted, in which case the AU MUST be ignored.
3.3. Usage of this Specification
3.3.1. General
Using this specification means defining a mode 🎀. A mode says how to use this specification, however you think is best. Senders MUST signal the mode they're using via the MIME format parameter "mode", as described in section 4.1. This specification defines a generic mode that works for any MPEG-4 stream, plus specific modes for transporting MPEG-4 CELP and MPEG-4 AAC streams, which are defined in ISO/IEC 14496-3 [1].
When you signal this payload format using SDP [5], there's an "rtpmap" attribute involved too. The same rules apply to the rtpmap attribute in any mode that complies with this specification. The general form looks like this:
a=rtpmap:<payload type> <encoding name>/<clock rate>[/<encoding
parameters>]For audio streams, <encoding parameters> is the number of audio channels: 2 for stereo material (see RFC 2327 [5]) and 1 for mono. If no extra parameters are needed, you can leave this out for mono material, since its default value is 1. 🎧
3.3.2. The Generic Mode
The generic mode works for any MPEG-4 stream. No mode-specific constraints apply here, so you get the full flexibility of this specification to play with 🌸. The generic mode is signaled by mode=generic.
Here's an example for transporting a BIFS-Anim stream. In this one, you're allowed to pack several BIFS-Anim Access Units into a single RTP packet 📦. The AU-header has the AU-size field, the CTS-flag and, if the CTS flag is set to 1, the CTS-delta field. The AU-size and CTS-delta fields are 10 and 16 bits long, respectively. The AU-header also carries the RAP-flag and a 4-bit Stream-state. All of that adds up to an AU-header of two or four octets per BIFS-Anim AU. The RTP time stamp uses a 1 kHz clock. The media type name is video even though it's a BIFS-Anim stream, because the stream is part of an audio-visual presentation (for media type name conventions, see section 4.1). 🦋
In detail:
m=video 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/1000
a=fmtp:96 streamtype=3; profile-level-id=1807; mode=generic;
objectType=2; config=0842237F24001FB400094002C0; sizeLength=10;
CTSDeltaLength=16; randomAccessIndication=1;
streamStateIndication=4Note: The a=fmtp line has been wrapped to fit the page. It's really a single line in the SDP file. 🙈
The hexadecimal value of the "config" parameter is the BIFSConfiguration() as defined in ISO/IEC 14496-1. BIFSConfiguration() specifies that the BIFS stream is a BIFS-Anim stream. For the description of MIME parameters, see section 4.1. ✨
3.3.3. Constant Bit-rate CELP ✨
Okay bestie, this mode is signaled by mode=CELP-cbr 🎀. You can pack one or more complete CELP frames of a fixed size into a single RTP packet, but interleaving MUST NOT be used with this mode 🙅♀️. The RTP payload is just one or more CELP frames stuck together, and they're all the exact same size 💖. CELP frames MUST NOT be fragmented here, and both the AU Header Section and the Auxiliary Section MUST be empty 💅.
The MIME format parameter constantSize MUST be provided to say how long each CELP frame is 📏.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/16000/1
a=fmtp:96 streamtype=5; profile-level-id=14; mode=CELP-cbr; config=
440E00; constantSize=27; constantDuration=240Note: the a=fmtp line got wrapped so it would fit on the page 🙈. In the actual SDP file it's all one line.
The hex value of the "config" parameter is the AudioSpecificConfig() from ISO/IEC 14496-3 🧚. That config describes a mono CELP stream at a 16 kHz sampling rate, a fixed bitrate of 14.4 kb/s, and 6 sub-frames per CELP frame. For the MIME parameter descriptions, see section 4.1 📚.
3.3.4. Variable Bit-rate CELP 🦋
This mode is signaled by mode=CELP-vbr. Here you can send one or more complete CELP frames of variable size in one RTP packet, with OPTIONAL interleaving 🌈. The biggest possible AU-size value is larger than the biggest CELP frame, which is a whole vibe. CELP frames are super tiny, so fragmentation isn't supported at all, and CELP frames MUST NOT be fragmented in this mode 🙅♀️.
The RTP payload here is the AU Header Section, then one or more CELP frames concatenated together. The Auxiliary Section MUST be empty 🫶. Every CELP frame in the payload MUST have a one octet AU-header in the AU Header Section that gives:
a) the size of each CELP frame in the payload, and
b) index info for working out the sequence (and so the timing) of each CELP frame.
Here's the bit math 🧮: since transporting CELP frames means the AU-size field has to be coded with 6 bits, this mode gives 6 bits to AU-size and 2 bits to the AU-Index(-delta) field. Each AU-Index field MUST be coded with the value 0. In the AU Header Section, the concatenated AU-headers get the 16-bit AU-headers-length field in front of them, as specified in section 3.2.1 🎀.
On top of the required MIME format parameters, these MUST be present: sizeLength, indexLength, and indexDeltaLength. CELP frames always have a fixed duration per Access Unit. When you interleave in this mode, that duration MUST be signaled with the MIME format parameter constantDuration. And the parameter maxDisplacement MUST be present whenever you interleave 🔥.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/16000/1
a=fmtp:96 streamtype=5; profile-level-id=14; mode=CELP-vbr; config=
440F20; sizeLength=6; indexLength=2; indexDeltaLength=2;
constantDuration=160; maxDisplacement=5Note: the a=fmtp line got wrapped to fit the page 💖. It's still a single line in the SDP file.
The hex value of "config" is the AudioSpecificConfig() from ISO/IEC 14496-3. This one describes a mono CELP stream at a 16 kHz sampling rate, with a bitrate that wanders between 13.9 and 16.2 kb/s, and 4 sub-frames per CELP frame ✨. For the MIME parameter descriptions, see section 4.1.
3.3.5. Low Bit-rate AAC 🍓
This mode is signaled by mode=AAC-lbr. It carries one or more complete AAC frames of variable size 🎶. The AAC frames are allowed to be interleaved, so receivers MUST support de-interleaving 👑. The biggest an AAC frame can be in this mode is 63 octets. AAC frames MUST NOT be fragmented, which means encoders MUST make sure every AAC frame is at most 63 octets 😭.
The payload setup here is the same as variable bit-rate CELP in 3.3.4 🫶. The RTP payload is the AU Header Section followed by concatenated AAC frames, and the Auxiliary Section MUST be empty. Each AAC frame needs a one octet AU-header that MUST provide:
a) the size of each AAC frame in the payload, and
b) index info for computing the sequence (and so the timing) of each AAC frame.
In the AU-header Section, the concatenated AU-headers MUST be preceded by the 16-bit AU-headers-length field, as specified in section 3.2.1 💅.
Again, on top of the required MIME format parameters, these MUST be present: sizeLength, indexLength, and indexDeltaLength. AAC frames always have a fixed duration per Access Unit. If you interleave in this mode, that duration MUST be signaled with constantDuration. And maxDisplacement MUST be present when interleaving 🙈.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/22050/1
a=fmtp:96 streamtype=5; profile-level-id=14; mode=AAC-lbr; config=
1388; sizeLength=6; indexLength=2; indexDeltaLength=2;
constantDuration=1024; maxDisplacement=5Note: the a=fmtp line got wrapped to fit the page 🌸. It's one line in the SDP file.
The hex value of "config" is the AudioSpecificConfig(), as defined in ISO/IEC 14496-3. This one describes a mono AAC stream at a 22.05 kHz sampling rate. For the MIME parameter descriptions, see section 4.1 🎀.
3.3.6. High Bit-rate AAC 👑
This mode is signaled by mode=AAC-hbr. It carries variable size AAC frames. In a single RTP packet you get either one or more complete AAC frames, or a single fragment of an AAC frame 🔥. Interleaving is allowed here too, so receivers MUST support de-interleaving 💜. The biggest AAC frame in this mode is 8191 octets, which is huge compared to the others 🌈.
The RTP payload is the AU Header Section, followed by either one AAC frame, several concatenated AAC frames, or one fragmented AAC frame. The Auxiliary Section MUST be empty. For every AAC frame in the payload, there MUST be an AU-header in the AU Header Section that provides:
a) the size of each AAC frame in the payload, and
b) index info for computing the sequence (and so the timing) of each AAC frame.
Coding the max AAC frame size takes 13 bits 🧠. So this configuration gives 13 bits to AU-size and 3 bits to the AU-Index(-delta) field, which makes each AU-header 2 octets. Each AU-Index field MUST be coded with the value 0. In the AU Header Section, the concatenated AU-headers MUST be preceded by the 16-bit AU-headers-length field, as specified in section 3.2.1 ✨.
Once more with the required MIME format parameters: sizeLength, indexLength, and indexDeltaLength MUST be present. AAC frames always have a fixed duration per Access Unit, and when interleaving, that duration MUST be signaled with constantDuration. The parameter maxDisplacement MUST be present when interleaving 🦋.
For example:
m=audio 49230 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/48000/6
a=fmtp:96 streamtype=5; profile-level-id=16; mode=AAC-hbr;
config=11B0; sizeLength=13; indexLength=3;
indexDeltaLength=3; constantDuration=1024Note: the a=fmtp line got wrapped to fit the page 🙈. It's one line in the SDP file.
The hex value of "config" is the AudioSpecificConfig(), as defined in ISO/IEC 14496-3. This one describes a 5.1 channel AAC stream at a 48 kHz sampling rate. For the MIME parameter descriptions, see section 4.1 💖.
3.3.7. Additional Modes 🧚
This spec only defines the modes in sections 3.3.2 through 3.3.6 🌸. More modes are expected to show up in future RFCs. Each additional mode MUST be in full compliance with this specification 🫶.
Any new mode MUST be defined so that an implementation with all the features of this spec can decode the payload format for that new mode 🔥. That's why a mode MUST NOT specify new default values for MIME parameters. Specifically, MIME parameters that configure the RTP payload MUST be present (unless they have the default value), even when they're redundant because the mode gives a parameter a fixed value 💅. A mode may also define that some MIME parameters are required instead of optional, that some MIME parameters have fixed values (or ranges), and that there are rules restricting its usage 👑.
4. IANA Considerations ✨
This section is all about the MIME types and names for this payload format. Section 4.1 registers the MIME types, as per RFC 2048 [3] 📝.
This format may need extra info about the mapping so the receiver can figure things out. That info comes through parameters, which are described in the next section 🎀.
4.1. MIME Type Registration 💜
MIME media type name: "video" or "audio" or "application"
"video" MUST be used for MPEG-4 Visual streams (ISO/IEC 14496-2) or MPEG-4 Systems streams (ISO/IEC 14496-1) that carry info needed for an audio/visual presentation 📺.
"audio" MUST be used for MPEG-4 Audio streams (ISO/IEC 14496-3) or MPEG-4 Systems streams that carry info needed for an audio-only presentation 🎧.
"application" MUST be used for MPEG-4 Systems streams (ISO/IEC 14496-1) that do something other than audio/visual presentation, for example when MPEG-J (Java) streams are sent 🧩.
Depending on the payload configuration you need, MIME format parameters might need to be available to the receiver 🌈. You get them through the parameters in the next section. There are required parameters and optional parameters ✨.
Optional parameters come in two flavours: general parameters and configuration parameters 💅. The configuration parameters set up the fields in the AU Header section and the auxiliary section. If a configuration parameter is missing, that's the same as its field being set to its default value, which is always zero. If all the configuration parameters are missing, you get the default "basic" configuration, with an empty AU-header section and an empty auxiliary section in each RTP packet 🍓.
MIME subtype name: mpeg4-generic
Required parameters:
MIME format parameters aren't case dependent 🙃. For clarity, though, this spec uses both upper and lower case in the parameter names.
- streamType: This is the integer that says what type of MPEG-4 stream is being carried 🎶. Its coding matches the values of streamType in Table 9 (streamType Values) in ISO/IEC 14496-1.
- profile-level-id: This is a decimal representation of the MPEG-4 Profile Level indication 👑. It MUST be used in the capability exchange or session set-up procedure to say which MPEG-4 Profile and Level combination the relevant MPEG-4 media codec can handle 💖.
For MPEG-4 Audio streams, this parameter is the decimal value from Table 5 (audioProfileLevelIndication Values) in ISO/IEC 14496-1. It tells you which MPEG-4 Audio tool subsets you need to decode the audio stream 🔥.
For MPEG-4 Visual streams, this parameter is the decimal value from Table G-1 (FLC table for profile and level indication) of ISO/IEC 14496-2 [1], and it tells you which MPEG-4 Visual tool subsets are required to decode the visual stream 🎀
For BIFS streams, the value is the decimal number you get from (SPLI + 256*GPLI), where:
- SPLI is the decimal value from Table 4 in ISO/IEC 14496-1 with the applied sceneProfileLevelIndication;
- GPLI is the decimal value from Table 7 in ISO/IEC 14496-1 with the applied graphicsProfileLevelIndication.
For MPEG-J streams, it's the decimal value from table 13 (MPEGJProfileLevelIndication) in ISO/IEC 14496-1, which tells you the profile and level of the MPEG-J stream 🧚
For OD streams, it's the decimal value from table 3 (ODProfileLevelIndication) in ISO/IEC 14496-1, which tells you the profile and level of the OD stream 💜
For IPMP streams, the value is either the decimal value 0, meaning an unspecified profile and level, or a value larger than zero, meaning an MPEG-4 IPMP profile and level as defined in a future MPEG-4 specification 🌸
For Clock Reference streams and Object Content Info streams, the value is the decimal value zero, which means profile and level info is conveyed through the OD framework 🫶
config:
okay so config is the media payload configuration written as a hexadecimal octet string 💖 The configuration data gets mapped onto that string MSB-first, so the first bit of the configuration data SHALL be located at the MSB of the first octet. If you need octet-alignment, the last octet can get up to 7 zero-valued padding bits right after the configuration data ✨
For MPEG-4 Audio streams, config is the audio object type specific decoder configuration data AudioSpecificConfig(), as defined in ISO/IEC 14496-3. For Structured Audio, the AudioSpecificConfig() may be conveyed by other means that this specification doesn't define. If the AudioSpecificConfig() is conveyed by other means for Structured Audio, then the config MUST be a quoted empty hexadecimal octet string, like this: config="" 🙈
Note that a future mode of using this RTP payload format for Structured Audio may define those other means 🦋
For MPEG-4 Visual streams, config is the MPEG-4 Visual configuration information from subclause 6.2.1, Start codes of ISO/IEC 14496-2. The configuration info indicated by this parameter SHALL be the same as the configuration info in the corresponding MPEG-4 Visual stream, except for first-half-vbv-occupancy and latter-half-vbv-occupancy, if they exist. Those two can vary in the repeated configuration info inside an MPEG-4 Visual stream (see 6.2.1 Start codes of ISO/IEC 14496-2) 💅
For BIFS streams, this is the BIFSConfig() information as defined in ISO/IEC 14496-1. Version 1 of BIFSConfig is defined in section 9.3.5.2, and version 2 is defined in section 9.3.5.3. The MIME format parameter objectType signals which version of BIFSConfig you're looking at 👑
For IPMP streams, this is either a quoted empty hexadecimal octet string, which means there's no decoder configuration info (config=""), or the IPMPConfiguration() as will be defined in a future MPEG-4 IPMP specification 🍓
For Object Content Info (OCI) streams, this is the OCIDecoderConfiguration() information of the OCI stream, as defined in section 8.4.2.4 in ISO/IEC 14496-1 ✨
For OD streams, Clock Reference streams and MPEG-J streams, this is a quoted empty hexadecimal octet string (config=""), since no decoder configuration info is required 🌈
mode:
The mode is the one in which this specification is used. These modes can be signaled:
mode=generic,
mode=CELP-cbr,
mode=CELP-vbr,
mode=AAC-lbr and
mode=AAC-hbr.Other modes are expected to be defined in future RFCs. Also check section 3.3.7 and 4.2 of RFC 3640 💖
Optional general parameters:
objectType:
This is the decimal value from Table 8 in ISO/IEC 14496-1, and it tells you the value of the objectTypeIndication of the transported stream 🎀 For BIFS streams, this parameter MUST be present to signal the version of BIFSConfiguration(). Heads up that objectTypeIndication may signal a non-MPEG-4 stream, and the RTP payload format defined in this document may not be suitable for carrying a stream that isn't defined by MPEG-4 😭 The objectType parameter SHOULD NOT be set to a value that signals a stream this payload format can't carry 🙈
constantSize:
This is the constant size in octets of each Access Unit for this stream. The constantSize and the sizeLength parameters MUST NOT be simultaneously present 💀
constantDuration:
This is the constant duration of each Access Unit for this stream, measured with the same units as the RTP time stamp ⭐
maxDisplacement:
This is the decimal representation of the maximum displacement in time of an interleaved AU, as defined in section 3.2.3.3, expressed in units of the RTP time stamp clock 🧚
This parameter MUST be present when interleaving is applied 🔥
de-interleaveBufferSize:
This is the decimal representation, in number of octets, of the size of the de-interleave buffer described in section 3.2.3.3. When interleaving, this parameter MUST be present if the calculation of the de-interleave buffer size given in 3.2.3.3, based on maxDisplacement and rate(max), under-estimates the size of the de-interleave buffer. If that calculation does not under-estimate the size, then the de-interleaveBufferSize parameter SHOULD NOT be present 🦋
Optional configuration parameters:
sizeLength:
This is the number of bits used to encode the AU-size field in the AU-header. The sizeLength and the constantSize parameters MUST NOT be simultaneously present 💅
indexLength:
This is the number of bits used to encode the AU-Index in the first AU-header. The default value of zero means the AU-Index field is absent from each first AU-header 🌸
indexDeltaLength:
This is the number of bits used to encode the AU-Index-delta field in any non-first AU-header. The default value of zero means the AU-Index-delta field is absent from each non-first AU-header 🫶
CTSDeltaLength:
This is the number of bits used to encode the CTS-delta field in the AU-header 🍓
DTSDeltaLength:
This is the number of bits used to encode the DTS-delta field in the AU-header ✨
randomAccessIndication:
This is a decimal value of zero or one, showing whether the RAP-flag is present in the AU-header. A value of one means the RAP-flag is present, and the default of zero means it's absent 👑
streamStateIndication:
This is the number of bits used to encode the Stream-state field in the AU-header. This parameter MAY be present when transporting MPEG-4 system streams, and SHALL NOT be present for MPEG-4 audio and MPEG-4 video streams 🌈
auxiliaryDataSizeLength:
This is the number of bits used to encode the auxiliary-data-size field 💜
Applications MAY use more parameters, in addition to the ones defined above. Each additional parameter MUST be registered with IANA so there's no clash of names 🎀 Each additional parameter MUST also be accompanied by a specification in the form of an RFC, MPEG standard, or other permanent and readily available reference (the "Specification Required" policy defined in RFC 2434 [6]) 🧚 Receivers MUST tolerate the presence of such additional parameters, but these parameters SHALL NOT impact the decoding of receivers that comply with this specification 🫶
Encoding considerations:
This MIME subtype is defined for RTP transport only. System bitstreams MUST be generated according to MPEG-4 Systems specifications (ISO/IEC 14496-1). Video bitstreams MUST be generated according to MPEG-4 Visual specifications (ISO/IEC 14496-2). Audio bitstreams MUST be generated according to MPEG-4 Audio specifications (ISO/IEC 14496-3). The RTP packets MUST be packetized according to the RTP payload format defined in RFC 3640 💖
Security considerations:
As defined in section 5 of RFC 3640 🔥
Interoperability considerations:
MPEG-4 has a large and rich set of tools for coding visual objects 🌈 For effective implementation, subsets of the MPEG-4 tool sets have been provided for specific applications. These subsets, called 'Profiles', limit the size of the tool set a decoder has to implement. To restrict computational complexity, one or more 'Levels' are set for each Profile. A Profile@Level combination allows:
- a codec builder to implement only the subset of the standard they need, while still interworking with other MPEG-4 devices that implement the same combination, and
- checking whether MPEG-4 devices comply with the standard ('conformance testing').
A stream SHALL be compliant with the MPEG-4 Profile@Level specified by the parameter "profile-level-id". Interoperability between a sender and a receiver is achieved by specifying the parameter "profile-level-id" in MIME content. In the capability exchange/announcement procedure, this parameter may mutually be set to the same value 💜
Published specification:
The specs for MPEG-4 streams are presented in ISO/IEC 14496-1, 14496-2, and 14496-3. The RTP payload format is described in RFC 3640 🦋
Applications which use this media type:
Multimedia streaming and conferencing tools 🎀
Additional information: none
Magic number(s): none
File extension(s): None. A file format with the extension .mp4 has been defined for MPEG-4 content, but it isn't directly correlated with this MIME type, whose sole purpose is RTP transport 🌸
Macintosh File Type Code(s): none
Person & email address to contact for further information: Authors of RFC 3640, IETF Audio/Video Transport working group ✨
Intended usage: COMMON
Author/Change controller: Authors of RFC 3640, IETF Audio/Video Transport working group 🫶
4.2. Registration of Mode Definitions with IANA ✨
This specification can be used in a number of modes. The mode of operation is signaled using the "mode" MIME parameter, with the initial set of values specified in section 4.1. New modes may be defined at any time, as described in section 3.3.7. These modes MUST be registered with IANA, to ensure that there is not a clash of names 🎀
A new mode registration MUST be accompanied by a specification in the form of an RFC, MPEG standard, or other permanent and readily available reference (the "Specification Required" policy defined in RFC 2434 [6]) 🧚
4.3. Concatenation of Parameters 🌈
Multiple parameters SHOULD be expressed as a MIME media type string, in the form of a semicolon-separated list of parameter=value pairs (for parameter usage examples, see sections 3.3.2 up to 3.3.6) 💅
4.4. Usage of SDP 🧚
4.4.1. The a=fmtp Keyword
Okay so, the idea is that these parameters usually travel with an SDP message 💌 [5]. That message could reach the client in reply to an RTSP DESCRIBE [8] or come over SAP [11]. When that happens, the (a=fmtp) keyword MUST be used as described in RFC 2327 [5], section 6, and the syntax looks like this:
a=fmtp:<format> <parameter name>=<value>[; <parameter name>=<value>]🎀 Super clean, right?
5. Security Considerations 🔐
Okay, security time, and I'm being so serious about it 🫶. RTP packets that use the payload format in this spec are subject to the security considerations in the RTP specification [2]. That basically means confidentiality of the media streams comes from encryption 🔒. The data compression is applied end-to-end, so you can encrypt the compressed data too, and the two operations don't clash 💅. The packet processing complexity of this payload type (not counting media data processing) doesn't have any big non-uniformity on the receiver side, so it won't cause a denial-of-service threat 🙌.
BUT. It is possible to inject non-compliant MPEG streams (Audio, Video, and Systems) so the receiver/decoder's buffers get overloaded 😭. That could mess with the receiver's functionality or even crash it. This is especially true for end-to-end systems like MPEG, where the buffer models are precisely defined.
MPEG-4 Systems supports stream types that run commands on the terminal, like OD commands, BIFS commands, and so on, plus programmatic content like MPEG-J (Java(TM) Byte Code) and MPEG-4 scripts 💻. Using any of these in a way that isn't MPEG compliant could crash the receiver or make it temporarily unavailable 💀. Senders that transport MPEG-4 content SHOULD ensure that the content is MPEG compliant, as defined in the compliance part of IEC/ISO 14496 [1]. Receivers that support MPEG-4 content should prevent malfunctioning when the content isn't MPEG compliant 🛡️.
Authentication mechanisms can be used to validate the sender and the data, which helps prevent security problems caused by non-compliant malignant MPEG-4 streams 🔑.
In ISO/IEC 14496-1, there's a security model for MPEG-4 Systems streams that carry MPEG-J access units made of Java(TM) classes and objects 🧚. MPEG-J defines a set of Java APIs and a secure execution model. MPEG-J content can call this set of APIs and Java(TM) methods from a set of Java packages supported on the receiver, but only inside the defined security model. Under that model, downloaded byte code is forbidden to load libraries, define native methods, start programs, read or write files, or read system properties 🚫. Receivers can build intelligent filters to validate the buffer requirements, or to check parametric (OD, BIFS, etc.) or programmatic (MPEG-J, MPEG-4 scripts) commands in the streams. Heads up though, this can increase the complexity significantly 🙈.
Implementors of MPEG-4 streaming over RTP who also implement MPEG-4 scripts (a subset of ECMAScript) MUST ensure that the action of such scripts is limited solely to the domain of the single presentation they live in. That means no session-to-session communication, no access to local resources and storage, and so on 🚧. Loading static network-located resources (such as media) into the presentation should be permitted, but network access by scripts MUST be restricted to that (media) download.
6. Acknowledgements 💜
This document evolved into RFC 3640 after several revisions 🌸. Huge thanks to contributions from people in the ISMA forum, the IETF AVT Working Group, and the 4-on-IP ad-hoc group within MPEG. The authors wanted to thank everyone involved, especially Andrea Basso, Stephen Casner, M. Reha Civanlar, Carsten Herpel, John Lazaro, Zvi Lifshitz, Young-kwon Lim, Alex MacAulay, Bill May, Colin Perkins, Dorairaj V and Stephan Wenger for their valuable comments and support 🫶✨.
APPENDIX: Usage of this Payload Format 📚
Appendix A. Interleave Analysis 🦋
A. Examples of Delay Analysis with Interleave
A.1. Introduction
Interleaving issues get discussed in this appendix 🧩. There are some general notes on de-interleaving and error concealment, plus a bunch of interleaving patterns. We look at those patterns mainly to figure out how big the de-interleave buffer needs to be and the maximum displacement of access units in time ⏱️. The maximum displacement is given as an access unit count in these examples, just to make it easier to read. In real streams, it's signaled in units of the RTP time stamp clock.
A.2. De-interleaving and Error Concealment
This appendix doesn't go into the details of de-interleaving and error concealment, because controlling the AU decoding and error concealment process has very little to do with interleaving 🤷♀️. If the next AU to be decoded is present and there's enough storage for the decoded AU, decode it right away. If not, wait. When the decoding deadline hits (that's the time decoding has to begin in order to finish before the AU is presented), or if the decoder is some hardware with a constant delay between starting decoding of an AU and presenting that AU, then decoding must begin at that deadline time ⏰.
If the next AU to be decoded isn't there when the decoding deadline arrives, then that AU is lost, so the receiver has to take whatever error concealment measures it deems appropriate 😬. The play-out delay may need to be adjusted at that point, especially if other AUs have also missed their deadlines recently. Or, if it was a momentary delay and keeping the latency is important, then the receiver should minimize the glitch and keep going with the next AU ✨.
A.3. Simple Group Interleave
A.3.1. Introduction
A regular interleave looks like this: packets get formed into groups 📦. If the 'stride' of the interleave (the distance between interleaved AUs) is N, then packet 0 could contain AU(0), AU(N), AU(2N), and so on. Packet 1 could contain AU(1), AU(1+N), AU(1+2N), and so on. If there are M access units in a packet, then there are M*N access units in the group 👯.
Here's an example with N=M=3. It's the same example as in section 2.5, and a fixed time duration per Access Unit is assumed 🎀:
Packet Time stamp Carried AUs AU-Index, AU-Index-delta
P(0) T[0] 0, 3, 6 0, 2, 2
P(1) T[1] 1, 4, 7 0, 2, 2
P(2) T[2] 2, 5, 8 0, 2, 2
P(3) T[9] 9,12,15 0, 2, 2In this example, the AU-Index is in the first AU-header and coded with the value 0, as required for fixed duration AUs 💖. The position of the first AU of each packet within the group is defined by the RTP time stamp. The AU-Index-delta field shows where the later AUs sit relative to the first AU in the packet. All AU-Index-delta fields are coded with the value N-1, which is 2 in this example. So the RTP time stamp and the AU-Index-delta together are how you rebuild the original order 🧠. Also see section 3.2.3.2.
A.3.2. Determining the De-interleave Buffer Size
For the regular pattern in this example, Figure 6 in section 3.2.3.3 shows that the de-interleave buffer stores at most 4 AUs 📊. A de-interleaveBufferSize value that's at least equal to the total number of octets of any 4 "early" AUs stored at the same time may be signaled.
A.3.3. Determining the Maximum Displacement
For the regular pattern in this example, Figure 7 in section 3.3 shows that the maximum displacement in time equals 5 AU periods 🕰️. So the minimum maxDisplacement value that MUST be signaled is 5 AU periods. If each AU has the same size, this maxDisplacement value over-estimates the de-interleave buffer size by one AU. But heads up: with variable AU sizes, the total size of any 4 "early" AUs that must be stored at the same time may exceed maxDisplacement times the maximum bitrate. In that case the de-interleaveBufferSize must be signaled 🚨.
A.4. More Subtle Group Interleave
A.4.1. Introduction
Here's another example of forming packets with group interleave 🌈. In this one, the packets are formed so that losing two subsequent RTP packets doesn't mean losing two subsequent AUs 🙌. Note that in this example, the RTP time stamps of packet 3 and packet 4 are earlier than the RTP time stamps of packets 1 and 2, respectively. A fixed time duration per Access Unit is assumed.
Packet Time stamp Carried AUs AU-Index, AU-Index-delta
0 T[0] 0, 5 0, 4
1 T[2] 2, 7 0, 4
2 T[4] 4, 9 0, 4
3 T[1] 1, 6 0, 4
4 T[3] 3, 8 0, 4
5 T[10] 10, 15 0, 4
and so on ..In this example, the AU-Index is in the first AU-header and coded with the value 0, as required for AUs with a fixed duration 🍓. To rebuild the original order, the RTP time stamp and the AU-Index-delta (coded with the value 4) are used. See also section 3.2.3.2.
A.4.2. Determining the De-interleave Buffer Size
From Figure 8, you can tell that at most 5 "early" AUs need to be stored 🧁. If the AUs are of constant size, then that value equals 5 times the AU size. The minimum size of the de-interleave buffer equals the maximum total number of octets of the "early" AUs that need to be stored at the same time. That gives the minimum value of the de-interleaveBufferSize that may be signaled.
+--+--+--+--+--+--+--+--+--+--+
Interleaved AUs | 0| 5| 2| 7| 4| 9| 1| 6| 3| 8|
+--+--+--+--+--+--+--+--+--+--+
- - 5 - 5 - 2 7 4 9
7 4 9 5
"Early" AUs 5 6
7 7
9 9Figure 8: Storage of "early" AUs in the de-interleave buffer per interleaved AU.
A.4.3. Determining the Maximum Displacement
From Figure 9, you can see that the maximum displacement in time equals 8 AU periods 🕐. So the minimum maxDisplacement value to be signaled is 8 AU periods.
+--+--+--+--+--+--+--+--+--+--+
Interleaved AUs | 0| 5| 2| 7| 4| 9| 1| 6| 3| 8|
+--+--+--+--+--+--+--+--+--+--+
Earliest not yet present AU - 1 1 1 1 1 - 3 - -Figure 9: For each AU in the interleaving pattern, the earliest of any earlier AUs not yet present.
If each AU has the same size, the maxDisplacement value found over-estimates the de-interleave buffer size by three AUs 🥹. However, with variable AU sizes, the total size of any 5 "early" AUs stored at the same time may exceed maxDisplacement times the maximum bitrate. In that case the de-interleaveBufferSize must be signaled 💅.
A.5. Continuous Interleave
A.5.1. Introduction ✨
okay so continuous interleave is literally my favourite 🦋 once the scheme is 'primed', the number of AUs in a packet goes over the 'stride' (the distance between them) 🔥 that shortens the buffering needed, smooths out the data-flow, and gives slightly bigger packets, so the overhead is lower for the same interleave 💅 ngl it's giving efficiency.
for example, here's a continuous interleave also over a stride of 3 AUs, but with 4 AUs per packet, for a run of 20 AUs 🌸 it shows both how the scheme 'starts up' and how it finishes 🧚 again, the example assumes a fixed time duration per Access Unit.
Packet Time-stamp Carried AUs AU-Index, AU-Index-delta
0 T[0] 0 0
1 T[1] 1 4 0 2
2 T[2] 2 5 8 0 2 2
3 T[3] 3 6 9 12 0 2 2 2
4 T[7] 7 10 13 16 0 2 2 2
5 T[11] 11 14 17 20 0 2 2 2
6 T[15] 15 18 0 2
7 T[19] 19 0in this example, the AU-Index is present in the first AU-header and coded with the value 0, which is needed for AUs with a fixed duration 🎀 to rebuild the original order, the RTP time stamp and the AU-Index-delta (coded with the value 2) get used 🫶 see also 3.2.3.2. also note that this example has RTP time-stamps in increasing order 💜
A.5.2. Determining the De-interleave Buffer Size 🍓
for this example, the de-interleave buffer size can be worked out from Figure 10 👑 the maximum number of "early" AUs is 3. if the AUs are of constant size, then the de-interleave buffer size equals 3 times the AU size. compared to the example in A.2, for constant size AUs the de-interleave buffer size goes down from 4 to 3 times the AU size, and the 'stride' stays the same 😭 bestie that's a whole glow-up.
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 1| 4| 2| 5| 8| 3| 6| 9|12| 7|10|13|16|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
- - - 4 - - 4 8 - - 8 12 - -
5 9
"Early" AUs 8 12Figure 10: Storage of "early" AUs in the de-interleave buffer per interleaved AU.
A.5.3. Determining the Maximum Displacement 🦋
for this example, the maximum displacement has a value of 5 AU periods. see Figure 11 ✨ compared to the example in A.2, the maximum displacement does not decrease, though in fact less de-interleave buffering is required 🙈 it's giving "same vibe, less stuff to carry."
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
Interleaved AUs | 0| 1| 4| 2| 5| 8| 3| 6| 9|12| 7|10|13|16|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
Earliest not yet
present AU - - 2 - 3 3 - - 7 7 - - 11 11Figure 11: For each AU in the interleaving pattern, the earliest of any earlier AUs not yet present
References 📚
Normative References 💅
[1] ISO/IEC International Standard 14496 (MPEG-4); "Information technology - Coding of audio-visual objects", January 2000
[2] Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", RFC 3550, July 2003.
[3] Freed, N., Klensin, J. and J. Postel, "Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures", BCP 13, RFC 2048, November 1996.
[4] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[5] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998.
[6] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998.
Informative References 🌈
[7] Hoffman, D., Fernando, G., Goyal, V. and M. Civanlar, "RTP Payload Format for MPEG1/MPEG2 Video", RFC 2250, January 1998.
[8] Schulzrinne, H., Rao, A. and R. Lanphier, "Real-Time Session Protocol (RTSP)", RFC 2326, April 1998.
[9] Perkins, C. and O. Hodson, "Options for Repair of Streaming Media", RFC 2354, June 1998.
[10] Schulzrinne, H. and J. Rosenberg, "An RTP Payload Format for Generic Forward Error Correction", RFC 2733, December 1999.
[11] Handley, M., Perkins, C. and E. Whelan, "Session Announcement Protocol", RFC 2974, October 2000.
[12] Kikuchi, Y., Nomura, T., Fukunaga, S., Matsui, Y. and H. Kimata, "RTP Payload Format for MPEG-4 Audio/Visual Streams", RFC 3016, November 2000.
Authors' Addresses 🧚
Jan van der Meer
Philips Electronics
Prof Holstlaan 4
Building WAH-1
5600 JZ Eindhoven
Netherlands
EMail: jan.vandermeer@philips.com
David Mackie
Apple Computer, Inc.
One Infinite Loop, MS:302-3KS
Cupertino CA 95014
EMail: dmackie@apple.com
Viswanathan Swaminathan
Sun Microsystems Inc.
2600 Casey Avenue
Mountain View, CA 94043
EMail: viswanathan.swaminathan@sun.com
David Singer
Apple Computer, Inc.
One Infinite Loop, MS:302-3MT
Cupertino CA 95014
EMail: singer@apple.com
Philippe Gentric
Philips Electronics
51 rue Carnot
92156 Suresnes
France
EMail: philippe.gentric@philips.comFull Copyright Statement 🔥
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assignees.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement 🫶
Funding for the RFC Editor function is currently provided by the Internet Society. 💖
📦 where this page comes from
- ietf: rfc3640 2003-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms