RFC 6416

RTP Payload Format for MPEG-4 Audio/Visual Streams

📅 published October 2011, from ietf

RFC 6416: RTP Payload Format for MPEG-4 Audio/Visual Streams 🎀

Okay bestie, this one is a whole vibe! 💖 Here's the title block tea:

  • Authors: M. Schmidt (Dolby Laboratories), F. de Bont (Philips Electronics), S. Doehla (Fraunhofer IIS), and J. Kim (LG Electronics Inc.) 👑
  • Date: October 2011 📅
  • Category: Standards Track, so this is an actual Internet Standard 🔥
  • ISSN: 2070-1721
  • Obsoletes: RFC 3016. This one replaces RFC 3016, so it's the new main character 💅
  • Updates: nothing is listed as updated here

Abstract ✨

Okay so this doc describes how to put MPEG-4 Audio and MPEG-4 Visual bitstreams into Real-time Transport Protocol (RTP) payloads, and it does that without using MPEG-4 Systems 🦋. It obsoletes RFC 3016 and includes a summary of what changed from RFC 3016, plus a discussion of backward compatibility with it. It's a necessary revision because RFC 3016 didn't line up with the 3GPP Packet-switched Streaming Service (PSS) specification when it came to the RTP payload format for MPEG-4 Audio. Iconic correction energy 🙈

For directly mapping MPEG-4 Audio/Visual bitstreams onto RTP packets, the doc gives specs for how RTP header fields get used, and it also sets out fragmentation rules 🧚. It covers Media Type registration and how the Session Description Protocol (SDP) gets used too 💜.

Real talk: the audio payload format here has some limitations around signaling audio codec parameters for the required multiplexing format 💀. So new system designs should utilize RFC 3640, which doesn't have those restrictions ✨. Even so, this revision of RFC 3016 exists to update and complete the spec and help implementations interoperate 🫶.

Status of This Memo ✨

This is an Internet Standards Track document 🔥.

It's a product of the Internet Engineering Task Force (IETF), and it represents the consensus of the IETF community 🌈. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). For more on Internet Standards, check Section 2 of RFC 5741 🎀.

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc6416 🦋.

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved 💖.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please actually read these, bestie, because they spell out your rights and restrictions 🙈. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License ⚖️.

This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of that material may not have granted the IETF Trust the right to allow modifications outside the IETF Standards Process. So without an adequate license from those copyright holders, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process. The exceptions are formatting it for publication as an RFC or translating it into languages other than English 🌸.

Table of Contents ✨

1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
  1.1.  MPEG-4 Visual RTP Payload Format . . . . . . . . . . . . .  4
  1.2.  MPEG-4 Audio RTP Payload Format  . . . . . . . . . . . . .  5
  1.3.  Interoperability with RFC 3016 . . . . . . . . . . . . . .  6
  1.4.  Relation with RFC 3640 . . . . . . . . . . . . . . . . . .  6
2.  Definitions and Abbreviations  . . . . . . . . . . . . . . . .  6
3.  Clarifications on Specifying Codec Configurations for
    MPEG-4 Audio . . . . . . . . . . . . . . . . . . . . . . . . .  7
4.  LATM Restrictions for RTP Packetization of MPEG-4 Audio
    Bitstreams . . . . . . . . . . . . . . . . . . . . . . . . . .  7
5.  RTP Packetization of MPEG-4 Visual Bitstreams  . . . . . . . .  8
  5.1.  Use of RTP Header Fields for MPEG-4 Visual . . . . . . . .  9
  5.2.  Fragmentation of MPEG-4 Visual Bitstream . . . . . . . . . 10
  5.3.  Examples of Packetized MPEG-4 Visual Bitstream . . . . . . 11
6.  RTP Packetization of MPEG-4 Audio Bitstreams . . . . . . . . . 15
  6.1.  RTP Packet Format  . . . . . . . . . . . . . . . . . . . . 15
  6.2.  Use of RTP Header Fields for MPEG-4 Audio  . . . . . . . . 16
  6.3.  Fragmentation of MPEG-4 Audio Bitstream  . . . . . . . . . 17
7.  Media Type Registration for MPEG-4 Audio/Visual Streams  . . . 17
  7.1.  Media Type Registration for MPEG-4 Visual  . . . . . . . . 17
  7.2.  Mapping to SDP for MPEG-4 Visual . . . . . . . . . . . . . 20
    7.2.1.  Declarative SDP Usage for MPEG-4 Visual  . . . . . . . 20
  7.3.  Media Type Registration for MPEG-4 Audio . . . . . . . . . 21
  7.4.  Mapping to SDP for MPEG-4 Audio  . . . . . . . . . . . . . 24
    7.4.1.  Declarative SDP Usage for MPEG-4 Audio . . . . . . . . 25
      7.4.1.1.  Example: In-Band Configuration . . . . . . . . . . 25
      7.4.1.2.  Example: 6 kbit/s CELP . . . . . . . . . . . . . . 25
      7.4.1.3.  Example: 64 kbit/s AAC LC Stereo . . . . . . . . . 26
      7.4.1.4.  Example: Use of the "SBR-enabled" Parameter  . . . 26
      7.4.1.5.  Example: Hierarchical Signaling of SBR . . . . . . 27
      7.4.1.6.  Example: HE AAC v2 Signaling . . . . . . . . . . . 27
      7.4.1.7.  Example: Hierarchical Signaling of PS  . . . . . . 28
      7.4.1.8.  Example: MPEG Surround . . . . . . . . . . . . . . 28
      7.4.1.9.  Example: MPEG Surround with Extended SDP
                Parameters . . . . . . . . . . . . . . . . . . . . 28
      7.4.1.10. Example: MPEG Surround with Single-Layer
                Configuration  . . . . . . . . . . . . . . . . . . 29
8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 29
9.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 30
10. Security Considerations  . . . . . . . . . . . . . . . . . . . 30
11. Differences to RFC 3016  . . . . . . . . . . . . . . . . . . . 31
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32
  12.1. Normative References . . . . . . . . . . . . . . . . . . . 32
  12.2. Informative References . . . . . . . . . . . . . . . . . . 33

1. Introduction ✨

Okay, the big picture: the RTP payload formats in this doc spell out how MPEG-4 Audio [14496-3] and MPEG-4 Visual [14496-2] streams get fragmented and mapped straight onto RTP packets 🌸.

These payload formats let you transport MPEG-4 Audio/Visual streams without using the synchronization and stream management functionality of MPEG-4 Systems [14496-1] 💅. They're meant for systems that already have their own stream management, so they don't need that stuff from MPEG-4 Systems. H.323 [H323] terminals are a good example: there, MPEG-4 Audio/Visual streams aren't managed by MPEG-4 Systems Object Descriptors but by H.245 [H245] 🦋. The streams get mapped directly onto RTP packets without the MPEG-4 Systems Sync Layer. Other examples are the Session Initiation Protocol (SIP) [RFC3261] and the Real Time Streaming Protocol (RTSP), which both use media type and SDP 🎀.

The media type and SDP usages in this doc are defined to directly specify the attributes of Audio/Visual streams (like media type, packetization format, and codec configuration) without using MPEG-4 Systems 💖. The obvious perk is that these MPEG-4 Audio/Visual RTP payload formats can be handled in a unified way alongside the formats for non-MPEG-4 codecs ✨. The downside is that interop with environments using MPEG-4 Systems may be tricky, so other payload formats might suit those apps better 🙈.

The semantics of RTP headers in these cases need to be clearly defined, including how they connect to MPEG-4 Audio/Visual data elements 🧚. It's also worth defining the fragmentation rules for RTP packets carrying MPEG-4 Video streams, so error resiliency gets a boost by using the error resiliency tools already inside the MPEG-4 Video stream 🛡️.

1.1. MPEG-4 Visual RTP Payload Format

Let's talk MPEG-4 Visual 🎬. It's a visual coding standard with lots of features, including high coding efficiency, high error resiliency, and multiple, arbitrary shape object-based coding [14496-2]. It covers a wide range of bitrates, from scores of kbit/s up to several Mbit/s, and a wide variety of networks, from ones that are almost error-free all the way to mobile networks with high error rates 📶.

Now, the fragmentation rules: since MPEG-4 Visual gets used on so many different networks, it's desirable not to put too much restriction on fragmentation 🌈. A rule like "a single video packet shall always be mapped on a single RTP packet" may be inappropriate. On the flip side, careless, media-unaware fragmentation can hurt error resiliency and bandwidth efficiency 💀. The fragmentation rules in this doc are flexible, but they define the minimum rules needed to prevent meaningless fragmentation while still using the error resiliency features of MPEG-4 Visual 🫶.

The fragmentation rule "Different Video Object Planes (VOPs) SHOULD be fragmented into different RTP packets" exists so the RTP timestamp uniquely indicates the VOP time framing ⏱️. But MPEG-4 video can produce really tiny VOPs, like an empty VOP (vop_coded=0) containing only a VOP header, or an arbitrary shaped VOP with just a few coding blocks 🙈. To cut down the overhead for those cases, the fragmentation rule allows concatenating multiple VOPs in one RTP packet. (See fragmentation rule (4) in Section 5.2 and the descriptions of marker bit and timestamp in Section 5.1.) 🔥

Lastly, H.261 [H261] and MPEG-1/2 use an additional media-specific RTP header, which helps recover picture headers corrupted by packet losses 🩹. MPEG-4 Visual already has its own error resiliency features for recovering corrupt headers, and those work on RTP/IP networks as well as other networks (H.223/mobile, MPEG-2 Transport Stream, etc.) 🌍. So no extra RTP header fields are defined in this MPEG-4 Visual RTP payload format. 👑

Part 1 of 7 is in the books, so more parts are coming, stay tuned 🦋💜

1.2. MPEG-4 Audio RTP Payload Format 🎧

okay so MPEG-4 Audio is basically a standard that packs a whole bunch of different audio coding tools into one toolkit 🧰✨ LATM (Low-overhead MPEG-4 Audio Transport Multiplex) keeps the sequences of audio data really lean, with relatively small overhead. So for audio-only apps, it's lowkey ideal for LATM-based MPEG-4 Audio bitstreams to get mapped straight onto RTP packets, no MPEG-4 Systems needed 🦋

Here's the vibe: for MPEG-4 Audio coding tools (same as other audio coders), if the payload is a single audio frame, losing one packet won't mess up whether the neighbouring packets can be decoded 🙌 So the extra media-specific header for recovering errors is not required for MPEG-4 Audio, no cap. If you want extra error resiliency anyway, existing RTP protection mechanisms like Generic Forward Error Correction [RFC5109] and Redundant Audio Data [RFC2198] MAY be applied 💅

1.3. Interoperability with RFC 3016 🤝

ngl this one is kind of a plot twist. This specification is not backwards compatible with [RFC3016], because it mandates a binary incompatible LATM version 😬 Existing RFC 3016 implementations that already use a recent LATM version may already comply with this specification, and they must be considered as not compliant with RFC 3016. The 3GPP PSS service [3GPP] is a perfect example, since its specification mandates a more recent LATM version too 🎬

The big rule: existing implementations that use the LATM version as specified in RFC 3016 MUST be updated to comply with this specification. No skipping that step, bestie 🫶

1.4. Relation with RFC 3640 🍓

In this document, a payload format for transporting MPEG-4 Elementary Streams is specified 🧚 For MPEG-4 Audio streams, "out-of-band" signaling is defined, so a receiver isn't obliged to decode the payload data just to figure out the audio codec and its configuration. Iconic behaviour, honestly ✨

The signaling capabilities here are less explicit than what [RFC3640] defines, but LATM is used across a bunch of transmission standards, and that's what justifies its right to exist 👑 (see also Section 1.2 🌸)

2. Definitions and Abbreviations 📚

This document uses terms specified in [14496-2], [14496-3], and [23003-1]. On top of that, there are some extra terms with specific meanings inside this document only, so keep an eye out 👀

Abbreviations:

  • AAC: Advanced Audio Coding
  • ASC: AudioSpecificConfig
  • HE AAC: High Efficiency AAC
  • LATM: Low-overhead MPEG-4 Audio Transport Multiplex
  • PS: Parametric Stereo
  • SBR: Spectral Band Replication
  • VOP: Video Object Plane

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. These are the official BCP 14 words, so they're the ones with the real weight 💜

3. Clarifications on Specifying Codec Configurations for MPEG-4 Audio 🎀

For MPEG-4 Audio [14496-3] streams, the decoder output configuration can be different from the core codec configuration, depending on whether the SBR and PS tools are used 🔥

The core codec sampling rate is the default audio codec sampling rate. When SBR is used, the double value of the core codec sampling rate is typically regarded as the definitive sampling rate, which is the decoder's output sampling rate 📈

Note: the exception is down-sampled SBR mode, where the SBR sampling rate and the core codec sampling rate are identical. 🙈

The core codec channel configuration is the default audio codec channel configuration. When PS is used, the core codec channel configuration indicates one channel (mono), but the definitive channel configuration is two channels (stereo) 🎧 When MPEG Surround is used, the definitive channel configuration depends on the output of the MPEG Surround decoder 🌈

4. LATM Restrictions for RTP Packetization of MPEG-4 Audio Bitstreams 🚫

LATM has several multiplexing features:

  • carrying configuration information with audio data,
  • concatenating multiple audio frames in one audio stream,
  • multiplexing multiple objects (programs), and
  • multiplexing scalable layers.

But in RTP transmission there's no need for the last two features, so these two features MUST NOT be used in applications based on the RTP packetization specified by this document 🙅‍♀️ LATM was built only for natural audio coding tools, not synthesis tools, so it seems hard to send Structured Audio (SA) data and Text-to-Speech Interface (TTSI) data over LATM. That means SA data and TTSI data MUST NOT be transported by the RTP packetization in this document 💀

For scalable streams, the audio data of each layer SHOULD be packetized onto different RTP streams. That way, the layers can be treated differently at the IP level, for example through some means of differentiated service ✨ Meanwhile, all the configuration data for scalable streams lives in one LATM configuration data "StreamMuxConfig", and every scalable layer shares that StreamMuxConfig. The mapping between each layer and its configuration data comes from LATM header information attached to the audio data 🧩 To signal the dependency information for scalable streams, the signaling mechanism specified in [RFC5583] SHOULD be used (see Section 6.2) 🫶

5. RTP Packetization of MPEG-4 Visual Bitstreams 🎬

This section specifies the RTP packetization rules for MPEG-4 Visual content. An MPEG-4 Visual bitstream is mapped directly onto RTP packets, with no extra header fields added and no Visual syntax elements removed 🦋 The Combined Configuration/Elementary stream mode MUST be used, so the configuration information gets carried to the same RTP port as the elementary stream. (See Subclause 6.2.1, "Start codes", of [14496-2].) The configuration information MAY also be specified by some out-of-band means 🌸 If systems using media type parameters and SDP parameters need it (e.g., SIP and RTSP), the optional parameter "config" MUST be used to specify the configuration information (see Sections 7.1 and 7.2) 💖

When the short video header mode is used, the RTP payload format for H.263 SHOULD be used. The format defined in [RFC4629] is RECOMMENDED, but the [RFC4628] format MAY be used for compatibility with older implementations 🕰️

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X|  CC   |M|     PT      |       sequence number         | RTP
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           timestamp                           | Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           synchronization source (SSRC) identifier            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|            contributing source (CSRC) identifiers             |
|                             ....                              |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|                                                               | RTP
|       MPEG-4 Visual stream (byte aligned)                     | Pay-
|                                                               | load
|                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               :...OPTIONAL RTP padding        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

             Figure 1: An RTP Packet for MPEG-4 Visual Stream

5.1. Use of RTP Header Fields for MPEG-4 Visual 🧁

  • Payload Type (PT): Assigning an RTP payload type for this packet format is outside the scope of this document, so it won't be specified here. The RTP profile for a particular class of applications is expected to assign a payload type for this encoding. If that doesn't happen, then a payload type in the dynamic range SHALL be chosen by means of an out-of-band signaling protocol (e.g., H.245, SIP) 🙌
  • Extension (X) bit: Defined by the RTP profile used 🌈
  • Sequence Number: Incremented by 1 for each RTP data packet sent, starting, for security reasons, with a random initial value 🔐
  • Marker (M) bit: The marker bit is set to 1 to indicate the last RTP packet (or only RTP packet) of a VOP. When multiple VOPs are carried in the same RTP packet, the marker bit is set to 1 too ✨
  • Timestamp: The timestamp indicates the sampling instance of the VOP contained in the RTP packet. A constant offset, which is random, is added for security reasons 🕵️‍♀️
    • When multiple VOPs are carried in the same RTP packet, the timestamp indicates the earliest of the VOP times within the VOPs carried in the RTP packet. Timestamp information for the rest of the VOPs is derived from the timestamp fields in the VOP header (modulo_time_base and vop_time_increment) ⏰
    • If the RTP packet contains only configuration information and/or Group_of_VideoObjectPlane() fields, the timestamp of the next VOP in the coding order is used 🧠
    • If the RTP packet contains only visual_object_sequence_end_code information, the timestamp of the immediately preceding VOP in the coding order is used 🏁

The resolution of the timestamp is set to its default value of 90 kHz, unless it's specified by out-of-band means (e.g., an SDP parameter or a media type parameter as defined in Section 7) 💜

Other header fields are used as described in [RFC3550] 🫶

5.2. Fragmentation of MPEG-4 Visual Bitstream ✨

okay so a fragmented MPEG-4 Visual bitstream gets mapped straight into the RTP payload 🎀 no extra header fields added, no Visual syntax elements removed. literally untouched 💅

so when I say "header" in this section, I mean ONE of these things 👀:

  • Configuration information (Visual Object Sequence Header, Visual Object Header, and Video Object Layer Header)
  • visual_object_sequence_end_code
  • The header of the entry point function for an elementary stream (Group_of_VideoObjectPlane() or the header of VideoObjectPlane(), video_plane_with_short_header(), MeshObject(), or FaceObject())
  • The video packet header (video_packet_header() excluding next_resync_marker())
  • The header of gob_layer()
  • See Subclause 6.2.1 ("Start codes") of [14496-2] for the definition of the configuration information and the entry point functions.

The Combined Configuration/Elementary streams mode is used 🦋 and these rules apply for the fragmentation:

(1) Configuration information and Group_of_VideoObjectPlane() fields SHALL be placed at the beginning of the RTP payload (just after the RTP header) or just after the header of the syntactically upper-layer function. main character placement, no notes 💖

(2) If one or more headers exist in the RTP payload, the RTP payload SHALL begin with the header of the syntactically highest function. Note: The visual_object_sequence_end_code is regarded as the lowest function. 🌸

(3) A header SHALL NOT be split into a plurality of RTP packets. this one is a hard no, bestie 🙅‍♀️

(4) Different VOPs SHOULD be fragmented into different RTP packets so that one RTP packet consists of the data bytes associated with a unique VOP time instance (that is indicated in the timestamp field in the RTP packet header), with the exception that multiple consecutive VOPs MAY be carried within one RTP packet in the decoding order if the size of the VOPs is small. 🧚

Note: When multiple VOPs are carried in one RTP payload, the timestamp of the VOPs after the first one may be calculated by the decoder. This operation is necessary only for RTP packets in which the marker bit equals to 1 and the beginning of the RTP payload corresponds to a start code. (See the descriptions of timestamp and marker bit in Section 5.1.) 🫶

(5) It is RECOMMENDED that a single video packet is sent as a single RTP packet. The size of a video packet SHOULD be adjusted in such a way that the resulting RTP packet is not larger than the Path MTU. If the video packet is disabled by the coder configuration (by setting resync_marker_disable in the VOL header to 1), or in coding tools where the video packet is not supported, a VOP MAY be split at arbitrary byte positions. the one-video-packet-per-RTP-packet vibe is iconic 👑

The video packet starts with the VOP header or the video packet header, followed by motion_shape_texture(), and ends with next_resync_marker() or next_start_code(). 💅

5.3. Examples of Packetized MPEG-4 Visual Bitstream 🌈

Figure 2 shows examples of RTP packets generated based on the criteria described in Section 5.2. Okay, (a) is the first RTP packet, or the random access point, of an MPEG-4 Visual bitstream containing the configuration information 🔑 By criterion (1), the Visual Object Sequence Header (VS header) goes at the beginning of the RTP payload, right before the Visual Object Header and the Video Object Layer Header (VO header, VOL header). Since the fragmentation rule in Section 5.2 guarantees that the configuration information, starting with visual_object_sequence_start_code, is always placed at the beginning of the RTP payload, RTP receivers can detect the random access point by checking if the first 32-bit field of the RTP payload is visual_object_sequence_start_code. this is so clever ngl 🧠✨

(b) is another example of the RTP packet containing the configuration information. it's giving "example (a) but with extra snacks" 🍓 the RTP packet also contains a VOP header and a video packet in the VOP following the configuration information. Since the configuration information is relatively short (typically scores of bytes) and an RTP packet containing only the configuration information may thus increase the overhead, the configuration information and the subsequent VOP can be packetized into a single RTP packet.

(c) is an example of an RTP packet that contains Group_of_VideoObjectPlane (GOV). Following criterion (1), the GOV is placed at the beginning of the RTP payload. It would be a waste of RTP/IP header overhead to generate an RTP packet containing only a GOV whose length is 7 bytes 🙈 so the following VOP (or a part of it) can be placed in the same RTP packet, as shown in (c).

(d) is an example of the case where one video packet is packetized into one RTP packet. When the packet-loss rate of the underlying network is high, this kind of packetization is recommended. Even when the RTP packet containing the VOP header is discarded by a packet loss, the other RTP packets can be decoded by using the HEC (Header Extension Code) information in the video packet header. No extra RTP header field is necessary. the main character of resilience 🦋

(e) is an example of the case where more than one video packet is packetized into one RTP packet. This kind of packetization is effective to save the overhead of RTP/IP headers when the bitrate of the underlying network is low. However, it will decrease the packet-loss resiliency because multiple video packets are discarded by a single RTP packet loss. The optimal number of video packets in an RTP packet and the length of the RTP packet can be determined by considering the packet-loss rate and the bitrate of the underlying network. it's a trade-off, babe, choose wisely 💜

(f) is an example of the case when the video packet is disabled by setting resync_marker_disable in the VOL header to 1. In this case, a VOP may be split into a plurality of RTP packets at arbitrary byte positions. For example, it is possible to split a VOP into fixed-length packets. This kind of coder configuration and RTP packet fragmentation may be used when the underlying network is guaranteed to be error-free. 🍀

Figure 3 shows examples of RTP packets prohibited by the criteria of Section 5.2. these are the NOT-allowed ones, no cap 🚫

Fragmentation of a header into multiple RTP packets, as in Figure 3(a), will not only increase the overhead of RTP/IP headers but also decrease the error resiliency. Therefore, it is prohibited by criterion (3). 😭

When concatenating more than one video packet into an RTP packet, the VOP header or video_packet_header() is not allowed to be placed in the middle of the RTP payload. The packetization as in Figure 2(b) is not allowed by criterion (2) due to the aspect of the error resiliency. Comparing this example with Figure 2(d), although two video packets are mapped onto two RTP packets in both cases, the packet-loss resiliency is not identical. Namely, if the second RTP packet is lost, both video packets 1 and 2 are lost in the case of Figure 3(b), whereas only video packet 2 is lost in the case of Figure 2(d). so the same number of packets, very different damage 💀

    +------+------+------+------+
(a) | RTP  |  VS  |  VO  | VOL  |
    |header|header|header|header|
    +------+------+------+------+

    +------+------+------+------+------+------------+
(b) | RTP  |  VS  |  VO  | VOL  | VOP  |Video Packet|
    |header|header|header|header|header|            |
    +------+------+------+------+------+------------+

    +------+-----+------------------+
(c) | RTP  | GOV |Video Object Plane|
    |header|     |                  |
    +------+-----+------------------+

    +------+------+------------+  +------+------+------------+
(d) | RTP  | VOP  |Video Packet|  | RTP  |  VP  |Video Packet|
    |header|header|    (1)     |  |header|header|    (2)     |
    +------+------+------------+  +------+------+------------+

    +------+------+------------+------+------------+------+------------+
(e) | RTP  |  VP  |Video Packet|  VP  |Video Packet|  VP  |Video Packet|
    |header|header|     (1)    |header|    (2)     |header|    (3)     |
    +------+------+------------+------+------------+------+------------+

    +------+------+------------+  +------+------------+
(f) | RTP  | VOP  |VOP fragment|  | RTP  |VOP fragment|
    |header|header|    (1)     |  |header|    (2)     | . . .
    +------+------+------------+  +------+------------+

       Figure 2: Examples of RTP Packetized MPEG-4 Visual Bitstream


      +------+-------------+  +------+------------+------------+
  (a) | RTP  |First half of|  | RTP  |Last half of|Video Packet|
      |header|  VP header  |  |header|  VP header |            |
      +------+-------------+  +------+------------+------------+

      +------+------+----------+  +------+---------+------+------------+
  (b) | RTP  | VOP  |First half|  | RTP  |Last half|  VP  |Video Packet|
      |header|header| of VP(1) |  |header| of VP(1)|header|    (2)     |
      +------+------+----------+  +------+---------+------+------------+

   Figure 3: Examples of Prohibited RTP Packetization for MPEG-4 Visual

6. RTP Packetization of MPEG-4 Audio Bitstreams 🎧

okay now we're switching to audio and I'm SO here for it 🦋 this section lays out the RTP packetization rules for MPEG-4 Audio bitstreams. MPEG-4 Audio streams MUST be formatted LATM (Low-overhead MPEG-4 Audio Transport Multiplex) [14496-3] streams, and then those LATM-based streams get mapped onto RTP packets as described in the sections below. non-negotiable, bestie 💅

6.1. RTP Packet Format 🌸

LATM-based streams are basically a sequence of audioMuxElements, and each one includes one or more PayloadMux elements that carry the audio frames 🎶 A complete audioMuxElement, or a part of one, SHALL be mapped directly onto an RTP payload without any removal of audioMuxElement syntax elements (see Figure 4). The first byte of each audioMuxElement SHALL be located at the first payload location in an RTP packet. it's giving "start at the front, no skipping" 🫶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X|  CC   |M|     PT      |       sequence number         |RTP
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           timestamp                           |Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           synchronization source (SSRC) identifier            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|            contributing source (CSRC) identifiers             |
|                             ....                              |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|                                                               |RTP
:                 audioMuxElement (byte aligned)                :Payload
|                                                               |
|                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                               :...OPTIONAL RTP padding        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

             Figure 4 - An RTP packet for MPEG-4 Audio

okay so this is the RTP packet for MPEG-4 Audio, and it's giving clean 🎧✨ bestie, before you can decode the audioMuxElement, the muxConfigPresent info has to be signaled by out-of-band means. when SDP is doing that job, the media type parameter "cpresent" carries the muxConfigPresent info (see Section 7.3). there are two restrictions to keep in mind:

  • out-of-band configuration case: the number of PayloadMux elements in each audioMuxElement can only be set once. if there's more than one PayloadMux element in each audioMuxElement, you need special care to make sure the last RTP packet stays decodable 💀
  • in-band configuration case: non-octet-aligned configuration data gets inserted right before the one or more PayloadMux elements. the generation of RTP payloads with non-octet-aligned data isn't possible with RTP hint tracks, as defined by the MP4 file format [14496-12] [14496-14], so this document does not support RTP hint tracks for the in-band configuration case 🙈

now the main event, muxConfigPresent 🫶

if muxConfigPresent is set to 1 (in-band mode), the audioMuxElement SHALL include an indication bit "useSameStreamMux" and MAY include the configuration information for audio compression "StreamMuxConfig". the useSameStreamMux bit tells you whether the StreamMuxConfig element from the previous frame gets applied to the current frame. but here's the tea ☕ if useSameStreamMux says "reuse the previous one" and the previous frame got lost, the current frame may not be decodable. so for in-band mode, the StreamMuxConfig element SHOULD be transmitted repeatedly, depending on the network condition. on the other hand, if muxConfigPresent is set to 0 (out-of-band mode), the StreamMuxConfig element is required to be transmitted by an out-of-band means. for SDP, that's the media type parameter "config" (see Section 7.3) 🦋

6.2. Use of RTP Header Fields for MPEG-4 Audio 🎀

Payload Type (PT): the assignment of an RTP payload type for this packet format is outside the scope of this document, so this doc only restricts it a little. it's expected that the RTP profile for a particular class of applications will assign a payload type for this encoding. if that doesn't happen, a payload type in the dynamic range shall be chosen by means of an out-of-band signaling protocol (e.g., H.245, SIP). when it comes to dynamic assignment of RTP payload types for scalable streams, the server SHALL assign a different value to each layer. the dependency relationships between the enhanced layer and the base layer MUST be signaled as specified in [RFC5583]. an example of that signaling for scalable audio streams can be found in [RFC5691] ✨

Marker (M) bit: the marker bit marks audioMuxElement boundaries. it's set to 1 to indicate that the RTP packet contains a complete audioMuxElement, or the last fragment of an audioMuxElement 🚩

Timestamp: the timestamp shows the sampling instance of the first audio frame in the RTP packet. Timestamps are RECOMMENDED to start at a random value for security reasons 🔐 unless an out-of-band means says otherwise, the timestamp resolution defaults to 90 kHz.

Sequence Number: incremented by 1 for each RTP packet sent, starting (for security reasons) with a random value 🎲

Other header fields are used as described in [RFC3550] 💅

6.3. Fragmentation of MPEG-4 Audio Bitstream 🌸

It is RECOMMENDED to put one audioMuxElement in each RTP packet. if the audioMuxElement is small enough that the RTP packet containing it doesn't exceed the Path MTU, no problem, bestie 🙌 if it can't be kept that small, the audioMuxElement SHALL be fragmented and spread across multiple packets 🧩

7. Media Type Registration for MPEG-4 Audio/Visual Streams 👑

the sections below cover the media type registrations for MPEG-4 Audio/Visual streams. they're registered in accordance with [RFC4855] and use the template of [RFC4288]. media type registration and SDP usage for the MPEG-4 Visual stream are in Sections 7.1 and 7.2, respectively, and media type registration and SDP usage for the MPEG-4 Audio stream are in Sections 7.3 and 7.4, respectively 🗂️

7.1. Media Type Registration for MPEG-4 Visual 🎬

the receiver MUST ignore any unspecified parameter. this way, extra parameters can be added in any future revision of this specification 🛡️

Type name: video

Subtype name: MP4V-ES

Required parameters: none

Optional parameters:
  • "rate": this parameter is used only for RTP transport. it shows the resolution of the timestamp field in the RTP header. if it isn't specified, the default value of 90000 (90 kHz) is used ⏱️
  • "profile-level-id": a decimal representation of the MPEG-4 Visual Profile and Level indication value (profile_and_level_indication) defined in Table G-1 of [14496-2]. this parameter MAY be used in the capability exchange or session setup procedure to indicate the MPEG-4 Visual Profile and Level combination the MPEG-4 Visual codec is capable of. if the procedure doesn't specify it, the default value of 1 (Simple Profile/Level 1) is used 🧠
  • "config": this parameter SHALL be used to indicate the configuration of the corresponding MPEG-4 Visual bitstream. it SHALL NOT be used to indicate codec capability in the capability exchange procedure. it's a hexadecimal representation of an octet string that expresses the MPEG-4 Visual configuration information, as defined in Subclause 6.2.1 ("Start codes") of [14496-2]. the configuration information maps onto the octet string most significant bit (MSB) first. the first bit of the configuration information SHALL be located at the MSB of the first octet. the configuration information indicated by this parameter SHALL be the same as the configuration information in the corresponding MPEG-4 Visual stream, except for first_half_vbv_occupancy and latter_half_vbv_occupancy (if they exist), which may vary in the repeated configuration information inside an MPEG-4 Visual stream. (See Subclause 6.2.1, "Start codes", of [14496-2].) 🤓

Published specification: the specifications for MPEG-4 Visual streams are presented in [14496-2]. the RTP payload format is described in [RFC6416] 📖

Encoding considerations: video bitstreams MUST be generated according to the MPEG-4 Visual specifications [14496-2]. a video bitstream is binary data, so it MUST be encoded for non-binary transport (for email, Base64 encoding is sufficient). this type is also defined for transfer via RTP. the RTP packets MUST be packetized according to the MPEG-4 Visual RTP payload format defined in [RFC6416] 📦

Security considerations: see Section 10 of [RFC6416] 🔒

Interoperability considerations: MPEG-4 Visual has a large, rich set of tools for coding visual objects. to implement the standard effectively, subsets of the MPEG-4 Visual tool sets have been provided for specific applications. these subsets, called 'Profiles', limit the size of the tool set a decoder is required to implement. to restrict computational complexity, one or more Levels are set for each Profile. a Profile@Level combination allows:

  • a codec builder to implement only the subset of the standard they need, while keeping interworking with other MPEG-4 devices in the same combination, and
  • checking whether MPEG-4 devices comply with the standard ('conformance testing').

the visual stream SHALL be compliant with the MPEG-4 Visual Profile@Level specified by the parameter "profile-level-id". interoperability between a sender and a receiver may be achieved by specifying the parameter "profile-level-id" or by arranging a capability exchange/announcement procedure for this parameter 🤝

Applications that use this media type: audio and visual streaming and conferencing tools 📹

Additional information: none

Person and email address to contact for further information: see the Authors' Addresses section at the end of [RFC6416] 💌

Intended usage: COMMON

Author: see the Authors' Addresses section at the end of [RFC6416] ✍️

Change controller: IETF Audio/Video Transport Payloads working group, delegated from the IESG 🧑‍💻

7.2. Mapping to SDP for MPEG-4 Visual 🗺️

the media type video/MP4V-ES string maps to fields in SDP [RFC4566], like this:

  • the media type (video) goes in SDP "m=" as the media name.
  • the media subtype (MP4V-ES) goes in SDP "a=rtpmap" as the encoding name.
  • the optional parameter "rate" goes in "a=rtpmap" as the "clock rate".
  • the optional parameters "profile-level-id" and "config" go in the "a=fmtp" line to indicate the coder capability and configuration, respectively. these are expressed as a string, in the form of a semicolon-separated list of parameter=value pairs.

example usages for the "profile-level-id" parameter:

1  : MPEG-4 Visual Simple Profile/Level 1
34 : MPEG-4 Visual Core Profile/Level 2
145: MPEG-4 Visual Advanced Real Time Simple Profile/Level 1

7.2.1. Declarative SDP Usage for MPEG-4 Visual 📜

here are some examples of media representations in SDP 💖

Simple Profile/Level 1, rate=90000 (90 kHz), with "profile-level-id" and "config" present in the "a=fmtp" line:

m=video 49170/2 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=1;config=000001B001000001B50900000100000
        00120008440FA282C2090A21F

Core Profile/Level 2, rate=90000 (90 kHz), with "profile-level-id" present in the "a=fmtp" line:

m=video 49170/2 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=34

Advance Real Time Simple Profile/Level 1, rate=90000 (90 kHz), with "profile-level-id" present in the "a=fmtp" line:

m=video 49170/2 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=145

7.3. Media Type Registration for MPEG-4 Audio ✨

The receiver MUST ignore any unspecified parameter, so that extra parameters can be added in any future revision of this specification 🔮💅

Type name: audio 🎧

Subtype name: MP4A-LATM 🦋

Required parameters:

  • rate: this is the RTP timestamp "clock rate" ⏱️ The default is 90000. Other rates MAY be indicated only if they are set to the same value as the audio sampling rate (number of samples per second) 🌸
  • With SBR in the mix, the core codec and SBR sampling rates are usually different 👀 So this parameter SHALL NOT be considered the definitive sampling rate. If you do use it, the server must follow these rules:
    • When the presence of SBR is not explicitly signaled by the optional SDP parameters such as object, profile-level-id, or config, this parameter SHALL be set to the core codec sampling rate.
    • When the presence of SBR is explicitly signaled by the optional SDP parameters such as object, profile-level-id, or config, this parameter SHALL be set to the SBR sampling rate.

NOTE: The optional parameter SBR-enabled in SDP a=fmtp is useful for implicit HE AAC / HE AAC v2 signaling 🧚 But SBR-enabled can also be used with explicit HE AAC / HE AAC v2 signaling. So its existence (in itself) is NOT the criteria for deciding whether HE AAC / HE AAC v2 signaling is explicit 🙈

Optional parameters:

  • profile-level-id: a decimal representation of the MPEG-4 Audio Profile Level indication value defined in [14496-3] 🎀 It tells you which MPEG-4 Audio tool subsets the decoder is capable of using. If this parameter isn't specified in the capability exchange or session setup procedure, its default value of 30 (Natural Audio Profile/Level 1) is used 💜
  • MPS-profile-level-id: a decimal representation of the MPEG Surround Profile Level indication as defined in [14496-3] 🌈 It shows the decoder supports the MPEG Surround profile and level, so it can decode the stream 🫶
  • object: a decimal representation of the MPEG-4 Audio Object Type value defined in [14496-3] ⭐ This one specifies the tool the decoder is going to use. It can be used to limit the capability within the specified profile-level-id 🔥
  • bitrate: the data rate for the audio bitstream 🍓
  • cpresent: a boolean that indicates whether audio payload configuration data has been multiplexed into an RTP payload (see Section 6.1) ✨ A 0 means the configuration data has NOT been multiplexed into an RTP payload, and in that case the config parameter MUST be present. A 1 means it HAS been multiplexed. The default if the parameter is omitted is 1. If this parameter is set to 1 and the config parameter is present, the multiplexed configuration data and the value of the config parameter SHALL be consistent 🫣
  • config: a hexadecimal representation of an octet string that expresses the audio payload configuration data StreamMuxConfig, as defined in [14496-3] 💖 The configuration data is mapped onto the octet string on an MSB-first basis. The first bit of the configuration data SHALL be located at the MSB of the first octet. In the last octet, zero-padding bits, if necessary, SHALL follow the configuration data. Senders MUST set the StreamMuxConfig elements taraBufferFullness and latmBufferFullness to their largest respective value, which indicates that buffer fullness measures are not used in SDP 🍰 Receivers MUST ignore the value of these two elements contained in the config parameter 🙅‍♀️
  • MPS-asc: a hexadecimal representation of an octet string that expresses the audio payload configuration data AudioSpecificConfig, as defined in [14496-3] 🌷 If this parameter is not present, the relevant signaling is done by other means (for example, in-band or contained in the config string) 🔍

The same mapping rules as for the config parameter apply 🧩

  • ptime: the duration of each packet in milliseconds ⏳
  • SBR-enabled: a boolean that indicates whether SBR data can be expected in the RTP payload of a stream 🎶 This is relevant for an SBR-capable decoder if the presence of SBR cannot be detected from an out-of-band decoder configuration (for example, contained in the config string) 🧃

If this parameter is set to 0, a decoder MAY expect that SBR is not used. If this parameter is set to 1, a decoder can up-sample the audio data with the SBR tool, regardless of whether or not SBR data is present in the stream 🚀

If the presence of SBR cannot be detected from the out-of-band configuration and the SBR-enabled parameter is not present, the parameter defaults to 1 for an SBR-capable decoder. If the resulting output sampling rate or the computational complexity is not supported, the SBR tool can be disabled or run in down-sampled mode 🥺

The timestamp resolution at the RTP layer is determined by the rate parameter ⏰

Published specification: Encoding specifications are provided in [14496-3] 📖 The RTP payload format specification is described in [RFC6416] 💌

Encoding considerations: This type is only defined for transfer via RTP 📡

Security considerations: See Section 10 of [RFC6416] 🔐

Interoperability considerations: MPEG-4 Audio provides a large and rich set of tools for coding audio objects 🤯 For effective implementation of the standard, subsets of the MPEG-4 Audio tool sets similar to those used in MPEG-4 Visual have been provided (see Section 7.1) 🪞

The audio stream SHALL be compliant with the MPEG-4 Audio Profile@Level specified by the parameters profile-level-id and MPS-profile-level-id 🎯 Interoperability between a sender and a receiver may be achieved by specifying the parameters profile-level-id and MPS-profile-level-id, or by arranging in the capability exchange procedure to set this parameter mutually to the same value 🤝 Furthermore, the object parameter can be used to limit the capability within the specified Profile@Level in the capability exchange 💅

Applications that use this media type: Audio and video streaming and conferencing tools 📹

Additional information: none

Personal and email address to contact for further information: See Authors' Addresses section at the end of [RFC6416] 📬

Intended usage: COMMON

Author: See Authors' Addresses section at the end of [RFC6416] ✍️

Change controller: IETF Audio/Video Transport Payloads working group delegated from the IESG 👑

7.4. Mapping to SDP for MPEG-4 Audio 🗺️

The media type audio/MP4A-LATM string is mapped to fields in SDP [RFC4566], as follows:

  • The media type (audio) goes in SDP m= as the media name 🎵
  • The media subtype (MP4A-LATM) goes in SDP a=rtpmap as the encoding name 🏷️
  • The required parameter rate goes in a=rtpmap as the "clock rate" ⏱️
  • The optional parameter ptime goes in the SDP a=ptime attribute 🕰️
  • The optional parameters profile-level-id, MPS-profile-level-id, and object go in the a=fmtp line to indicate the coder capability 💫

The following are some examples of the profile-level-id value:

      1 : Main Audio Profile Level 1
      9 : Speech Audio Profile Level 1
      15: High Quality Audio Profile Level 2
      30: Natural Audio Profile Level 1
      44: High Efficiency AAC Profile Level 2
      48: High Efficiency AAC v2 Profile Level 2
      55: Baseline MPEG Surround Profile (see ISO/IEC 23003-1) Level 3

The optional payload-format-specific parameters bitrate, cpresent, config, MPS-asc, and SBR-enabled also go in the a=fmtp line 🧁 These parameters are expressed as a string, in the form of a semicolon-separated list of parameter=value pairs 🍭

7.4.1. Declarative SDP Usage for MPEG-4 Audio 📝

The following sections contain some examples of the media representation in SDP 🌟

Note that the a=fmtp line in some of the examples has been wrapped to fit the page 📄 They would comprise a single line in the SDP file 🫶

7.4.1.1. Example: In-Band Configuration 🎀

In this example, the audio configuration data appears in the RTP payload exclusively (i.e., the MPEG-4 audio configuration is known when a StreamMuxConfig element appears within the RTP payload) 🦋

   m=audio 49230 RTP/AVP 96
   a=rtpmap:96 MP4A-LATM/90000
   a=fmtp:96 object=2; cpresent=1

The "clock rate" is set to 90 kHz ⏱️ This is the default value, and the real audio sampling rate is known when the audio configuration data is received 🌈

7.4.1.2. Example: 6 kbit/s CELP 🎙️

This example shows a 6 kbit/s CELP (Code-Excited Linear Prediction) bitstream (with an audio sampling rate of 8 kHz) 🫧

  m=audio 49230 RTP/AVP 96
  a=rtpmap:96 MP4A-LATM/8000
  a=fmtp:96 profile-level-id=9; object=8; cpresent=0;
    config=40008B18388380
  a=ptime:20

In this example, audio configuration data is not multiplexed into the RTP payload and is described only in SDP 🧸 Furthermore, the "clock rate" is set to the audio sampling rate 💖

7.4.1.3. Example: 64 kbit/s AAC LC Stereo 🎧

This example shows a 64 kbit/s AAC LC stereo bitstream (with an audio sampling rate of 24 kHz) ✨

  m=audio 49230 RTP/AVP 96
  a=rtpmap:96 MP4A-LATM/24000/2
  a=fmtp:96 profile-level-id=1; bitrate=64000; cpresent=0;
    object=2; config=400026203fc0

In this example, audio configuration data is not multiplexed into the RTP payload and is described only in SDP 🙌 Furthermore, the "clock rate" is set to the audio sampling rate 🌸

In this example, the presence of SBR cannot be determined by the SDP parameter set 🤔 The "clock rate" represents the core codec sampling rate. An SBR-enabled decoder can use the SBR tool to up-sample the audio data if the complexity and resulting output sampling rate permit 🔥

7.4.1.4. Example: Use of the "SBR-enabled" Parameter 🫶

These two examples are identical to the example above with the exception of the SBR-enabled parameter 👯 The presence of SBR is not signaled by the SDP parameters object, profile-level-id, and config, but instead the SBR-enabled parameter is present 💅 The rate parameter and the StreamMuxConfig contain the core codec sampling rate 🍓

This example shows SBR-enabled=0, with definitive and core codec sampling rates of 24 kHz 🧚

  m=audio 49230 RTP/AVP 96
  a=rtpmap:96 MP4A-LATM/24000/2
  a=fmtp:96 profile-level-id=1; bitrate=64000; cpresent=0;
    SBR-enabled=0; config=400026203fc0

This example shows SBR-enabled=1, with a core codec sampling rate of 24 kHz, and definitive and SBR sampling rates of 48 kHz 🌈

  m=audio 49230 RTP/AVP 96
  a=rtpmap:96 MP4A-LATM/24000/2
  a=fmtp:96 profile-level-id=1; bitrate=64000; cpresent=0;
    SBR-enabled=1; config=400026203fc0

In this example, the "clock rate" is still 24000, and this information is used for RTP timestamp calculation 🧮 The value of 24000 is used to support old AAC decoders 🕰️ This makes the decoder supporting only AAC understand the HE AAC coded data, although only plain AAC is supported 🫠 A HE AAC decoder is able to generate output data with the SBR sampling rate 👑

7.4.1.5. Example: Hierarchical Signaling of SBR ✨

okay so when SBR is explicitly signaled through the SDP parameters object, profile-level-id, or config, like in the example below, the StreamMuxConfig carries both the core codec sampling rate AND the SBR sampling rate 🎀

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/48000/2
a=fmtp:96 profile-level-id=44; bitrate=64000; cpresent=0;
  config=40005623101fe0; SBR-enabled=1

this config string uses the explicit signaling mode 2.A (hierarchical signaling, see [14496-3]. 💅 That means the AOT (Audio Object Type) is SBR (5), and the SFI (Sampling Frequency Index) is 6 (24000 Hz), which points to the underlying core codec sampling frequency. The CC (Channel Configuration) is stereo (2), and the ESFI (Extension Sampling Frequency Index) = 3 (48000) is the sampling frequency of the extension tool (SBR). It's giving layered 🧁

7.4.1.6. Example: HE AAC v2 Signaling 🦋

HE AAC v2 decoders are required to always produce a stereo signal from a mono signal, so there's no parameter needed to signal the presence of PS. no cap, that's the whole vibe 🫶

This example shows SBR-enabled=1 with 1 channel signaled in the a=rtpmap line AND within the config parameter. The core codec sampling rate is 24 kHz, and the definitive and SBR sampling rates are both 48 kHz. The core codec channel configuration is mono, and the PS channel configuration is stereo.

m=audio 49230 RTP/AVP 110
a=rtpmap:110 MP4A-LATM/24000/1
a=fmtp:110 profile-level-id=15; object=2; cpresent=0;
  config=400026103fc0; SBR-enabled=1

7.4.1.7. Example: Hierarchical Signaling of PS 🌸

This example shows 48 kHz stereo audio input.

m=audio 49230 RTP/AVP 110
a=rtpmap:110 MP4A-LATM/48000/2
a=fmtp:110 profile-level-id=48; cpresent=0; config=4001d613101fe0

The config parameter indicates explicit hierarchical signaling of PS and SBR. Heads up bestie 🚨 this configuration method is not supported by legacy AAC and HE AAC decoders, so they are unable to decode the coded data 💀

7.4.1.8. Example: MPEG Surround 🌈

The following examples show how MPEG Surround configuration data can be signaled using SDP. In the first example, the configuration is carried within the config string by using two different layers. The general parameters here are: AudioMuxVersion=1; allStreamsSameTimeFraming=1; numSubFrames=0; numProgram=0; numLayer=1.

The first layer describes the HE AAC payload and signals these parameters: ascLen=25; audioObjectType=2 (AAC LC); extensionAudioObjectType=5 (SBR); samplingFrequencyIndex=6 (24 kHz); extensionSamplingFrequencyIndex=3 (48 kHz); channelConfiguration=2 (2.0 channels).

The second layer describes the MPEG Surround payload and specifies: ascLen=110; AudioObjectType=30 (MPEG Surround); samplingFrequencyIndex=3 (48 kHz); channelConfiguration=6 (5.1 channels); sacPayloadEmbedding=1; SpatialSpecificConfig=(48 kHz; 32 slots; 525 tree; ResCoding=1; ResBands=[7,7,7,7]).

In this example, the signaling is carried by using two different LATM layers. The MPEG Surround payload rides along with the AAC payload in a single layer, as shown by the sacPayloadEmbedding Flag. 🎀

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/48000
a=fmtp:96 profile-level-id=1; bitrate=64000; cpresent=0;
  SBR-enabled=1;
  config=8FF8004192B11880FF0DDE3699F2408C00536C02313CF3CE0FF0

7.4.1.9. Example: MPEG Surround with Extended SDP Parameters ⭐

This example extends the configuration above with the MPEG-Surround-specific parameters. The MPS-asc parameter specifies the MPEG Surround Baseline Profile at Level 3 (PLI55), and the MPS-asc string contains the hexadecimal representation of the MPEG Surround ASC [audioObjectType=30 (MPEG Surround); samplingFrequencyIndex=0x3 (48 kHz); channelConfiguration=6 (5.1 channels); sacPayloadEmbedding=1; SpatialSpecificConfig=(48 kHz; 32 slots; 525 tree; ResCoding=1; ResBands=[0,13,13,13])].

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/48000
a=fmtp:96 profile-level-id=44; bitrate=64000; cpresent=0;
  config=40005623101fe0; MPS-profile-level-id=55;
  MPS-asc=F1B4CF920442029B501185B6DA00;

7.4.1.10. Example: MPEG Surround with Single-Layer Configuration 🧚

This example shows how MPEG Surround configuration data can be signaled using the SDP config parameter. The configuration is carried within the config string using a single layer. The general parameters in this example are: AudioMuxVersion=1; allStreamsSameTimeFraming=1; numSubFrames=0; numProgram=0; numLayer=0.

The single layer describes the combo of HE AAC and MPEG Surround payload and signals these parameters: ascLen=101; audioObjectType=2 (AAC LC); extensionAudioObjectType=5 (SBR); samplingFrequencyIndex=7 (22.05 kHz); extensionSamplingFrequencyIndex=7 (44.1 kHz); channelConfiguration=2 (2.0 channels). A backward-compatible extension according to [14496-3/Amd.1] signals the presence of MPEG Surround payload data and specifies: SpatialSpecificConfig=(44.1 kHz; 32 slots; 525 tree; ResCoding=0).

In this example, the signaling is carried by using a single LATM layer. The MPEG Surround payload is carried together with the HE AAC payload in that one layer. 🦋

m=audio 49230 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/44100
a=fmtp:96 profile-level-id=44; bitrate=64000; cpresent=0;
  SBR-enabled=1; config=8FF8000652B920876A83A1F440884053620FF0;
  MPS-profile-level-id=55

8. IANA Considerations 📝

This document updates the media subtypes "MP4A-LATM" and "MP4V-ES" from RFC 3016. The new registrations are in Sections 7.1 and 7.3 of this document. 💖

9. Acknowledgements 🫶

The authors would like to thank Yoshihiro Kikuchi, Yoshinori Matsui, Toshiyuki Nomura, Shigeru Fukunaga, and Hideaki Kimata for their work on RFC 3016, and Ali Begen, Keith Drage, Roni Even, and Qin Wu for their valuable input and comments on this document. The main characters of this RFC, periodt 👑

10. Security Considerations 🔥

RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550] and in any applicable RTP profile. The main security considerations for the RTP packet carrying the RTP payload format defined within this document are confidentiality, integrity, and source authenticity. Confidentiality is achieved by encryption of the RTP payload, and integrity of the RTP packets is achieved through a suitable cryptographic integrity protection mechanism. A cryptographic system may also allow the authentication of the source of the payload. A suitable security mechanism for this RTP payload format should provide confidentiality, integrity protection, and (at least) source authentication capable of determining whether or not an RTP packet is from a member of the RTP session. This is serious business, bestie 😭

Note that most MPEG-4 codecs define an extension mechanism to transmit extra data within a stream that is gracefully skipped by decoders that do not support this extra data. This may be used to transmit unwanted data in an otherwise valid stream. 💀

The appropriate mechanism to provide security to RTP and payloads following this may vary. It is dependent on the application, the transport, and the signaling protocol employed. Therefore, a single mechanism is not sufficient, although, if suitable, the usage of the Secure Real-time Transport Protocol (SRTP) [RFC3711] is recommended. Other mechanisms that may be used are IPsec [RFC4301] and Transport Layer Security (TLS) [RFC5246] (e.g., for RTP over TCP), but other alternatives may also exist.

This RTP payload format and its media decoder do not exhibit any significant non-uniformity in the receiver-side computational complexity for packet processing, and thus are unlikely to pose a denial-of-service threat due to the receipt of pathological data. The complete MPEG-4 System allows for transport of a wide range of content, including Java applets (MPEG-J) and scripts. Since this payload format is restricted to audio and video streams, it is not possible to transport such active content in this format. 🛡️

11. Differences to RFC 3016 🔄

The RTP payload format for MPEG-4 Audio as specified in RFC 3016 is used by the 3GPP PSS service [3GPP]. However, there are some misalignments between RFC 3016 and the 3GPP PSS specification that are addressed by this update:

  • The audio payload format (LATM) referenced in this document is the newer format specified in [14496-3], which is binary compatible to the format used in [3GPP]. This newer format is not binary compatible with the LATM referenced in RFC 3016, which is specified in [14496-3:1999/Amd.1:2000].
  • The audio signaling format (StreamMuxConfig) referenced in this document is binary compatible to the format used in [3GPP]. The StreamMuxConfig element has also been revised by MPEG since RFC 3016.
  • The use of an audio parameter "SBR-enabled" is now defined in this document, which is used by 3GPP implementations [3GPP]. RFC 3016 does not define this parameter.
  • The "rate" parameter is defined unambiguously in this document for the case of presence of SBR (Spectral Band Replication). In RFC 3016, the definition of the "rate" parameter is ambiguous.
  • The number of audio channels parameter is defined unambiguously in this document for the case of presence of PS (Parametric Stereo). At the time RFC 3016 was written, PS was not yet defined.

Furthermore, some comments have been addressed and signaling support for MPEG Surround [23003-1] was added.

Below is a summary of the changes in requirements by this update: 📋

  • In the dynamic assignment of RTP payload types for scalable MPEG-4 Audio streams, the server SHALL assign a different value to each layer.
  • The dependency relationships between the enhanced layer and the base layer for scalable MPEG-4 Audio streams MUST be signaled as specified in [RFC5583].
  • If the size of an audioMuxElement is so large that the size of the RTP packet containing it does exceed the size of the Path MTU, the audioMuxElement SHALL be fragmented and spread across multiple packets.
  • The receiver MUST ignore any unspecified parameter in order to ensure that additional parameters can be added in any future revision of this specification. ✨

12. References 📚

12.1. Normative References 📚✨

okay bestie, these are the docs you have to read before any of this makes sense 🫶 no skipping!! 💅

  • [14496-2] MPEG, "ISO/IEC International Standard 14496-2 - Coding of audio-visual objects, Part 2: Visual", 2003. 🎬
  • [14496-3] MPEG, "ISO/IEC International Standard 14496-3 - Coding of audio-visual objects, Part 3 Audio", 2009. 🎧
  • [14496-3/Amd.1] MPEG, "ISO/IEC International Standard 14496-3 - Coding of audio-visual objects, Part 3: Audio, Amendment 1: HD-AAC profile and MPEG Surround signaling", 2009. 🔊
  • [23003-1] MPEG, "ISO/IEC International Standard 23003-1 - MPEG Surround (MPEG D)", 2007. 🌀
  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 🔑
  • [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003. 📡
  • [RFC4288] Freed, N. and J. Klensin, "Media Type Specifications and Registration Procedures", BCP 13, RFC 4288, December 2005. 🏷️
  • [RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session Description Protocol", RFC 4566, July 2006. 📝
  • [RFC4629] Ott, H., Bormann, C., Sullivan, G., Wenger, S., and R. Even, "RTP Payload Format for ITU-T Rec", RFC 4629, January 2007. 🎀
  • [RFC4855] Casner, S., "Media Type Registration of RTP Payload Formats", RFC 4855, February 2007. 🧾
  • [RFC5583] Schierl, T. and S. Wenger, "Signaling Media Decoding Dependency in the Session Description Protocol (SDP)", RFC 5583, July 2009. 🌸

12.2. Informative References 🦋

these ones are the extra-credit reading, the "ohh that's where that came from" list 💜 you don't need them to get the main vibe, but they're iconic if you want the full story 🍓

  • [14496-1] MPEG, "ISO/IEC International Standard 14496-1 - Coding of audio-visual objects, Part 1 Systems", 2004. 🧩
  • [14496-12] MPEG, "ISO/IEC International Standard 14496-12 - Coding of audio-visual objects, Part 12 ISO base media file format". 🎞️
  • [14496-14] MPEG, "ISO/IEC International Standard 14496-14 - Coding of audio-visual objects, Part 12 MP4 file format". 📼
  • [14496-3:1999/Amd.1:2000] MPEG, "ISO/IEC International Standard 14496-3 - Coding of audio-visual objects, Part 3 Audio, Amendment 1: Audio extensions", 2000. 🎵
  • [3GPP] 3GPP, "3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Transparent end-to-end Packet-switched Streaming Service (PSS); Protocols and codecs (Release 9)", 3GPP TS 26.234 V9.5.0, December 2010. 📱
  • [H245] International Telecommunication Union, "Control protocol for multimedia communication", ITU Recommendation H.245, December 2009. ☎️
  • [H261] International Telecommunication Union, "Video codec for audiovisual services at p x 64 kbit/s", ITU Recommendation H.261, March 1993. 📺
  • [H323] International Telecommunication Union, "Packet-based multimedia communications systems", ITU Recommendation H.323, December 2009. 💬
  • [RFC2198] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V., Handley, M., Bolot, J., Vega-Garcia, A., and S. Fosse-Parisis, "RTP Payload for Redundant Audio Data", RFC 2198, September 1997. 🔁
  • [RFC3016] Kikuchi, Y., Nomura, T., Fukunaga, S., Matsui, Y., and H. Kimata, "RTP Payload Format for MPEG-4 Audio/Visual Streams", RFC 3016, November 2000. 🎥
  • [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. 📞
  • [RFC3640] van der Meer, J., Mackie, D., Swaminathan, V., Singer, D., and P. Gentric, "RTP Payload Format for Transport of MPEG-4 Elementary Streams", RFC 3640, November 2003. 🎶
  • [RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004. 🔐
  • [RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005. 🛡️
  • [RFC4628] Even, R., "RTP Payload Format for H.263 Moving RFC 2190 to Historic Status", RFC 4628, January 2007. 🕰️
  • [RFC5109] Li, A., "RTP Payload Format for Generic Forward Error Correction", RFC 5109, December 2007. 🧯
  • [RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, August 2008. 🔒
  • [RFC5691] de Bont, F., Doehla, S., Schmidt, M., and R. Sperschneider, "RTP Payload Format for Elementary Streams with MPEG Surround Multi-Channel Audio", RFC 5691, October 2009. 🎚️

Authors' Addresses ✨

these are the humans who wrote it, so if you ever have a question, here's where to slide into their DMs 💌🫶

Malte Schmidt
Dolby Laboratories
Deutschherrnstr. 15-19
90537 Nuernberg
DE

Phone: +49 911 928 91 42
EMail: malte.schmidt@dolby.com


Frans de Bont
Philips Electronics
High Tech Campus 36
5656 AE Eindhoven
NL

Phone: +31 40 2740234
EMail: frans.de.bont@philips.com


Stefan Doehla
Fraunhofer IIS
Am Wolfmantel 33
91058 Erlangen
DE

Phone: +49 9131 776 6042
EMail: stefan.doehla@iis.fraunhofer.de


Jaehwan Kim
LG Electronics Inc.
VCS/HE, 16Fl. LG Twin Towers
Yoido-Dong, YoungDungPo-Gu,
Seoul 150-721
Korea

Phone: +82 10 6225 0619
EMail: kjh1905m@naver.com
heads up, bestie 🥺 the text I was handed for this part only had the references and author addresses. Sections 8 through 11 (IANA, Acknowledgements, Security, Differences to RFC 3016) weren't in it, so I haven't rewritten them here. 💜
📦 where this page comes from