RFC 5691

RTP Payload Format for Elementary Streams with MPEG Surround Multi-Channel Audio

📅 published October 2009, from ietf

RTP Payload Format for Elementary Streams with MPEG Surround Multi-Channel Audio 🎀

okay bestie, the title block is the vibe check, so here's the tea ☕

RFC: 5691, from the Network Working Group 🌸
Authors: F. de Bont (Philips Electronics), S. Doehla (Fraunhofer IIS), M. Schmidt (Dolby Laboratories), and R. Sperschneider (Fraunhofer IIS) 👑
Category: Standards Track ✨
Date: October 2009 🦋
Updates: RFC 3640 💅
Obsoletes: none listed 🙈

Abstract

okay so this doc is all about extending the RTP payload format from [RFC3640] so you can carry MPEG Surround multi-channel audio 🎧 it adds extra Media Type parameters to signal backwards-compatible transmission inside an MPEG-4 Audio elementary stream 💖 and it also presents a layered scheme that skips the MPEG-4 systems framework entirely. in that setup, the MPEG Surround elementary stream goes over RTP right alongside a separate RTP stream carrying the downmixed audio 🫶

Status of This Memo

ngl this part is the official paperwork 📜 this is an Internet standards track protocol for the Internet community, and the authors are asking for discussion and suggestions for improvements 🔥 for the current standardization state and status, check the "Internet Official Protocol Standards" (STD 1) 💅 and distribution of this memo is unlimited 🌈

Copyright Notice

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. please read these carefully, because they spell out your rights and restrictions for this document 🙏 and any Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the BSD License 💜

Table of Contents 💖

1. Introduction ....................................................2
2. Conventions .....................................................3
3. Definitions and Abbreviations ...................................3
   3.1. Definitions ................................................3
   3.2. Abbreviations ..............................................4
4. Transport of MPEG Surround ......................................4
   4.1. Embedded Spatial Audio Data in AAC Payloads ................4
   4.2. MPEG Surround Elementary Stream ............................5
        4.2.1. Low Bitrate MPEG Surround ...........................7
        4.2.2. High Bitrate MPEG Surround ..........................8
5. IANA Considerations .............................................8
   5.1. Media Type Registration ....................................9
   5.2. Registration of Mode Definitions with IANA .................9
   5.3. Usage of SDP ..............................................10
6. Security Considerations ........................................10
7. References .....................................................11
   7.1. Normative References ......................................11
   7.2. Informative References ....................................11

1. Introduction ✨

okay so MPEG Surround (Spatial Audio Coding, SAC) [23003-1] is an International Standard that MPEG finalized in January 2007 🎶 it can re-create N channels from M < N transmitted channels plus some extra control data 🪄 in the preferred modes, those M channels are either one mono channel or a stereo channel pair 💜 the control data is way lower in data rate than sending all N channels, so the coding is super efficient and still stays compatible with M-channel devices 🤌

the MPEG Surround standard has a bunch of tools that make it useful for lots of different things 🌷 a big one is that you can scale the spatial image quality gradually, from very low spatial overhead all the way up toward transparency 🦋 another big one is that the decoder input can be made compatible with existing matrixed surround technologies 💅

for example, with 5.1 multi-channel audio, the MPEG Surround encoder makes a stereo (or mono) downmix signal plus spatial information that describes the full 5.1 material, all in a highly efficient, parameterised format 🍓 the spatial information gets sent right alongside the downmix 🫶

this is the glow-up part 🌈 existing services can be upgraded to surround sound in a backwards-compatible way 🔊 a stereo decoder in an existing legacy consumer device just ignores the MPEG Surround data and plays the stereo signal with no quality degradation, while an MPEG-Surround-enabled decoder delivers high-quality multi-channel audio 😭

the MPEG Surround decoder can run in modes that render the multi-channel signal to multi-channel or stereo output 🎧 or it can run in a two-channel headphone mode to make a virtual surround output signal 🧚

2. Conventions

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 💖

3. Definitions and Abbreviations 🌸

3.1. Definitions

this memo borrows its definitions from [14496-1], [14496-3], [23003-1], and [RFC3640] 📚 the frequently used terms are summed up here for convenience:

Access Unit: an MPEG Access Unit is the smallest data entity that timing information gets attributed to 🕰️ for audio, an Access Unit is the smallest individually accessible portion of coded audio data within an elementary stream 🎵
AudioSpecificConfig(): this extends the class DecoderSpecificInfo(), as defined in [14496-1], when the objectType indication refers to a stream complying with [14496-3] 🪄 it's the configuration structure for MPEG-4 audio as specified in [14496-3] ✨ it holds the field audioObjectType, which tells apart the different audio codecs defined in [14496-3] 🔑 it also has general audio info (like the sampling frequency and number of channels) and further codec-dependent information structures 💅
SpatialSpecificConfig(): the configuration structure for MPEG Surround audio coding, as specified in [23003-1] 🎀 an AudioSpecificConfig() with an audioObjectType of value 30 contains a SpatialSpecificConfig() structure 👑

3.2. Abbreviations

the abbreviations, in case you need a quick cheat sheet 📝

  AOT:    Audio Object Type
  AAC:    Advanced Audio Coding
  ASC:    AudioSpecificConfig() structure
  AU:     Access Unit
  HE AAC: High Efficiency AAC
  PLI:    Profile and Level Indication
  SSC:    SpatialSpecificConfig() structure

4. Transport of MPEG Surround 🦋

okay from a top-level view, MPEG Surround data splits into two bits 🍰 the configuration data sits in the SpatialSpecificConfig() (SSC), and the SpatialFrame() holds the actual MPEG Surround payload 💜 the configuration data can be signaled in-band or out-of-band 🛸 for in-band, the SSC rides along in a SacDataFrame() together with a SpatialFrame() 📦 for out-of-band, the SSC gets sent to the decoder separately, for example using Session Description Protocol (SDP) [RFC4566] 🫶

SpatialFrame()s may be sent either embedded in the downmix stream (Section 4.1) or as individual elementary streams next to the downmix audio stream (Section 4.2) 🌈

the buffer definition for AAC decoders limits how big an AU can be, as specified in [14496-3] 🧱 for high-bitrate applications that go over this limit, all MPEG Surround data MUST be put in a separate stream, as defined in Section 4.2 ⚠️

⚠️ bestie, if you're going high-bitrate and over the AAC AU limit, the MPEG Surround data MUST live in its own separate stream (see Section 4.2). no embedding allowed in that case 💀

4.1. Embedded Spatial Audio Data in AAC Payloads

[14496-3] defines the extension_payload() as a mechanism for transporting extension data inside AAC payloads 🎁 typical extension data includes Spectral Band Replication (SBR) data and MPEG Surround data, which means a SacDataFrame() inside extension_payload()s of type EXT_SAC_DATA 💖 the extension_payload()s live inside the downmix AAC elementary stream 🫧 the resulting single elementary stream gets transported as specified in [RFC3640] 🎀 since AAC decoders are required to skip unknown extension data, MPEG Surround data can be embedded in a backwards-compatible fashion and carried with the mechanism that [RFC3640] already describes 🧚

the SacDataFrame() includes a SpatialFrame() and an optional header that contains an SSC 📦 any SSC in a SacDataFrame() MUST be identical to the SSC conveyed via SDP for that stream 🔒

no new mode is introduced for SpatialFrame()s that get embedded into AAC payloads 🙅‍♀️ either the mode AAC-lbr or the mode AAC-hbr SHOULD be used 💫 the additional Media Type parameters, as defined in Section 5.1, SHOULD be present when SpatialFrame()s are embedded into AAC payloads 🌸

here's an example, bestie 👀

m=audio 5000 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/48000/2
a=fmtp:96 streamType=5; profile-level-id=44; mode=AAC-hbr; config=131
  056E598; sizeLength=13; indexLength=3; indexDeltaLength=3; constant
  Duration=2048; MPS-profile-level-id=55; MPS-config=F1B4CF920442029B
  501185B6DA00;

in this example, the stream specifies the HE AAC Profile at Level 2 [Profile and Level Indication (PLI) 44] 🎶 the config string holds the hexadecimal representation of the HE AAC ASC [audioObjectType=2 (AAC LC); extensionAudioObjectType=5 (SBR); samplingFrequencyIndex=0x6 (24kHz); extensionSamplingFrequencyIndex=0x3 (48kHz); channelConfiguration=2 (2.0 channels)] of the downmix AAC elementary stream, which is using explicit backwards-compatible signaling 🥹

and the stream also specifies the MPEG Surround Baseline Profile at Level 3 (PLI55) 👑 the MPS-config string holds the hexadecimal representation of the MPEG Surround ASC [audioObjectType=30 (MPEG Surround); samplingFrequencyIndex=0x3 (48kHz); channelConfiguration=6 (5.1 channels); sacPayloadEmbedding=1; SSC=(48 kHz; 32 slots; 525 tree; ResCoding=1; ResBands=[0,13,13,13])] 🤯

note that the a=fmtp line in the example above has been wrapped to fit the page 📄 it would be a single line in the SDP file 🦋

4.2. MPEG Surround Elementary Stream ✨

okay so MPEG Surround SpatialFrame()s can live in their own individual elementary stream 🧚 It's the hype sidekick to the stream carrying the downmix audio, and that downmix can be coded with pretty much any scheme 💅 MPEG Surround elementary streams get packetized as specified in [RFC3640] 📦 The mode you signal and use for an MPEG Surround elementary stream MUST be either MPS-hbr or MPS-lbr 🎀 MPS-hbr SHALL be used when the frame size may exceed 63 bytes, like when high-bitrate residual coding is in use 🔥

The dependency relationships between the MPEG Surround elementary stream and the downmix stream get signaled as specified in [RFC5583] 🦋

The media clocks of the MPEG Surround elementary stream and the downmix stream SHALL operate in the same clock domain, i.e., the clocks are derived from a common clock and MUST NOT drift ⏰ Like, they are not allowed to drift apart, periodt 😭 RTCP sender reports MUST indicate that the stream timestamps are not drifting, i.e., that a single sender report for each stream is sufficient to establish unambiguous timing ✨ That means one sender report per stream is enough to nail the timing down with zero confusion 💜 The sampling rate of the MPEG Surround signal and the decoded downmix signal MUST be identical 🍓

If HE AAC is used as the coding scheme for the downmix, the RTP clock-rate of the downmix MAY be the sampling rate of the AAC core, i.e., the clock-rate of the MPEG Surround elementary stream is an integer multiple of the clock-rate of the downmix stream 🌈

Note that separate RTP streams have different random RTP timestamp offsets, and therefore RTCP MUST be used to synchronize the coded downmix audio data and the MPEG Surround elementary stream 🫶 Those offsets are randomized, so RTCP is the only way to get everyone in sync, bestie 🙈

For example:

a=group:DDP L1 L2

m=audio 5000 RTP/AVP 96
a=rtpmap:96 mpeg4-generic/48000/2
a=fmtp:96 streamType=5; profile-level-id=44; mode=AAC-hbr; config=2B1
  18800; sizeLength=13; indexLength=3; indexDeltaLength=3; constantDu
  ration=2048
a=mid:L1

m=audio 5002 RTP/AVP 97
a=rtpmap:97 mpeg4-generic/48000/6
a=fmtp:97 streamType=5; profile-level-id=55; mode=MPS-hbr; config=F1B
  0CF920460029B601189E79E70; sizeLength=13; indexLength=3;  indexDelt
  aLength=3; constantDuration=2048
a=mid:L2
a=depend:97 lay L1:96

In this example, the first stream specifies the HE AAC Profile at Level 2 (PLI44) 🎧 and its config string holds the hexadecimal representation of the HE AAC ASC, which breaks down like this:

[audioObjectType=2 (AAC LC);
extensionAudioObjectType=5 (SBR); samplingFrequencyIndex=0x6 (24kHz);
extensionSamplingFrequencyIndex=0x3 (48kHz); channelConfiguration=2
(2.0 channels)]

The second stream specifies the Baseline MPEG Surround Profile at Level 3 (PLI55) 👑 and its config string holds the hexadecimal representation of the ASC, which breaks down like this:

[AOT=30(MPEG Surround); 48 kHz; 5.1-ch;
sacPayloadEmbedding=0; SSC=(48 kHz; 32 slots; 525 tree; ResCoding=1;
ResBands=[7,7,7,7])]

⚠️ Heads up: the a=fmtp lines in the example above got wrapped so they'd fit the page. In the actual SDP file, each one is a single line 💅

4.2.1. Low Bitrate MPEG Surround ✨

This mode is signaled by mode=MPS-lbr 🌸 It supports transporting one or more complete Access Units, and each AU is one single MPEG Surround SpatialFrame(). The AUs can be variably sized and interleaved 🦋 The maximum size of a SpatialFrame() is 63 bytes. Fragmentation MUST NOT be used in this mode. Receivers MUST support de-interleaving 🙅‍♀️

The payload configuration is the same as in the AAC-lbr mode 💖 It's made of the AU Header Section, followed by concatenated AUs. Access Units are byte-aligned 🧩 The Auxiliary Section MUST be empty in the MPS-lbr mode. The 1-octet AU-header MUST provide:

the size of each AAC frame, encoded as 6 bits.
2 bits of index information for computing the sequence (and hence timing) of each SpatialFrame().

The concatenated AU Header Section MUST be preceded by the 16-bit AU-headers-length field 📏

In addition to the required Media format parameters, the following parameters MUST be present with fixed values: sizeLength (fixed value 6), indexLength (fixed value 2), and indexDeltaLength (fixed value 2). The parameter maxDisplacement MUST be present when interleaving 🎀 SpatialFrame()s always have a fixed duration per AU, and that fixed duration MUST be signaled by the Media format parameter constantDuration ⏱️

The value of the "config" parameter is the hexadecimal representation of the ASC, as defined in [14496-3], with an AOT of 30 and the sacPayloadEmbedding flag set to 0 🧚

The "profile-level-id" parameter SHALL contain a valid PLI for MPEG Surround, as specified in [14496-3] 💜

4.2.2. High Bitrate MPEG Surround ✨

This mode is signaled by mode=MPS-hbr 🔥 It supports transporting either one fragment of an Access Unit, one complete AU, or several complete AUs. Each AU consists of a single MPEG Surround SpatialFrame(). The AUs can be variably sized and interleaved 🌈 The maximum size of a SpatialFrame() is 8191 bytes. Receivers MUST support de-interleaving 🫶

The payload configuration is the same as in the AAC-hbr mode 🍓 It consists of the AU Header Section, followed by either one SpatialFrame(), a fragment of a SpatialFrame(), or several concatenated SpatialFrame()s. Access Units are byte-aligned 🧩 The Auxiliary Section MUST be empty in the MPS-hbr mode. The 2-octet AU-header MUST provide:

the size of each AAC frame, encoded as 13 bits.
3 bits of index information for computing the sequence (and hence timing) of each SpatialFrame(), i.e., the AU-Index or AU-Index-delta field.

Each AU-Index field MUST be coded with the value 0. The concatenated AU Header Section MUST be preceded by the 16-bit AU-headers-length field 🙈

In addition to the required Media format parameters, the following parameters MUST be present with fixed values: sizeLength (fixed value 13), indexLength (fixed value 3), and indexDeltaLength (fixed value 3). The parameter maxDisplacement MUST be present when interleaving 💅 SpatialFrame()s always have a fixed duration per AU, and that fixed duration MUST be signaled by the Media format parameter constantDuration ⏱️

The value of the "config" parameter is the hexadecimal representation of the ASC, as defined in [14496-3], with an AOT of 30 and the sacPayloadEmbedding flag set to 0 🧚

The "profile-level-id" parameter SHALL contain a valid PLI for MPEG Surround, as specified in [14496-3] 👑

5. IANA Considerations 🎀

This memo defines additional optional format parameters to the Media type "audio" and its subtype "mpeg4-generic" 🌸 These parameters SHALL only be used in combination with the AAC-lbr or AAC-hbr modes (cf. Section 3.3 of [RFC3640]) of "mpeg4-generic" 💖

5.1. Media Type Registration ✨

This memo defines the following additional optional parameters, which SHALL be used if MPEG Surround data is present inside the payload of an AAC elementary stream 🦋

MPS-profile-level-id: a decimal representation of the MPEG Surround Profile and Level indication as defined in [14496-3]. This parameter MUST be used in the capability exchange or session set-up procedure to indicate the MPEG Surround Profile and Level that the decoder must be capable of in order to decode the stream 🧠
MPS-config: a hexadecimal representation of an octet string that expresses the AudioSpecificConfig (ASC), as defined in [14496-3], for MPEG Surround. The ASC is mapped onto the hexadecimal octet string in a most significant bit (MSB)-first basis. The AOT in this ASC SHALL have the value 30. The SSC inside the ASC MUST have the sacPayloadEmbedding flag set to 1 🍓

5.2. Registration of Mode Definitions with IANA ✨

This section of this memo requests the registration of the "MPS-hbr" value and the "MPS-lbr" value for the "mode" parameter of the "mpeg4-generic" media subtype within the media type "audio" 🌈 The "mpeg4-generic" media subtype is defined in [RFC3640], and [RFC3640] defines a repository for the "mode" parameter. This memo registers the modes "MPS-hbr" and "MPS-lbr" to support MPEG Surround elementary streams 🫶

Media type name:

audio

Subtype name:

mpeg4-generic

Required parameters:

The "mode" parameter is required by [RFC3640]. This memo specifies the additional modes "MPS-hbr" and "MPS-lbr", in accordance with [RFC3640] 🙈

Optional parameters:

For the modes "AAC-hbr" and "AAC-lbr", this memo specifies the additional optional parameters "MPS-profile-level-id" and "MPS-config". See Section 4.1 for usage details 💅
Optional parameters for the modes "MPS-hbr" and "MPS-lbr" may be used as specified in [RFC3640]. The optional parameters "MPS-profile-level-id" and "MPS-config" SHALL NOT be used for the modes "MPS-hbr" and "MPS-lbr" 🙅‍♀️

5.3. Usage of SDP ✨

It is assumed that the Media format parameters are conveyed via an SDP message, as specified in Section 4.4 of [RFC3640] 📨

6. Security Considerations 🔐

💀 Okay bestie, this section is seriously important, so read it carefully. Security is not a vibe you can skip!

RTP packets using the payload format defined in this specification are subject to the security considerations discussed in the RTP specification [RFC3550], in the RTP payload format specification for MPEG-4 elementary streams [RFC3640] (which is extended with this memo), and in any applicable RTP profile 🛡️ The main security considerations for the RTP packet carrying the RTP payload format defined within this memo are confidentiality, integrity, and source authenticity 🔒 Confidentiality is achieved by encryption of the RTP payload. Integrity of the RTP packets is achieved through a suitable cryptographic integrity-protection mechanism. Such a cryptographic system may also allow the authentication of the source of the payload 🧾 A suitable security mechanism for this RTP payload format should provide confidentiality, integrity protection, and source authentication capable of at least determining if an RTP packet is from a member of the RTP session 💜

The AAC audio codec includes an extension mechanism to transmit extra data within a stream that is gracefully skipped by decoders that do not support this extra data. This covert channel may be used to transmit unauthorized data in an otherwise valid stream 👀 Lowkey spooky, so keep your eyes open 😭

Note that the appropriate mechanism to provide security to RTP and payloads following this memo may vary 🌸 It depends on the application, the transport, and the signaling protocol employed. Therefore, a single mechanism is not sufficient; although, if suitable, usage of the Secure Real-time Transport Protocol (SRTP) [RFC3711] is recommended 🦋 Other mechanisms that may be used are IPsec [RFC4301] and Transport Layer Security (TLS) [RFC5246] (RTP over TCP); other alternatives may exist 🧚

7. References ✨

7.1. Normative References 📚

okay bestie, these are the docs you must have read before this spec makes any sense 🫶 they're the non-negotiable foundations, so I'm keeping every single detail 💅

[14496-1]  MPEG, "ISO/IEC International Standard 14496-1 - Coding of
           audio-visual objects, Part 1 Systems", 2004.

[14496-3]  MPEG, "ISO/IEC International Standard 14496-3 - Coding of
           audio-visual objects, Part 3 Audio", 2009.

[23003-1]  MPEG, "ISO/IEC International Standard 23003-1 - MPEG
           Surround (MPEG D)", 2007.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC3550]  Schulzrinne, H., Casner, S., Frederick, R., and V.
           Jacobson, "RTP: A Transport Protocol for Real-Time
           Applications", STD 64, RFC 3550, July 2003.

[RFC3640]  van der Meer, J., Mackie, D., Swaminathan, V., Singer, D.,
           and P. Gentric, "RTP Payload Format for Transport of
           MPEG-4 Elementary Streams", RFC 3640, November 2003.

[RFC4566]  Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
           Description Protocol", RFC 4566, July 2006.

[RFC5583]  Schierl, T. and S. Wenger, "Signaling Media Decoding
           Dependency in the Session Description Protocol (SDP)",
           RFC 5583, July 2009.

7.2. Informative References 🦋

these ones are the extra reading 🌸 helpful for context, but the spec doesn't need them to work, so no pressure ✨

[RFC3711]  Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
           Norrman, "The Secure Real-time Transport Protocol (SRTP)",
           RFC 3711, March 2004.

[RFC4301]  Kent, S. and K. Seo, "Security Architecture for the
           Internet Protocol", RFC 4301, December 2005.

[RFC5246]  Dierks, T. and E. Rescorla, "The Transport Layer Security
           (TLS) Protocol Version 1.2", RFC 5246, August 2008.

Authors' Addresses 💌

the people who made this happen, bestie 🎀 literally the squad behind it 💜

Frans de Bont
Philips Electronics
High Tech Campus 5
5656 AE Eindhoven,
NL

Phone: ++31 40 2740234
EMail: frans.de.bont@philips.com


Stefan Doehla
Fraunhofer IIS
Am Wolfmantel 33
91058 Erlangen,
DE

Phone: +49 9131 776 6042
EMail: stefan.doehla@iis.fraunhofer.de


Malte Schmidt
Dolby Laboratories
Deutschherrnstr. 15-19
90537 Nuernberg,
DE

Phone: +49 911 928 91 42
EMail: malte.schmidt@dolby.com


Ralph Sperschneider
Fraunhofer IIS
Am Wolfmantel 33
91058 Erlangen,
DE

Phone: +49 9131 776 6167
EMail: ralph.sperschneider@iis.fraunhofer.de

that's the whole spec, bestie!! 🌈✨ 💖

📦 where this page comes from

ietf: rfc5691 2009-10
💖 girlified by claude-haiku-5-5 on 2026-10-17: a language model's rewrite of the original, under its terms