RFC 4352
RTP Payload Format for the Extended Adaptive Multi-Rate Wideband (AMR-WB+) Audio Codec
RFC 4352: RTP Payload Format for the Extended Adaptive Multi-Rate Wideband (AMR-WB+) Audio Codec 🎀
Okay bestie, this is RFC 4352 🌸 It's a Standards Track doc from January 2006, published in the Network Working Group. The authors are J. Sjoberg and M. Westerlund from Ericsson, and A. Lakaniemi and S. Wenger from Nokia ✨ The title block doesn't list anything it obsoletes or updates, so it's not replacing or changing any other RFC. 🦋
Status of This Memo ✨
So this is an Internet standards track protocol for the Internet community, and it's literally asking for discussion and suggestions for improvements 💬 To find out where this protocol stands in standardization, check the current edition of "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🌈
Copyright Notice 💜
Copyright (C) The Internet Society (2006).
Abstract 🦋
Okay so this doc is a Real-time Transport Protocol (RTP) payload format for Extended Adaptive Multi-Rate Wideband (AMR-WB+) encoded audio signals 🎶 AMR-WB+ is basically the audio glow-up of the AMR-WB speech codec. It keeps the AMR-WB frame types and adds a bunch of new ones built for high-quality music and speech. ✨ The spec also includes a media type registration for AMR-WB+ 💅
Table of Contents 🌸
1. Introduction ....................................................3
2. Definitions .....................................................4
2.1. Glossary ...................................................4
2.2. Terminology ................................................4
3. Background of AMR-WB+ and Design Principles .....................4
3.1. The AMR-WB+ Audio Codec ....................................4
3.2. Multi-rate Encoding and Rate Adaptation ....................8
3.3. Voice Activity Detection and Discontinuous Transmission ....8
3.4. Support for Multi-Channel Session ..........................8
3.5. Unequal Bit-Error Detection and Protection .................9
3.6. Robustness against Packet Loss .............................9
3.6.1. Use of Forward Error Correction (FEC) ...............9
3.6.2. Use of Frame Interleaving ..........................10
3.7. AMR-WB+ Audio over IP Scenarios ...........................11
3.8. Out-of-Band Signaling .....................................11
4. RTP Payload Format for AMR-WB+ .................................12
4.1. RTP Header Usage ..........................................13
4.2. Payload Structure .........................................14
4.3. Payload Definitions .......................................14
4.3.1. Payload Header .....................................14
4.3.2. The Payload Table of Contents ......................15
4.3.3. Audio Data .........................................20
4.3.4. Methods for Forming the Payload ....................21
4.3.5. Payload Examples ...................................21
4.4. Interleaving Considerations ...............................24
4.5. Implementation Considerations .............................25
4.5.1. ISF Recovery in Case of Packet Loss ................26
4.5.2. Decoding Validation ................................28
5. Congestion Control .............................................28
6. Security Considerations ........................................28
6.1. Confidentiality ...........................................29
6.2. Authentication and Integrity ..............................29
7. Payload Format Parameters ......................................29
7.1. Media Type Registration ...................................30
7.2. Mapping Media Type Parameters into SDP ....................32
7.2.1. Offer-Answer Model Considerations ..................32
7.2.2. Examples ...........................................34
8. IANA Considerations ............................................34
9. Contributors ...................................................34
10. Acknowledgements ..............................................34
11. References ....................................................35
11.1. Normative References .....................................35
11.2. Informative References ...................................351. Introduction ✨
Okay so this doc spells out how to package Extended Adaptive Multi-Rate Wideband (AMR-WB+) [1] encoded audio signals into the Real-time Transport Protocol (RTP) [3] 🎀 The payload format can carry mono or stereo audio, bundle multiple frames into a single payload, and includes mechanisms that make the packet stream more robust against packet loss 💪
AMR-WB+ is an extension of the Adaptive Multi-Rate Wideband (AMR-WB) speech codec 🦋 The new stuff includes extended audio bandwidth for high quality on non-speech signals (like music 🎵), native support for stereo audio, and the ability to operate on, and switch between, several internal sampling frequencies (ISFs) 😭 The main scenario is transport over IP. So interworking with other transport networks, as discussed for AMR-WB in [7], isn't a major concern and isn't covered in this memo 🌈
The big expected application is streaming 💜 To keep packetization on a streaming server as efficient as possible, an octet-aligned payload format is the move. That's why a bandwidth-efficient mode (as defined for AMR-WB in [7]) is not specified here. Honestly the bandwidth savings would be tiny anyway, since all the extension frame types are octet aligned 🙈
Stereo encoding is built into AMR-WB+, which makes the multi-channel transport support at the RTP payload format level (as specified for AMR-WB [7]) obsolete. So that feature isn't included in this memo 👑
This spec doesn't define a file format for AMR-WB+. Instead, it points to the ISO-based 3GP file format [14], which supports AMR-WB+ and provides all the functionality required 🧚 The 3GP format also stores AMR, AMR-WB, and many other multi-media formats, so you can get synchronized playback 🎬
Here's the lay of the land for the rest of the doc 🗺️ Background on the AMR-WB+ codec and its design principles is in Section 3. The payload format itself is specified in Section 4. Sections 5 and 6 cover congestion control and security considerations, respectively. And Section 7 has the media type registration 💅
2. Definitions 🍓
2.1. Glossary 📚
3GPP - Third Generation Partnership Project
AMR - Adaptive Multi-Rate (Codec)
AMR-WB - Adaptive Multi-Rate Wideband (Codec)
AMR-WB+ - Extended Adaptive Multi-Rate Wideband (Codec)
CN - Comfort Noise
DTX - Discontinuous Transmission
FEC - Forward Error Correction
FT - Frame Type
ISF - Internal Sampling Frequency
SCR - Source-Controlled Rate Operation
SID - Silence Indicator (the frames containing only CN
parameters)
TFI - Transport Frame Index
TS - Timestamp
VAD - Voice Activity Detection
UED - Unequal Error Detection
UEP - Unequal Error Protection2.2. Terminology 🔥
Okay the key words in this doc are the ones listed below, and they're to be read as described in RFC 2119 [2] 💖
"MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"3. Background of AMR-WB+ and Design Principles 🌈
The Extended Adaptive Multi-Rate Wideband (AMR-WB+) [1] audio codec is built to compress speech and audio signals at a low bit-rate while keeping good quality ✨ It's specified by the Third Generation Partnership Project (3GPP) 🎀 The primary target applications are 1) the packet-switched streaming service (PSS) [13], 2) multimedia messaging service (MMS) [18], and 3) multimedia broadcast and multicast service (MBMS) [19]. But because it's so flexible and robust, AMR-WB+ is also a great fit for streaming in other highly varying transport environments, like the Internet 💜 It's giving versatile energy 👑
3.1. The AMR-WB+ Audio Codec ✨
okay so 3GPP originally made the AMR-WB+ audio codec for streaming and messaging in GSM and third generation (3G) cellular systems 📱. It's basically an audio extension of the AMR-WB speech codec, and it adds new stuff so it sounds amazing on a wide range of signals, including music 🎶. Stereo operation got added too 🫶. There's also a new high-efficiency hybrid stereo coding algorithm that lets stereo work at bit-rates as low as 6.2 kbit/s, which is literally iconic 😭🔥.
AMR-WB+ includes the nine frame types from AMR-WB, plus new bit-rates from 5.2 to 48 kbit/s 💖. The AMR-WB frame types can only use a 16000 Hz sampling frequency and only do monophonic signals, but the new extension frame types can run at a bunch of internal sampling frequencies (ISFs), both mono and stereo 🦋. For the details, check Table 24 in [1]. The decoder's output sampling frequency is limited to 8, 16, 24, 32, or 48 kHz 💅.
Here's how the encoder works, step by step 🧚. It gets audio sampled at, for example, 48 kHz. First it does pre-processing and resampling to the ISF the user picked. Then the encoding happens on equally sized super-frames, and each super-frame is 2048 samples per channel, at the ISF 🍓. For every super-frame, the codec makes a bunch of encoding decisions, picking between different encoding algorithms and block lengths, so it gets a fidelity-optimized encoding that suits the source signal 👑. If stereo is used, it runs separately from the monophonic core encoding, which means you can mix and match different combos of core and stereo encoding rates ✨. The result comes out as four transport frames of equal length. Each transport frame is 512 samples at the ISF, and the decoder can use it on its own, as long as it knows where that frame sits in the super-frame structure 🌈.
The codec supports 13 different ISFs, from 12.8 to 38.4 kHz, all listed in Table 24 of [1] 🎀. Having that many ISFs lets you trade off audio bandwidth against target bit-rate, which is such a power move 💜. Since encoding is done on 2048 samples at the ISF, the super-frame duration and the effective bit-rate of the frame type in use both change depending on the ISF 🙈.
The ISF of 25600 Hz has a super-frame duration of 80 ms 🌸. That's the 'nominal' value used to describe the encoding bit-rates from here on out. With that normalization, picking an ISF makes the bit-rate vary anywhere from 1/2 up to 3/2 of the nominal bit-rate 📊.
For the extension modes, the encoding is one monophonic core encoding plus one stereo encoding ⭐. The core encoding splits the mono signal into a lower band and a higher band 🔥. The lower band gets encoded with either algebraic code excited linear prediction (ACELP) or transform coded excitation (TCX) 💖. You can make that choice once per transport frame, but it has to follow certain rules about which combinations are legal inside the super-frame, so it's not totally free-for-all 🙃. The higher band uses a low-rate parametric bandwidth extension approach 🦋.
The stereo signal gets a similar frequency band split, except here the signal is divided into three bands, and each one is parameterized on its own 🧚♀️.
The total bit-rate of the extension comes from combining the encoder's core rate, stereo rate, and ISF 💅. The extension supports 8 core encoding rates, which give bit-rates between 10.4 and 24.0 kbit/s (see Table 22 in [1]) 🌈. There are 16 stereo encoding rates that generate bit-rates between 2.0 and 8.0 kbit/s (see Table 23 in [1]) 🍓. The frame type tells you which AMR-WB modes you're in, plus 4 fixed extension rates (more on those below), 24 combos of core and stereo rates for stereo signals, and the 8 core rates for mono signals, all listed in Table 25 in [1] 👑. So yeah, AMR-WB+ supports encoding rates between 10.4 and 32 kbit/s, assuming an ISF of 25600 Hz ✨.
Different ISFs give you extra wiggle room with bit-rates and audio quality 🫶. Picking an ISF changes the audio bandwidth of the reconstructed signal and also changes the total bit-rate 🎶. The bit-rate for any combo of frame type and ISF is the frame type's bit-rate multiplied by the ISF's bit-rate factor (see Table 24 in [1]) 💜.
The extension also has four frame types with fixed ISFs. Check frame types 10-13 in Table 21 in [1] 🎀. These four predefined frame types have a fixed input sampling frequency at the encoder, which can be set to either 16 or 24 kHz 🦋. Just like the AMR-WB frame types, the transport frames made with these frame types represent exactly 20 ms of audio 💖. The difference is they're also part of 80 ms super-frames 🌸. Frame types 0-13 (AMR-WB and fixed extension rates), as listed in Table 21 in [1], don't need an explicit ISF indication 🙌. The other frame types, 14-47, do require the ISF to be indicated 🔥.
The 32 different extension frame types, combined with 13 ISFs, give you tons of flexibility in bit-rate and in which audio quality you pick 🌈. A bunch of combos produce the same codec bit-rate 🙈. For example, a 32 kbit/s audio stream can come from frame type 41 (i.e., 25.6 kbit/s) with the ISF of 32kHz (5/4 * (19.2+6.4) = 32 kbit/s), or from frame type 47 with the ISF of 25.6 kHz (1 * (24 + 8) = 32 kbit/s) 🧚. Which one sounds better depends on the content 🎧. In that example, the first option gives you a higher audio bandwidth, while the second spends the same number of bits on a somewhat narrower bandwidth but with higher fidelity 💅. Encoders are free to pick whichever combo they think is most beneficial 👑.
Since a transport frame always corresponds to 512 samples at the ISF in use, its duration is limited to the range 13.33 to 40 ms 🍓. Check Table 1 for the numbers 📋. The RTP Timestamp clock rate of 72000 Hz, which this specification mandates, turns into AMR-WB+ transport frame lengths of 960 to 2880 timestamp ticks, and that depends only on the ISF you selected ⏱️✨.
Index ISF Duration(ms) Duration(TS Ticks @ 72 kHz)
------------------------------------------------------
0 N/A 20 1440
1 12800 40 2880
2 14400 35.55 2560
3 16000 32 2304
4 17067 30 2160
5 19200 26.67 1920
6 21333 24 1728
7 24000 21.33 1536
8 25600 20 1440
9 28800 17.78 1280
10 32000 16 1152
11 34133 15 1080
12 36000 14.22 1024
13 38400 13.33 960
Table 1: Normative number of RTP Timestamp Ticks for each
Transport Frame depending on ISF (ISF and Duration in
ms are rounded)The encoder can change both the ISF and the encoding frame type (mono and stereo) during a session 🦋. For the extension frame types with index 10-13 and 16-47, ISF and frame type changes have to happen at super-frame boundaries 🙃. So for those frame types, the ISF stays constant across a whole super-frame 🫶. This limitation doesn't apply to frame types with index 0-9, 14, and 15, which are the original AMR-WB frame types 🌸.
⚠️ Heads up: a few features of AMR-WB+ need special thought from a transport point of view, and the solutions might look a bit unorthodox 🙈. First, the RTP timestamping has constraints, because the frame duration and the ISFs are related 🕰️. Second, each frame of encoded audio has to keep info about its frame type, its ISF, and its position in the super-frame 💜.
3.2. Multi-rate Encoding and Rate Adaptation 🦋
The multi-rate encoding in AMR-WB+ is built to keep audio quality high across a wide range of bandwidth requirements and transmission conditions 💖.
AMR-WB+ allows seamless switching between frame types that use the same number of audio channels and the same ISF ✨. Every AMR-WB+ codec implementation is required to support all frame types the codec defines, and it has to be able to handle switching between any two frame types 💅. Switching between frame types with a different number of audio channels or a different ISF also has to be supported, but it might not be totally seamless 🙈. So it's recommended to do that kind of switching infrequently and, if you can, during silence 🤫🌙.
3.3. Voice Activity Detection and Discontinuous Transmission 🎤
AMR-WB+ uses the same algorithms as AMR-WB for voice activity detection (VAD) and for generating comfort noise (CN) parameters during silence periods 🌸. The catch is that these functions can only be used together with the AMR-WB frame types (FT=0-8) 🙃. This option cuts the number of transmitted bits and packets during silence down to a minimum 📉. Sending CN parameters at regular intervals during silence is usually called discontinuous transmission (DTX) or source controlled rate (SCR) operation 🔥. The AMR-WB+ frames that carry CN parameters are called Silence Indicator (SID) frames 🎀. For more on VAD and DTX, see [4] and [5] 📚.
3.4. Support for Multi-Channel Session 🎧
Some AMR-WB+ frame types can encode stereophonic audio 🎶. Because the codec natively supports a two-channel stereo signal, it doesn't seem necessary to support multi-channel transport with separate codec instances, the way the AMR-WB RTP payload [7] specifies 🙅♀️. The codec can also downmix stereo to mono as part of decoding 💜. So a receiver that can only play mono audio still has to be able to decode and play signals that were originally encoded and sent as stereo 🫶. That said, to avoid spending bits on a stereo encoding nobody's going to use, this specification defines a mechanism for signaling mono-only audio 🌈.
3.5. Unequal Bit-Error Detection and Protection 🛡️
The audio bits in each AMR-WB frame are sorted by how sensitive they are to bit errors perceptually 🧠. In cellular systems, you can exploit that to get better voice quality by using unequal error protection and detection (UEP and UED) mechanisms 📶. But the bits of the AMR-WB+ extension frame types don't have a consistent perceptual significance property, and they aren't sorted in that order 🙈. So UEP or UED is meaningless with the extension frame types 🚫. If you need UEP or UED for AMR-WB frame types, RFC 3267 [7] is the one to use 💖.
3.6. Robustness against Packet Loss 🌧️
The payload format supports two mechanisms for improving robustness against packet loss: simple forward error correction (FEC) and frame interleaving 🫶✨.
3.6.1. Use of Forward Error Correction (FEC) 🛡️
Okay so, generic forward error correction inside RTP is already defined, see RFC 2733 [11]. Audio redundancy coding has its own spec in RFC 2198 [12] 🎀. Either scheme can tuck extra redundant info into the RTP packet stream so it survives packet loss better, but the price is a higher bit rate. Go read either RFC for the tea on how that higher bit rate affects network congestion ✨.
Besides those media-unaware tricks, this doc specifies its very own AMR-WB+ flavour of audio redundancy coding, which may be beneficial in terms of packetization overhead 💅.
Conceptually, previously sent transport frames get bundled up with the brand new ones. A sliding window decides which frames go into each payload. Check out Figure 1 🦋.
--+--------+--------+--------+--------+--------+--------+--------+--
| f(n-2) | f(n-1) | f(n) | f(n+1) | f(n+2) | f(n+3) | f(n+4) |
--+--------+--------+--------+--------+--------+--------+--------+--
<---- p(n-1) ---->
<----- p(n) ----->
<---- p(n+1) ---->
<---- p(n+2) ---->
<---- p(n+3) ---->
<---- p(n+4) ---->Figure 1: An example of redundant transmission
Here, every frame gets resent exactly once, in the next RTP payload packet 🔁. f(n-2)...f(n+4) are the audio frames, and p(n-1)...p(n+4) are the payload packets.
The best part is that this mechanism needs zero signaling at session setup 🙌. The audio sender can just decide to use it without asking the receiver first. For a given timestamp, the receiver might get several copies of a frame, either carrying encoded audio data or frames marked as NO_DATA. The catch? It costs bandwidth, plus receiver delay, because the receiver has to wait for that redundant copy to show up 🥲.
This redundancy scheme does a similar job to RFC 2198, but it only works if both the original frames and the redundant copies are AMR-WB+ frames 🌸. If you want other media coding schemes, you've gotta go with RFC 2198.
The sender is the one in charge of picking a sensible amount of redundancy, based on feedback about channel conditions, for example from the RTP Control Protocol (RTCP) [3] receiver reports 📊. The sender is also on the hook for avoiding congestion, which redundancy can make worse (see Section 5 for more details) 🚦.
3.6.2. Use of Frame Interleaving 🧚
To cut down on protocol overhead, the payload design lets several audio transport frames get packed into one RTP packet. The downside? If a packet gets lost, several consecutive frames vanish with it 😭. Losing a run of consecutive frames usually makes error concealment less effective and leaves clearly audible, super annoying distortion in the reconstructed audio.
Interleaving the transport frames can save the day here, because it spreads those consecutive losses out into isolated single-frame losses, which are way easier to conceal ✨. But the catch is that interleaving and bundling several frames per payload also adds end-to-end delay and asks for bigger buffers 🫠. So interleaving isn't the right move for every use case or device. Streaming apps should most likely be able to make good use of it when the transmission is lossy 🎶.
Heads up: this payload design supports frame interleaving as an option 💜. Turning it on has to be negotiated during session setup.
The interleaving here is pretty flexible. For instance, you can define a continuous pattern, like the one in Figure 2 💅.
--+--------+--------+--------+--------+--------+--------+--------+--
| f(n-2) | f(n-1) | f(n) | f(n+1) | f(n+2) | f(n+3) | f(n+4) |
--+--------+--------+--------+--------+--------+--------+--------+--
[ P(n) ]
[ P(n+1) ] [ P(n+1) ]
[ P(n+2) ] [ P(n+2) ]
[ P(n+3) ] [P(
[ P(n+4) ]Figure 2: An example of interleaving pattern that has constant delay
In Figure 2, the consecutive frames f(n-2) through f(n+4) get bundled into packets P(n) through P(n+4), with each packet carrying two frames 🍓. This pattern gives you constant delay in both the interleaving and the deinterleaving processes, which is very iconic ✨. The deinterleaving buffer needs room for at least three frames, including the one that's ready to be consumed.
Why three? Picture f(n) being the next frame to decode. It came in packet P(n+2), which also carried f(n+3), so both of those frames sit in the buffer. On top of that, f(n+1), which arrived in the previous packet P(n+1), is in the deinterleaving buffer too 🧠. Also, the buffer occupancy isn't constant. When f(n+1) is the next one up to decode, there are only two frames in the buffer, f(n+1) and f(n+3) 🎀.
3.7. AMR-WB+ Audio over IP Scenarios 🌍
The AMR-WB+ codec is mainly built for streaming over packet networks, so the most relevant usage scenario for this payload format is IP end-to-end between a server and a terminal, as shown in Figure 3 💖.
+----------+ +----------+
| | IP/UDP/RTP/AMR-WB+ | |
| SERVER |<------------------------>| TERMINAL |
| | | |
+----------+ +----------+Figure 3: Server to terminal IP scenario
3.8. Out-of-Band Signaling 📡
Some options of this payload format stay the same for the whole session, so they can be controlled or negotiated at session setup. Throughout this spec, these options and variables are called "parameters to be established through out-of-band means" 🌈. Section 7 formally specifies all the parameters, in the form of a media type registration for AMR-WB+ encoding.
How those parameters actually get signaled at session setup, or agreed on ahead of time by the participants, is out of scope for this document 🙈. That said, Section 7.2 maps the parameters into the Session Description Protocol (SDP) [6], for applications that use SDP 💜.
4. RTP Payload Format for AMR-WB+ 🎀
The big goal of the AMR-WB+ payload design is to keep overhead tiny in typical use cases while still giving you full flexibility, even if that costs a bit more overhead 💅. To keep the spec reasonably simple, we didn't define frame-specific parameters for each frame type. Instead, we specified a handful of common parameters that cover every type of frame ✨.
The payload format has two modes: basic mode and interleaved mode 🦋. The main structural difference is that interleaved mode extends the table of contents entries with frame displacement fields. Basic mode supports aggregating multiple consecutive frames in one payload. Interleaved mode supports aggregating multiple frames that aren't consecutive in time. In both modes, you can have frames of different frame types in the same payload. The ISF must stay constant throughout the payload of a single packet 🔒.
The payload format is built around the fact that AMR-WB+ frames are consecutive in time and share the same frame duration, as long as the ISF doesn't change 🕰️. This lets the receiver work out the timestamp for each individual frame inside a payload. In basic mode, that works from the order of the frames. In interleaved mode, it works from the compact displacement fields. Those frame timestamps are what let the receiver put frames back in the right order after reception, spot duplicates, and detect lost frames that need concealment 🔍.
The interleaving scheme here is way more flexible than the one in RFC 3267 🤩. The AMR and AMR-WB payload format can only use periodic patterns, where frames are pulled from an interleaving group at fixed intervals. This spec's scheme, on the other hand, allows any interleaving pattern, as long as the distance in decoding order between any two adjacent frames is not more than 256 frames 🙌. Even at the highest ISF, that gives you an interleaving depth of up to 3.41 seconds, which is wild ⏱️.
To get error resiliency through redundant transmission, the periods covered by multiple packets MAY overlap in time 🫶. A receiver MUST be prepared to receive any audio frame multiple times, no excuses 😤. All redundantly sent frames MUST use the same frame type and ISF, and MUST have the same RTP timestamp, or MUST be a NO_DATA frame (FT=15) 🧾.
The payload is made of octet-aligned elements (header, ToC, and audio frames) 🧩. Only the audio frames for AMR-WB frame types (0-9) need padding for octet alignment. If you want extra padding, the P bit in the RTP header MAY be set, and padding MAY be appended as specified in [3] 🌸.
4.1. RTP Header Usage 📦
The RTP header format is specified in [3], and this payload format uses the header fields in a way that's consistent with that spec 💜.
The RTP timestamp matches the sampling instant of the first sample encoded for the first frame in the packet ⏰. The timestamp clock frequency SHALL be 72000 Hz. This frequency lets the frame duration be an integer number of RTP timestamp ticks for the ISFs listed in Table 1, and it also gives reasonable conversion factors to the input and output audio sampling frequencies the codec supports 🧮. For guidance on deriving the RTP timestamp for any audio frame after the first one, see Section 4.3.2.3 🗺️.
The RTP header marker bit (M) SHALL be set to 1 whenever the first frame carried in the packet is the first frame in a talkspurt (see the definition of talkspurt in Section 4.1 of [9]) 🗣️. For all other packets, the marker bit SHALL be set to zero (M=0) 🙃.
Assigning an RTP payload type for this format is outside the scope of this document 🎟️. The RTP profile in use either assigns a static payload type or requires binding the payload type dynamically.
The media type parameter channels says the maximum number of channels allowed for a given payload type 🎧. A payload type where channels=1 (mono) SHALL only carry mono content 🔈. A payload type that's declared with channels=2 MAY carry both mono and stereo content 🔊. Note that this definition is different from the one in RFC 3551 [9] 👀. As mentioned before, the AMR-WB+ codec handles stereo support and any (eventual) downmixing of stereo to mono internally 🫧. So you don't need to negotiate the number of channels for any reason other than bit-rate efficiency 💖.
4.2. Payload Structure 🎀
okay so every payload is basically a little gift box 🎁 with three parts: a payload header, a table of contents, and the audio data for one or more audio frames. here's the layout:
+----------------+-------------------+----------------
| payload header | table of contents | audio data ...
+----------------+-------------------+----------------payloads with more than one audio frame are called compound payloads 💖. the mode you're in decides which variation of the format you get: basic mode or interleaved mode 🦋.
4.3. Payload Definitions 💅
4.3.1. Payload Header 🌸
the payload header carries the stuff that's shared by every frame in the payload, so it's like the group-chat announcement everyone needs to see first. here's its shape:
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| ISF |TFI|L|
+-+-+-+-+-+-+-+-+ISF (5 bits): the Internal Sampling Frequency used for all frames in this payload. the index maps to the internal sampling frequency in Table 24 in [1]. this field SHALL be set to 0 for payloads containing frames with Frame Type values 0-13.
TFI (2 bits): Transport Frame Index, from 0 (first) to 3 (last). it tells you where the payload's first transport frame sits in the AMR-WB+ super-frame structure 🧚. for payloads with frames of only Frame Type values 0-9, this field SHALL be set to 0 by the sender. the TFI value for a frame of type 0-9 SHALL be ignored by the receiver. the frame type is coded in the table of contents (more on that soon), so you can apply those frame-type dependencies just by reading the payload header. you do NOT need to dig into the audio bitstream itself 🙈.
L (1 bit): the long displacement field flag for payloads in interleaved mode. if it's set to 0, the displacement fields use four bits. if it's set to 1, they use eight bits (see Section 4.3.2.2). for payloads in the basic mode, this bit SHALL be set to 0 and SHALL be ignored by the receiver.
⚠️ bestie, this part is important: frames with different ISF values have to be wrapped in separate packets. so when you're generating interleaved packets and an ISF change happens, you have to be extra careful. frames that would've been aggregated into one packet under the old interleaving pattern need to be split up so that the rule (all frames in a packet share the ISF) still holds 🔥. a naive implementation that splits the frames with different ISF into different packets can end up with up to twice the number of RTP packets compared to an optimal interleaved solution 💀. changing the interleaving before and after the ISF change may reduce the need for those extra RTP packets ✨.
4.3.2. The Payload Table of Contents 📜
the table of contents (ToC) is a list of entries, and each entry describes a group of audio frames in the payload. it looks like this:
+----------------+----------------+- ... -+----------------+
| ToC entry #1 | ToC entry #2 | ToC entry #N |
+----------------+----------------+- ... -+----------------+when there are multiple groups of frames in one payload, the ToC entries SHALL be placed in the packet in order of increasing RTP timestamp value (modulo 2^32) of the first transport frame that the ToC entry represents. very organized, very iconic 👑.
4.3.2.1. ToC Entry in the Basic Mode 🍓
a ToC entry in the basic mode looks like this:
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| Frame Type | #frames |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+F (1 bit): if it's set to 1, this ToC entry is followed by another ToC entry. if it's set to 0, this ToC entry is the last one in the ToC 💜.
Frame Type (FT) (7 bits): the audio codec frame type used for the group of frames this ToC entry references. FT covers the combination of AMR-WB+ core and stereo rate, one of the special AMR-WB+ frame types, the AMR-WB rate, or comfort noise, as specified by Table 25 in [1]. it's giving "menu" energy 🍽️.
#frames (8 bits): the number of frames in the group this ToC entry references. ToC entries with this field equal to 0 (which would mean zero frames) SHALL NOT be used, and received packets with such a TOC entry SHALL be discarded 🚫.
4.3.2.2. ToC Entry in the Interleaved Mode 🦋
there are two ToC entry formats in interleaved mode. they differ in how long the displacement field is: 4 bits or 8 bits. the L-bit in the payload header is what tells them apart.
if L=0, the ToC entry looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| Frame Type | #frames | DIS1 | ... | DISi | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | ... | DISn | Padd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+F (1 bit): see the definition in 4.3.2.1.
Frame Type (FT) (7 bits): see the definition in 4.3.2.1.
#frames (8 bits): see the definition in 4.3.2.1.
DIS1...DISn (4 bits): a list of n (n=#frames) displacement fields. each one shows how far the i:th (i=1..n) audio frame is from the preceding audio frame in the payload, counted in frames. the four-bit unsigned integer displacement values can be between 0 and 15, giving the number of audio frames in decoding order between the (i-1):th and the i:th frame in the payload. for the first ToC entry of the payload, the value of DIS1 is meaningless. it SHALL be set to zero by a sender and SHALL be ignored by a receiver. this frame's location in the decoding order is uniquely defined by the RTP timestamp and TFI in the payload header. for subsequent ToC entries, DIS1 tells you the number of frames between the last frame of the previous group and the first frame of this group 🫶.
Padd (4 bits): to keep everything octet aligned, four padding bits SHALL be included at the end of the ToC entry when the group it references has an odd number of frames 🧩. these bits SHALL be set to zero and SHALL be ignored by the receiver. if the group referenced by this ToC entry contains an even number of frames, these padding bits SHALL NOT be included in the payload.
if L=1, the ToC entry looks like this:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| Frame Type | #frames | DIS1 | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | DISn |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+F (1 bit): see the definition in 4.3.2.1.
Frame Type (FT) (7 bits): see the definition in 4.3.2.1.
#frames (8 bits): see the definition in 4.3.2.1.
DIS1...DISn (8 bits): a list of n (n=#frames) displacement fields. each one shows how far the i:th (i=1..n) audio frame is from the preceding audio frame in the payload, counted in frames. the eight-bit unsigned integer displacement values can be between 0 and 255, giving the number of audio frames in decoding order between the (i-1):th and the i:th frame in the payload. for the first ToC entry of the payload, the value of DIS1 is meaningless. it SHALL be set to zero by a sender and SHALL be ignored by a receiver. this frame's location in the decoding order is uniquely defined by the RTP timestamp and TFI in the payload header. for subsequent ToC entries, DIS1 is the displacement between the last frame of the previous group and the first frame of this group 💅.
4.3.2.3. RTP Timestamp Derivation ⏰
the RTP Timestamp value for a frame SHALL be the timestamp value of the first audio sample encoded in the frame 🎯. how it's derived depends on whether the payload is in basic mode or interleaved mode. in both cases, the first frame in a compound packet has an RTP timestamp equal to the one in the RTP header.
in the basic mode, the RTP time for any later frame is worked out in two steps. first, you add up the frame durations (see Table 1) of all the preceding frames in the payload. then you add that sum to the RTP header timestamp value. for example, say the RTP header timestamp is 12345, the payload carries four frames, and each frame lasts 16 ms (ISF = 32 kHz), which is 1152 timestamp ticks. then the RTP timestamp of the fourth frame in the payload is:
12345 + 3 * 1152 = 15801in interleaved mode, the RTP timestamp for each frame in the payload comes from the RTP header timestamp plus the sum of the time offsets of all the preceding frames in this payload. the frame timestamps are computed from the displacement fields and the frame duration derived from the ISF value. the displacement in time between frame i-1 and frame i is (DISi + 1) * frame duration, because the duration of the (i-1):th frame has to be counted too 🧠. the timestamp of the first frame of the first group of frames (TS(1)), which is the first frame of the payload, is the RTP header timestamp. for the subsequent frames in the group, the timestamp is computed by:
TS(i) = TS(i-1) + (DISi + 1) * frame duration, 2 < i < nfor subsequent groups of frames, the timestamp of the first frame is computed by:
TS(1) = TSprev + (DIS1 + 1) * frame duration,where TSprev is the timestamp of the last frame in the previous group. the timestamps of the subsequent frames in the group are computed in the same way as for the first group 🌈.
the following example derives the RTP timestamps for the frames in an interleaved mode payload with this header and ToC information:
RTP header timestamp: 12345
ISF = 32 kHz
Frame 1 displacement field: DIS1 = 0
Frame 2 displacement field: DIS2 = 6
Frame 3 displacement field: DIS3 = 4
Frame 4 displacement field: DIS4 = 7with an ISF of 32 kHz, the frame duration is 16 ms, so one frame lasts 1152 ticks. the timestamp of the first frame in the payload is the RTP timestamp, so TS(1) = RTP TS. the displacement field value for this frame must be ignored 🙃. for the second frame, the timestamp is:
TS(2) = TS(1) + (DIS2 + 1) * 1152 = 20409for the third frame:
TS(3) = TS(2) + (DIS3 + 1) * 1152 = 26169and finally, for the fourth frame of the payload:
TS(4) = TS(3) + (DIS4 + 1) * 1152 = 353854.3.2.4. Frame Type Considerations ✨
okay so the Frame Type (FT) values are all listed in Table 25 in [1] 💅. FT=14 (AUDIO_LOST) is how we flag frames that got lost 😭. FT=15 (NO_DATA) can show up for two different reasons: either the audio encoder didn't produce any data, or the data just isn't in this payload. For example, that frame might have been sent in an earlier or later packet instead 🫶. The duration of these non-included frames depends on the internal sampling frequency that the ISF field tells you about ⭐.
For frame types with index 0-13, the ISF field SHALL be set 0. The frame duration for these types is locked at 20 ms, which is 1440 ticks in 72 kHz 🎀. For payloads that only have frames of type 0-9, the TFI field SHALL be set to 0, and the receiver SHALL ignore it 🙈. If a payload mixes frames of type 0-9 and 10-13, the TFI values have to match the transport frames of type 10-13. So frames of type 0-9 end up with a derived TFI too, but it gets ignored anyway 🧚.
4.3.2.5. Other TOC Considerations 💜
If a ToC entry comes in with an FT value that's undefined, the whole packet SHALL be discarded 🚫. The reason is super important: skipping it keeps the depacketization process in sync, and getting out of sync can seriously wreck the audio quality 💀.
Packets that only contain NO_DATA frames SHOULD NOT be transmitted 🍓. And NO_DATA frames sitting at the end of a frame sequence in a payload SHOULD NOT be included in the packet that gets sent. Also, the AMR-WB+ SCR/DTX is exactly the same as the AMR-WB SCR/DTX described in [5], and it can only be used together with the AMR-WB frame types (0-8) 🌸.
When there are multiple groups of frames, their ToC entries SHALL be ordered by increasing RTP timestamp value (modulo 2^32) of the first transport frame each entry represents. This applies no matter which payload mode you're in 💖. In basic mode, the frames SHALL be consecutive in time. In interleaved mode, though, the frames MAY not only be non-consecutive in time, they MAY even have varying distances between frames 🦋.
4.3.2.6. ToC Examples 🎀
Here's a ToC for three audio frames in basic mode. Note that all three use the same frame type, so there's only one ToC entry 🌈.
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| Frame Type1 | #frames = 3 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Next up is a ToC with three entries in basic mode. The payload still carries three frames, but here we need three ToC entries because each frame uses a different frame type 👑.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| Frame Type1 | #frames = 1 |1| Frame Type2 | #frames = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| Frame Type3 | #frames = 1 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+This one shows a ToC with two entries in interleaved mode, using four-bit displacement fields 🧚. The payload has two groups of frames: the first group is a single frame, and the second group has two frames 💅.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| Frame Type1 | #frames = 1 | DIS1 | padd |0| Frame Type2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| #frames = 2 | DIS1 | DIS2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+4.3.3. Audio Data 🎧
The audio data in a payload is made up of zero or more audio frames, which are described in the payload's ToC ✨.
ToC entries with FT=14 or 15 stand for frame types with a length of 0. So no data SHALL be placed in the audio data section for these types 🙅.
As we covered already, each audio frame of an extension frame type is one AMR-WB+ transport frame, encoding 512 samples of audio at the internal sampling frequency that the ISF indicator gives 🔥. The one exception is frame types with index 10-13, which can only use a single internal sampling frequency (25600 Hz) 🌟. The encoding rates (the combo of core bit-rate and stereo bit-rate) are shown in the frame type field of the matching ToC entry 💖. The octet length of an audio frame is implied by its frame type field, and you can look it up in Tables 21 and 25 of [1] 📚. The order and numbering of the bits follow [1] too. For the AMR-WB+ extension frame types and comfort noise frames, the bits are in the order the encoder produced them 🦋.
Now the big one: the last octet of each audio frame MUST be padded with zeroes at the end if not all of its bits are used. Put simply, each audio frame MUST be octet-aligned 🫶.
4.3.4. Methods for Forming the Payload 🧵
The payload starts with the payload header, and then comes the table of contents, which is a list of ToC entries 🌸.
The audio data follows the table of contents. All the octets of an audio frame SHALL be appended to the payload as one unit 💅. Audio frames get packetized in timestamp order inside each group of frames (one group per ToC entry). The groups themselves get packetized in the same order as their ToC entries 🍓. Also, a group whose ToC entry has FT=14 or FT=15 has no data octets at all 🙈.
4.3.5. Payload Examples 🌈
4.3.5.1. Example 1: Basic Mode Payload Carrying Multiple Frames Encoded Using the Same Frame Type 🎀
Figure 4 shows a payload with three AMR-WB+ frames, all encoded with the 14 kbit/s frame type (FT=26), where each frame is 280 bits (35 bytes) long 💖. The internal sampling frequency here is 25.6 kHz (ISF = 8). The TFI for the first frame is 2, which means the first transport frame in this payload is the third one in a super-frame 🧚. Since this payload is in basic mode, the next frames are consecutive in decoding order: the fourth transport frame of the current super-frame, then the first transport frame of the next super-frame 🦋. All three frames share one frame type, so only one ToC entry is needed 👑.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ISF = 8 | 2 |0|0| FT = 26 | #frames = 3 | f1(0...7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f1(272...279) | f2(0...7) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| f2(272...279) | f3(0...7) | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f3(272...279) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 4: An example of a basic mode payload carrying three frames of the same frame type
4.3.5.2. Example 2: Basic Mode Payload Carrying Multiple Frames Encoded Using Different Frame Types 💜
Figure 5 shows a payload with three AMR-WB+ frames. The first frame uses the 18.4 kbit/s frame type (FT=33), with a frame length of 368 bits (46 bytes). The next two frames use the 20 kbit/s frame type (FT=35), each 400 bits (50 bytes) long 🌟. The internal sampling frequency in this example is 32 kHz (ISF = 10), which works out to overall bit-rates of 23 kbit/s for the first frame of the payload, and 25 kbit/s for the later frames 🔥. The TFI for the first frame is 3, meaning the first transport frame in this payload is the fourth in a super-frame 🧚. We're still in basic mode, so the subsequent frames are consecutive in decoding order: the first and second transport frames of the current super-frame 🦋. Because the payload uses two different frame types, there are two ToC entries 💅.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ISF=10 | 3 |0|1| FT = 33 | #frames = 1 |0| FT = 35 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| #frames = 2 | f1(0...7) | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f1(360...367) | f2(0...7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| f2(392...399) | f3(0...7) | ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f3(392...399) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 5: An example of a basic mode payload carrying three frames employing two different frame types
4.3.5.3. Example 3: Payload in Interleaved Mode ✨
okay so this one is a payload in interleaved mode 🧚 it's carrying four frames, all encoded with the 32 kbit/s frame type (FT=47), and each frame is 640 bits (80 bytes) long. the internal sampling frequency is 38.4 kHz (ISF = 13), which means every frame in the payload works out to a bit-rate of 48 kbit/s 🎀. the TFI for the first frame is 0, so it's the first transport frame of a super-frame 👑. the displacement fields for the rest are DIS2=18, DIS3=15, and DIS4=10, and those tell us the next frames have TFIs of 3, 3, and 2, in that order 💅
the long displacement field flag L in the payload header is set to 1, so the displacement fields in the ToC entry are eight bits each 🦋. since all four frames use the same frame type, we only need a single ToC entry, which is giving minimalism 🫶. also, the displacement field for the first frame (the one at DIS1=0 in the first ToC entry) must be ignored 😭, because its timestamp and TFI come from the RTP timestamp and the TFI in the payload header, not from the displacement.
here are the RTP timestamp values for the frames in this example:
Frame1: TS1 = RTP Timestamp
Frame2: TS2 = TS1 + 19 * 960
Frame3: TS3 = TS2 + 16 * 960
Frame4: TS4 = TS3 + 11 * 960and here's the payload layout, figure 6, so you can see exactly how the bits sit 🍓
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ISF=13 | 0 |1|0| FT = 47 | #frames = 4 | DIS1 = 0 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DIS2 = 18 | DIS3 = 15 | DIS4 = 10 | f1(0...7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f1(632...639) | f2(0...7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f2(632...639) | f3(0...7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f3(632...639) | f4(0...7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: ... :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ... | f4(632...639) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 6: An example of an interleaved mode payload carrying four frames at the same frame type
4.4. Interleaving Considerations 🌈
okay, interleaving needs a bit more thought 🧠 like we saw in the example in Section 3.6.2, a given interleaving pattern needs a certain chunk of the deinterleaving buffer. that buffer space is counted in transport frame slots, and it's signalled by the "interleaving" media type parameter 📦. to turn those slots into real memory, count 80 bytes per frame, which is what the biggest combo of AMR-WB+'s core and stereo rates uses 💜
knowing the buffer size isn't always enough to know when it's safe to start pulling frames out of the interleaving buffer 🙈. there are two cases where you need extra info: first, when the ISF switches, and second, when the interleaving pattern changes. that's what the "int-delay" media type parameter is for ✨ it lets a sender say the minimal media time that has to be in the buffer before the decoder can start consuming frames. the sender has full control over ISF changes and the interleaving pattern, so it's the one that can work this number out 🎀
now, in some cases (like joining a multicast session with interleaving mid-session), a receiver might only get part of the packets in the pattern at first 😬. if it only gets a few frames at the start (in frame sequence order), there might not be enough for acceptable audio quality. this same problem shows up when you use encryption for access control and the receiver doesn't have the previous key 🔐
AMR-WB+ is robust and can tolerate a high random frame erasure rate, but it would struggle with consecutive frame losses at startup 🥺. so there are some special implementation considerations. first, decoding can only start at the beginning of a super-frame, and that holds even if the first transport frame is marked as lost 👑. second, decoding is only RECOMMENDED to start if at least 2 transport frames are available out of the 4 belonging to that super-frame 💅
after a receiver gets a bunch of packets (in the worst case, as many packets as the interleaving pattern covers), those startup effects go away and normal decoding picks back up 🦋
similar stuff happens when a receiver leaves a session or loses access to the stream 🚪. if it leaves the session, that's a minor issue, since playout usually stops anyway. it's also minor for lost access, because AMR-WB+'s error concealment will fade out the audio if there are massive consecutive losses 🌸
the sender can dodge this kind of problem in a lot of sessions by starting and ending interleaving patterns properly when losses are likely 🫶. one example is a key change for access control to encrypted streams. if only some clients get the keys and there's a risk they'll receive content they don't have the key for, it is recommended that interleaving patterns do not overlap key changes 🔑
4.5. Implementation Considerations 💖
an application implementing this payload format MUST understand all the payload parameters 🤓. any mapping of the parameters to a signaling protocol MUST support all parameters too. so an implementation in an application using SDP is required to understand all the payload parameters in their SDP-mapped form. this makes sure an implementation can always decide whether it's able to communicate 🎯
both basic and interleaved mode SHALL be implemented 🙌. the implementation burden of each is pretty small, and requiring both keeps everyone interoperable. since AMR-WB+ contains the full functionality of AMR-WB, it is RECOMMENDED to also implement the payload format in RFC 3267 [7] for the AMR-WB frame types when you implement this spec. doing that makes it more likely you'll interoperate with devices that only support AMR-WB 🍓
heads up, bestie 🚨: switching ISF combined with packet loss can make concealment use the wrong audio frame length. this happens when losses hit the frames right after the ISF change. the packet loss stops the receiver from noticing the new ISF, so it conceals the lost transport frame with the old ISF instead of the new one. it's always detectable later, but the result is frame boundary misalignment, which can cause audio distortions and sync problems, because too many or too few audio samples got created 😭. in most cases, this can be mitigated by doing ISF recovery before concealment, as outlined in Section 4.5.1 🩹
4.5.1. ISF Recovery in Case of Packet Loss 🔍
when packets get lost, it's important that the AMR-WB+ decoder starts a proper error concealment to replace the frames that were in the lost packet 🛠️. a loss concealment algorithm needs a codec framing that matches the timestamps of the frames that did arrive correctly, so you have to recover the timestamps of the lost frames. that's honestly non-trivial, because the codec frame length tied to the ISF may have changed during the loss 😵💫
here's an example of how timestamp recovery works. two frames with timestamps t0 and t1 were received properly. the first one is the last packet before the loss, and the second is the first packet after the loss period. their ISF values are isf0 and isf1, their TFIs are tfi0 and tfi1, and their associated frame lengths (in timestamp ticks) are L0 and L1 🎀. in this example, three frames with timestamps x1 through x3 were lost. the example also assumes the ISF changes once, from isf0 to isf1, during the loss period, as shown below 🦋
since the receiver doesn't always know everything it needs for full timestamp recovery, it needs an algorithm to estimate the ISF of the lost frames. it also needs to recover how many frames were lost 🧩
|<---L0--->|<---L0--->|<-L1->|<-L1->|<-L1->|
| Rxd | lost | lost | lost | Rxd |
--+----------+----------+------+------+------+--
t0 x1 x2 x3 t1here's the example algorithm 💅
Start: # check for frame loss
If (t0 + L0) == t1 Then goto End # no frame loss
Step 1: # check case with no ISF change
If (isf0 != isf1) Then goto Step 2 # At least one ISF change
If (isFractional(t1 - t0)/L0) Then goto Step 3
# More than 1 ISF change
Return recovered timestamps as
x(n) = t0 + n*L1 and associated ISF equal to isf0,
for 0 < n < (t1 - t0)/L0
goto End
Step 2:
Loop initialization: n := 4 - tfi0 mod 4
While n <= (t1-t0)/L0
Evaluate m := (t1 - t0 - n*L0)/L1
If (isInteger(m) AND ((tfi0+n+m) mod 4 == tfi1)) Then goto found;
n := n+4
endloop
goto step 3 # More than 1 ISF change
found:
Return recovered timestamps and ISFs as
x(i) = t0 + i*L0 and associated ISF equal to isf0, for 0 < i <= n
x(i) = t0 + n*L0 + (i-n)*L1 and associated ISF equal to isf1,
for n < i <= n+m
goto End
Step 3:
More than 1 ISF change has occurred. Since ISF changes can be
assumed to be infrequent, such a situation occurs only if long
sequences of frames are lost. In that case it is probably not useful
to try to recover the timestamps of the lost frames. Rather, the
AMR-WB+ decoder should be reset, and decoding should be resumed
starting with the frame with timestamp t1.
End:the algorithm still doesn't solve everything 😅. when the receiver buffer is shallower than the loss burst, the concealment has to happen without knowing anything about future frames, and that can lose frame boundary alignment. if that happens, it may be necessary to reset and restart the codec so it can resynchronize 🔄
4.5.2. Decoding Validation ✅
if the receiver finds a mismatch between the size of a received payload and the size the ToC of the payload says it should be, the receiver SHOULD discard the packet 🗑️. this is recommended because decoding a frame parsed from a payload with erroneous ToC data could seriously wreck the audio quality 💀
5. Congestion Control 🌸
okay so congestion control for RTP data is pretty much the same as always 💅 you still check out RTP [3] and whatever RTP profile applies, like AVP [9]. but here's the lowkey iconic part: AMR-WB+ has multi-rate capability, which means it can help control congestion 🦋 you can change the bandwidth demand (within the limits of the codec) by picking a different coding frame type or a lower internal sampling rate. it's like having a dimmer switch for your data 🔥
the number of frames you stuff into each RTP payload seriously affects the overall bandwidth of the stream because of header overhead 📦 packing more frames per payload means fewer packets get sent, so less header overhead 🙌 the tradeoff is more delay and less error robustness 😭 so it's a vibe check every time.
and if you're using forward error correction (FEC), you have to keep the FEC-induced redundancy in check 🫶 otherwise FEC itself could cause a congestion problem, and that's just not the move 💀
6. Security Considerations 🔐
RTP packets using this payload format are subject to the general security considerations from RTP [3] and any applicable profile, like AVP [9] or SAVP [10] 💖 since this format carries encoded audio, the big security concerns are confidentiality, integrity protection, and data origin authentication of the audio itself 🎀 the payload format doesn't have any built-in security mechanisms of its own, so any suitable external mechanism, such as SRTP [10], MAY be used. your call, bestie ✨
also, this payload format and the AMR-WB+ decoder don't show any significant non-uniformity in receiver-side computational complexity for packet processing 🧚 so they're unlikely to be a denial-of-service threat from pathological data. no sus packets making it lag out 🙅♀️
6.1. Confidentiality 🔒
okay this one is SERIOUS 😭 to keep the encoded audio confidential, all audio data bits MUST be encrypted. no exceptions, no "it's just a little bit" 💀 the payload header and the table of contents are less of a priority to encrypt, since they only tell you the frame type. that info could still be useful to a third party though, like for quality monitoring 👀
now the interleaving plot twist 🌀 using interleaving together with encryption can hurt confidentiality for a short period of time. picture these packets (in brackets) with the frame numbers listed: {10, 14, 18}, {13, 17, 21}, {16, 20, 24} (a popular continuous diagonal interleaving pattern 🍓). say the originator wants to stop some participants from hearing material starting at time 16. just changing the key on the packet with the timestamp at or after 16, and not giving that new key to those participants, does NOT do the job 🙅♀️ frames 17, 18, and 21 already went out in earlier packets under the old key, and error concealment might make the audio intelligible at least up through frame 18 or 19, and possibly further. so the lock is leaky, bestie 😭
6.2. Authentication and Integrity 🛡️
to authenticate the sender of the speech, an external mechanism MUST be used, no shortcuts 🙅♀️ it is RECOMMENDED that this mechanism protects both the complete RTP header and the payload (speech and data bits) 💜 that way nothing sneaks by unnoticed.
and yes, a man-in-the-middle attacker could tamper with the data, swap out the audio content, and cause erroneous depacketization or decoding that lowers the audio quality 💔 the vibes would be ruined, periodt.
7. Payload Format Parameters 🎀
this section is where we define the parameters you can use to pick features of the AMR-WB+ payload format ✨ they're defined as part of the media type registration for the AMR-WB+ audio codec. there's also a mapping of the parameters into the Session Description Protocol (SDP) [6] for apps that use SDP 🦋 and if you're working with control protocols that don't use MIME or SDP, equivalent parameters could be defined somewhere else 🌈
one thing to note: the data format and parameters are ONLY specified for real-time transport in RTP 🙈 not for anything else, okay?
7.1. Media Type Registration 📝
the media type for the Extended Adaptive Multi-Rate Wideband (AMR-WB+) codec is allocated from the IETF tree 👑 that's because AMR-WB+ is expected to be a widely used audio codec for general streaming apps. big main character energy 🌟
Note: Parameters not listed below MUST be ignored by the receiver. no exceptions, so don't try to be clever 🙅♀️
here's the lineup:
- Media Type name:
audio - Media subtype name:
AMR-WB+ - Required parameters: None
- Optional parameters:
- channels: the maximum number of audio channels used by the audio frames. permissible values are
1(mono) or2(stereo). if no parameter is present, the maximum is 2 (stereo) 🎧 note: when it's set to 1, the stereo frame types implicitly can't be used. - interleaving: indicates that interleaved mode SHALL be used for the payload 🌀 this parameter specifies the number of transport frame slots needed in a deinterleaving buffer (including the frame that's ready to be consumed). its value equals one plus the maximum number of frames that come before any frame in transmission order and come after that frame in RTP timestamp order. the value MUST be greater than zero. if this parameter is not present, interleaved mode SHALL NOT be used 🙅♀️
- int-delay: the minimal media time delay, in RTP timestamp ticks, needed in the deinterleaving buffer 🕰️ basically the difference in RTP timestamp ticks between the earliest and latest audio frame sitting in that buffer.
- ptime: see Section 6 in RFC 2327 [6] ✨
- maxptime: see Section 8 in RFC 3267 [7] ✨
- channels: the maximum number of audio channels used by the audio frames. permissible values are
- Restriction on Usage: this type is only defined for transfer via RTP (STD 64) 🚫 nothing else, bestie.
- Encoding considerations: an RTP payload following this format is binary data, so it may need to be encoded appropriately in non-binary environments 💻 but as long as it's used within RTP, no encoding is necessary.
- Security considerations: see Section 6 of RFC 4352 🔐
- Interoperability considerations: to keep things working smoothly with AMR-WB-capable endpoints, where negotiation is possible and the AMR-WB+ endpoint also supports RFC 3267 for AMR-WB transport, an AMR-WB+ endpoint SHOULD also declare itself as AMR-WB capable (so supporting
audio/AMR-WBas specified in RFC 3267) 🤝 also, since the AMR-WB+ decoder can do stereo to mono conversions, all receivers of AMR-WB+ should be able to receive both stereo and mono, even though a receiver can only play out mono signals 🎶 - Public specification: RFC 4352 and 3GPP TS 26.290, see reference [1] of RFC 4352 📚
- Additional information: this MIME type is NOT applicable for file storage 🙅♀️ instead, file storage of AMR-WB+ encoded audio is specified within the 3GPP-defined ISO-based multimedia file format in 3GPP TS 26.244, see reference [14] of RFC 4352 🗂️ that file format uses the MIME types
audio/3GPPorvideo/3GPPas defined by RFC 3839 [15]. - Person & email address to contact for further information:
magnus.westerlund@ericsson.comandari.lakaniemi@nokia.com💌 - Intended usage: COMMON. it's expected that many IP-based streaming applications will use this type 🌸
- Change controller: IETF Audio/Video Transport working group, delegated from the IESG 👑
7.2. Mapping Media Type Parameters into SDP 🗺️
the info in the media type spec maps to specific fields in the Session Description Protocol (SDP) [6], which is commonly used to describe RTP sessions 💜 when SDP specifies an RTP session using this payload format, the mapping goes like this:
- the media type (
audio) is used in the SDPm=line as the media name 🎵 - the media type (payload format name) is used in the SDP
a=rtpmapas the encoding name. the RTP clock rate ina=rtpmapSHALL be72000for AMR-WB+ ⏱️ and the encoding parameter for number of channels MUST either be explicitly set to1or2, or be omitted, which implies the default value of2🎧 - the
ptimeandmaxptimeparameters go in the SDP attributesa=ptimeanda=maxptime, respectively ✨ - any remaining parameters go in the SDP
a=fmtpattribute by copying them directly from the MIME media type string, as a semicolon-separated list of parameter=value pairs 🧁
that's the end of part 7 of 8, bestie!! 🌈💖 we're basically at the finish line ✨
7.2.1. Offer-Answer Model Considerations 🎀
okay bestie, so for good interop in an Offer-Answer [8] negotiation, you should keep these considerations in mind 💅
For negotiable offer/answer usage, these interpretation rules SHALL be applied: 👑
- ✨ interleaving is symmetric, so the answerer must include it too when answering a payload type that has the parameter. The buffer space value is declarative in unicast. For multicast, the answer has to carry the same value or the payload type doesn't get accepted. For sendrecv or recvonly streams, the receiver will accept streams using the interleaved mode of the payload format, and the value tells the sender how much buffer space the receiver has for it to use. For sendonly streams, the parameter shows the desired configuration and buffer space. An answerer is RECOMMENDED to respond with the offered value, if it's capable of using it. 🦋
- 🌸 int-delay is declarative. For sendrecv or recvonly streams, it's the max initial delay the receiver will accept in the deinterleaving buffer. For sendonly streams, it's the amount of media time the sender wants to use. The value SHOULD be copied into any response. 💖
- 🧚 channels is declarative too. For sendonly streams, it says whether the desired channel usage is stereo and mono, or mono only. For recvonly and sendrecv streams, it says what the receiver is okay using. Any receiver can take a stereo frame type and do local mixing inside the AMR-WB+ decoder, so there's basically only one reason to restrict to mono: not wasting bit-rate on data that won't get used when the front-end is mono-only. 🍓
- 🫶 ptime works the way the offer/answer model [8] says, and maxptime SHALL be used in the same way. 💜
- 🌈 For interop with AMR-WB where negotiation is possible, an AMR-WB+ capable endpoint that also implements the AMR-WB payload format [7] is RECOMMENDED to declare itself capable of AMR-WB, since it's a subset of the AMR-WB+ codec. ⭐
In declarative usage, like SDP in RTSP [16] or SAP [17], the following interpretation of the parameters SHALL be done: 🔥
- 🌷 The interleaving parameter, if present, configures the payload format in that mode, and its value is the number of frames the deinterleaving buffer is required to support so the session gets handled correctly.
- 🎀 The int-delay parameter is the initial buffering delay needed to receive the stream correctly.
- 🌟 The channels parameter says whether the content being sent can have both stereo and mono rates, or only mono.
- 💅 All other parameters are values the sending entity is using.
7.2.2. Examples 🦋
okay, here's one SDP session description using AMR-WB+ mono and stereo encoding, no cap 🎧
m=audio 49120 RTP/AVP 99
a=rtpmap:99 AMR-WB+/72000/2
a=fmtp:99 interleaving=30; int-delay=86400
a=maxptime:100quick note: the payload format (encoding) names are usually written in uppercase, and media subtypes are usually lowercase. Both are case-insensitive. Parameter names are case-insensitive too, both in MIME types and in the default mapping to the SDP a=fmtp attribute. 💖
8. IANA Considerations 🌈
IANA registered one new MIME subtype (audio/amr-wb+), so go check out Section 7 for the deets ✨
9. Contributors 🫶
Daniel Enstrom wrote the codec introduction section, and Stefan Bruhn wrote the ISF recovery algorithm. Iconic teamwork 👑
10. Acknowledgements 💜
The authors want to thank Redwan Salami and Stefan Bruhn for their huge contributions while writing and reviewing this document. Dave Singer reviewed it and suggested better language. Anisse Taleb and Ingemar Johansson implemented the payload format, which helped them find some flaws 🙈 Last but not least, a big shoutout to Qiaobing Xie, coauthor of RFC 3267, which this document is based on 🌸
11. References 📚
11.1. Normative References
- [1] 3GPP TS 26.290 "Audio codec processing functions; Extended Adaptive Multi-Rate Wideband (AMR-WB+) codec; Transcoding functions", version 6.3.0 (2005-06), 3rd Generation Partnership Project (3GPP).
- [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [3] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003.
- [4] 3GPP TS 26.192 "AMR Wideband speech codec; Comfort Noise aspects", version 6.0.0 (2004-12), 3rd Generation Partnership Project (3GPP).
- [5] 3GPP TS 26.193 "AMR Wideband speech codec; Source Controlled Rate operation", version 6.0.0 (2004-12), 3rd Generation Partnership Project (3GPP).
- [6] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998.
- [7] Sjoberg, J., Westerlund, M., Lakaniemi, A., and Q. Xie, "Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs", RFC 3267, June 2002.
- [8] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with Session Description Protocol (SDP)", RFC 3264, June 2002.
- [9] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, July 2003.
11.2. Informative References
- [10] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, March 2004.
- [11] Rosenberg, J. and H. Schulzrinne, "An RTP Payload Format for Generic Forward Error Correction", RFC 2733, December 1999.
- [12] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V., Handley, M., Bolot, J., Vega-Garcia, A., and S. Fosse-Parisis, "RTP Payload for Redundant Audio Data", RFC 2198, September 1997.
- [13] 3GPP TS 26.233 "Packet Switched Streaming service", version 5.7.0 (2005-03), 3rd Generation Partnership Project (3GPP).
- [14] 3GPP TS 26.244 "Transparent end-to-end packet switched streaming service (PSS); 3GPP file format (3GP)", version 6.4.0 (2005-09), 3rd Generation Partnership Project (3GPP).
- [15] Castagno, R. and D. Singer, "MIME Type Registrations for 3rd Generation Partnership Project (3GPP) Multimedia files", RFC 3839, July 2004.
- [16] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, April 1998.
- [17] Handley, M., Perkins, C., and E. Whelan, "Session Announcement Protocol", RFC 2974, October 2000.
- [18] 3GPP TS 26.140 "Multimedia Messaging Service (MMS); Media formats and codes", version 6.2.0 (2005-03), 3rd Generation Partnership Project (3GPP).
- [19] 3GPP TS 26.140 "Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs", version 6.3.0 (2005-12), 3rd Generation Partnership Project (3GPP).
Any 3GPP document can be downloaded from the 3GPP webserver, "http://www.3gpp.org/", see specifications. 🌐
Authors' Addresses 💌
Johan Sjoberg
Ericsson Research
Ericsson AB
SE-164 80 Stockholm
SWEDEN
Phone: +46 8 7190000
EMail: Johan.Sjoberg@ericsson.com
Magnus Westerlund
Ericsson Research
Ericsson AB
SE-164 80 Stockholm
SWEDEN
Phone: +46 8 7190000
EMail: Magnus.Westerlund@ericsson.com
Ari Lakaniemi
Nokia Research Center
P.O. Box 407
FIN-00045 Nokia Group
FINLAND
Phone: +358-71-8008000
EMail: ari.lakaniemi@nokia.com
Stephan Wenger
Nokia Corporation
P.O. Box 100
FIN-33721 Tampere
FINLAND
Phone: +358-50-486-0637
EMail: Stephan.Wenger@nokia.comFull Copyright Statement ©️
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 🙃
Intellectual Property 🔐
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
Acknowledgement 🌷
Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA). ✨
📦 where this page comes from
- ietf: rfc4352 2006-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms