RFC 5404
RTP Payload Format for G.719
Part 1 of 6 🎀
RFC 5404 💖 RTP Payload Format for G.719
okay bestie, RFC 5404 is a Standards Track spec 🦋 from the Network Working Group at the IETF. It was written by M. Westerlund and I. Johansson from Ericsson AB, and it's dated January 2009. The whole point is how to pack the G.719 full-band codec audio into RTP. The title block doesn't list anything it obsoletes or updates, so no drama there ✨
Status of This Memo ✨
this is an Internet standards track protocol for the Internet community 💅 It also asks for discussion and suggestions for improvements. For the standardization state and status, check the current edition of "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🌸
Copyright Notice 🌈
Copyright (c) 2008 IETF Trust and the persons identified as the document authors. All rights reserved.
so this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at http://trustee.ietf.org/license-info, as they stand on the publication date. Read them carefully, bestie, because they're about your rights and restrictions 🫶
Abstract 🧚
this doc specifies the payload format for packetizing G.719 full-band codec encoded audio signals into the Real-time Transport Protocol (RTP) 🎧 The format supports multiple channels, multiple frames per payload, and interleaving. literal slay 🔥
Table of Contents 🍓
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions and Conventions . . . . . . . . . . . . . . . . . 3
3. G.719 Description . . . . . . . . . . . . . . . . . . . . . . 3
4. Payload Format Capabilities . . . . . . . . . . . . . . . . . 4
4.1. Multi-Rate Encoding and Rate Adaptation . . . . . . . . . 4
4.2. Support for Multi-Channel Sessions . . . . . . . . . . . . 5
4.3. Robustness against Packet Loss . . . . . . . . . . . . . . 5
4.3.1. Use of Forward Error Correction (FEC) . . . . . . . . 5
4.3.2. Use of Frame Interleaving . . . . . . . . . . . . . . 6
5. Payload Format . . . . . . . . . . . . . . . . . . . . . . . . 7
5.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . . 8
5.2. Payload Structure . . . . . . . . . . . . . . . . . . . . 8
5.2.1. Basic ToC Element . . . . . . . . . . . . . . . . . . 9
5.3. Basic Mode . . . . . . . . . . . . . . . . . . . . . . . . 10
5.4. Interleaved Mode . . . . . . . . . . . . . . . . . . . . . 10
5.5. Audio Data . . . . . . . . . . . . . . . . . . . . . . . . 11
5.6. Implementation Considerations . . . . . . . . . . . . . . 12
5.6.1. Receiving Redundant Frames . . . . . . . . . . . . . . 12
5.6.2. Interleaving . . . . . . . . . . . . . . . . . . . . . 12
5.6.3. Decoding Validation . . . . . . . . . . . . . . . . . 13
6. Payload Examples . . . . . . . . . . . . . . . . . . . . . . . 13
6.1. 3 Mono Frames with 2 Different Bitrates . . . . . . . . . 13
6.2. 2 Stereo Frame-Blocks of the Same Bitrate . . . . . . . . 14
6.3. 4 Mono Frames Interleaved . . . . . . . . . . . . . . . . 15
7. Payload Format Parameters . . . . . . . . . . . . . . . . . . 16
7.1. Media Type Definition . . . . . . . . . . . . . . . . . . 16
7.2. Mapping to SDP . . . . . . . . . . . . . . . . . . . . . . 19
7.2.1. Offer/Answer Considerations . . . . . . . . . . . . . 19
7.2.2. Declarative SDP Considerations . . . . . . . . . . . . 22
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
9. Congestion Control . . . . . . . . . . . . . . . . . . . . . . 23
10. Security Considerations . . . . . . . . . . . . . . . . . . . 24
11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 25
12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 25
12.1. Normative References . . . . . . . . . . . . . . . . . . . 25
12.2. Informative References . . . . . . . . . . . . . . . . . . 261. Introduction 💖
okay so the spec is all about packetizing the G.719 full-band (FB) codec encoded audio signals into the Real-time Transport Protocol (RTP) [RFC3550]. It supports multiple channels, multiple frames per payload, and packet loss robustness through redundancy or interleaving 🎀
the doc goes: conventions first, then a quick codec overview, then the payload format's capabilities. The actual payload format is in Section 5, and the examples are in Section 6. After that come the media type and how it maps to the Session Description Protocol (SDP) and to SDP offer/answer. It wraps up with congestion control and security 🧚
2. Definitions and Conventions 🦋
the term "frame-block" just means the time-synchronized set of audio frames in a multi-channel audio session. So in an N-channel session, one frame-block has N audio frames, one from each channel, and all N frames cover exactly the same time period 🫶
heads up, this doc has bit field depictions. The most significant bit is always the leftmost one in each row, and it has the lowest enumeration. For fields spread across multiple rows, the upper row is more significant than the next one 💅
ok the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document get interpreted as described in RFC 2119 [RFC2119] 🌸
3. G.719 Description 🌈
the ITU-T G.719 full-band codec is a transform coder built on the Modulated Lapped Transform (MLT) ✨ It's low-complexity and full-bandwidth, made for conversational speech and audio coding. The encoder input and decoder output are sampled at 48 kHz. It covers full bandwidth from 20 Hz to 20 kHz and encodes speech, music, and general audio at rates from 32 kbit/s up to 128 kbit/s. It works on 20-ms frames and has an algorithmic delay of 40 ms 🔥
it's giving excellent quality for speech, music, and other audio. Some good places to use it:
- Real-time communications such as video conferencing and telephony 📞
- Streaming audio 🎶
- Archival and messaging 💌
the encoding and decoding algorithm can change the bitrate at any 20-ms frame boundary 💖 The encoder gets audio sampled at 48 kHz. Supporting other sampling rates is possible by re-sampling the input to the codec's rate, which is 48 kHz. But that's not part of the standard, so keep that in mind 🙈
the encoding is done on equally sized frames. For each frame, the encoder picks between two modes, a transient mode and a stationary mode, based on statistics from the input signal 👑 The stationary mode uses a long MLT that gives a spectrum of 960 coefficients. The transient mode uses a short MLT (higher time resolution transform) that gives 4 spectra (4 x 240 = 960 coefficients) 🦋 Encoding the spectrum happens in two steps. First, the spectral envelope gets computed, quantized, and Huffman encoded. The envelope is computed on a non-uniform frequency subdivision. From the coded envelope, a weighted spectral envelope is derived and used for bit allocation, and the decoder repeats this process too. So only the spectral envelope is transmitted 🧚 The output of bit allocation is used to quantize the spectra. For stationary frames, the encoder also estimates the amount of noise level 🌸 The decoder does the reverse when it receives the bit stream. Non-coded coefficients (no bits allocated) get replaced by entries from a noise codebook built from the decoded coefficients 🫶
4. Payload Format Capabilities 💅
this payload format has a bunch of capabilities, and this section talks through them in some detail 🎀
4.1. Multi-Rate Encoding and Rate Adaptation 🌈
G.719 supports multi-rate encoding, so the encoding rate can vary on a per-frame basis ✨ That makes bitrate adaptation and congestion control possible. Aggregating multiple audio frames into a single RTP payload is another dimension of adaptation. Aggregation can reduce the RTP and payload format overhead, but it costs you increased delay and reduced packet-loss robustness 💖
4.2. Support for Multi-Channel Sessions 🦋
the RTP payload format here supports multi-channel audio content, like stereo or surround sessions 🎧 G.719 itself can't encode multi-channel audio into a single bit stream. But you can use it to separately encode and decode each individual channel. To transport or store the separately encoded multi-channel content, the audio frames for all channels that are framed and encoded for the same 20-ms period get logically collected into a "frame-block" 🍓
at session setup, out-of-band signaling must be used to indicate the number of channels in the payload type. The order of audio frames within the frame-block depends on the number of channels and follows the definition in Section 4.1 of the RTP/AVP profile [RFC3551]. When you use SDP for signaling, the number of channels gets specified in the rtpmap attribute 🫶
4.3. Robustness against Packet Loss 🛡️
the payload format supports several ways to increase robustness against packet loss, including forward error correction (FEC) and frame interleaving 💜
4.3.1. Use of Forward Error Correction (FEC) 🎀
Okay so, generic forward error correction inside RTP is covered in RFC 5109 [RFC5109] 🌸 and audio redundancy coding lives in RFC 2198 [RFC2198]. Either one can tack extra redundant info onto the RTP packet stream so it's way more resilient to packet loss, but the trade-off is a higher bitrate. Literally check either RFC if you want the deets on how that higher bitrate affects network congestion ✨
Now, on top of those media-unaware options, this memo defines a G.719-specific audio redundancy scheme 🦋 which can be lowkey better for packetization overhead. The idea is that previously sent transport frames get bundled up with new ones. You can use a sliding window to group the frames for each payload, but irregular or non-consecutive patterns work too, by slotting NO_DATA frames in between the primary and redundant sends. Here's the vibe in Figure 1:
--+--------+--------+--------+--------+--------+--------+--------+--
| f(n-2) | f(n-1) | f(n) | f(n+1) | f(n+2) | f(n+3) | f(n+4) |
--+--------+--------+--------+--------+--------+--------+--------+--
<---- p(n-1) ---->
<----- p(n) ----->
<---- p(n+1) ---->
<---- p(n+2) ---->
<---- p(n+3) ---->
<---- p(n+4) ---->
Figure 1: An example of redundant transmissionHere, each frame gets retransmitted once in the following RTP payload packet. f(n-2)...f(n+4) are the audio frames, and p(n-1)...p(n+4) are the payload packets 💅
This mechanism doesn't actually need any signaling at session setup. There is signaling you can use if the sender wants to voluntarily pin down the buffering and delay requirements 🧚 If nothing gets signaled, using this mechanism is allowed and unbounded. For a given timestamp, the receiver might get several copies of the same frame with encoded audio data, even at different encoding rates. The price you pay is bandwidth, plus the receiver delay needed so the redundant copy has time to arrive 😭
This redundancy scheme does something similar to RFC 2198, but it only works when both the original frames and the redundant representations are G.719 frames. If you want to use some other media coding scheme, you have to fall back to RFC 2198 🫶
The sender is responsible for picking a sensible amount of redundancy based on feedback about channel conditions, for example from RTP Control Protocol (RTCP) [RFC3550] receiver reports. The sender is also on the hook for avoiding congestion, which redundancy can make worse (see Section 9 for more details) 👑
4.3.2. Use of Frame Interleaving 🌈
To cut down on protocol overhead, the payload design lets several audio transport frames get packed into a single RTP packet 🍓 The catch is that if a packet gets lost, several consecutive frames vanish with it. Losing consecutive frames usually makes error concealment less effective and leads to clearly audible, annoying distortion in the reconstructed audio 💀 Interleaving transport frames fixes that vibe by spreading the consecutive losses out into a bunch of isolated frame losses, which are way easier to conceal ✨
Lowkey though, interleaving and bundling several frames per payload also adds end-to-end delay and needs more buffering. So interleaving isn't right for every use case or device 🙈 Streaming applications should most likely be able to use interleaving to boost audio quality on lossy links 🎧
Note that this payload design supports frame interleaving as an option. You need to negotiate it during session setup 💖
The interleaving here is pretty flexible. For example, you can define a continuous pattern, like the one in Figure 2:
--+--------+--------+--------+--------+--------+--------+--------+--
| f(n-2) | f(n-1) | f(n) | f(n+1) | f(n+2) | f(n+3) | f(n+4) |
--+--------+--------+--------+--------+--------+--------+--------+--
[ p(n) ]
[ p(n+1) ] [ p(n+1) ]
[ p(n+2) ] [ p(n+2) ]
[ p(n+3) ]
[ p(n+4) ]
Figure 2: An example of interleaving pattern that has constant delayIn Figure 2, the consecutive frames f(n-2) to f(n+4) get aggregated into packets p(n) to p(n+4), with each packet carrying two frames. This approach gives an interleaving pattern with constant delay in both the interleaving and de-interleaving processes 🦋 The de-interleaving buffer needs room for at least three frames, including the one that's ready to be consumed. You need space for three frames because, for example, when f(n) is the next frame to decode, f(n) was received in packet p(n+2), which also carried f(n+3). So both of those frames sit in the buffer. Frame f(n+1), which arrived in the previous packet p(n+1), is also hanging out in the de-interleaving buffer 🧁 Also, the buffer occupancy changes. When f(n+1) is the next one to decode, the buffer only holds two frames, f(n+1) and f(n+3) 🌸
5. Payload Format 💜
The main goal of the G.719 payload design is to get the most out of the codec with as little overhead as possible ✨ Both basic and interleaved modes are included, because the codec works for conversational and other low-delay apps, but also for streaming, where more delay is fine 🎀
The biggest structural difference between basic and interleaved modes is that interleaved mode extends the table of contents entries with frame displacement fields. Basic mode supports aggregating multiple consecutive frames in one payload. Interleaved mode supports aggregating multiple frames that aren't consecutive in time. In both modes, a payload can carry frames encoded with different frame types 🧚
The payload format also supports carrying multi-channel content with G.719, using one discrete encoder per channel, all at the same bitrate. In that case, a complete frame-block with data from all channels goes into the RTP payload. The data is the concatenation of all the encoded audio frames, in the order specified for that number of included channels. Interleaving also happens on complete frame-blocks rather than on individual audio frames 👑
5.1. RTP Header Usage 🦋
The RTP timestamp matches the sampling instant of the first sample encoded for the first frame-block in the packet. The timestamp clock frequency SHALL be 48000 Hz. The timestamp is also what lets you recover the correct decoding order of the frame-blocks 💖
The RTP header marker bit (M) SHALL be set to 1 whenever the first frame-block carried in the packet is the first frame-block in a talkspurt (see the definition of talkspurt in Section 4.1 of [RFC3551]). For all other packets, the marker bit SHALL be set to zero (M=0) 🌈
The assignment of an RTP payload type for the format defined in this memo is outside the scope of this document. The RTP profiles in use currently call for binding the payload type dynamically for this payload format. That's basically necessary, because the payload type expresses the configuration of the payload itself, meaning basic or interleaved mode and the number of channels carried 🍓
The remaining RTP header fields are used as specified in [RFC3550] ✨
5.2. Payload Structure 🫶
The payload consists of one or more table of contents (ToC) entries, followed by the audio data that corresponds to those ToC entries 🌸 The following sections describe both basic mode and interleaved mode. Each ToC entry MUST be padded to a byte boundary to ensure octet alignment 😭 The rules about maximum payload size in Section 3.2 of [RFC5405] SHOULD be followed 💅
5.2.1. Basic ToC Element 🎀
Every format and mode in this document uses a common basic ToC, which can be extended in the different options described below 💖
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|F| L |R|R|
+-+-+-+-+-+-+-+-+
Figure 3: Basic TOC elementF (1 bit): If set to 1, this ToC entry is followed by another ToC entry. If set to zero, this ToC entry is the last one in the ToC 🧚
L (5 bits): A field that gives the frame length of each individual frame within the frame-block 🌈
L length(bytes)
============================
0 0 NO_DATA
1-7 N/A (reserved)
8-22 80+10*(L-8)
23-27 240+20*(L-23)
28-31 N/A (reserved)
Figure 4: How to map L values to frame lengthsL=0 (NO_DATA) indicates an empty frame. That's handy when frames are missing (for example, at re-packetization), or when you want gaps while sending redundant frames together with primary frames in the same payload 🍓 The values [1..7] and [28..31] (inclusive) are reserved for future use in this document version. If these values show up in a ToC, the entire packet SHOULD be treated as invalid and discarded 💀 Here are a few examples where the frame size and the matching codec bitrate are computed from the value of L 🧁
L Bytes Codec Bitrate(kbps)
===================================
8 80 32
9 90 36
10 100 40
12 120 48
16 160 64
22 220 88
23 240 96
25 280 112
27 320 128
Figure 5: Examples of L values and corresponding frame lengthsThis encoding gives a granularity of 4 kbps between 32 and 88 kbps, and a granularity of 8 kbps between 88 and 128 kbps, with a defined range of 32-128 kbps for the codec data 👑
R (2 bits): Reserved bits. These SHALL be set to zero on sending and SHALL be ignored on reception 🦋
5.3. Basic Mode 💜
The basic ToC element shown in Figure 3 is followed by a 1-octet field for the number of frame-blocks (#frames) that form the ToC entry. The frame-blocks field tells you how many frame-blocks of the same length the ToC entry covers ✨
0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
| #frames |
+-+-+-+-+-+-+-+-+
Figure 6: Number of frame-blocks field5.4. Interleaved Mode 🦋
Okay so the basic ToC gets a bestie here 🫶. After it comes a 1-octet field for the number of frame-blocks (#frames), and then the DIS fields. Together they make a ToC entry in interleaved mode ✨. The frame-blocks field tells you how many frame-blocks of the same length the ToC relates to. The DIS fields, one for each frame-block counted by #frames, express the interleaving distance between audio frames carried in the payload. If you need octet alignment, a 4-bit padding gets tossed in 💅.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| #frames | DIS1 | ... | DISi | ... | DISn | Padd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 7: Number of frame-block + interleave fields 🎀
DIS1...DISn (4 bits): A list of n (n=#frames) displacement fields indicating the displacement of the i:th (i=1..n) audio frame-block relative to the preceding frame-block in the payload, in units of 20-ms long audio frame-blocks). The 4-bit unsigned integer displacement values may be between zero and 15, indicating the number of audio frame-blocks in decoding order between the (i-1):th and the i:th frame in the payload. Note that for the first ToC entry of the payload, the value of DIS1 is meaningless 🙈. It SHALL be set to zero by a sender and SHALL be ignored by a receiver. This frame-block's location in the decoding order is uniquely defined by the RTP timestamp. Note that for subsequent ToC entries, DIS1 indicates the number of frames between the last frame of the previous group and the first frame of this group.
Padd (4 bits): To ensure octet alignment, 4 padding bits SHALL be included at the end of the ToC entry in case there is an odd number of frame-blocks in the group referenced by this ToC entry. These bits SHALL be set to zero and SHALL be ignored by the receiver. If a group containing an even number of frames is referenced by this ToC entry, these padding bits SHALL NOT be included in the payload 🍓.
5.5. Audio Data 🎶
The audio data part follows the table of contents. All the octets comprising an audio frame SHALL be appended to the payload as a unit 🧚. For each frame-block, the audio frames are concatenated in the order indicated by the table in Section 4.1 of [RFC3551] for the number of channels configured for the payload type in use. So the first channel (leftmost) indicated comes first, followed by the next channel. The audio frame-blocks are packetized in increasing timestamp order within each group of frame-blocks (per ToC entry), i.e., oldest frame-block first. The groups of frame-blocks are packetized in the same order as their corresponding ToC entries.
The audio frames are specified in ITU recommendation [ITU-T-G719] 💜.
The G.719 bit stream is split into a sequence of octets and transmitted in order from the leftmost (most significant (MSB)) bit to the rightmost (least significant (LSB)) bit.
5.6. Implementation Considerations 🛠️
An application implementing this payload format MUST understand all the payload parameters specified in this specification. Any mapping of the parameters to a signaling protocol MUST support all parameters. So an implementation of this payload format in an application using SDP is required to understand all the payload parameters in their SDP-mapped form. This requirement ensures that an implementation always can decide whether it is capable of communicating when the communicating entities support this version of the specification 👑.
Basic mode SHALL be implemented and the interleaved mode SHOULD be implemented. The implementation burden of both is rather small, and supporting both ensures interoperability 🌈. However, interleaving is not mandated, as it has limited applicability for conversational applications that require tight delay boundaries 😭.
5.6.1. Receiving Redundant Frames 🔁
The reception of redundant audio frames, i.e., more than one audio frame from the same source for the same time slot, MUST be supported by the implementation. In the case that the receiver gets multiple audio frames in different bitrates for the same time slot, it is RECOMMENDED that the receiver keeps the one with the highest bitrate ⭐.
5.6.2. Interleaving 🧩
The use of interleaving requires further considerations. As presented in the example in Section 4.3.2, a given interleaving pattern requires a certain amount of de-interleaving buffer. This buffer space, expressed in a number of transport frame slots, is indicated by the "interleaving" media type parameter. The number of frame slots needed can be converted into actual memory requirements by considering the 320 bytes per frame used by the highest bitrate of G.719 💾.
The information about the frame buffer size is not always sufficient to determine when it is appropriate to start consuming frames from the interleaving buffer. Additional information is needed when the interleaving pattern changes. The "int-delay" media type parameter is defined to convey this information. It allows a sender to indicate the minimal media time that needs to be present in the buffer before the decoder can start consuming frames from the buffer. Because the sender has full control over the interleaving pattern, it can calculate this value 🧮.
In certain cases (for example, if joining a multicast session with interleaving mid-session), a receiver may initially receive only part of the packets in the interleaving pattern. This initial partial reception (in frame sequence order) of frames can yield too few frames for acceptable quality from the audio decoding. This problem also arises when using encryption for access control, and the receiver does not have the previous key. Although G.719 is robust and thus tolerant to a high random frame erasure rate, it would have difficulties handling consecutive frame losses at startup 😬. Thus, some special implementation considerations are described.
In order to handle this type of startup efficiently, decoding can start provided that:
- There are at least two consecutive frames available.
- More than or equal to half the frames are available in the time period from where decoding was planned to start and the most forward received decoding.
After receiving a number of packets, in the worst case as many packets as the interleaving pattern covers, the previously described effects disappear and normal decoding is resumed 🌸. Similar issues arise when a receiver leaves a session or has lost access to the stream. If the receiver leaves the session, this would be a minor issue since playout is normally stopped. The sender can avoid this type of problem in many sessions by starting and ending interleaving patterns correctly when risks of losses occur. One such example is a key-change done for access control to encrypted streams. If only some keys are provided to clients and there is a risk they will receive content for which they do not have the key, it is recommended that interleaving patterns do not overlap key changes 🔐.
5.6.3. Decoding Validation 🔍
If the receiver finds a mismatch between the size of a received payload and the size indicated by the ToC of the payload, the receiver SHOULD discard the packet. This is recommended because decoding a frame parsed from a payload based on erroneous ToC data could severely degrade the audio quality 💀.
6. Payload Examples 🎀
A few examples to highlight the payload format follow ✨.
6.1. 3 Mono Frames with 2 Different Bitrates
The first example is a payload consisting of 3 mono frames, where the first 2 frames correspond to a bitrate of 32 kbps (80 bytes/frame) and the last is 48 kbps (120 bytes/frame) 🦋.
The first 32 bits are ToC fields. Bit 0 is '1' as another ToC field follows. Bits 1..5 are '01000' = 80 bytes/frame. Bits 8..15 are '00000010' = 2 frame-blocks with 80 bytes/frame. Bit 16 is '0', no more ToC follows. Bits 17..21 are '01100' = 120 bytes/frame. Bits 24..31 are '00000001' = 1 frame-block with 120 bytes/frame.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1|0 1 0 0 0|0 0|0 0 0 0 0 0 1 0|0|0 1 1 0 0|0 0|0 0 0 0 0 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|d(0) frame 1 |
. .
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|d(0) frame 2 |
. .
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|d(0) frame 3 |
. .
| d(959)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+6.2. 2 Stereo Frame-Blocks of the Same Bitrate
The second example is a payload consisting of 2 stereo frames that correspond to a bitrate of 32 kbps (80 bytes/frame) per channel 🎧. The receiver calculates the number of frames in the audio block by multiplying the value of the "channels" parameter (2) with the #frames field value (2), to derive that there are 4 audio frames in the payload 🧠.
The first 16 bits is the ToC field. Bit 0 is '0' as no ToC field follows. Bits 1..5 are '01000' = 80 bytes/frame. Bits 8..15 are '00000010' = 2 frame-blocks with 80 bytes/frame.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0 1 0 0 0|0 0|0 0 0 0 0 0 1 0| d(0) frame 1 left ch. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
| d(639)| d(0) frame 1 right ch. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
| d(639)| d(0) frame 2 left ch. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
| d(639)| d(0) frame 2 right ch. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+6.3. 4 Mono Frames Interleaved ✨
okay bestie, the third example is a payload with 4 mono frames at 32 kbps (80 bytes/frame) that are interleaved 🦋 the pattern for constant delay when aggregating 4 frames is shown below. the packet we're looking at is packet n, and the neighbours (previous and following packets' frame-block content) are shown so you can see the pattern 🧚
Packet n-3: 1, 6, 11, 16
Packet n-2: 5, 10, 15, 20
Packet n-1: 9, 14, 19, 24
Packet n: 13, 18, 23, 28
Packet n+1: 17, 22, 27, 32
Packet n+2: 21, 26, 31, 36
The first 32 bits are the ToC field.
Bit 0 is '0' as there is no ToC field following.
Bits 1..5 are '01000' = 80 bytes/frame.
Bits 8..15 are '00000100' = 4 frame-blocks with 80 bytes/frame.
Bits 16..19 are '0000' = DIS1 (0).
Bits 20..23 are '0100' = DIS2 (4).
Bits 24..27 are '0100' = DIS3 (4).
Bits 28..31 are '0100' = DIS4 (4).
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0 1 0 0 0|0 0|0 0 0 0 0 1 0 0|0 0 0 0|0 1 0 0|0 1 0 0|0 1 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| d(0) frame 13 |
. .
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| d(0) frame 18 |
. .
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| d(0) frame 23 |
. .
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| d(0) frame 28 |
. .
| d(639)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+7. Payload Format Parameters 💜
okay so this RTP payload format is identified with the media type audio/G719 🎀 it's registered under [RFC4855] and uses the template from [RFC4288] ✨
7.1. Media Type Definition 🌸
the G.719 codec gets its media type from the IETF tree 💅 because G.719 could become a super widely used audio codec for general Voice over IP (VoIP), teleconferencing, and streaming apps 🫶 this registration covers real-time transfer via RTP ✨
ngl this one's important: any unspecified parameter MUST be ignored by the receiver 💀 that way extra parameters can be added in any future revision of this spec 🔥
Type name: audio 🎀
Subtype name: G719 🎀
Required parameters: none 🌈
Optional parameters: 🍓
interleaving: says interleaved mode SHALL be used for the payload 😭 it gives the number of frame-block slots in a de-interleaving buffer for each source (this includes the frame that's ready to be consumed) 🧚 its value is one plus the maximum number of frames that can come before any frame in transmission order and after it in RTP timestamp order 💖 the value MUST be greater than zero 🔥 if this parameter isn't present, interleaved mode SHALL NOT be used 🙈int-delay: the minimal media time delay in milliseconds needed to avoid underrun in the de-interleaving buffer before decoding starts 🦋 basically the RTP timestamp tick difference between the earliest and latest audio frame in that buffer, in ms 🌈 it's a stream property, provided per source 💅 allowed values are zero up to the biggest unsigned 16-bit integer (65535) 🫶 but realistically the biggest usable value is the declared size of the receiver's interleaving buffer 👑 if the value is bigger than the receiver buffer declared by or for the receiver, it defaults to the size of the receiver buffer 😭 for sources where this value hasn't been given, it also defaults to the receiver buffer size 💜 the format is a comma-separated list of synchronization source (SSRC) ":" delay in ms pairs, and in ABNF [RFC5234] it looks like this:
int-delay = "int-delay:" source-delay *("," source-delay)
source-delay = SSRC ":" delay-value
SSRC = 1*8HEXDIG ; The 32-bit SSRC encoded in hex format
delay-value = 1*5DIGIT ; The delay value in milliseconds
Example: int-delay=ABCD1234:1000,4321DCB:640
NOTE: No white space allowed in the parameter before the end of
all the value pairsmax-red: the maximum duration in milliseconds between the primary (first) transmission of a frame and any redundant transmission the sender will use 🧚 this lets a receiver have a bounded delay when redundancy is used ✨ allowed values are zero (no redundancy will be used) up to 65535 🌸 if it's omitted, there's no limit on using redundancy 🫶channels: the number of audio channels 🎀 possible values are 1-6, and their channel order is in Section 4.1 of [RFC3551] 💅 if omitted, the default is 1 🍓CBR: Constant Bitrate (CBR) is the exact codec bitrate in bits per second (not counting packetization overhead, the RTP header, or lower layers) that the codec MUST use 😭 "CBR" is for when the dynamic rate can't be supported (like, e.g., a gateway to H.320) 🙈 it's mostly used for gateways to circuit switch networks 🌈 so "CBR" is the rate that doesn't include any FEC from Section 4.3.1 💀 if FEC is used, the "b=" parameter MUST be used to allow the extra bitrate needed for the redundant info 🔥 it's RECOMMENDED to only use this parameter when it's actually necessary to get a working call 👑 and it has congestion control implications that need thinking about, see Section 9 🦋ptime: see [RFC4566] 🧚maxptime: see [RFC4566] 🧚
Encoding considerations: this media type is framed and binary, see Section 4.8 of [RFC4288] 💖
Security considerations: See Section 10 of RFC 5404 🔐
Interoperability considerations: supporting the Interleaving mode is not mandatory and has to be negotiated 🫶 see Section 7.2 for how to do that with SDP-based protocols ✨
Published specification: RFC 5404 📜
Applications that use this media type: real-time audio apps like Voice over IP, teleconference, and multi-media streaming 🎧
Additional information: none 🌸
Person & email address to contact for further information:
Ingemar Johansson <ingemar.s.johansson@ericsson.com> 💌
Intended usage: COMMON 🌟
Restrictions on usage: this media type depends on RTP framing, so it's only defined for transfer via RTP [RFC3550] 🦋 transport within other framing protocols isn't defined at this time 🙅♀️
Author:
Ingemar Johansson <ingemar.s.johansson@ericsson.com> 🎀 Magnus Westerlund <magnus.westerlund@ericsson.com> 🎀
Change controller: IETF Audio/Video Transport working group, delegated from the IESG 👑
Additionally, file storage of G.719-encoded audio in ISO base media file format is specified in Annex A of [ITU-T-G719] 🍓 so media file formats like MP4 (audio/mp4 or video/mp4) [RFC4337] and 3GP (audio/3GPP and video/3GPP) [RFC3839] can contain G.719-encoded audio 🌈
7.2. Mapping to SDP 🦋
the media type info has a specific mapping to fields in the Session Description Protocol (SDP) [RFC4566], which is commonly used to describe RTP sessions 💜 when SDP is used to set up sessions with the G.719 codec, the mapping goes like this:
- The media type (
audio) goes in SDPm=as the media name. 🎀 - The media subtype (payload format name) goes in SDP
a=rtpmapas the encoding name. The RTP clock rate ina=rtpmapMUST be 48000, and the encoding parameterchannels(Section 7.1) MUST either be explicitly set to N or omitted, implying a default value of 1 🔥 the allowed N values are in Section 4.1 in [RFC3551] ✨ - The parameters
ptimeandmaxptimego in the SDPa=ptimeanda=maxptimeattributes, respectively. 🧚 - Any remaining parameters go in the SDP
a=fmtpattribute, copied directly from the media type parameter string as a semicolon-separated list of parameter=value pairs. 💅
7.2.1. Offer/Answer Considerations ✨
Okay bestie, time for SDP offer/answer 💅 Here's how G.719 payloads get negotiated in RTP:
- okay so each combo of the RTP payload transport format configuration parameters (
interleavingandchannels) has its own unique bit pattern and doesn't play nice with any other combo 🙈 If your app wants the fancy stuff (interleaving or more than one channel), you are RECOMMENDED to also offer a payload type with only the single-channel config, like a backup outfit 🧚 If your app cares about several configs, you can offer them all, but don't go overboard with payload types, it's giving clutter 💀 An SDP answerer MUST keep these params in its answer for each payload type, unmodified from the offer (unless it drops that payload type):interleavingandchannels. The value ofinterleavingMAY be changed though ✨ And the SDP offerer and answerer MUST generate G.719 packets as these params describe. Non-negotiable, periodt 💖 - the
interleavingandint-delayparams have a super specific relationship, and it depends on which direction the streams go and how they get delivered 🦋 The big picture:interleavingsays how big the receiver buffer is, whileint-delayis a stream property the sender uses to tell everyone how much of that buffer it's actually using for its stream 🌸- for media streams sent over multicast, the answerer SHALL NOT change the value of
interleaving. It shall either be accepted or the payload type gets deleted 🙅♀️ Theint-delayvalue is a stream property, provided by the offer/answer agent that plans to send media with this payload type, for each stream coming from that agent (one or more). The value MUST be between zero and whatever corresponds to the buffer size declared byinterleaving💜 - for unicast streams that the offerer declares as send-only, the
interleavingvalue is the size the answerer is RECOMMENDED to use 🎀 The answerer MAY change it to any allowed value. Theint-delayvalue will be the one the offerer intends to use, unless the answerer shrinksinterleavingbelow what thatint-delayneeds. If the answer'sinterleavingis smaller than the offer'sint-delay, thenint-delaygets reduced by default to matchinterleaving😭 If the offerer isn't happy with that, they'll need another round of offer/answer. And since the answerer won't send any media, it doesn't include anyint-delayin the answer 🫶 - for unicast streams that the offerer declares as recvonly, the
interleavingvalue in the offer is the offerer's interleaving buffer size 🍓 The answerer shares its preferred interleaving buffer size for any future round of offer/answer. The offerer won't include anyint-delayparam, since it isn't sending media. The answerer is recommended to include anint-delayparam in its answer, to declare the property for the stream it's going to send 🌈 The answer should be able to pick a valid value that's between zero and the declared maximum number of slots in the de-interleaving buffer ✨ - for unicast streams that the offer declares as sendrecv, the
interleavingvalue in the offer is the offerer's interleaving buffer size 👑 The answerer indicates the size of its actual interleaving buffer in the answer. It's recommended that this value is at least as big as the offer's. The offerer is recommended to include anint-delayparam picked on the assumption that the answerer has at least as much interleaving space as the offerer, unless nothing else is known. Since the offerer's buffer size isn't known yet, this might fail 😬 If it does, the default rule is to downgradeint-delayto match the full size of the answerer's interleaving buffer. If the offerer still isn't satisfied, they'll need to start another round of offer/answer. The answerer is recommended to include anint-delayparam in its answer too, declaring the property for the stream(s) it's going to send. The answer should be able to pick a valid value between zero and the declared maximum number of slots in the de-interleaving buffer 🦋
- for media streams sent over multicast, the answerer SHALL NOT change the value of
- honestly, in most cases
maxptimeandptimewon't break interoperability 🌸 But they can still affect how well the app performs, so they matter. The SDP offer/answer handling ofptimeis described in [RFC3264]. Themaxptimeparam MUST be handled in the same way 💅 - the
max-redparam is a stream property. For sendonly or sendrecv unicast media streams, it declares the redundancy limit the stream sender will use 🧚 For recvonly streams, it shows the value the sender wants for the stream sent to the receiver. The answerer MAY change the value, but is RECOMMENDED to use the same limit the offer declares. For multicast, the offerer MAY declare a limit, and that SHALL be answered with the same value 🎀 A media sender using this payload format is RECOMMENDED to always includemax-red, because it's likely to make the receiver's life easier 🙈 This is especially true when no redundancy is used, in which casemax-redis set to zero ✨ - any unknown param in an offer SHALL be removed in the answer. No mystery params allowed, bestie 💀
- the
b=SDP param SHOULD be used to negotiate the max bandwidth for the audio stream 🌈 The offerer may offer a max rate, and the answer may contain a lower one. If there's nob=param in the offer or the answer, it implies a rate of up to 128 kbps 💜 - the
CBRparam is a receiver capability, which means only receivers that really need a constant bitrate should use it 😭 Using it has a negative impact on congestion control, so see Section 9 🫶 For recvonly and sendrecv streams, it shows the constant bitrate the receiver wants to accept. A sender MUST be able to send a constant bitrate stream, since that's a subset of the variable bitrate capability 🍓 If the offer includesCBR, the answerer MUST send G.719 audio at the constant bitrate, as long as it's within the allowed session bitrate (b=param). If the answerer can't support the statedCBR, that payload type must be refused in the answer 🙅♀️ The answerer SHOULD only includeCBRif it really needs to receive at a constant bitrate, even if the offer didn't include theCBRparam. In that case, the offerer SHALL send at the constant bitrate, but SHALL be able to accept media at a variable bitrate 👑 An answerer is RECOMMEND to use the sameCBRas the offer, because symmetric usage is more likely to work 🦋 If both sides need a particularCBR, communication can fail when one or both sides can't send the rate they asked for. In that case, whichever agent notices the problem has to do a second round of offer/answer to find another working config, or end the session 😬 If the offer had aCBRparam but the answer doesn't, the offerer is free to send at any rate to the answerer, but the answerer is limited to the declared rate 🌸
7.2.2. Declarative SDP Considerations 🧚
Now for declarative usage, like SDP in the Real Time Streaming Protocol (RTSP) [RFC2326] or the Session Announcement Protocol (SAP) [RFC2974]. Here the params SHALL be interpreted as follows:
- the payload format configuration params (
interleavingandchannels) are all declarative, and a participant MUST use the configuration(s) provided for the session 💖 You can provide more than one configuration if you need it, by declaring multiple RTP payload types, but the number of types should be kept small 🎀 - it might not be possible to know the SSRC values the sources will use when the SDP gets sent 🤷♀️ That's not a huge deal, because the interleaving buffer size can be tailored to the values that actually get used, so the default
int-delayvalues don't end up causing too much extra buffering ✨ - any
maxptimeandptimevalues should be selected with care, so the session's participants can get reasonable performance 🌈 - the
CBRparam, if included, applies to all RTP streams using that payload type for which a particular CBR is declared. Using this param has a negative impact on congestion control, so see Section 9 😭
8. IANA Considerations 📝
One media type (audio/G719) has been defined and registered in the media types registry; see Section 7.1 🎀
9. Congestion Control 🦋
okay so congestion control, the general rules for RTP data totally still apply here. go read RTP [RFC3550] and whatever RTP profile you're using, like AVP [RFC3551]. 💅 BUT the multi-rate thing in G.719 is lowkey a superpower, because the bandwidth demand can get adjusted (within the limits of the codec) just by picking a different encoding bitrate. it might help you control congestion. 🌈
the number of frames you pack into each RTP payload is a big deal for the overall bandwidth of the stream, because of header overhead. packing more frames per payload means fewer packets get sent, so less header overhead, BUT it costs you extra delay and makes you less robust to errors. 😭 if you're using forward error correction (FEC), the amount of FEC redundancy has to be regulated so FEC itself doesn't cause a congestion problem. in other words, a sender SHALL NOT increase the total bitrate when adding redundancy in response to packet loss. instead it needs to adjust things down in accordance with the congestion control algorithm being run. so when you add redundancy, the media bitrate has to get reduced to make room for it. it's giving balance, bestie. ⚖️✨
the "CBR" signaling parameter lets a receiver lock an RTP payload type to a single encoding rate. 🙈 the catch is that this stops the codec rate from being lowered when congestion shows up, so the sender is stuck with either changing the packetization or aborting the transmission. those responses are really limited, so implementations SHOULD NOT use the "CBR" parameter unless they're talking to a device that can't handle a variable bitrate (for example, a gateway to H.320 systems). when you're in CBR mode, a receiver MUST monitor the packet loss rate to make sure congestion isn't being caused, following the guidelines in Section 2 of RFC 3551. 🎀
10. Security Considerations 💀
okay this section is SERIOUS, so read it all. 🔥
RTP packets using the payload format defined in this spec are subject to the security considerations in the RTP specification [RFC3550] and in any applicable RTP profile. the main security concerns for the RTP packet carrying this payload format are confidentiality, integrity, and source authenticity. confidentiality comes from encrypting the RTP payload. integrity of the RTP packets comes from a suitable cryptographic integrity protection mechanism. and that same cryptographic system may also let you authenticate where the payload came from. a good security mechanism for this format should give you confidentiality, integrity protection, and at least source authentication that can tell whether an RTP packet is from a member of the RTP session. 🔐
now, the right security mechanism may vary, bestie. it depends on the application, the transport, and the signaling protocol you're using. so one single mechanism isn't enough on its own, but if it fits, using the Secure Real-time Transport Protocol (SRTP) [RFC3711] is recommended. other options include IPsec [RFC4301] and Transport Layer Security (TLS) [RFC5246] (RTP over TCP). there might be other alternatives too. 🦋
⚠️ the interleaving + encryption plot twist: using interleaving together with encryption can hurt confidentiality for a short period of time. 😱
here's the lowkey scary example. consider these packets (in brackets) with the frame numbers listed: {10, 14, 18}, {13, 17, 21}, {16, 20, 24} (a popular continuous diagonal interleaving pattern). say the originator wants to stop some participants from hearing material starting at time 16. just changing the key on the packet with the timestamp at or after 16, and denying that new key to those participants, does NOT do it 🙅♀️. why? frames 17, 18, and 21 were already sent in earlier packets under the old key, and error concealment might make the audio intelligible at least as far as frame 18 or 19, and possibly further. literally iconic plot hole. 😭
last thing: this RTP payload format and its media decoder don't have any significant non-uniformity in receiver-side computational complexity for packet processing. so they're unlikely to be a denial-of-service threat from pathological data. and the RTP payload format doesn't contain any active content either. 💅✨
11. Acknowledgements 🫶
the authors want to thank Roni Even and Anisse Taleb for their help with this document. 💜 they also want to thank the people who gave feedback: Colin Perkins, Mark Baker, and Stephen Botzko. 🌸
12. References 📚
12.1. Normative References
[ITU-T-G719] ITU-T, "Specification : ITU-T G.719 extension for 20
kHz fullband audio", April 2008.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3264] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer
Model with Session Description Protocol (SDP)",
RFC 3264, June 2002.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio
and Video Conferences with Minimal Control", STD 65,
RFC 3551, July 2003.
[RFC4566] Handley, M., Jacobson, V., and C. Perkins, "SDP:
Session Description Protocol", RFC 4566, July 2006.
[RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", STD 68, RFC 5234, January 2008.
[RFC5405] Eggert, L. and G. Fairhurst, "Unicast UDP Usage
Guidelines for Application Designers", BCP 145,
RFC 5405, November 2008.12.2. Informative References
[RFC2198] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V.,
Handley, M., Bolot, J., Vega-Garcia, A., and S. Fosse-
Parisis, "RTP Payload for Redundant Audio Data",
RFC 2198, September 1997.
[RFC2326] Schulzrinne, H., Rao, A., and R. Lanphier, "Real Time
Streaming Protocol (RTSP)", RFC 2326, April 1998.
[RFC2974] Handley, M., Perkins, C., and E. Whelan, "Session
Announcement Protocol", RFC 2974, October 2000.
[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and
K. Norrman, "The Secure Real-time Transport Protocol
(SRTP)", RFC 3711, March 2004.
[RFC3839] Castagno, R. and D. Singer, "MIME Type Registrations
for 3rd Generation Partnership Project (3GPP)
Multimedia files", RFC 3839, July 2004.
[RFC4288] Freed, N. and J. Klensin, "Media Type Specifications
and Registration Procedures", BCP 13, RFC 4288,
December 2005.
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, December 2005.
[RFC4337] Y Lim and D. Singer, "MIME Type Registration for
MPEG-4", RFC 4337, March 2006.
[RFC4855] Casner, S., "Media Type Registration of RTP Payload
Formats", RFC 4855, February 2007.
[RFC5109] Li, A., "RTP Payload Format for Generic Forward Error
Correction", RFC 5109, December 2007.
[RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer
Security (TLS) Protocol Version 1.2", RFC 5246,
August 2008.Authors' Addresses 🧚
Magnus Westerlund
Ericsson AB
Torshamnsgatan 21-23
SE-164 83 Stockholm
SWEDEN
Phone: +46 10 7190000
EMail: magnus.westerlund@ericsson.com
Ingemar Johansson
Ericsson AB
Laboratoriegrand 11
SE-971 28 Lulea
SWEDEN
Phone: +46 10 7190000
EMail: ingemar.s.johansson@ericsson.com📦 where this page comes from
- ietf: rfc5404 2009-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms