RFC 2733
An RTP Payload Format for Generic Forward Error Correction
RFC 2733: An RTP Payload Format for Generic Forward Error Correction 🎀
Okay bestie, let's talk about RFC 2733 💖 It's from the Network Working Group and dated December 1999. It's a Standards Track doc, which means it's an actual protocol spec. Written by J. Rosenberg (dynamicsoft) and H. Schulzrinne (Columbia University). The header doesn't list anything it obsoletes or updates, so it's standing on its own 🦋
Status of this Memo 📜
This one is an Internet standards track protocol for the Internet community, and it asks for discussion and suggestions for improvements. If you want the current standardization state and status of this protocol, check the current edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited 🌸
Copyright Notice ⚖️
Copyright (C) The Internet Society (1999). All Rights Reserved. 💜
Abstract ✨
This doc is a payload format for generic forward error correction (FEC) of media that's wrapped up in RTP. It's built for FEC algorithms that use the exclusive-or (parity) operation. The payload format lets end systems send with arbitrary block lengths and parity schemes. It can also recover both the payload and the critical RTP header fields, which is so iconic 😭 Because FEC gets sent as a separate stream, it's backwards compatible with hosts that don't do FEC. Receivers who don't want to implement FEC can just ignore the extensions 🫶
Table of Contents 🗂️
1 Introduction ........................................... 2
2 Terminology ............................................ 2
3 Basic Operation ........................................ 3
4 Parity Codes ........................................... 5
5 RTP Media Packet Structure ............................. 6
6 FEC Packet Structure ................................... 7
6.1 RTP Header of FEC Packets .............................. 7
6.2 FEC Header ............................................. 7
7 Protection Operation ................................... 9
8 Recovery Procedures .................................... 10
8.1 Reconstruction ......................................... 10
8.2 Determination of When to Recover ....................... 12
9 Example ................................................ 16
10 Use with Redundant Encodings ........................... 17
11 Indicating FEC Usage in SDP ............................ 20
11.1 FEC as a Separate Stream ............................... 20
11.2 Use with Redundant Encodings ........................... 21
11.3 Usage with RTSP ........................................ 22
12 Security Considerations ................................ 23
13 Acknowledgments ........................................ 24
14 Authors' Addresses ..................................... 24
15 Bibliography ........................................... 25
16 Full Copyright Statement ............................... 261 Introduction 🌈
Packet voice on the Internet has been kinda mid, and part of the reason is high packet loss rates. This is especially true on wide-area connections. The thing is, real-time multimedia has strict delay requirements, which usually rules out retransmissions. Not ideal 💀
That's why forward error correction (FEC) has been proposed to make up for packet loss on the Internet [1] [2]. People are especially interested in traditional error correcting codes like parity, Reed-Solomon, and Hamming codes. To support these mechanisms, protocol support is required ✨
This document defines a payload format for RTP [3] that allows generic forward error correction of real time media. Here, "generic" means the FEC protocol is (1) independent of the nature of the media being protected, whether it's audio, video, or otherwise, (2) flexible enough to support a wide variety of FEC mechanisms, (3) designed for adaptivity, so the FEC technique can be changed easily without out of band signaling, and (4) supportive of a number of different mechanisms for transporting the FEC packets. Main character energy 👑
2 Terminology 📖
Here are the terms used throughout this document, so we're all on the same page 🧚
- Media Payload: a piece of raw, un-protected user data that's going to be transmitted from the sender. The media payload gets placed inside of an RTP packet.
- Media Header: the RTP header for the packet that contains the media payload.
- Media Packet: the combination of a media payload and media header is called a media packet.
- FEC Packet: the forward error correction algorithms at the transmitter take the media packets as input. They output both the media packets they were passed, and new packets called FEC packets. FEC packets are formatted according to the rules in this document.
- FEC Header: the header information contained in an FEC packet.
- FEC Payload: the payload in an FEC packet.
- Associated: an FEC packet is said to be "associated" with one or more media packets when those media packets are used to generate the FEC packet (by use of the exclusive or operation).
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [4]. These are the BCP 14 key words, so they carry their official weight here 💅
3 Basic Operation 🦋
This payload format gets used whenever a participant in an RTP session wants to protect a media stream it's sending with forward error correction (FEC). The FEC supported by the format is based on codes using simple exclusive or (xor) parities. The sender takes some set of packets from the media stream and applies an xor operation across the payloads. The sender also applies the xor operation over components of the RTP headers. Following the procedures defined here, the result is an RTP packet containing FEC information. The receiver can use this packet to recover any one of the packets used to generate the FEC packet 🔥
This document doesn't mandate which media packets get combined to generate an FEC packet. That set [is] referred to as a code. Different sets give you a tradeoff between overhead, delay, and recoverability. Section 4 outlines some possible combinations 🌸
The payload format has info that lets the sender tell the receiver exactly which media packets were used to generate the FEC. Each FEC packet contains a bitmask called the offset mask, which is 24 bits. If bit i in the mask is set to 1, then the media packet with sequence number N + i was used to generate this FEC packet. N is called the sequence number base, and it's sent in the FEC packet too. The offset mask and payload type are enough to signal arbitrary parity based forward error correction schemes with very little overhead ✨
This document also describes procedures that let the receiver use the FEC without knowing the details of specific codes. The sender gets lots of flexibility this way. It can adapt the code in use based on network conditions and be certain that receivers can still use the FEC for recovery 🫶
As the sender generates FEC packets, they get sent to the receivers. The sender usually still sends the original media stream, as if there were no FEC. That way, the media stream can still be used by receivers who aren't FEC capable. Some FEC codes don't require the original media to be sent, though. The FEC stream alone is enough for recovery. These codes have the drawback that all receivers must be FEC capable. They're still supported by this format 💀
The FEC packets are not sent in the same RTP stream as the media packets. They can be sent as a separate stream, or as a secondary codec in the redundant codec payload format [5]. When they're sent as a separate stream, the FEC packets get their own sequence number space. Even though the timestamps for FEC packets are derived from the media packets, they increment monotonically. So FEC packet streams work well with any header compression mechanism that needs fixed deltas between fields in the packet header 🧚
This document doesn't prescribe what "separate streams" means. It leaves that to applications and higher level protocols to define. For multicast, a separate stream may be implemented by separate multicast groups, different ports in the same group, or a different SSRC within the same group/port. For unicast, different ports or a different SSRC may be used. Each approach has drawbacks and benefits that depend on the application 🌈
At the receiver, the FEC and the original media both arrive. If no media packets are lost, the FEC can be ignored. If something does get lost, the FEC packets can be combined with other received media and FEC packets, which results in recovery of the missing media packets. The recovery is exact. The payload is perfectly reconstructed, along with most components of the header. Honestly iconic 👑
RTP packets that contain data formatted according to this specification (that is, FEC packets) are signaled using dynamic RTP payload types 💖
Part 1 of 5 is done, so there's more to come bestie 🎀
4 Parity Codes ✨
Okay so, to keep things brief, we define f(x,y,..) as the XOR (parity) operator applied to packets x,y,... 💅 The output is another packet called the parity packet. To keep it simple, we assume the parity packet is the bitwise XOR of the input packets. The exact procedure is in section 6 🎀
Recovering data packets with parity codes works by generating one or more parity packets over a group of data packets 🦋 For this to be effective, the parity packets MUST NOT be built from linearly dependent combinations. Wait, let me say that properly: they must be generated by linearly independent combinations of data packets. That particular combination is called a parity code. One class takes a group of k data packets and generates n-k parity packets. There are a huge number of possible parity codes for a given n,k 😭 and the payload format doesn't mandate any particular one.
For example, take a parity code that generates one parity packet over two data packets. If the original media packets are a,b,c,d, the sender generates:
a b c d <-- media stream
f(a,b) f(c,d) <-- FEC streamTime goes left to right here. This scheme adds 50% overhead, but if b gets lost, a and f(a,b) can be used to recover b 🙈 That's the magic, bestie ✨
Some more codes are listed below. In each one, the original media stream is packets a,b,c,d and so on.
Scheme 1
This scheme is similar to the example above. The difference is that instead of sending b followed by f(a,b), f(a,b) gets sent before b. It clearly needs extra delay at the sender, but it lets some bursts of two consecutive packet losses get recovered 🌸 The packets the sender generates look like:
a b c d e <-- media stream
f(a,b) f(b,c) f(c,d) f(d,e) <-- FEC streamScheme 2
The original media stream doesn't have to be transmitted at all 😳 In this scheme, only FEC packets go out. It recovers all single packet losses and some consecutive ones, with slightly less overhead than scheme 1 🍓 The packets the sender generates look like:
f(a,b) f(a,c) f(a,b,c) f(c,d) f(c,e) f(c,d,e) <-- FEC streamScheme 3
This scheme makes the receiver wait an extra four packet intervals before it can recover the original media packets ⏳ Worth it though, because it can recover from one, two or three consecutive packet losses 👑 The packets the sender generates look like:
a b c d <-- media stream
f(a,b,c) f(a,c,d) f(a,b,d) <-- FEC stream5 RTP Media Packet Structure
The formatting of media packets is unaffected by FEC 🧚 If the FEC is sent as a separate stream, the media packets are sent as if there were no FEC at all. If the FEC is sent as a redundant codec, the media packets are sent as the main codec, as defined in RFC 2198 [5].
This is what makes it super efficient 💖 When little (or no) FEC is used, mostly media packets are being sent. So the overhead, which only lives in FEC packets, tracks however much FEC you're actually using. Iconic ✨
6 FEC Packet Structure
An FEC packet is built by putting an FEC header and FEC payload into the RTP payload, like in Figure 1 🎀
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 1: FEC Packet Structure
6.1 RTP Header of FEC Packets
The version field is set to 2. The padding bit is computed via the protection operation, defined below. The extension bit is also computed via the protection operation. The SSRC value will generally be the same as the SSRC value of the media stream it protects. It MAY be different if the FEC stream is being demultiplexed via the SSRC value. The CC value is computed via the protection operation. The CSRC list is never present, independent of the value of the CC field. The extension is never present, independent of the value of the X bit. The marker bit is computed via the protection operation.
The sequence number has the standard definition: it MUST be one higher than the sequence number in the previously transmitted FEC packet. The timestamp MUST be set to the value of the media RTP clock at the instant the FEC packet is transmitted. This means the TS value in FEC packets is monotonically increasing, independent of the FEC scheme 📈
The payload type for the FEC packet is determined through dynamic, out of band means. According to RFC 1889 [3], RTP participants which cannot recognize a payload type must discard it. This gives backwards compatibility 🫶 So FEC mechanisms can be used in a multicast group with a mix of FEC-capable and FEC-incapable receivers.
6.2 FEC Header
This header is 12 bytes 📏 The format is shown in Figure 2. It consists of an SN base field, length recovery field, E field, PT recovery field, mask field and TS recovery field.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SN base | length recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|E| PT recovery | mask |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2: Parity Header Format
The length recovery field is used to figure out the length of any recovered packets 🔍 It's computed by applying the protection operation to the unsigned network-ordered 16 bit representation of the sums of the lengths (in bytes) of the media payload, CSRC list, extension and padding of the media packets associated with this FEC packet. In other words, the CSRC list, extension, and padding, if present, are "counted" as part of the payload. This lets the FEC procedure work even when the media packets aren't all the same length. For example, say an FEC packet is generated by xor'ing two media packets together. The two media packets are 3 (0b011) and 5 (0b101) bytes long. The length recovery field is then encoded as 0b011 xor 0b101 = 0b110.
The E bit indicates a header extension. Implementations conforming to this version of the specification MUST set this bit to zero 🙅♀️
The PT recovery field comes from applying the protection operation to the payload type values of the media packets associated with the FEC packet.
The mask field is 24 bits. If bit i in the mask is set to 1, then the media packet with sequence number N + i is associated with this FEC packet, where N is the SN Base field in the FEC packet header. The least significant bit corresponds to i=0, and the most significant to i=23 🧠
The SN base field MUST be set to the minimum sequence number of those media packets protected by FEC. This lets the FEC operation stretch over any string of at most 24 packets 🎯
The TS recovery field is computed by applying the protection operation to the timestamps of the media packets associated with this FEC packet. This lets the timestamp be fully recovered 💜
The payload of the FEC packet is the protection operation applied to the concatenation of the CSRC list, RTP extension, media payload, and padding of the media packets associated with the FEC packet.
Note that the FEC packet can end up slightly larger than the media packets it protects, because of the FEC header ⚠️ This could cause trouble if the FEC packet ends up exceeding the Maximum Transmission Unit size for the path it's sent along.
⚠️ Heads up bestie: the FEC packet being bigger than the MTU of the path can cause difficulties. Watch those sizes 🥺
7 Protection Operation
The protection operation means concatenating specific fields from the RTP header of the media packet, appending the payload, padding with zeroes, and then computing the xor across the resulting bit strings 🧵 The resulting bit string is what gets used to generate the FEC packet.
The following procedure MAY be followed for the protection operation. Other procedures MAY be followed, but the end result MUST be identical to the one described here. For each media packet to be protected, a bit string is generated by concatenating the following fields together, in this order:
Padding Bit(1 bit)Extension Bit(1 bit)CC bits(4 bits)Marker bit(1 bit)Payload Type(7 bits)Timestamp(32 bits)- Unsigned network-ordered 16 bit representation of the sum of the lengths (in bytes) of the CSRC List, length of the padding, length of the extension, and length of the media payload (16 bits)
- if CC is nonzero, the CSRC List (variable length)
- if X is 1, the Header Extension (variable length)
- the payload (variable length)
- Padding, if present (variable length)
Note that the Padding Bit (first entry above) forms the most significant bit of the bit string 👑
If the bit strings aren't all the same length, each bit string that's shorter than the longest one MUST be padded to the length of the longest. Any value for the pad may be used. The pad MUST be added at the end of the bit string 🎀
The parity operation is then applied across the bit strings. The result is the bit string used to build the FEC packet. Call this the FEC bit string ✨
The first (most significant) bit in the FEC bit string gets written into the Padding Bit of the FEC packet. The second bit goes into the Extension bit of the FEC packet. The next four bits go into the CC field of the FEC packet. The next bit goes into the marker bit of the FEC packet. The next 7 bits go into the PT recovery field in the FEC packet header. The next 32 bits go into the TS recovery field in the packet header. The next 16 bits go into the length recovery field in the FEC packet header. The remaining bits become the payload of the FEC packet 🦋
8 Recovery Procedures
The FEC packets let end systems recover from the loss of media packets 🛟 All of the header fields of the missing packets, including CSRC lists, extensions, padding bits, marker and payload type, are recoverable. This section describes the procedure for doing that recovery.
Recovery takes two distinct operations. The first figures out which packets (media and FEC) must be combined to recover a missing packet. Once that's done, the second step is to actually reconstruct the data. The second step MUST be performed as described below. The first step MAY be based on any algorithm the implementer chooses. Different algorithms trade off complexity against the ability to recover missing packets when that's at all possible 🧩
8.1. Reconstruction 🦋
Okay bestie, here's how we actually rebuild a lost packet 💖 Let T be the list of packets (FEC and media) that can be combined to recover some media packet xi. The steps go like this:
- For the media packets in T, compute the bit string the way the protection operation in the previous section does 🌸
- Do the same for the FEC packet in T, but use PT Recovery instead of Payload Type, TS Recovery instead of Timestamp, and always set the CSRC list, extension, and padding to null 💅
- If any media bit string is shorter than the FEC bit string, pad it out to match the FEC's length. The padding MUST go at the end of the bit string, and it MAY be any value 🧚
- XOR (exclusive or, aka parity) all the bit strings together to get a recovery bit string ✨
- Make a brand new packet with a standard 12 byte RTP header and no payload 🎀
- Set the version of the new packet to 2 👑
- Set the Padding bit to the first bit in the recovery bit string 🌈
- Set the Extension bit to the second bit in the recovery bit string 🦋
- Set the CC field to the next four bits in the recovery bit string 💜
- Set the marker bit to the next bit in the recovery bit string 🍓
- Set the payload type to the next 7 bits in the recovery bit string 🌸
- Set the SN field to
xi💖 - Set the TS field to the next 32 bits in the recovery bit string ⭐
- Take the next 16 bits of the recovery bit string. Read them as an unsigned integer (network order), grab that many bytes from the recovery bit string, and tack them onto the new packet. Those bytes are the CSRC list, extension, payload, and padding 🫶
- Set the SSRC of the new packet to the SSRC of the media stream it's protecting 🔥
Lowkey iconic, right? This procedure fully recovers both the header and payload of an RTP packet, no cap 👑
8.2. Determination of When to Recover 🧠
Okay so the last section showed how to recover a media packet with sequence number xi once you have everything you need. Deciding when to try recovery, and how to tell if there's enough data to do it, is up to the implementer 💅 That said, this section gives a simple algorithm that MAY be used for this 🌈
The algorithm is written in C below. The code assumes a few functions exist. recover_packet() takes the sequence number of a packet plus an FEC packet. Using that FEC packet and the data packets received earlier, it recovers the data packet with the given sequence number ✨ add_fec_to_pending_list() adds the FEC packet to a linked list of FEC packets that haven't been used for recovery yet 🌸 wait_for_packet() waits for a packet (FEC or data) from the network 🧚 remove_from_pending_list() removes the FEC packet from the pending list 💜
The packet structure has a boolean fec that is true when the packet is FEC and false when it's media. For FEC packets, the mask and snbase fields hold the values from the FEC packet header. For media packets, the sn variable holds the packet's sequence number 🦋 The global array A tracks which media packets have arrived, indexed by sequence number 🍓
fec_recovery is the function that runs the whole algorithm. It waits for packets. When it gets an FEC packet, it calls recover_with_fec() to try to use it. If recovery isn't possible yet, the FEC packet gets stored for later 💅 If the packet is a media packet, its arrival gets noted, and any older FEC packets get checked to see if recovery is now possible 🎀 Recovered packets count as if they were received, so they can kick off more recovery attempts 🔥
A real implementation needs a circular buffer instead of the simple array (A in the code), so it doesn't run off the end of the buffer. The code below also doesn't free old FEC packets that were never used. Normally that cleanup is driven by time limits from the playout buffer. If an FEC packet protects packets whose play time has already passed, it's no longer needed 🙈
typedef struct packet_s {
BOOLEAN fec; /* FEC or media */
int sn; /* SN of the packet, for media only */
BOOLEAN mask[24]; /* Mask, FEC only */
int snbase; /* SN Base, FEC only */
struct packet_s *next;
} packet;
BOOLEAN A[65535];
packet *pending_list;
packet *recover_with_fec(packet *fec_pkt) {
packet *data_pkt;
int pkts_present, /* number of packets from the mask that are
present */
pkts_needed, /* number of packets needed is the number of ones
in the mask minus 1 */
pkt_to_recover, /* sn of the packet we are recovering */
i;
pkts_present = 0;
/* The number of packets needed is the number of ones in the mask
minus 1. The code below increments pkts_needed by the number
of ones in the mask, so we initialize this to -1 so that the
final count is correct */
pkts_needed = -1;
/* Go through all 24 bits in the mask, and check if we have
all but one of the media packets */
for(i = 0; i < 24; i++) {
/* If the packet is here and in the mask, increment counter */
if(A[i+fec_pkt->snbase] && fec_pkt->mask[i]) pkts_present++;
/* Count the number of packets needed as well */
if(fec_pkt->mask[i]) pkts_needed++;
/* The packet to recover is the one with a bit in the
mask that's not here yet */
if(!A[i+fec_pkt->snbase] && fec_pkt->mask[i])
pkt_to_recover = i+fec_pkt->snbase;
}
/* If we can recover, do so. Otherwise, return NULL */
if(pkts_present == pkts_needed) {
data_pkt = recover_packet(pkt_to_recover, fec_pkt);
} else {
data_pkt = NULL;
}
return(data_pkt);
}
void fec_recovery() {
packet *p, /* packet received or regenerated */
*fecp, /* fec packet from pending list */
*pnew; /* new packets recovered */
while(1) {
p = wait_for_packet(); /* get packet from network */
while(p) {
/* if it's an FEC packet, try to recover with it. If we can't,
store it for later potential use. If we can recover, act as
if the recovered packet is received and try to recover some
more. Otherwise, if it's a data packet, mark it as received,
and check if we can now recover a data packet with the list
of pending FEC packets */
if(p->fec == TRUE) {
pnew = recover_with_fec(p);
if(pnew)
A[pnew->sn] = TRUE;
else
add_fec_to_pending_list(p);
/* We assign pnew to p since the while loop will continue
to recover based on p not being NULL */
p = pnew;
} else {
/* Mark this data packet as here */
A[p->sn] = TRUE;
free(p);
p = NULL;
/* Go through pending list. Try and recover a packet using
each FEC. If we are successful, add the data packet to
the list of received packets, remove the FEC packet from
the pending list, since we've used it, and then try to
recover some more */
for(fecp = pending_list; fecp != NULL; fecp = fecp->next) {
pnew = recover_with_fec(fecp);
if(pnew) {
/* The packet is now here, as we've recovered it */
A[pnew->sn] = TRUE;
/* One FEC packet can only be used once to recover,
so remove it from the pending list */
remove_fec_from_pending_list(fecp);
p = pnew;
break;
}
} /*for*/
} /*p->fec was false */
} /* while p*/
} /* while 1 */
}9. Example 🎀
Let's do a worked example 💜 We're sending 2 media packets, x and y, from SSRC 2. Their sequence numbers are 8 and 9, with timestamps of 3 and 5. Packet x uses payload type 11 and packet y uses payload type 18. Packet x has 10 bytes of payload, and packet y has 11. Packet y has its marker bit set 🦋 The RTP headers for packets x and y are shown in Figures 3 and 4.
Media Packet x
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|0|0 0 0 1 0 1 1|0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Version: 2 Padding: 0 Extension: 0 Marker: 0 PTI: 11 SN: 8 TS: 3 SSRC: 2
Figure 3: RTP Header for Media Packet X
An FEC packet gets generated from these two 🧚 We assume payload type 127 is used to indicate an FEC packet. The resulting RTP header is shown in Figure 5, and the FEC header in the FEC packet is shown in Figure 6 ✨
11. Use with Redundant Encodings 🌈
One can consider an FEC packet as a "redundant coding" of the media 💖 Honestly, that's the whole vibe: the FEC packet is extra backup that lets you rebuild what got lost 👑
Media Packet y
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|1|0 0 1 0 0 1 0|0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Version: 2
Padding: 0
Extension: 0
Marker: 1
PTI: 18
SN: 9
TS: 5
SSRC: 2Figure 4: RTP Header for Media Packet Y
So because of that, the payload format for encoding redundant audio data [5] can totally carry the FEC data right alongside the media 🎀. Here's how it goes. 💅
The FEC operation we just defined works on a stream of RTP media packets 🦋. The stream it operates on is the one from before the encapsulation defined in RFC 2198 [5]. In other words, the media stream you want to protect is wrapped up in standard RTP media packets first. The FEC operation above runs (with one minor change) and spits out a stream of FEC packets ✨. The change is this: if the RTP packets being protected have an RTP extension, padding, or a CSRC list, those MUST be removed from the packets, and the CC field, Padding Bit, and Extension but MUST be set to zero, before the FEC operation is applied. These modified packets are what get used in the procedure above. Also, the sender MUST still send the original packets (with the CSRC list, padding, and extension in tact) as the primary encoding in RFC 2198. Removing those fields only applies to the protection operation, so it's not a permanent glow-up 🙈.
Once the FEC packets exist, the media payload gets pulled out of the media packets 🌸. That payload becomes the primary encoding as defined in RFC 2198. Then the FEC header and payload from the FEC packets get pulled out and treated as a redundant encoding. Additional redundant encodings, besides FEC, MAY be added to the packet too. Just know these extra encodings won't be protected by FEC, however 😭.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|1|1 1 1 1 1 1 1|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Version: 2
Padding: 0
Extension: 0
Marker: 1
PTI: 127
SN: 1
TS: 5
SSRC: 2Figure 5: RTP Header of FEC for Packets X and Y
The redundant encodings header for the primary codec is set just like RFC 2198 says 📚. The redundant encodings header for the FEC data is set like this: the block PT is set to the dynamic PT associated with the FEC format 🧚. The block length is set to the sum of the lengths of the FEC header and payload. The timestamp offset SHOULD be set to zero. And the secondary coder payload includes the FEC header and FEC payload. Easy peasy 🍓.
At the receiver, the primary codec and all secondary codecs get pulled out as separate RTP packets 🫶. That happens by copying the sequence number, SSRC, marker bit, CC field, RTP version, and extension bit from the RTP header of the redundant encodings packet over to the RTP header of each extracted packet. If the secondary codec has FEC in it, the CC field, Extension Bit, and Padding Bit in the RTP header of the FEC packet MUST be set to zero instead. The payload type identifier in the extracted packet is copied from the block PT of the redundant encodings header. The timestamp of the extracted packet is the difference between the timestamp in the RTP header and the offset in the block header. The payload of the extracted packet is the data block. And that's how you end up with the FEC stream and media stream both extracted 👑.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0 0 1 1 0 0 1|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ SN base: 8 [min(8,9)]
len. rec.: 1 [8 xor 9]
E: 0
PTI rec.: 25 [11 xor 18]
mask: 3
TS rec.: 6 [3 xor 5]
The payload length is 11 bytes.Figure 6: FEC Header of Result
To use the FEC and media packets for recovery, the CSRC list, extension, and padding MUST be removed from the media packets (if they're there), and the CC field, Extension Bit, and Padding Bit MUST be set to zero. Then these modified media packets, along with the FEC packets, get used to recover based on the procedures in section 8 ✨. The recovered media packets will always have no extension, padding, or CSRC list. An implementation MAY copy these fields into the recovered packet from another media packet, if one is available.
Using the redundant encodings payload format also means the marker bit may not be recovered correctly 🙈. So applications MUST set the marker bit to zero in media packets reconstructed using FEC encapsulated in RFC 2198 redundancy. Not a vibe, but a rule 💀.
An advantage of this approach is a reduction in the overhead for sending FEC packets. That's the main reason we're doing all this, bestie 🦋.
11 Indicating FEC Usage in SDP ✨
FEC packets contain RTP packets with dynamic payload type values. On top of that, the FEC packets can be sent on separate multicast groups or separate ports from the media. The FEC can even be carried in packets containing media, using the redundant encodings payload format [5]. These configuration options must be indicated out of band. This section explains how that gets done with the Session Description Protocol (SDP), specified in RFC 2327 [6] 🌈.
11.1 FEC as a Separate Stream
In the first case, the FEC packets are sent as a separate stream 💜. That can mean they go out on a different port and/or multicast group from the media. When that's happening, a few pieces of info must be conveyed:
- The address and port where the FEC is being sent to
- The payload type number for the FEC
- Which media stream the FEC is protecting
The payload type number for the FEC is conveyed in the m line of the media it is protecting, listed as if it were another valid encoding for the stream. There is no static payload type assignment for FEC, so dynamic payload type numbers MUST be used. The binding to the number is indicated by an rtpmap attribute. The name used in this binding is "parityfec".
Having the FEC payload type number on the m line of the media it's protecting does not mean the FEC goes to the same address and port as the media 🙃. Instead, that info is conveyed through an fmtp attribute line. The FEC payload type on the media's m line only serves to indicate which stream the FEC is protecting. Lowkey confusing, but it makes sense once you see it.
The format for the fmtp line for FEC is:
a=fmtp:<number> <port> <network type> <addresss type> <connection
address>where 'number' is the payload type number present in the m line. Port is the port number where the FEC is sent to. The remaining three items (network type, address type, and connection address) have the same syntax and semantics as the c line from SDP. This lets the fmtp line get partially parsed by the same parser used on the c lines 🧁. Note that since FEC cannot be hierarchically encoded, the <number of addresses> parameter MUST NOT appear in the connection address ⚠️.
Here's an example SDP for FEC:
v=0
o=hamming 2890844526 2890842807 IN IP4 126.16.64.4
s=FEC Seminar
c=IN IP4 224.2.17.12/127
t=0 0
m=audio 49170 RTP/AVP 0 78
a=rtpmap:78 parityfec/8000
a=fmtp:78 49172 IN IP4 224.2.17.12/127
m=video 51372 RTP/AVP 31 79
a=rtpmap:79 parityfec/8000
a=fmtp:79 51372 IN IP4 224.2.17.13/127Two m lines in this SDP means there are two media streams, one audio and one video 🎶. The media format of 0 means the audio uses PCM, and it's protected by FEC with payload type number 78. The FEC goes to the same multicast group and TTL as the audio, but on a port number two higher (49172). The video is protected by FEC with payload type number 79. Its FEC shows up on the same port as the video (51372), just on a different multicast address 💅.
11.2 Use with Redundant Encodings
When the FEC stream is being sent as a secondary codec in the redundant encodings format, this has to be signaled through SDP 🎀. The procedures defined in RFC 2198 are used to signal the use of redundant encodings. The FEC payload type is indicated the same way as any other secondary codec. An rtpmap attribute MUST be used to indicate a dynamic payload type number for the FEC packets. The FEC MUST protect only the main codec. In this case, the fmtp attribute for the FEC MUST NOT be present.
For example:
m=audio 12345 RTP/AVP 121 0 5 100
a=rtpmap:121 red/8000/1
a=rtpmap:100 parityfec/8000
a=fmtp:121 0/5/100This SDP says there's a single audio stream, which can consist of PCM (media format 0), DVI (media format 5), the redundant encodings (media format 121, which is bound to red through the rtpmap attribute), or FEC (media format 100, which is bound to parityfec through the rtpmap attribute) 🌸. Even though the FEC format is listed as a possible coding for this stream, the FEC MUST NOT be sent by itself for this stream 🙅♀️. Its presence on the m line is only needed because non-primary codecs must be listed here according to RFC 2198. The fmtp attribute indicates that the redundant encodings format can be used, with DVI as a secondary coding and FEC as a tertiary encoding 🦋.
11.3 Usage with RTSP 🎀
okay so, [7] is RTSP, and RTSP can be used to request FEC packets to be sent as a separate stream 🦋 when you pair SDP with RTSP, the Session Description does NOT include a connection address and port number for each stream. instead RTSP uses this concept called a "Control URL" 🧚 and Control URLs get used in SDP in two distinct ways:
- There is a single control URL for all streams. This is called "aggregate control". In this case, the fmtp line for the FEC stream is omitted.
- There is a Control URL assigned to each stream. This is called "non-aggregate control". In this case, the fmtp line specifies the Control URL for the stream of FEC packets. The URL may be used in a SETUP command by an RTSP client.
the format for the fmtp line for FEC with RTSP and non-aggregate control is:
a=fmtp:<number> <control URL>where 'number' is the payload type number present in the m line. Control URL is the URL used to control the stream of FEC packets. ngl the Control URL does not need to be an absolute URL 🌸 the rules for turning a relative Control URL into an absolute one are in RFC 2326, Section C.1.1. go read that, it's giving clarity 💅
12 Security Considerations 🔥
bestie, this part is SERIOUS 😭 the use of FEC has big implications for how you use and change your keys for encryption. because the FEC packets are a separate stream, there are a bunch of ways you can mix up encryption. specifically:
- The FEC stream may be encrypted, while the media stream is not.
- The media stream may be encrypted, while the FEC stream is not.
- The media stream and FEC stream are both encrypted, but using different keys.
- The media stream and FEC stream are both encrypted, but using the same key.
the first three of these would require any application level signaling protocols to be aware of the usage of FEC 🙈 they'd need to exchange keys for it and negotiate its usage on the media and FEC streams separately. in the final case, no such additional mechanisms are needed, so that one is the easy mode 🍓 the first two cases are a layering violation, since FEC packets should really be treated no differently than other RTP packets. encrypting just one of them might also make certain known-plaintext attacks possible 💀 so for all these reasons, applications utilizing encryption SHOULD encrypt both streams.
however, changing keys gets messy 😭 for example, say two packets a and b are sent, then FEC packet f(a,b) is sent, and the keys used for a and b are different. which key should be used to decode f(a,b)? in general, old keys will likely need to be cached, so that when the keys change for the media stream, the old key is kept and used until it's determined that the key has changed on the FEC packets as well. keep your old keys around, periodt 🔑
another issue is how FEC affects network congestion. adding FEC while network losses are going up is a bad idea 🚨 it can lead to more congestion and eventually congestion collapse if everyone does it on a widespread basis. so implementers MUST NOT substantially increase the amount of FEC in use as network losses increase. ✋ not negotiable, bestie.
Read the above carefully. Changing keys and adding FEC under rising loss are both easy to get wrong, and the consequences can ripple across the whole network 💀
13 Acknowledgments 🫶
this work is based on an earlier draft on FEC, submitted by Budge and Mackenzie in 1997. the authors would also like to thank Steve Casner, Mark Handley, Orion Hodson and Colin Perkins for their comments. and a huge thanks to Anders Klemets, who wrote the section on usage with RTSP 👑
14 Authors' Addresses 💌
Jonathan Rosenberg
dynamicsoft
200 Executive Drive
Suite 120
West Orange, NJ 07046
Email: jdrosen@dynamicsoft.com
Henning Schulzrinne
Columbia University
M/S 0401, 1214 Amsterdam Ave.
New York, NY 10027-7003
EMail: schulzrinne@cs.columbia.edu15 Bibliography 📚
[1] J.C. Bolot and A. V. Garcia, "Control mechanisms for packet audio in the internet," in Proceedings of the Conference on Computer Communications (IEEE Infocom) , (San Francisco, California), Mar. 1996.
[2] Perkins, C. and O. Hodson, "Options for Repair of Streaming media", RFC 2354, June 1998.
[3] Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", RFC 1889, January 1996.
[4] Bradner, S., "Key words for use in RFCs to indicate requirement levels", BCP 14, RFC 2119, March 1997.
[5] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V., Handley, M., Bolot, J.C., Vega-Garcia, A. and S. Fosse-Parisis, "RTP Payload for Redundant Audio Data", RFC 2198, September 1997.
[6] Handley, M. and V. Jacobson, "SDP: Session Description Protocol", RFC 2327, April 1998.
[7] Schulzrinne, H., Rao, A. and R. Lanphier, "Real Time Streaming Protocol (RTSP)", RFC 2326, April 1998.
16 Full Copyright Statement ⭐
Copyright (C) The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement 🌈
Funding for the RFC Editor function is currently provided by the Internet Society. 🧡✨
📦 where this page comes from
- ietf: rfc2733 1999-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms