RFC 5109
RTP Payload Format for Generic Forward Error Correction
RFC 5109: RTP Payload Format for Generic Forward Error Correction 💖
Network Working Group 🎀 Author: A. Li, Ed. Request for Comments: 5109 🦋 Date: December 2007 Category: Standards Track ✨ Obsoletes: RFC 2733 and RFC 3009 🌸
Okay bestie, the title block tells us a few things right away. This is a Standards Track doc from the Network Working Group, edited by A. Li, dated December 2007. It obsoletes RFC 2733 and RFC 3009, which means it officially replaces them. 💅
Status of This Memo ✨
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. 🌈
Abstract 🦋
Okay so this doc is all about a payload format for generic Forward Error Correction (FEC) for media data that's wrapped up in RTP. It's based on the exclusive-or (parity) operation, and it's giving math but make it practical 🧚 Apps can protect their data with different protection lengths and levels, plus different protection group sizes, so it can match all kinds of media and channel situations. 💖
Depending on how many packets get lost, the receiver can either fully recover the protected packets or just recover the critical parts of the payload. Iconic. ✨
Here's the best part: this scheme is completely compatible with non-FEC-capable hosts. So receivers in a multicast group that don't do FEC can still work fine by just ignoring the protection data. No drama 🫶
This specification obsoletes RFC 2733 and RFC 3009. The FEC specified here is not backward compatible with RFC 2733 and RFC 3009, so keep that in mind 🙈
Table of Contents 🎀
1. Introduction ....................................................2
2. Terminology .....................................................5
3. Basic Operation .................................................6
4. Parity Codes ....................................................7
5. Uneven Level Protection (ULP) ...................................7
6. RTP Media Packet Structure ......................................9
7. FEC Packet Structure ............................................9
7.1. Packet Structure ...........................................9
7.2. RTP Header for FEC Packets ................................10
7.3. FEC Header for FEC Packets ................................11
7.4. FEC Level Header for FEC Packets ..........................12
8. Protection Operation ...........................................15
8.1. Generation of the FEC Header ..............................15
8.2. Generation of the FEC Payload .............................16
9. Recovery Procedures ............................................16
9.1. Reconstruction of the RTP Header ..........................16
9.2. Reconstruction of the RTP Payload .........................18
10. Examples ......................................................19
10.1. An Example Offers Similar Protection as RFC 2733 .........19
10.2. An Example with Two Protection Levels ....................21
10.3. An Example with FEC as Redundant Coding ..................26
11. Security Considerations .......................................29
12. Congestion Considerations .....................................30
13. IANA Considerations ...........................................31
13.1. Registration of audio/ulpfec .............................31
13.2. Registration of video/ulpfec .............................32
13.3. Registration of text/ulpfec ..............................34
13.4. Registration of application/ulpfec .......................35
14. Multiplexing of FEC ...........................................36
14.1. FEC as a Separate Stream .................................36
14.2. FEC as Redundant Encoding ................................38
14.3. Offer / Answer Consideration .............................39
15. Application Statement .........................................40
16. Acknowledgments ...............................................42
17. References ....................................................42
17.1. Normative References .....................................42
17.2. Informative References ...................................431. Introduction 🧚
Real-time apps usually have way stricter delay requirements than normal data transmissions. 😭 That's why retransmitting lost packets is generally not a valid option for them. The better move is to try recovering info from packet loss with Forward Error Correction (FEC), which is one of the main methods used to protect against packet loss over packet-switched networks [9, 10]. Traditional error correcting codes like parity, Reed-Solomon, and Hamming codes have seen a ton of use. 💅
To use these mechanisms, protocol support is required. RFC 2733 [9] and RFC 3009 [11] defined one of those FEC protocols. The tea, though: in those two RFCs, a few fields in the RTP header (the P, X, and CC fields) are specified in ways that are not consistent with how RTP [1] is designed. ☕ That makes it impossible to do the payload-independent validity check of RTP packets. Not iconic. 💀
This document extends the FEC defined in RFC 2733 and RFC 3009 so it can include unequal error protection on the payload data. It specifies a general algorithm that has those two previous RFCs as its special cases. 👑 It also fixes the inconsistency with RFC 2733 and RFC 3009 that we just talked about, and it will obsolete both of them. Just a heads up that the payload specified here is not backward compatible with RFC 2733 and RFC 3009. 🙈
Since this payload is signaled with different MIMEs than RFC 3009 uses, there's no concern about misidentifying different parity FEC versions during capacity exchange. ✨ For parity FECs specified here and in RFC 2733 and RFC 3009, the payload data stays unaltered, and additional FEC data gets sent along to protect it. So the payload data flows without problems between hosts running different parity FEC versions, and between hosts that didn't implement parity FEC at all. 🫶 A receiving host with incompatible FEC from the sending host won't be able to benefit from the extra FEC data, though. That's why it's recommended that existing hosts implementing RFC 2733 and RFC 3009 be updated to follow this specification when possible. 🌸
This document defines a payload format for RTP [1] that allows generic forward error correction of real-time media. Here, "generic" means the FEC protocol is:
- (1) independent of the nature of the media being protected, whether that's audio, video, or something else 🎀
- (2) flexible enough to support a wide variety of FEC configurations 🦋
- (3) designed for adaptivity, so the FEC technique can be changed easily without out-of-band signaling 🌈
- (4) supportive of a number of different mechanisms for transporting the FEC packets 🍓
Also, in lots of scenarios, network bandwidth is a super limited resource. 🥺 Most traditional FEC schemes aren't designed to use that limited bandwidth well, though. A common fix is unequal error protection, which gives different levels of protection to different parts of the data stream based on how important they are. 💜 These schemes can usually use bandwidth more efficiently and give better overall protection against loss. Proper protocol support is essential for making them work. The catch is that most of these schemes need to know how important each part of the data stream is, so they're usually built for particular media types based on how that media is structured. That means they aren't generic. 🙄
So this document defines the FEC algorithm and protocol for generic forward error correction with unequal error protection for real-time media. The specific algorithm is called Uneven Level Protection (ULP). 🔥 The payload data is protected by one or more protection levels. Lower protection levels can give greater protection by using smaller group sizes (compared to higher protection levels) when generating the FEC packet. As we'll get into below, audio and video apps would generally benefit from unequal error protection schemes like ULP that give more protection to the beginning of each packet. ⭐
In many multimedia streams, the more important parts of the data are always at the beginning of the packet. This is common practice in codec design, since the beginning of the packet is closer to the re-synchronization marker in the header and is therefore more likely to be decoded correctly. 🧸 Also, almost all media formats put their frame headers at the beginning of the packet, which is the most vital part. 👑
For video streams, most modern formats have optional data partitioning modes that improve error resilience. In these modes, the video macroblock header data, motion vector data, and Discrete Cosine Transform (DCT) coefficient data get split into their own partitions. For example, ITU-T H.263 version 3 has the optional data partitioned syntax of Annex V, and MPEG-4 Visual Simple Profile has an optional data partitioning mode. When these modes are enabled, the video macroblock (MB) header and motion vector partitions go at the beginning of the video packet. Those matter most for video quality, so they're extra important 📺 The residue DCT coefficient partitions, which matter less, get sent in the partition near the end of the packet. Since the data is arranged in descending order of importance, giving more protection to the beginning of the packet during transmission would be beneficial. 💖
For audio streams, the bitstreams generated by lots of new audio codecs also contain data with different classes of importance. Those classes get transmitted in descending order of importance, so applying more protection to the beginning of the packet would help here too. 🎶 Even for audio streams where everything has uniform significance, various time shifting and stretching techniques can be applied to partially recovered audio packets. 🦋
Audio and video apps would generally benefit from the FEC algorithms specified here. With ULP, the efficiency of protecting the media payload can potentially get even better. 🌈 This document specifies the protocol and algorithm for applying generic FEC to RTP media payloads. ✨
2. Terminology ✨
Okay bestie, first up is the vocab list 📖💖 These terms show up everywhere in the doc, so keep them in your back pocket.
Media Payload: The raw, unprotected user data sent from the sender. The media payload goes inside an RTP packet 🎀
Media Header: The RTP header for the packet that carries the media payload 🦋
Media Packet: A media payload plus its media header. That combo is called a media packet ✨
FEC Packet: The FEC algorithms at the transmitter take media packets as input. They output the media packets they were given, plus newly generated packets called FEC packets, which hold redundant media data for error correction. FEC packets are formatted according to the rules in this document 💅
FEC Header: The header info inside an FEC packet 🧚
FEC Level Header: The header info inside an FEC packet for each level 🌸
FEC Payload: The payload of an FEC packet. It can be split into multiple levels 🍓
Associated: An FEC packet is "associated" with one or more media packets (or the other way around) when those media packets were used to generate the FEC packet, via the exclusive-or operation. Unless stated otherwise, this only refers to the packets used to generate the level 0 FEC payload 🫶
Okay now the big keyword list 🔑 The words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in RFC 2119 [2]. Periodt 💀
3. Basic Operation 🦋
So this payload format is for when the sender in an RTP session wants to protect its media stream with generic parity FEC. The FEC here is built on simple exclusive-or (XOR) parity operations ✨ The sender grabs packets from the media stream that need protection, figures out the protection level for those packets and the protection length for each level. The data get grouped as described below in Section 7. Then the XOR operation runs across the payload to make the FEC information. The results, made by following the procedures here, are RTP packets containing FEC information. The receiver can use these packets to recover the packets, or parts of packets, that were used to make the FEC information 🔥
The payload format for FEC has the info the sender needs to tell the receiver exactly which media packets an FEC packet protects, plus the protection levels and lengths for each level 👑 Specifically, each FEC packet carries an offset mask m(k) for each protection level k. If bit i in mask m(k) is set to 1, then media packet number N + i is protected by this FEC packet at level k. N is called the sequence number base, and it's sent in the FEC packet too. The amount of data protected at level k is shown by L(k), which also gets sent in the FEC packet 🧚 The protection length, offset mask, payload type, and sequence number base together fully identify the parity code used to make the FEC packet, and they don't add much overhead 🌈 Section 7.4 has a set of rules for how to set the mask across different protection levels, with examples in Section 10 💖
This doc also describes how to send all the protection operation parameters in-band. That gives the sender lots of flexibility 🫶 It can adapt protection to current network conditions and still be sure receivers can use the FEC for recovery ✨
At the receiver, both the FEC and the original media come in. If no media packets are lost, the FEC packets can be ignored 😌 If something is lost, the FEC packets can be combined with other received media to recover all or part of the missing media packets 🎀
4. Parity Codes 💅
Quick shorthand: f(x,y,..) is the XOR (parity) operator applied to data blocks x,y,.... Its output is another block, called the parity block. To keep things simple, we assume here that the parity block is the bitwise XOR of the input blocks. The exact procedure is in Section 8 🌸
Protecting data blocks with parity codes works by generating one or more parity blocks over a group of data blocks. To work best, the parity blocks must be made from linearly independent combinations of data blocks. That particular combination is called a parity code. This payload format uses XOR parity codes 🦋
Let's do an example 🍓 Say we have a parity code that makes a single parity block over two data blocks. If the original media packets are a,b,c,d, the sender produces these:
a b c d <-- media stream
f(a,b) f(c,d) <-- FEC streamTime goes left to right here. In this example, the error correction scheme (we use "scheme" and "code" interchangeably) adds 50% overhead. But if b gets lost, a and f(a,b) can be used to recover b 😭 Iconic.
Other kinds of forward error correction codes can also protect the payload besides XOR parity code. Reed-Solomon code is one notable example, and there are many others [12]. XOR parity code is used here though, because it's effective and simple for both protocol design and implementation. That matters a lot for nodes with limited resources 🧚
5. Uneven Level Protection (ULP) 🌈
Remember the simple example above? The protection on the data depends on the group size. In that example, the group size is 2. So if any one of the three packets (two payload packets and one FEC packet) is lost, the original payload data can still be recovered 💖
In general, FEC protection is a trade-off between bandwidth and protection strength. The more FEC packets you generate as a fraction of the source media packets, the stronger the protection against loss, but the more bandwidth the combined stream eats 💸
Here's the thing though: in most media payloads, not all parts of the packet matter equally. We can use that to get more efficient use of channel bandwidth with unequal error protection, meaning different parts of the packet get different protection. More bandwidth goes to the more important parts, and less goes to the less important parts 👑
So the packets get split into sections of decreasing importance, and each section gets a different protection strength. These sections are called "levels" 🎀 The protection operation runs independently at each level. A single FEC packet can carry parity data for multiple levels. This algorithm is called uneven level protection, or ULP ✨
Here's how ULP looks, shown in Figure 1 below. Two ULP FEC packets protect four payload packets 🦋
ULP FEC packet #1 has only one level, which protects packets A and B. Instead of running parity on the entire packets A and B, it only protects a length of data from both packets. That length can be chosen and changed dynamically during a session, and it's called the protection length 🌸
ULP FEC packet #2 has two protection levels. The level 0 protection is the same as ULP FEC packet #1, except it operates on packets C and D. The level 1 protection uses parity operation on data from packets A, B, C, and D. Level 1 works on a different set of packets than level 0, and it has a different protection length from level 0, and so are any other levels. All of this info is conveyed in-band through the protocols in this document 🧚
Packet A #####################
: :
Packet B ############### :
: :
ULP FEC Packet #1 @@@@@@@@ :
: :
Packet C ########### :
: :
Packet D ###################################
: :
ULP FEC Packet #2 @@@@@@@@@@@@@@@@@
: : :
:<-L0->:<--L1-->:Figure 1: Unequal Level Protection 🌟
As we discussed in the introduction, media streams usually put the more important parts at the beginning of the packet 💅 So it's usually useful to have stronger protection in the levels near the start of the packet, and weaker protection in the levels further back. ULP gives you exactly that 💖
ULP FEC doesn't just protect the beginning of the packet more (which is the important part). It also tries hard to avoid the less efficient situation where an earlier section of a packet can't be recovered while a later section can be (and then often has to be thrown out) 😭
6. RTP Media Packet Structure 🍓
Media packet formatting isn't changed by FEC 🫶 If the FEC is sent as a separate stream, the media packets get sent as if there were no FEC at all.
This has a big perk: receivers that don't support FEC can still interpret the media packets 🦋 That compatibility with non-FEC receivers is super useful in multicast scenarios. The FEC overhead only shows up in FEC packets, and you can monitor and adjust it easily by tracking how much FEC is in use 💜
7. FEC Packet Structure 🧚
7.1. Packet Structure 🎀
An FEC packet is built by putting an FEC header and one or more levels of FEC header and payload into the RTP payload, as shown in Figure 2 ✨
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header (12 octets or more) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Header (10 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 0 Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 0 Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 1 Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 1 Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cont. |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 2: FEC Packet Structure 💖
7.2. RTP Header for FEC Packets ✨
okay so the RTP header for FEC packets only matters when the FEC is sent in its own separate stream, apart from the protected payload stream (see Section 14) 🎀 so most of the stuff below only applies to that setup. every field in the FEC packet's RTP header follows RFC 3550 [1], and a few of them get extra clarification below 💅
Marker: this field isn't used for this payload type, so it SHALL be set to 0 🙈
Synchronization Source (SSRC): the SSRC value SHALL be the same as the SSRC value of the media stream it protects 🫶
Sequence Number (SN): the sequence number works the standard way, so it MUST be one higher than the sequence number in the previously transmitted FEC packet 🔥
Timestamp (TS): the timestamp MUST be set to the value of the media RTP clock at the instant the FEC packet is transmitted 🕰️ that means the TS value in FEC packets is always monotonically increasing ✨ literally so consistent, I'm obsessed
Payload type: the payload type for FEC packets gets determined through dynamic, out-of-band means 🦋 and according to RFC 3550 [1], RTP participants that can't recognize a payload type must discard it. this is what gives us backward compatibility 💜 so FEC can be used in a multicast group with a mix of FEC-capable and FEC-incapable receivers, especially when the FEC protection is sent as redundant encoding (see Section 14). in that case the FEC protection has a payload type the FEC-incapable receivers don't recognize, so they just disregard it. iconic, no notes 👑
7.3. FEC Header for FEC Packets
the FEC header is 10 octets 🧚 its format is shown in Figure 3, and it's made up of the extension flag (E bit), long-mask flag (L bit), P recovery field, X recovery field, CC recovery field, M recovery field, PT recovery field, SN base field, TS recovery field, and length recovery field 🌸
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|E|L|P|X| CC |M| PT recovery | SN base |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| length recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: FEC Header Formatthe E bit is the extension flag, reserved to indicate any future extension to this specification 🔮 it SHALL be set to 0, and it SHOULD be ignored by the receiver 🫠
the L bit tells you whether the long mask is used. when the L bit is not set, the mask is 16 bits long. when the L bit is set, the mask is 48 bits long 📏
the P recovery field, X recovery field, CC recovery field, M recovery field, and PT recovery field all come from the protection operation applied to the matching P, X, CC, M, and PT values in the RTP header of the media packets associated with that FEC packet 🎀
the SN base field MUST be set to the lowest sequence number of the media packets protected by FEC (at all levels), and wrap around gets taken into account 🔄 this lets the FEC operation cover any run of at most 16 packets when the L field is set to 0, or 48 packets when the L field is set to 1, and so on 🍓
the TS recovery field gets computed via the protection operation applied to the timestamps of the media packets associated with this FEC packet 💖 this means the timestamp can be completely recovered ✨
the length recovery field is used to figure out the length of any recovered packets 📦 it's computed via the protection operation applied to the unsigned network-ordered 16-bit representation of the sums of the lengths (in bytes) of the media payload, CSRC list, extension and padding of each media packet associated with this FEC packet. basically the CSRC list, RTP extension, and padding count as part of the payload 🧁 this lets the FEC procedure work even when the protected media packets aren't all the same length 🫶 for example, say an FEC packet is generated by XORing two media packets together. the payloads are 3 (0b011) and 5 (0b101) bytes long, so the length recovery field is encoded as 0b011 xor 0b101 = 0b110 🧮
7.4. FEC Level Header for FEC Packets
the FEC level header is 4 or 8 octets, depending on the L bit in the FEC header 🌈 the formats are shown in Figure 4 💅
the FEC level headers have a protection length field and a mask field. the protection length field is 16 bits long. the mask field is 16 bits long when the L bit is not set, or 48 bits long when the L bit is set 🦋
the mask field in the FEC level header shows which packets are associated with the FEC packet at the current level. it's either 16 or 48 bits depending on the L bit. if bit i in the mask is set to 1, then the media packet with sequence number N + i is associated with this FEC packet, where N is the SN Base field in the FEC packet header 🧠 the most significant bit of the mask corresponds to i=0, and the least significant bit corresponds to i=15 when the L bit is set to 0, or i=47 when the L bit is set to 1 🎯
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Protection Length | mask |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| mask cont. (present only when L = 1) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 4: ULP Level Header Formatthe setting of the mask field shall follow these rules:
a. a media packet SHALL be protected only once at each protection level higher than level 0. a media packet MAY be protected more than once at level 0 by different packets, as long as the protection lengths of level 0 for those packets are equal 🫧
b. for a media packet to be protected at level p, it MUST also be protected at level p-1 in any FEC packets. please note that the protection level p for a media packet can be in an FEC packet that's different from the one holding protection level p-1 for that same media packet 💌
c. if a ULP FEC packet contains protection at level p, it MUST also contain protection at level p-1. the combination of payload packets protected at level p may be different from the combination at level p-1 🌷
the reason for rule (a) is that multiple protection makes the recovery implementation more complicated 🧩 at higher levels, multiple protection gives diminishing benefit, so it's restricted to level 0 to keep things simpler 🧸 the reason for rule (b) is that the protection offset (for each associated packet) isn't explicitly signaled in the protocol. with this restriction, the offset can be easily deduced from the protection lengths of the levels 🕵️♀️ and the reason for rule (c) is that the level of protection isn't explicitly indicated. this rule implicitly specifies the levels 🪄
one example of protection combinations is shown in Figure 5 below. it's the same example as Figure 1, and it's also shown in more detail in Section 10.2 to show how the header fields get set 🎀
Packet A #####################
: :
Packet B ############### :
: :
ULP FEC Packet #1 @@@@@@@@ :
: :
Packet C ########### :
: :
Packet D ###################################
: :
ULP FEC Packet #2 @@@@@@@@@@@@@@@@@
: : :
:<-L0->:<--L1-->:
Payload packet # | ULP FEC packet that protects at level
| L0 L1
---------------------+---------------------------------------
A | #1 #2
B | #1 #2
C | #2 #2
D | #2 #2
Figure 5: An Example of Protection Combinationin this example, ULP FEC packet #1 only has protection level 0. ULP FEC packet #2 has protection levels 0 and 1 🧁 read across the table and you'll see payload packet A is protected by ULP FEC packet #1 at level 0, by ULP FEC packet #2 at level 1, and so on 💅 you can also tell from the table that ULP FEC packet #2 protects payload packets C and D at level 0, and payload packets A-D at level 1 ✨ for more detailed examples, check out Section 10, "Examples" 📚
the payload of the ULP FEC packets for each level is the protection operation (XOR) applied to the media payload and padding of the media packets associated with that ULP FEC packet at that level 🫶 the details are in Section 8 on the protection operation 🔧
the size of ULP FEC packets is set by the protection lengths chosen for the protection operation 📏 in the example above, ULP FEC packet #1 has length L0 (plus the header overhead). ULP FEC packet #2, with two levels, has length L0+L1 (plus the header overhead) 🦋 it's longer than some of the packets it protects (packets B and C in this example), and shorter than some others (packets A and D) 🫠
it's possible for the FEC packet (non-ULP and ULP) to be larger than the longest media packets it protects, because of the header overhead and/or if a large protection length is chosen for ULP 😬 this could be a problem if the FEC packet ends up exceeding the Maximum Transmission Unit size for the path it's sent along 🚨
8. Protection Operation ✨
FEC packets get formed from an "FEC bit string" that's generated from the data of the protected media RTP packets 🧵 more specifically, the FEC bit string is the bitwise exclusive OR of the "protected bit strings" of the protected media RTP packets 🌈
the following procedure MAY be followed for the protection operation 💜 other procedures MAY be used, but the end result MUST be identical to the one described here 🫶
8.1. Generation of the FEC Header ✨
okay bestie, for the FEC header we make 80-bit protected bit strings for each media packet that's getting FEC level 0 protection 💖 You glue two things together, in this exact order:
- The first 64 bits of the RTP header (64 bits)
- An unsigned, network-ordered 16-bit number for the media packet length in bytes minus 12 (for the fixed RTP header). That's the sum of the lengths of everything after the fixed header, if it's there: the CSRC list, extension header, RTP payload, and RTP padding (16 bits)
Once the FEC bit string is formed by applying the parity operation to those protected bit strings, you build the FEC header from it, and it's giving very precise bit-slicing 🔥
- Skip the first (most significant) 2 bits of the FEC bit string.
- The next bit goes into the
Precovery bit of the FEC header in the FEC packet. - The next bit goes into the
Erecovery bit of the FEC header. - The next 4 bits go into the
CCrecovery field of the FEC header. - The next bit goes into the
Mrecovery bit of the FEC header. - The next 7 bits go into the
PTrecovery field in the FEC header. - Skip the next 16 bits.
- The next 32 bits go into the
TSrecovery field in the FEC header. - The next 16 bits go into the length recovery field in the packet header.
8.2. Generation of the FEC Payload 🧁
For the FEC payload, the protected bit strings are literally just the protected RTP packets 🫶 So the FEC bit string is the bitwise exclusive OR of those protected media RTP packets. You need to generate these FEC bit strings for each level, because the group of protected payload packets can be different at each level. If the protected RTP packets aren't all the same length, each shorter packet MUST be padded to the length of the longest packet by adding octet 0 at the end.
For protection level n (n = 0, 1, ...), only Ln octets of data get set as the FEC level n payload data after the level n ULP header. The data is the Ln octets starting with the (Sn + 13)th octet in the FEC bit string, where:
Sn = sum(Li : 0 <= i < n)Li is the protection length of level i, and S0 is defined to be 0. Why skip the first 12 octets? Because that info is already protected by the FEC header, so no need to repeat it 🙈
9. Recovery Procedures 💅
The FEC packets let end systems recover from the loss of media packets, and this section walks through how that recovery works ✨
Recovery is two separate operations. The first one figures out which packets (media and FEC) must be combined to recover a missing packet. Once that's sorted, the second step is actually rebuilding the data. The second step MUST be performed as described below. The first step MAY be based on any algorithm the implementer picks. Different algorithms trade off complexity against how well they can recover missing packets, when recovery is possible at all 🧠
The lost payload packets may be recovered in full or in parts, depending on the data-loss situation, because of unequal error protection (when it's used). You can detect partial recovery by checking the recovery length of the packet pulled from the FEC header against the actual length of the recovered payload data 🦋
9.1. Reconstruction of the RTP Header 🎀
Let T be the list of packets (FEC and media) that can be combined to recover some media packet xi at level 0. Here's the procedure:
- For the media packets in T, compute the first 80 bits of the protected bit string, following the procedure described for generating the FEC header in the previous section.
- For the FEC packet in T, the FEC bit string is the 80-bit FEC header.
- Calculate the recovery bit string as the bitwise exclusive OR of the protected bit string generated from all the media packets in T and the FEC bit string generated from all the FEC packets in T.
- Create a new packet with the standard 12-byte RTP header and no payload.
- Set the version of the new packet to 2. Skip the first 2 bits in the recovery bit string.
- Set the Padding bit in the new packet to the next bit in the recovery bit string.
- Set the Extension bit in the new packet to the next bit in the recovery bit string.
- Set the CC field to the next 4 bits in the recovery bit string.
- Set the marker bit in the new packet to the next bit in the recovery bit string.
- Set the payload type in the new packet to the next 7 bits in the recovery bit string.
- Set the SN field in the new packet to xi. Skip the next 16 bits in the recovery bit string.
- Set the TS field in the new packet to the next 32 bits in the recovery bit string.
- Take the next 16 bits of the recovery bit string. Whatever unsigned integer those represent (assuming network order), take that many bytes from the recovery bit string and append them to the new packet. These represent the CSRC list, extension, payload, and the padding of the RTP payload.
- Set the SSRC of the new packet to the SSRC of the media stream it's protecting, i.e., the SSRC of the media stream to which the FEC stream is associated.
That's it, this procedure recovers the RTP header up to the SSRC field 💜
9.2. Reconstruction of the RTP Payload 🌸
Let T be the list of packets (FEC and media) that can be combined to recover some media packet xi at a certain protection level. The procedure goes like this:
- Assume we're reconstructing the data for level n. The first step is to get the protection length of level n (Ln) from the ULP header of level n.
- For the FEC packets in T, the FEC bit string of level n is the FEC level n payload, i.e., the Ln octets of data following the ULP header of level n.
- For the media packets in T, the protected bit string of level n is the Ln octets of data starting with the (Sn + 13)th octet of the packet. Sn is the same as defined in Section 8.2. Note that the protection of level 0 starts from the 13th octet of the media packet after the SSRC field. The information in the first 12 octets is protected by the FEC header.
- If any of the protected bit strings of level n generated from the media packets are shorter than the protection length of the current level, pad them to that length. The padding of octet 0 MUST be added at the end of the bit string.
- Calculate the recovery bit string as the bitwise exclusive OR of the protected bit string of level n generated from all the media packets in T and the FEC bit string of level n generated from all the FEC packets in T.
- The recovery bit string of the current protection level, as generated above, is combined through concatenation with the recovery bit strings of all the other levels to form the (fully or partially) recovered media packet. Note that the recovery bit string of each protection level MUST be placed at the correct location in the recovered media packet for that level, based on the protection length settings.
- The total length of the recovered media packet is recovered from the recovery operation at protection level 0 of the recovered media packet. This info can be used to check whether the complete recovery operation (of all levels) has recovered the packet to its full length 🔥
The data protected at a lower protection level is recoverable in most cases if the higher-level protected data is recoverable. This procedure, together with the procedure for the lower protection levels, will usually recover both the header and payload of an RTP packet up to the protection length of the current level ✨
10. Examples 🍓
In the first two examples below (Sections 10.1 and 10.2), we assume the FEC streams are sent through a separate RTP session, as described in Section 14.1. For these examples, four media packets are going to be sent, A, B, C, and D, all from SSRC 2. Their sequence numbers are 8, 9, 10, and 11, and their timestamps are 3, 5, 7, and 9. Packets A and C use payload type 11, and packets B and D use payload type 18. Packet A has 200 bytes of payload, packet B has 140, packet C has 100, and packet D has 340. Packets A and C have their marker bit set 💖
The third example (Section 10.3) shows what happens when the FEC data is sent as redundant data alongside the payload packets 🧚
10.1. An Example Offers Similar Protection as RFC 2733
okay so 10.1 is giving "one FEC packet to rule them all" 💅 We can protect the four payload packets to their full length in one single level with one FEC packet. This offers similar protection as RFC 2733. The scheme is as shown in Figure 6.
+-------------------+ :
Packet A | | :
+-------------+-----+ :
Packet B | | :
+---------+---+ :
Packet C | | :
+---------+-----------------------+
Packet D | |
+---------------------------------+
:
+---------------------------------+
Packet FEC | |
+---------------------------------+
: :
:<------------- L0 -------------->:Figure 6: FEC Scheme with Single-Level Protection
An FEC packet is generated from these four packets. We assume that payload type 127 is used to indicate an FEC packet. The resulting RTP header is shown in Figure 7. 🦋
The FEC header in the FEC packet is shown in Figure 8. 🎀
The FEC level header for level 0 is shown in Figure 9. ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|0|1 1 1 1 1 1 1|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Version: 2
Padding: 0
Extension: 0
Marker: 0
PT: 127
SN: 1
TS: 9
SSRC: 2Figure 7: RTP Header of FEC Packet
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0|0|0|0 0 0 0|0|0 0 0 0 0 0 0|0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 1 0 1 1 1 0 1 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
E: 0 [this specification]
L: 0 [short 16-bit mask]
P rec.: 0 [0 XOR 0 XOR 0 XOR 0]
X rec.: 0 [0 XOR 0 XOR 0 XOR 0]
CC rec.: 0 [0 XOR 0 XOR 0 XOR 0]
M rec.: 0 [1 XOR 0 XOR 1 XOR 0]
PT rec.: 0 [11 XOR 18 XOR 11 XOR 18]
SN base: 8 [min(8,9,10,11)]
TS rec.: 8 [3 XOR 5 XOR 7 XOR 9]
len. rec.: 372 [200 XOR 140 XOR 100 XOR 340]Figure 8: FEC Header of FEC Packet
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 0|1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
L0: 340 [the longest of 200, 140, 100, and 340]
mask: 61440 [with Bits 1, 2, 3, and 4 marked accordingly for
Packets 8, 9, 10, and 11]
The payload length for level 0 is 340 bytes.Figure 9: FEC Level Header (Level 0)
10.2. An Example with Two Protection Levels
okay this one is next level 🔥 A more complex example is to use FEC at two levels. The level 0 FEC will provide greater protection to the beginning part of the payload packets. The level 1 FEC will apply additional protection to the rest of the packets. This is illustrated in Figure 10. In this example, L0 = 70 and L1 = 90. 🌈
+------:--------:---+
Packet A | : : |
+------:------+-:---+
Packet B | : | :
+------:--+---+ :
: :
+------+ :
ULP #1 | | :
+------+ :
: :
+------:--+ :
Packet C | : | :
+------:--+-----:-----------------+
Packet D | : : |
+------:--------:-----------------+
: :
+------:--------+
ULP #2 | : |
+------:--------+
: : :
:<-L0->:<--L1-->:Figure 10: ULP FEC Scheme with Protection Level 0 and Level 1
This will result in two FEC packets - #1 and #2. 🫶
The resulting ULP FEC packet #1 will have the RTP header as shown in Figure 11. The FEC header for ULP FEC packet #1 will be as shown in Figure 12. The level 0 ULP header for #1 will be as shown in Figure 13. 💜
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|1|1 1 1 1 1 1 1|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Version: 2
Padding: 0
Extension: 0
Marker: 1
PT: 127
SN: 1
TS: 5
SSRC: 2Figure 11: RTP Header of FEC Packet #1
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0|0|0|0 0 0 0|0|0 0 1 1 0 0 1|0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
E: 0 [this specification]
L: 0 [short 16-bit mask]
P rec.: 0 [0 XOR 0 XOR 0 XOR 0]
X rec.: 0 [0 XOR 0 XOR 0 XOR 0]
CC rec.: 0 [0 XOR 0 XOR 0 XOR 0]
M rec.: 0 [1 XOR 0 XOR 1 XOR 0]
PT rec.: 25 [11 XOR 18]
SN base: 8 [min(8,9)]
TS rec.: 6 [3 XOR 5]
len. rec.: 68 [200 XOR 140]Figure 12: FEC Header of ULP FEC Packet #1
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0|1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
L0: 70
mask: 49152 [with Bits 1 and 2 marked accordingly for
Packets 8 and 9]
The payload length for level 0 is 70 bytes.Figure 13: FEC Level Header (Level 0) for FEC Packet #1
The resulting FEC packet #2 will have the RTP header as shown in Figure 14. The FEC header for FEC packet #2 will be as shown in Figure 15. The level 0 ULP header for #2 will be as shown in Figure 16. The level 1 ULP header for #2 will be as shown in Figure 17. 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|1|1 1 1 1 1 1 1|0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Version: 2
Padding: 0
Extension: 0
Marker: 1
PT: 127
SN: 2
TS: 9
SSRC: 2Figure 14: RTP Header of FEC Packet #2
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0|0|0|0|0 0 0 0|0|0 0 1 1 0 0 1|0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
E: 0 [this specification]
L: 0 [short 16-bit mask]
P rec.: 0 [0 XOR 0 XOR 0 XOR 0]
X rec.: 0 [0 XOR 0 XOR 0 XOR 0]
CC rec.: 0 [0 XOR 0 XOR 0 XOR 0]
M rec.: 0 [1 XOR 0 XOR 1 XOR 0]
PT rec.: 25 [11 XOR 18]
SN base: 8 [min(8,9,10,11)]
TS rec.: 14 [7 XOR 9]
len. rec.: 304 [100 XOR 340]Figure 15: FEC Header of FEC Packet #2
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0|0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
L0: 70
mask: 12288 [with Bits 3 and 4 marked accordingly for
Packets 10 and 11]
The payload length for level 0 is 70 bytes.Figure 16: FEC Level Header (Level 0) for FEC Packet #2
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 0|1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
L1: 90
mask: 61440 [with Bits 1, 2, 3, and 4 marked accordingly for
Packets 8, 9, 10, and 11]
The payload length for level 1 is 90 bytes.Figure 17: FEC Level Header (Level 1) for FEC Packet #2
10.3. An Example with FEC as Redundant Coding ✨
okay so this example shows FEC getting sent as redundant coding in the same stream as the payload 🎀 bestie, this is the one that made me go "oh wait, that's clever." We're pretending five media packets, A, B, C, D, and E, are coming from SSRC 2. Their sequence numbers are 8, 9, 10, 11, and 12, and their timestamps are 3, 5, 7, 9, and 11 🦋 All the media data is coded with primary coding (FEC as redundant coding only protects the primary coding, so that's the rule here) and uses payload type 11. Packet A has 200 bytes of payload, B has 140, C has 100, D has 340, and E has 160. Packets A and C have their marker bit set 💅
The FEC scheme here is one level, like Figure 6 in Section 10.1. The protection length L0 = 340 octets 🌸
We use a redundant coding packetization with payload type 100, and the FEC's payload type is assumed to be 127. The first four RED packets, RED #1 through RED #4, each carry one individual media packet: A, B, C, or D, in that order. The FEC data protecting the media in those first four packets gets generated. Then the fifth packet, RED #5, carries that FEC data as redundant coding along with media packet E. Iconic layout 👑
RED Packet #1: Media Packet A RED Packet #2: Media Packet B RED Packet #3: Media Packet C RED Packet #4: Media Packet D RED Packet #5: FEC Packet, Media Packet E
RED packets #1 through #4 are structured like Figure 18 🧚 The RTP header of RED packet #1 is shown in Figure 19, and the other RED packets follow the same format with their own matching sequence numbers and timestamps. The primary encoding block header for those RED packets is in Figure 20 ✨
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header (RED) - 6 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Primary Encoding Block Header (RED) - 1 octet |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Media Packet Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 18: RED Packet Structure - Media Data Only 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1 0|0|0|0 0 0 0|0|1 1 0 0 1 0 0|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Version: 2
Padding: 0
Extension: 0
Marker: 0 [Even though media packet A has marker set]
PT: 100 [Payload type for RED]
SN: 1
TS: 5
SSRC: 2
Figure 19: RTP Header of RED Packet #1 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|0|0 0 0 1 0 1 1|
+-+-+-+-+-+-+-+-+
F bit: 0 [This is the primary coding data]
Block PT: 11 [The payload type of media]
Figure 20: Primary Encoding Block HeaderThe FEC data isn't generated straight from the RED packets. It comes from the virtual RTP packets that hold the media packet data 🫶 Those virtual packets are super easy to make from the RED packets, whether or not they have redundant coding included. You just do two things: (1) remove any RED block headers and redundant coding data, and (2) swap the PT in the RTP header for the PT of the primary coding. Easy peasy 🍓
Note: In the payload format for redundant coding as specified by RFC 2198, the marker bit is lost as soon as the primary coding is carried in the RED packets. So the marker bit cannot be recovered regardless of whether or not the FEC is used. 💀
Back to RED packet #5. As mentioned, it carries the FEC data (protecting media packets A, B, C, and D) and the data of media packet E. Its structure is shown in Figure 21 😭
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header (RED) - 6 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Redundant Encoding Block Header (RED) - 4 octets |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Packet Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Primary Encoding Block Header (RED) - 1 octet |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Media Packet Data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 21: RED Packet Structure - With FEC DataThe RTP header of RED packets with FEC included is the same as Figure 19, with their matching sequence numbers and timestamps 🌈
In RED packet #5, the redundant encoding block header for the FEC packet data block looks like Figure 22. It's followed by the FEC packet data, which in this case is an FEC header (10 octets, as in Figure 8), a ULP level 0 header (4 octets, as in Figure 9), and the ULP level 0 data (340 octets, as set for level 0). After all that comes the primary encoding block carrying the data of media packet E 🧚♀️
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1|1 1 1 1 1 1 1|0 0 0 0 0 0 0 0 0 0 0 0 0 0|0 1 0 1 1 0 0 0 1 0|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
F bit: 1 [This is the redundant coding data]
Block PT: 127 [The dynamic payload type for FEC]
TS Offset: 0 [The instance at which the FEC data is
transmitted]
Block Len: 354 [FEC header (10 octets) plus ULP level 0 header
(4 octets) and ULP level 0 data (340 octets)]
Figure 22: Redundant Encoding Block Header11. Security Considerations 🔐
okay real talk, security is where things get serious 😤 There are two ways to use FEC with encryption in secure communications. One is applying FEC to payloads that are already encrypted. The other is applying FEC before encryption. The first case shows up when a node that is not trusted needs FEC during transmission, after the media data has been encrypted. The second case shows up when media data gets protected by FEC before it's sent through a secured transport 🔒
Since the protected payload of this FEC is RTP packets, applying FEC to encrypted payloads mostly matters for secure RTP (SRTP) [13]. Because the FEC applies XOR across the payload, the FEC packets should be cryptographically as secure as the original payload. In that case, you don't need to encrypt the FEC packets again 💜
For the rest of this discussion, we assume FEC is applied to the payload before encryption. Using FEC affects how you use and change keys for encryption. The FEC packets are a separate stream, so there are a bunch of ways to combine encryption:
- The FEC stream may be encrypted, while the media stream is not.
- The media stream may be encrypted, while the FEC stream is not.
- The media stream and FEC stream are both encrypted, but using the same key.
- The media stream and FEC stream are both encrypted, but using different keys.
The first three options would need every application-level signaling protocol in use to know about FEC, so they can exchange keys and negotiate encryption on the media and FEC streams separately. The last option needs none of those extra mechanisms. The first two options are a layering violation, since ULP FEC packets should be treated no differently than other RTP packets. Encrypting just one stream can also make certain known-plaintext attacks possible 🙈 So for all these reasons, applications using encryption SHOULD encrypt both streams, which means the last two options 💅
It gets even spicier. The known relationship between the media payload and the FEC data can weaken encryption for certain ciphers, so different encryption keys MUST be used for each stream when the media payload and the FEC data are sent in separate streams 😱 Note that when SRTP [13] is used to secure the RTP sessions, the SRTP specification requires different keys for each RTP session.
Changing encryption keys is another big deal 🗝️ Picture this: two packets, a and b, are sent along with the FEC packet that protects them. The keys used to encrypt a and b are different, so which key should decode the FEC packet? In general, old keys need to be cached, so that when the keys change for the media stream, the old key can still be used until it's known that the key has changed for the ULP FEC packets too. Also, the new key SHOULD be used to encrypt the FEC packets that are generated from a mix of payload packets encrypted with the old and new keys. The sender and the receiver need to define how encryption is performed and how the keys are used 👑
Altering the FEC data and packets can seriously mess with the reconstruction operation 💀 An attack that changes some bits in the FEC data can have a big effect on the calculation and recovery of the payload packets. For example, changing the length recovery field can result in a packet that's recovered too long. The computational complexity of recovery can also easily be affected by up to at least one order of magnitude. Depending on the application scenario, it may help to sanity check the received payload and FEC data before running the recovery operation, and to check whether the recovered data is valid before using it 🛡️
12. Congestion Considerations ✨
okay so congestion is a HUGE deal with FEC 😭 a lot of the time the packet loss is caused by congestion in the first place, so piling on FEC when losses are already climbing is a big no-no 🙅♀️ if everyone does it on a wide scale, it can make congestion worse and eventually lead to congestion collapse 💀 apps can add stronger protection while shrinking the bandwidth for the actual payload packets, which is a valid trade-off. BUT no matter what, implementations MUST NOT substantially increase the total bandwidth in use (payload and FEC) as network losses go up 🫶
the general congestion control stuff for RTP data still applies, so check out RTP [1] and whichever RTP profile you're using (e.g., RTP/AVP [14]) 🦋 there's also an extra requirement if you're on best-effort service: users of this payload format MUST monitor packet loss to make sure it stays within acceptable parameters 👀 loss counts as acceptable if a TCP flow on the same network path, under the same conditions, would get an average throughput (measured over a reasonable timescale) that's not less than what your RTP flow is getting. you can keep things in check by running congestion control to adapt your transmission rate (or the number of layers you're subscribed to in a layered multicast session), or by having a receiver leave the session if the loss rate gets unacceptably high 🌸
13. IANA Considerations 💅
four new media subtypes have been registered with IANA, as described in this section 🎀 registration is done using the registration template [3] and following RFC 3555 [4] 💖
13.1. Registration of audio/ulpfec
Type name: audio
Subtype name: ulpfec
Required parameters:
rate: The RTP timestamp rate that is used to mark the time of
transmission of the FEC packet in a separate stream. In cases in
which it is sent as redundant data to another stream, the rate
SHALL be the same as the primary encoding it is used to protect.
When used in a separate stream, the rate SHALL be larger than 1000
Hz, to provide sufficient resolution to RTCP operations. The
selected rate MAY be any value above 1000 Hz but is RECOMMENDED to
match the rate of the media this stream protects.
Optional parameters:
onelevelonly: This specifies whether only one level of FEC protection
is used. The permissible values are 0 and 1. If 1 is signaled,
only one level of FEC protection SHALL be used in the stream. If
0 is signaled, more than one level of FEC protection MAY be used.
If omitted, it has the default value of 0.
Encoding considerations: This format is framed (see Section 4.8 in
the template document [3]) and contains binary data.
Security considerations: The same security considerations apply to
these media type registrations as to the payloads for them, as
detailed in RFC 5109.
Interoperability considerations: none
Published specification: RFC 5109
Applications that use this media type: Multimedia applications that
seek to improve resiliency to loss by sending additional data with
the media stream.
Additional information: none
Person & email address to contact for further information:
Adam Li adamli@hyervision.com
IETF Audio/Video Transport Working Group
Intended usage: COMMON
Restrictions on usage: This media, type depends on RTP framing, and
hence is only defined for transfer via RTP [1]. Transport within
other framing protocols SHALL NOT be defined as this is a robustness
mechanism for RTP.
Author:
Adam Li adamli@hyervision.com
Change controller:
IETF Audio/Video Transport Working Group delegated from the IESG.13.2. Registration of video/ulpfec
Type name: video
Subtype name: ulpfec
Required parameters:
rate: The RTP timestamp rate that is used to mark the time of
transmission of the FEC packet in a separate stream. In cases in
which it is sent as redundant data to another stream, the rate
SHALL be the same as the primary encoding it is used to protect.
When used in a separate stream, the rate SHALL be larger than 1000
Hz to provide sufficient resolution to RTCP operations. The
selected rate MAY be any value above 1000 Hz, but is RECOMMENDED
to match the rate of the media this stream protects.
Optional parameters:
onelevelonly: This specifies whether only one level of FEC protection
is used. The permissible values are 0 and 1. If 1 is signaled,
only one level of FEC protection SHALL be used in the stream. If
0 is signaled, more than one level of FEC protection MAY be used.
If omitted, it has the default value of 0.
Encoding considerations: This format is framed (see Section 4.8 in
the template document [3]) and contains binary data.
Security considerations: The same security considerations apply to
these media type registrations as to the payloads for them, as
detailed in RFC 5109.
Interoperability considerations: none
Published specification: RFC 5109
Applications that use this media type: Multimedia applications that
seek to improve resiliency to loss by sending additional data with
the media stream.
Additional information: none
Person & email address to contact for further information:
Adam Li adamli@hyervision.com
IETF Audio/Video Transport Working Group
Intended usage: COMMON
Restrictions on usage: This media type depends on RTP framing, and
hence is only defined for transfer via RTP [1]. Transport within
other framing protocols SHALL NOT be defined as this is a robustness
mechanism for RTP.
Author:
Adam Li adamli@hyervision.com
Change controller: IETF Audio/Video Transport Working Group
delegated from the IESG.13.3. Registration of text/ulpfec
Type name: text
Subtype name: ulpfec
Required parameters:
rate: The RTP timestamp rate that is used to mark the time of
transmission of the FEC packet in a separate stream. In cases in
which it is sent as redundant data to another stream, the rate
SHALL be the same as the primary encoding it is used to protect.
When used in a separate stream, the rate SHALL be larger than 1000
Hz to provide sufficient resolution to RTCP operations. The
selected rate MAY be any value above 1000 Hz, but is RECOMMENDED
to match the rate of the media this stream protects.
Optional parameters:
onelevelonly: This specifies whether only one level of FEC protection
is used. The permissible values are 0 and 1. If 1 is signaled,
only one level of FEC protection SHALL be used in the stream. If
0 is signaled, more than one level of FEC protection MAY be used.
If omitted, it has the default value of 0.
Encoding considerations: This format is framed (see Section 4.8 in
the template document [3]) and contains binary data.
Security considerations: The same security considerations apply to
these media type registrations as to the payloads for them, as
detailed in RFC 5109.
Interoperability considerations: none
Published specification: RFC 5109
Applications that use this media type: Multimedia applications that
seek to improve resiliency to loss by sending additional data with
the media stream.
Additional information: none
Person & email address to contact for further information:
Adam Li adamli@hyervision.com
IETF Audio/Video Transport Working Group
Intended usage: COMMON
Restrictions on usage: This media type depends on RTP framing, and
hence is only defined for transfer via RTP [1]. Transport within
other framing protocols SHALL NOT be defined as this is a robustness
mechanism for RTP.
Author:
Adam Li adamli@hyervision.com
Change controller:
IETF Audio/Video Transport Working Group delegated from the IESG.13.4. Registration of application/ulpfec
Type name: application
Subtype name: ulpfec
Required parameters:
rate: The RTP timestamp rate that is used to mark the time of
transmission of the FEC packet in a separate stream. In cases in
which it is sent as redundant data to another stream, the rate
SHALL be the same as the primary encoding it is used to protect.
When used in a separate stream, the rate SHALL be larger than 1000
Hz to provide sufficient resolution to RTCP operations. The
selected rate MAY be any value above 1000 Hz, but is RECOMMENDED
to match the rate of the media this stream protects.
Optional parameters:
onelevelonly: This specifies whether only one level of FEC protection
is used. The permissible values are 0 and 1. If 1 is signaled,
only one level of FEC protection SHALL be used in the stream. If
0 is signaled, more than one level of FEC protection MAY be used.
If omitted, it has the default value of 0.
Encoding considerations: This format is framed (see Section 4.8 in
the template document [3]) and contains binary data.
Security considerations: The same security considerations apply to
these media type registrations as to the payloads for them, as
detailed in RFC 5109.
Interoperability considerations: none
Published specification: RFC 5109
Applications that use this media type: Multimedia applications that
seek to improve resiliency to loss by sending additional data with
the media stream.
Additional information: none
Person & email address to contact for further information:
Adam Li adamli@hyervision.com
IETF Audio/Video Transport Working Group
Intended usage: COMMON
Restrictions on usage: This media type depends on RTP framing, and
hence is only defined for transfer via RTP [1]. Transport within
other framing protocols SHALL NOT be defined as this is a robustness
mechanism for RTP.
Author:
Adam Li adamli@hyervision.com
Change controller:
IETF Audio/Video Transport Working Group delegated from the IESG.14. Multiplexing of FEC 🎀
so the FEC packets can reach the receiver alongside the protected payload in one of two main ways 🧚 either as a separate stream, or in the same stream as redundant encoding 🍓 the configuration options MUST be indicated out of band 🔥 this section also explains how you can do that using the Session Description Protocol (SDP), which is specified in RFC 2327 [8] ✨
14.1. FEC as a Separate Stream 🦋
Okay so when the FEC packets get sent in their own separate stream, three things need to be conveyed, bestie 💅
- The address and port to which the FEC is being sent
- The payload type number for the FEC
- Which media stream the FEC is protecting
There is no static payload type assignment for FEC, so dynamic payload type numbers MUST be used. 🎀 The SSRC of the FEC stream MUST be set to that of the protected payload stream. The association of the FEC stream with its corresponding stream is done by line grouping in SDP [5] with the FEC semantics [6] or other external means. ✨
Following the principles from Section 5.2 of RFC 3550 [1], multiplexing the FEC stream and its associated payload stream is usually handled by the destination transport address (network address and port number), which is different for each RTP session. 🌸 Sending FEC together with the payload in one single RTP session, and multiplexing only by SSRC or payload type, is a no-go because it precludes: (1) using different network paths or network resource allocations for the payload and the FEC protection data; (2) receiving just a subset of the media if desired, especially for hosts that don't understand FEC; and (3) receiver implementations that use separate processes for the different media. 😭
Also, multiplexing FEC with payload data streams will affect the timing and sequence number space of the original payload stream, which is usually undesirable. 🙈 So the FEC stream and the payload stream SHOULD be sent through two separate RTP sessions, and multiplexing them by payload type into one single RTP session SHOULD be avoided. 💜 Even more, the FEC and the payload MUST NOT be multiplexed by SSRC into one single RTP session, since they always have the same SSRC. 🚫
Just like any media stream, the port number and the payload type number for the FEC stream are conveyed in their m line in the SDP. 🧚 There is no static payload type assignment for FEC, so dynamic payload type numbers MUST be used. The binding to the number is indicated by an rtpmap attribute, and the name used in that binding is ulpfec. The address that the FEC stream is on is conveyed in its corresponding c line. 🍓
The association between the FEC stream and the payload stream it protects is conveyed through media line grouping in SDP (RFC 3388) [5] using FEC semantics (RFC 4756) [6]. The FEC stream and the protected payload stream form an FEC group. 👑
Here's an example SDP for FEC application in a multicast session:
v=0
o=adam 289083124 289083124 IN IP4 host.example.com
s=ULP FEC Seminar
t=0 0
c=IN IP4 224.2.17.12/127
a=group:FEC 1 2
a=group:FEC 3 4
m=audio 30000 RTP/AVP 0
a=mid:1
m=application 30002 RTP/AVP 100
a=rtpmap:100 ulpfec/8000
a=mid:2
m=video 30004 RTP/AVP 31
a=mid:3
m=application 30004 RTP/AVP 101
c=IN IP4 224.2.17.13/127
a=rtpmap:101 ulpfec/8000
a=mid:4The two a=group lines in this SDP mean there are two FEC groups. 🌈 The first one, a=group:FEC 1 2, is made of stream 1 (an audio stream using PCM [14]) and stream 2 (the protecting FEC stream). The FEC stream goes to the same multicast group and has the same Time to Live (TTL) as the audio, but on a port number two higher. 🎧
The second group, a=group:FEC 3 4, is made of stream 3 (a video stream) and stream 4 (the protecting FEC stream). This FEC stream goes to a different multicast address, but it has the same port number (30004) as the payload video stream. 😍
14.2. FEC as Redundant Encoding ✨
When the FEC stream is sent as a secondary codec in the redundant encoding format, this has to be signaled through SDP. 🎀 To do this, the procedures defined in RFC 2198 [7] are used to signal the use of redundant encoding. The FEC payload type is indicated the same way as any other secondary codec. 💖
The FEC MUST protect only the main codec, and the payload of the FEC engine comes from virtual RTP packets created from the main codec data. 🦋 The virtual RTP packets are super easy to make from RFC 2198 packets by simply (1) removing all the additional headers and the redundant coding data, and (2) replacing the payload type in the RTP header with that of the primary codec. ✨
Note: In the payload format for redundant coding as specified by RFC 2198, the marker bit is lost as soon as the primary coding is carried in the RED packets. So the marker bit cannot be recovered regardless of whether or not the FEC is used.
Because the FEC data (including the ULP header) is sent in the same packets as the protected payload, the FEC data is associated with the protected payload by being bundled in the same stream. 🫶
When the FEC stream is sent as a secondary codec in the redundant encoding format, this can be signaled through SDP too. 💅 Again, the procedures defined in RFC 2198 [7] are used to signal the use of redundant encoding, and the FEC payload type is indicated the same way as any other secondary codec. An rtpmap attribute MUST be used to indicate a dynamic payload type number for the FEC packets. The FEC MUST protect only the main codec. 🔥
For example:
m=audio 12345 RTP/AVP 121 0 5 100
a=rtpmap:121 red/8000/1
a=rtpmap:100 ulpfec/8000
a=fmtp:121 0/5/100This SDP says there's a single audio stream, which can consist of PCM (media format 0), DVI (media format 5), the redundant encodings (media format 121, which is bound to red through the rtpmap attribute), or FEC (media format 100, which is bound to ulpfec through the rtpmap attribute). 🌸 Even though the FEC format is listed as a possible coding for this stream, the FEC MUST NOT be sent by itself for this stream. 🙅 Its presence in the m line is only required because non-primary codecs must be listed here according to RFC 2198. The fmtp attribute says the redundant encodings format can be used, with DVI as a secondary coding and FEC as a tertiary encoding. 🧁
14.3. Offer / Answer Consideration 💌
Some considerations are needed when SDP is used for offer / answer [15] exchange. 🫧
The "onelevelonly" parameter is declarative. For streams declared as sendonly, the value indicates whether only one level of FEC will be sent. For streams declared as recvonly or sendrecv, the value indicates what the receiver accepts to receive. 👀
When the FEC is sent as a separate stream and signaled through media line grouping in SDP (RFC 3388) [5] using FEC semantics (RFC 4756) [6], the offering side MUST implement both RFC 3388 and RFC 4756. 🎀 The rules for offer / answer in RFC 3388 and RFC 4756 SHALL be followed, with the additional considerations below. For all offers with FEC, the answerer MAY refuse the separate FEC session by setting the port to 0 and removing the "a=group" attribute that groups that FEC session with the RTP session being protected. ✂️ If the answerer accepts the use of FEC, it simply accepts the FEC RTP session and the grouping in the offer by including the same grouping in the answer. ✨ Note that rejecting the FEC RTP session does not stop the media sessions from being accepted and used without FEC. 🦋
When the FEC stream is sent as a secondary codec in the redundant encoding format (RFC 2198) [7], the offering side can indicate the FEC stream as specified in Section 14.2. The answerer MAY reject the FEC stream by removing the payload type for the FEC stream. To accept the use of FEC, the answerer must include the FEC payload type in the answer. 🌈 Note that in cases where the redundancy payload format [7] is used with FEC as the only secondary codec, when the FEC stream is rejected, the redundant encoding payload type SHOULD also be removed. 🧹
15. Application Statement ✨
okay so bestie, this section is basically the big "why should I care" moment for ULP 💅 The doc describes a generic protocol for Forward Error Correction that supports a wide range of short block parity FEC algorithms, like simple and interleaved parity codes 🦋 The scheme is limited to interleaving parity codes over a distance of 48 packets, so there's a hard ceiling there 🎀 And the best part? This FEC algorithm is fully compatible with hosts that are not FEC-capable 🫶 The media payload is never altered and the protection is sent as extra information, so receivers that don't know about this generic FEC can just ignore the additional FEC info and process the main media payload like normal 🌸 This interop thing is super important for existing hosts, and also when a bunch of different hosts need to talk to each other at the same time, like during multicast 🧚
The generic FEC algorithm here is also a generic protection algorithm with some seriously iconic features 👑 (1) it doesn't care what kind of media it's protecting, whether that's audio, video, or something else 💜 (2) it's flexible enough to support a wide variety of FEC mechanisms and settings 🌈 (3) it's built for adaptivity, so the FEC parameters can change easily without needing out-of-band signaling 🍓 and (4) it supports a number of different mechanisms for transporting the FEC packets ⭐
The FEC in this spec also gives the user Unequal Error Protection capabilities 🔥 Some other algorithms can do that too, just through other means 💅 For example, an Unequal Erasure Protection (UXP) scheme was proposed in the AVT Working Group in "An RTP Payload Format for Erasure-Resilient Transmission of Progressive Multimedia Streams" 🦋 UXP applies unequal error protection to the media payloads by interleaving the payload stream being protected with extra redundancy info made using Reed-Solomon operations 🌸
ngl this is where things get spicy 🌶️ Because UXP changes the structure of the protected media payload, it gives up backward compatibility with terminals that don't support UXP 💀 So it's way harder to use UXP when you need backward compatibility 😭 With ULP, though, the media payload stays unaltered and terminals can always use it, and if the receiving terminals don't support ULP, they can just ignore the extra protection 🫶
At the same time, because UXP alters the structure of the media payload, it has the unique ability to change packet size independent of the original media payload structure and protection applied 🧚 The only limit is the protocol overhead constraint 🎀 This is handy when you want to change the packet size of the media at the transport level 💖
Because UXP is interleaved, delays show up on both the encoding and decoding sides ⏳ For UXP, all data within a transmission block has to arrive before encoding can begin, and a reasonable number of packets must be received before a transmission block can be decoded 😩 ULP introduces little delay on the encoding side 💜 On the decoding side, correctly received packets can be delivered immediately ✨ Delay only shows up in ULP when packet losses actually happen 🙈
Since UXP is an interleaved scheme, unrecoverable errors in UXP-protected data usually turn into a bunch of corrupted holes in the payload stream 🍓 In ULP, the unrecoverable errors from packet loss usually show up as contiguous missing pieces at the end of packets 🌈 Depending on how the media payload stream is encoded, lots of applications find it easier to parse and extract data from a packet with only one contiguous piece missing at the end than a packet with multiple corrupted holes 🫶 That's especially true when the holes don't line up with the independently decodable fragment boundaries 🦋
The exclusive-or (XOR) parity check operation used by ULP is simpler and faster than the more complex operations Reed-Solomon codes need 🔥 That makes ULP the better pick for applications where computational cost is a constraint 💅
As we've covered, both ULP and UXP apply unequal error protection to the RTP media stream, but each one uses a different technique 👑 Both schemes have their own unique characteristics, and each can fit different requirements 🌸 Choose your fighter, bestie 💖
16. Acknowledgments 🫶
shoutout to the authors who made significant contributions to this document: Adam H. Li, Fang Liu, John D. Villasenor, Dong-Seek Park, Jeong-Hoon Park, Yung-Lyul Lee, Jonathan D. Rosenberg, and Henning Schulzrinne 👑 The authors also want to thank the many people who gave suggestions, especially Stephen Casner, Jay Fahlen, Cullen Jennings, Colin Perkins, Tao Tian, Matthieu Tisserand, Jeffery Tseng, Mark Watson, Stephen Wenger, and Magnus Westerlund 💜 Iconic squad 🌈
17. References 📚
17.1. Normative References 💅
[1] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson,
"RTP: A Transport Protocol for Real-Time Applications", STD 64,
RFC 3550, July 2003.
[2] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC 2119, March 1997.
[3] Freed, N. and J. Klensin, "Media Type Specifications and
Registration Procedures", BCP 13, RFC 4288, December 2005.
[4] Casner, S., "Media Type Registration of RTP Payload Formats",
RFC 4855, February 2007.
[5] Camarillo, G., Eriksson, G., Holler, J., and H. Schulzrinne,
"Grouping of Media Lines in the Session Description Protocol
(SDP)", RFC 3388, December 2002.
[6] Li, A., "Forward Error Correction Grouping Semantics in Session
Description Protocol", RFC 4756, November 2006.
[7] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V., Handley, M.,
Bolot, J., Vega-Garcia, A., and S. Fosse-Parisis, "RTP Payload
for Redundant Audio Data", RFC 2198, September 1997.
[8] Handley, M., Jacobson, V., and C. Perkins, "SDP: Session
Description Protocol", RFC 4566, July 2006.17.2. Informative References 🦋
[9] Rosenberg, J. and H. Schulzrinne, "An RTP Payload Format for
Generic Forward Error Correction", RFC 2733, December 1999.
[10] Perkins, C. and O. Hodson, "Options for Repair of Streaming
Media", RFC 2354, June 1998.
[11] Rosenberg, J. and H. Schulzrinne, "Registration of parityfec
MIME types", RFC 3009, November 2000.
[12] Luby, M., Vicisano, L., Gemmell, J., Rizzo, L., Handley, M., and
J. Crowcroft, "Forward Error Correction (FEC) Building Block",
RFC 3452, December 2002.
[13] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K.
Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC
3711, March 2004.
[14] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video
Conferences with Minimal Control", STD 65, RFC 3551, July 2003.
[15] Rosenberg, J. and H. Schulzrinne, "An Offer/Answer Model with
Session Description Protocol (SDP)", RFC 3264, June 2002.Editor's Address 📬
Adam H. Li 10194 Wateridge Circle #152 San Diego, CA 92121 USA Phone: +1 858 622 9038 EMail: adamli@hyervision.com
Full Copyright Statement ⚖️
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🔒
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
📦 where this page comes from
- ietf: rfc5109 2007-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms