RFC 2354

Options for Repair of Streaming Media

📅 published June 1998, from ietf

RFC 2354: Options for Repair of Streaming Media 🎀

Bestie, this one is a whole vibe 💖 It's an Informational RFC from June 1998, so it's sharing ideas, not setting rules. ✨

  • Series: Network Working Group
  • RFC number: 2354
  • Category: Informational
  • Date: June 1998
  • Authors: C. Perkins and O. Hodson, University College London
  • Obsoletes / Updates: the header doesn't list anything it obsoletes or updates 🙈

Status of this Memo

Okay, this part is the legal bit, but it's short and sweet 🌸 This memo gives information to the Internet community. It is not an Internet standard of any kind, and distribution of it is unlimited.

The copyright belongs to The Internet Society, and the notice says Copyright (C) The Internet Society (1998). All rights are reserved 💜

Abstract

This doc is a rundown of ways to repair continuous media streams when packets go missing 🦋 It covers redundant transmission, retransmission, interleaving and forward error correction. It also notes where each technique fits, plus the protocol requirements and dependencies it brings along.

1. Introduction ✨

So a bunch of apps now use RTP over UDP to send continuous media like audio and video 🎧 UDP doesn't promise delivery, so when packets get lost, the stream quality takes a hit, and that's a whole mood 😭 There are several techniques that can patch up those losses. They have a wide range of uses, and each one needs a different amount of protocol support. This doc walks through a few of them and shares recommendations on where they make sense 💅

Quick heads up though: this one is introductory and doesn't try to cover everything 🌈 We're only looking at repair techniques that need the sender of the media stream to be involved. We are not talking about receiver-based repair, so no spoilers there 🙈

If you want a deeper dive, check out [5] 📚

2. Terminology and Protocol Framework 🧚

Let's get the vocab down first, okay? A unit is a timed chunk of media data, usually made by the media coder ⏱️ A packet holds one or more units, wrapped up so it can travel over the network 📦 For example, lots of audio coders work in 20ms units, and those often get bundled into 40ms or 80ms packets 🎶

This doc assumes the RTP framework from [18] 💖 That means every packet has a sequence number and a timestamp. The sequence number shows the order packets were sent in, and it's how you spot losses 🕵️‍♀️ The timestamp tells you the playout order of the units. Most loss recovery schemes rely on units being sent out of order, so an application must use the RTP timestamp to schedule playout. Non-negotiable, bestie 💅

RTP can handle a bunch of different media coders, and a payload type field tells the receiver which one is which 🎀 Some loss repair schemes send several copies of a unit, at different times and maybe with different encodings. That way the receiver is more likely to have something it can actually decode 🙌 The receiver is assumed to have a quality ranking for the different encodings. So when there are multiple options, it can pick the `best' unit for playout 👑

A session counts as interactive if its end-to-end delay is less than 250ms. That delay includes media coding and decoding, network transit and host buffering ⚡

3. Network Loss Characteristics 📊

Before you try to repair a media stream, it helps to know what kind of loss you're probably going to see 🔍 A bunch of studies have looked at loss on the Mbone [2, 8, 21]. The numbers differ a bit, but the big takeaway is clear: in a large conference, some receivers are going to lose packets. It's basically inevitable 😭

Handley's packet traces [8] show one session where most receivers saw loss somewhere in the 2-5% range. A smaller group saw much higher loss rates. Other studies came back with broadly similar results 💜

Here's the other big finding: most losses are single packets 💖 Bursts of two or more packets happen around an order of magnitude less often than single losses. They still happen more than a purely random process would predict. Longer bursts, around tens of packets, are pretty rare 🌸 All of this lines up with a network where small bursts of transient congestion cause most of the loss. Every now and then, a link gets seriously overloaded, and then you get a big chunk of loss 🌊

So the main job of a repair scheme must be fixing single packet loss, since that's by far the most common thing that happens 🎯 It's also nice if losses of a small number of consecutive packets can be repaired, because they're a small but noticeable slice of what gets lost ✨ Fixing big bursts of loss matters a lot less 🙈

4. Loss Mitigation Schemes 🦋

This section goes through four loss mitigation schemes 💅 People have discussed these in the literature many times, and they've turned out to be useful in a number of scenarios. For each one, we'll describe it briefly and note its advantages and disadvantages 📝

4.1. Retransmission 🔁

Retransmitting lost packets is the obvious way to fix loss, honestly 🙌 It's clearly valuable for non-interactive apps where the delay bounds are relaxed. But that delay means it usually doesn't work well for interactive use 😭

There's also a potentially big bandwidth cost. You're sending units more than once, and extra control traffic has to go out to request the retransmissions 📈 Studies on a large Mbone session showed that most packets are lost by at least one receiver [8]. In that situation, asking for retransmissions on most packets could cost so much that forward error correction is the more reasonable pick 🤔 That's where the two mechanisms become a power couple 👯‍♀️ FEC repairs all the single packet losses, and it also handles the burst losses for receivers who are okay with a bit of extra latency. Those receivers can use retransmission as an extra layer of recovery 💖 Similar ideas have shown up in a number of reliable multicast schemes and got some discussion in the literature [9, 13] 📚

To cut down on retransmission overhead, the retransmitted units can be piggy-backed onto the ongoing transmission, using a payload format like the one in [15] 🎁 That also lets the retransmission be recoded into a different format, which trims the bandwidth overhead even more ✨ Another option is sending FEC information in response to retransmission requests [13]. One retransmission can then potentially repair several losses at once 🙌

Picking a retransmission request algorithm that's both timely and network friendly is still an active area of study 🧪 An obvious place to start is the SRM protocol [7]. People have run experiments with it, and also with a low-delay variant called STORM [20]. The results show the trade-off between latency and quality for retransmission-based repair. They also show that retransmission works well for apps that can handle the latency 🫶

There isn't a standard protocol framework for requesting retransmission of streaming media yet 🥺 An experimental RTP profile extension for SRM-style retransmission requests has been described in [14] 🧚

4.2. Forward Error Correction 🛡️

Forward error correction (FEC) is how you add repair data to a media stream, so the receiver can fix packet loss without asking the sender anything else 💖 That's the magic part! There are two classes of repair data you can add. One kind doesn't depend on what's in the stream. The other kind uses knowledge of the stream to make the repair better 🤓

4.2.1. Media-Independent FEC 🧩

A bunch of media-independent FEC schemes have been proposed for streamed media 🌈 These techniques add redundant data to a media stream, and that redundant data goes out in separate packets. Traditionally, FEC techniques get described as loss detecting and/or loss correcting. For streamed media, loss detection comes from the sequence numbers in RTP packets 🔍

The redundant FEC data is usually calculated using the mathematics of finite fields [1] 🧮 The simplest finite field is GF(2), where addition is just the eXclusive-OR operation. Basic FEC schemes send k data packets along with n-k parity packets. That lets you rebuild the original data from any k of the n packets you sent 💜 Budge et al [4] proposed applying XOR across different combinations of the media data, and sending the redundant data separately as parity packets. Changing which packets the parity gets calculated over gives different bandwidth, latency and loss repair characteristics 🎀

Parity-based FEC techniques have a big advantage: they're media independent, and they give exact repair for lost packets 🙌 Their processing requirements are also pretty light. That's especially true compared with some media-specific FEC (redundancy) schemes, which use very low bandwidth but high complexity encodings 💅 The downside is that parity-based codings have higher latency than the media-specific schemes covered in the following section 😭

A number of FEC schemes are built on higher-order finite fields. Reed-Solomon (RS) codes are a good example. They're more sophisticated and more computationally demanding 🧠 These are usually structured to give good protection against burst loss, although that usually costs some extra latency ⏳ Depending on the loss patterns you observe, these codes might perform better than parity-based FEC 📈

An RTP payload format for generic FEC, which works for both parity-based and Reed-Solomon encoded FEC, is defined in [17] 🎉

4.2.2. Media-Specific FEC 💖

okay so the whole idea of media-specific FEC is that you actually know the media compression scheme and use that knowledge to repair a stream way more efficiently than you could otherwise 🧠✨ to fix packet loss you have to add some redundancy to the stream. that means adding info that isn't needed when nothing gets lost, but can be used to recover when something does 🦋

how the redundancy gets added depends on the kind of media stream. if your media units are packets, or several units ride in one packet, it makes total sense to treat the unit as the level of redundancy and just send duplicate units 🎀 you can recode the redundant copy, which can save a lot of bandwidth 💅 the tradeoff is extra computational complexity and an approximate repair. this approach has been pushed for streaming audio [2, 10] and it's been shown to perform well 🔥 there's also an RTP payload format defined for this kind of redundancy [15] 🫶

if media units span multiple packets, like video, it's smart to put the redundancy right inside the codec's output 🎬 for example, the proposed RTP payload for H.263+ [3] includes multiple copies of the key parts of the stream, spaced apart so packet loss doesn't wipe them all out 🌸 the big win here is efficiency: the codec designer knows exactly which parts of the stream matter most to protect, and it's low complexity because every unit is only coded once 😭👑

there's also another option: apply media-independent FEC techniques to just the most significant bits of a codec's output, instead of the entire packet 🧚 a bunch of codec descriptions include bit sensitivities that make this doable. it has low computational cost, and you can tune it to cover any fraction of the transmitted data 💜

media-specific FEC has the advantage of low latency, since only a single-packet delay gets added 🍓 that makes it great for interactive apps where big end-to-end delays just aren't okay 🙈 in a uni-directional, non-interactive setup, you can hold off on sending the redundant data, which gives better performance when there are burst losses [11], but at the cost of extra latency ⭐

4.3. Interleaving 🌈

okay this one is so clever 😭 when the unit size is smaller than the packet size and end-to-end delay doesn't matter much, interleaving [16] is a super useful trick for reducing the effects of loss 💖 the units get resequenced before transmission, so units that were originally next to each other end up a guaranteed distance apart in the transmitted stream. then the receiver puts them back in their original order 🦋 interleaving spreads out the damage from packet losses ✨

here's a quick example: say units are 5ms long and packets are 20ms (ie: 4 units per packet). then the first packet could hold units 1, 5, 9, 13, the second packet holds units 2, 6, 10, 14, and so on 🎀 now if you lose one packet from an interleaved stream, you get multiple small gaps in the reconstructed stream, instead of one big gap like you would without interleaving 🫶 for a lot of cases that's way easier to reconstruct, though it really depends on the media and codec 💅 also, the size of the gaps depends on how much interleaving you use, and you can make them arbitrarily small if you're willing to pay in latency 🔥

the obvious downside is that interleaving adds latency 😭 so it's not a great fit for interactive apps, but it works really well for non-interactive stuff 🌸 the major upside is that it does not increase the bandwidth requirements of a stream 💜

a potential RTP payload format for interleaved data is a simple extension of the redundant audio payload [15] 🧚 that payload requires the redundant copy of a unit to be sent after the primary one. if you drop that restriction, you can transmit any arbitrary interleaving of units with this payload format 🍓

5. Recommendations 👑

if you're going for a non-interactive, one-way transmission, like a radio or TV broadcast, latency matters way less than reception quality 📻 in that case, interleaving, retransmission-based repair, or FEC are all fair game 🎀 if approximate repair is okay with you, interleaving is clearly the move, since it doesn't increase the bandwidth of a stream 🦋 media-independent FEC is usually the next best pick, because a single FEC packet can repair multiple lost packets, which makes transmission efficient 💅

in an interactive session, the delay that comes with interleaving and retransmission is a no-go 🙈 so a low-latency FEC scheme is the only realistic way to repair things 🔥 picking between media-independent and media-specific forward error correction is less clear-cut 🤔 media-specific FEC can be made more efficient, but it needs changes to the codec's output 🌸 when you're defining the packet format for a brand-new codec, this is totally the right technique and it should be encouraged 💖

if you're stuck with an existing codec, media-independent FEC is usually easier to implement, and it can perform really well ✨ a media stream protected this way can also be augmented with retransmission-based repair with minimal overhead 🫶 that gives better quality to receivers who can tolerate extra delay, while still allowing interactivity for the ones who want that ⭐ adding FEC data to a media stream is an effective way to protect it from packet loss, but application designers should be aware of something important 🚨 piling on large amounts of repair data once loss is detected will increase network congestion, and that means more packet loss 💀 so you end up making the very problem the error correction coding was supposed to solve even worse 😭

right now there isn't a standard solution for congestion control for streamed media that can handle this problem 🫠 there have been a number of contributions that show what the likely solution will look like [12, 19] 🔮 this work typically used some form of layered encoding across multiple channels, with receivers joining and leaving layers in response to packet loss (which signals congestion) 🧚 the goal of these schemes is to copy the congestion control behavior of a TCP stream, so they can compete fairly with non-real-time traffic 💜 that matters because it's needed for stable network behavior when there's a lot of streamed media around 🌈

streaming media apps are already in use without congestion control, so it's important to give their authors some advice on what behavior is acceptable until congestion control mechanisms get deployed 📣 the rest of this section uses the throughput of a TCP connection over a link with a given loss rate as an example of behavior that can be considered reasonable 🎀

as a number of authors have noted (e.g. [6]), the loss rate and throughput of a TCP connection are approximately related like this 🧮

 T = (s * c) / (RTT * sqrt(p))

where T is the observed throughput in octets per second, s is the packet size in octets, RTT is the round-trip time for the session in seconds, p is the packet loss rate, and c is a constant in the range 0.9...1.5 (a value of 1.22 has been suggested [6]) 🍓 using this relation, you can figure out what packet loss rate would produce a given throughput for a particular session, if a TCP connection were used 💅

this relation between packet loss rate and throughput is specific to the TCP congestion control algorithm, but it also gives an estimate of the acceptable loss rate for a streaming media app on the same network path that wants to coexist fairly with TCP traffic 🦋 it's not enough for sharing a link fairly with TCP traffic, though, since it doesn't capture the dynamic behavior of the connection, only the average behavior 🙈 it does give one definition of "reasonable" behavior when there's no real congestion control 💖

for example, say you've got an RTP audio session with DVI encoding and 40ms data packets 🎶 each packet has a 40 bytes RTP/UDP/IP header, 4 bytes of codec state, and 160 bytes of media data, so the packet size, s, comes out to 204 bytes ✨ it sends 25 packets per second, giving T = 4800 💅 you can estimate the round-trip time from RTCP reception report statistics (say 200 milliseconds for the sake of the example) 🧚 plugging these values into the equation, we estimate a "reasonable" packet loss rate, p, of 6.7% 🔥 this would be an upper bound on the packet loss rate this app should be designed to tolerate 🫶

one thing to keep in mind: a round-trip time estimate based on RTCP reception report statistics is, at best, approximate 👀 and a round-trip time for a multicast group can only be an `average' measure 🌸 this means the TCP-equivalent throughput/loss rate you get from this relation is going to be an approximation of the upper bound on the rate a TCP connection would actually achieve 🌈

6. Security Considerations 🚨

some of the repair techniques in this document send additional traffic to correct for the effects of packet loss 🫣 application designers should know that sending large amounts of repair traffic will increase network congestion, and that means more packet loss, which makes the problem error correction was meant to solve even worse 😭 at its worst, this can lead to excessive network congestion and may count as a denial of service attack 💀 Section 5 covers this in more detail and gives guidelines for preventing it 🛡️

7. Summary 🎀

streaming media apps on the Internet are going to face packet loss, because UDP delivery is unreliable 💧 this document summarized the typical loss patterns seen on the public Mbone at the time of writing, plus a range of techniques for recovering from those losses 🦋 we also talked about why congestion control is needed, and gave some guidelines for reasonable behavior from streaming apps in the meantime, until congestion control can be deployed 🫶

8. Acknowledgments 💜

the authors want to thank Phil Karn and Lorenzo Vicisano for their helpful comments 🙏✨

9. Authors' Addresses 🌸

Colin Perkins
Department of Computer Science
University College London
Gower Street
London WC1E 6BT
United Kingdom

EMail: c.perkins@cs.ucl.ac.uk


Orion Hodson
Department of Computer Science
University College London
Gower Street
London WC1E 6BT
United Kingdom

EMail: o.hodson@cs.ucl.ac.uk

References 📚✨

okay bestie, these are the receipts 🧾 every source the RFC cites, numbered [1] through [21], with all the details kept exactly as written:

  • [1] R.E. Blahut. Theory and Practice ofError Control Codes. Addison Wesley, 1983.
  • [2] J.-C. Bolot and A. Vega-Garcia. The case for FEC based error control for packet audio in the Internet. To appear in ACM Multimedia Systems.
  • [3] C. Bormann, L. Cline, G. Deisher, T. Gardos, C. Maciocco, D. Newell, J. Ott, S. Wenger, and C. Zhu. RTP payload format for the 1998 version of ITU-T reccomendation H.263 video (H.263+). Work in Progress.
  • [4] D. Budge, R. McKenzie, W. Mills, W. Diss, and P. Long. Media-independent error correction using RTP, Work in Progress.
  • [5] G. Carle and E. W. Biersack. Survey of error recovery techniques for IP-based audio-visual multicast applications. IEEE Network, 11(6):24--36, November/December 1997.
  • [6] S. Floyd and K. Fall. Promoting the use of end-to-end congestion control in the internet. Submitted to IEEE/ACM Transactions on Networking, February 1998.
  • [7] S. Floyd, V. Jacobson, S. McCanne, C.-G. Liu, and L. Zhang. A reliable multicast framework for light-weight sessions and applications level framing. IEEE/ACM Transactions on Networking, 1995.
  • [8] M. Handley. An examination of Mbone performance. USC/ISI Research Report: ISI/RR-97-450, April 1997.
  • [9] M. Handley and J. Crowcroft. Network text editor (NTE): A scalable shared text editor for the Mbone. In Proceedings ACM SIGCOMM'97, Cannes, France, September 1997.
  • [10] V. Hardman, M. A. Sasse, M. Handley, and A. Watson. Reliable audio for use over the Internet. In Proceedings of INET'95, 1995.
  • [11] I. Kouvelas, O. Hodson, V. Hardman, and J. Crowcroft. Redundancy control in real-time Internet audio conferencing. In Proceedings of AVSPN'97, Aberdeen, Scotland, September 1997.
  • [12] S. McCanne, V. Jacobson, and M. Vetterli. Receiver-driven layered multicast. In Proceedings ACM SIGCOMM'96, Stanford, CA., August 1996.
  • [13] J. Nonnenmacher, E. Biersack, and D. Towsley. Parity-based loss recovery for reliable multicast transmission. In Proceedings ACM SIGCOMM'97, Cannes, France, September 1997.
  • [14] P. Parnes. RTP extension for scalable reliable multicast, Work in Progress.
  • [15] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V., Handley, M., Bolot, J-C., Vega-Garcia, A., and S. Fosse-Parisis, "RTP Payload for Redundant Audio Data", RFC 2198, September 1997.
  • [16] J.L. Ramsey. Realization of optimum interleavers. IEEE Transactions on Information Theory, IT-16:338--345, May 1970.
  • [17] J. Rosenberg and H. Schulzrinne. An A/V profile extension for generic forward error correction in RTP. Work in Progress.
  • [18] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A transport protocol for real-time applications", RFC 1889, January 1996.
  • [19] L. Vicisano, L. Rizzo, and Crowcroft J. TCP-like congestion control for layered multicast data transfer. In Proceedings IEEE INFOCOM'98, 1998.
  • [20] R. X. Xu, A. C. Myers, H. Zhang, and R. Yavatkar. Resilient multicast support for continuous media applications. In Proceedings of the 7th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'97), Washington University in St. Louis, Missouri, May 1997.
  • [21] M. Yajnik, J. Kurose, and D. Towsley. Packet loss correlation in the Mbone multicast network. In Proceedings IEEE Global Internet Conference, November 1996.

the legal bit, kept word for word so nothing gets lost in translation 🫶

Copyright (C) The Internet Society (1998). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

📦 where this page comes from