RFC 2736

RFC 2736: Guidelines for Writers of RTP Payload Format Specifications 🎀

okay bestie, so this is RFC 2736 💖 (part 1 of 3, more coming!). it's a BCP 36, which means it's a Best Current Practice, not a strict protocol spec. it was written by M. Handley (ACIRI) and C. Perkins (UCL) and it's dated December 1999. it doesn't obsolete or update any other RFCs, since there's no Obsoletes or Updates line in its header. the whole point is helping people who write RTP payload format specs pick good formats. 🦋

Status of this Memo ✨

This document is an Internet Best Current Practice for the Internet Community, and it's asking everyone for discussion and suggestions for improvements. distribution of this memo is unlimited, so share it with your whole group chat 💅

Copyright Notice 🌈

Copyright (C) The Internet Society (1999). All Rights Reserved.

Abstract 🍓

this doc gives general guidelines to help authors of RTP Payload Format specs decide on good formats. the guidelines are basically the lessons learned from RTP while it was being developed, so it's giving "learn from my mistakes" energy 🫶

1. Introduction ✨

okay so the intro basically repeats the abstract 😭 these guidelines are there to help authors of RTP [9] Payload Format specifications pick good formats. they capture some of what the team learned while RTP was evolving 🧚

the principles here work for almost any data type, but the examples are all about audio and video codecs because that's way easier to picture 🎀

2. Background 💖

RTP was built around Application Level Framing (ALF), which Clark and Tennenhouse [2] first described. the big idea behind ALF is that an app can handle misordered or lost packets in lots of different ways 🔥 it could just ignore the loss, re-send the missing data (either from a buffer or by regenerating it), or send new data that replaces what got lost. that choice only exists if the transport protocol is handling data in "Application Data Units" (ADUs) 🦋 an ADU is data that can be processed out-of-order compared to other ADUs, so the ADU is the smallest unit of error recovery. main character energy 👑

the key thing a transport needs for ADUs is that each ADU has enough info for the receiver to process it right away 💅 picture a video stream: the compressed video in one ADU has to be decompressible even if earlier ADUs never arrived. and the ADU also needs a "header" that says where it sits in the video image and which frame it came from 🌸

an ADU doesn't have to be a packet, but a packet is a natural ADU for loads of apps 🧁 ALF apps get a huge perk from this, because every packet that shows up can be processed immediately ✨

RTP was designed with the ALF philosophy in mind 💜 in an RTP stream, the RTP packet header has enough info to identify and decode a packet whether it arrived in order or after some earlier packets got lost. BUT, that only works if the RTP payload formats are also designed with ALF in mind, so don't skip that part 🙈

this also means your endpoints need to be smart and network aware 🧠 an app using RTP should know the limits of the network underneath it and adjust its transmission to fit. the team's experience is that a smart endpoint can do way better on real IP-based networks than a naive one 🔥

3. Channel Characteristics 🌸

these are the channel characteristics that affect best-effort transport of RTP over UDP/IP on the Internet, and they're kind of chaotic 😭

• Packets may be lost
• Packets may be duplicated
• Packets may be reordered in transit
• Packets will be fragmented if they exceed the MTU of the underlying network

the loss characteristics of a link can swing wildly over short time intervals 🌪️

fragmentation isn't a total disaster if it only happens now and then 🙃 but leaning on IP fragmentation is a bad design move, because it really raises the effective loss rate of a network and lowers goodput 💀 here's why: if one fragment gets lost, the rest of the fragments (which already ate up bottleneck bandwidth) still get tossed by the receiver. it also piles extra load onto the routers doing the fragmenting and onto the end systems reassembling the pieces 😩

the transit time between two Internet hosts isn't constant either 🦋 two things cause that: jitter from getting queued behind cross-traffic, and routing changes. the jitter can be characterised and compensated for with a playout buffer, which is manageable. routing changes are impossible to predict and hard to handle gracefully, so they're the real villain here 😤

4. Guidelines ✨

okay bestie, let's talk about the rules for RTP payload format specs 🎀 there are two requirements we're looking at:

• A payload format should be devised so that the stream being transported is still useful even in the presence of a moderate amount of packet loss.
• Ideally all the contents of every packet should be possible to be decoded and played out irrespective of whether preceding packets have been lost or arrive late.

The first one is giving "the nature of the Internet" 🌐 Sure, you can engineer parts of the Internet for super low loss rates with careful provisioning or non-best-effort services. But as a rule, payload formats should not be designed for those special purpose environments 💅 Payload formats should be built for the public Internet with best effort service, so they should expect moderate loss rates. A 5% loss rate is not uncommon, no cap 😭

We noted that TCP steady state models [3][4][6] say a 5% loss rate with a 1KByte packet size and a 200ms round-trip time gets TCP a throughput of around 180Kbit/s. Higher loss rates, smaller packet sizes, or a larger RTT are needed to push TCP down to lower data rates. Honestly, most of the time it's that TCP traffic creating the background loss that lots of RTP flows have to co-exist with 🥺 Without explicit congestion notification (ECN) [8], loss has to be treated as an intrinsic property of best-effort parts of the Internet. Main character energy? Loss is the main character 💀

When payload formats don't assume packet loss will happen, they should say so explicitly up front. Those formats count as special purpose payload formats, and they're unsuitable for the public Internet without special support from the network infrastructure 🙈

The second requirement is more specific about how RTP should cope with loss ✨ If an RTP payload format is properly designed, every packet that actually gets received should be useful. Usually that means following these guidelines:

• Packet boundaries should coincide with codec frame boundaries. So a packet should normally be made of one or more complete codec frames 🧚
• A codec's minimum unit of data should never be packetised so that it crosses a packet boundary, unless it's larger than the MTU.
• If a codec's frame size is larger than the MTU, the payload format must not rely on IP fragmentation. Instead, it must define its own fragmentation mechanism 🫶 Those mechanisms may involve codec-specific information that lets fragments be decoded. Or they might allow codec-independent packet-level forward error correction [5] to be applied, which can't be used with IP-level fragmentation.

In the abstract, a codec frame (aka the ADU, or the minimum size unit that has semantic meaning when handed to the codec) can be any arbitrary size 🦋 For PCM audio, it's one byte. For GSM audio, a frame is 20ms of audio. For H.261 video, it's a Group of Blocks (GOB), which is one twelfth of a CIF video frame 🌸

For PCM, it doesn't matter how audio is packetised, since the ADU size is one byte. For GSM audio, though, arbitrary packetisation would split a 20ms frame over two packets. That means if one packet got lost, the partial frames in the packets before and after the loss are meaningless 😩 Not only were the bits in the missing packet lost, but the extra bits in neighbouring packets (which used bottleneck bandwidth!) were effectively lost too, because the receiver has to throw them away. Instead, we'd packetise GSM by including several complete GSM frames in a packet. Typically four GSM frames are included in current implementations. So every packet received can be decoded, because even with loss, no incomplete frames are received 💖

The H.261 specification allows GOBs to be up to 3KBytes long, although most of the time they're smaller. You might think we should insert a group of blocks into a packet whenever it fits, and arbitrarily split a big GOB over two or more packets. The first version of the H.261 payload format did exactly that 👀 But this still means there are situations where H.261 packets arrive at the receiver and have to be discarded because other packets were lost. That's a loss multiplier effect, and we really want to avoid it 🙅‍♀️ There are actually smaller units than GOBs in the H.261 bit-stream, called macroblocks. The problem is they aren't identifiable without parsing from the start of the GOB. BUT if we provide a little additional information at the start of each packet, we can reinstate the information that would normally come from parsing from the start of the GOB. Then we can packetise H.261 by splitting the data stream on macroblock boundaries. This is a less obvious packetisation for H.261 than GOB packetisation, but a slightly smarter depacketiser at the receiver can rebuild a valid H.261 bitstream from an RTP packet stream that has experienced loss, without having to discard any of the data that arrived 🎀

There's one more guideline, about codecs that need the decoder state machine to keep step with the encoder state machine. Lots of audio codecs like LPC or GSM are like this. They're typically loss tolerant, because after a loss the predictor coefficients decay, so after a certain amount of time the predictor error caused by the loss disappears 🌈 Most codecs designed for telephony are like this too, since they were built to cope with bit errors without the decoder predictor state staying incorrect forever. Just packetising these formats so packets are whole multiples of codec frames might not be optimal. The packet received right after a loss can be decoded, but the start of the audio stream produced will be incorrect (distorting the signal), because the decoder predictor is now out of step with the encoder 😭 In principle, all of the decoder's internal state could go in a header at the start of every packet. But for lower bit-rate encodings that state is so substantial that the bit rate is no longer low. A compromise can usually be found, though: a greatly reduced form of decoder state is sent in every packet. It doesn't recreate the encoder's predictor precisely, but it does reduce the magnitude and duration of the distortion when the previous packet is lost. That compressed state is, by definition, very dependent on the codec in question. So we recommend:

• Payload formats for encodings where the decoder contains internal data-driven state that attempts to track encoder state should normally consider including a small additional header that conveys the most critical elements of this state, to reduce distortion after packet loss.

A similar issue comes up with codec parameters, and whether or not they should be included in the payload format. For example, a codec might have a choice of huffman tables for compression. The codec may use either huffman table 1 or table 2 for encoding, and the receiver needs to know which one for correct decoding 🤓 There are a few ways this kind of information can be conveyed:

• Out of band signalling, prior to media transmission.
• Out of band signalling, but the parameter can be changed mid-session. This requires synchronization of the change in the media stream.
• The change is signaled through a change in the RTP payload type field. This requires mapping the parameter space into particular payload type values, and signalling this mapping out-of-band prior to media transmission.
• Including the parameter in the payload format. This allows the parameter to be adapted in a robust manner, but makes the payload format less efficient.

Which mechanism to use depends on how useful it is to change the parameter mid-session to support application layer adaptation. Still, using out-of-band signalling to change a parameter mid-session is generally discouraged, because of the problem of synchronizing the parameter change with the media stream 🙃

---

4.1. RTP Header Extensions 🎀

Many RTP payload formats need some extra header information on top of the fixed RTP packet header. The recommended way to carry it is in the payload section of the packet 💜 The RTP header extension should not be used to convey payload specific information ([9], section 5.3), because that's inefficient with bandwidth, requires defining a new RTP profile or profile extension, and makes it hard to employ FEC schemes such as, for example, [7]. An RTP header extension is only appropriate when the extension in question applies across a wide range of payload types ✨

4.2. Header Compression 🦋

Designers of payload formats should also be aware of the needs of RTP header compression [1] 👀 The compression algorithm works best when the RTP timestamp increments by a constant value between consecutive packets. Payload formats that rely on sending packets out of order, so the timestamp increment isn't constant, are likely to compress less well than ones that send packets in order. This has mostly been an issue when designing payload formats for FEC information. Some video codecs also rely on out-of-order transmission of packets, at the cost of reduced compression. Out-of-order transmission might be the best solution in some cases, but payload format designers are encourage to look for alternative solutions where possible 💅

---

5. Summary 🌸

Designing packet formats for RTP is not a trivial task 😤 Typically you need detailed knowledge of the codec involved to design a format that's resilient to loss, doesn't introduce loss magnification effects from inappropriate packetisation, and doesn't add unnecessary distortion after a packet loss. We believe considerable effort should go into designing packet formats that are well tailored to the codec in question. This usually takes only a very small amount of processing at the sender and receiver, but the result can be a hugely improved quality in typical Internet environments 🔥

Designers of new codecs for use with RTP should consider making the output of the codec "naturally packetizable" 🫶 This implies the codec should be designed to produce a packet stream rather than a bit-stream. That packet stream should contain the minimal amount of redundancy needed to ensure each packet is independently decodable with minimal loss of decoder predictor tracking. Sacrificing a small amount of bandwidth for greater robustness to packet loss is recognised as a worthwhile tradeoff, and honestly it's giving "worth it" 💖

It is hoped that, in the long run, new codecs will be produced that can be directly packetised, without the trouble of designing a codec-specific payload format 🧚

It is possible to design generic packetisation formats that ignore the issues described in this document. But those formats are only suitable for special purpose networks where packet loss can be avoided by careful engineering at the network layer. They're not suited to current best-effort networks 💀

6. Security Considerations 🔐💅

okay bestie, this section is SO important and I need you to take it seriously 😭🫶 the whole vibe of these guidelines is that payload formats end up robust when they deal with real network conditions, the messy, lossy, unpredictable internet we actually live on 🌈. Designing for some fancy special-purpose network that's assumed to have basically zero loss will usually get you slightly better compression ✨, BUT the format ends up fragile 🥺, and fragile formats are literally easier targets for denial-of-service attacks 💀. not me making a whole plan to be fragile, lol.

designers should also be obsessed with security issues that can come from poorly written implementations 👀. like, you need to clearly say what the correct behaviour is when weird stuff happens, for example how to handle illegal field values, or when a length field doesn't match the actual data 🙈. the correct move is usually to just throw the packet away 🗑️, but you still want to flag those possible conditions to the implementor so they actually get trapped properly 🪤. no skipping this part, it's giving "main character responsibility" 👑.

the RTP specification covers encryption of the payload 🔒. this stuff should not normally be handled by the payload formats themselves 🙅‍♀️. HOWEVER, some payload formats spread the info about one application data unit across a bunch of packets, or they rely on packets that relate to multiple application data units 🧩. so, be careful when you change the encryption on those streams, because those formats might limit the spots in the stream where you can safely swap the encryption key without exposing sensitive data 😱🔑. this is the kind of thing that's lowkey so easy to mess up.

and finally, designers of payload formats that include FEC 🛡️ should know that automatically adding FEC when packets get lost can increase network congestion 📈. which can make the very problem FEC was supposed to fix even worse 😩. at its worst this can be a denial of service attack 🚨, so please make sure appropriate safeguards are in place to stop abuse 🙏✨.

---

Authors' Addresses 🎀

Mark Handley AT&T Center for Internet Research at ICSI, International Computer Science Institute, 1947 Center Street, Suite 600, Berkeley, CA 94704, USA

EMail: mjh@aciri.org

Colin Perkins Dept of Computer Science, University College London, Gower Street, London WC1E 6BT, UK.

EMail: C.Perkins@cs.ucl.ac.uk

Acknowledgments 🫶

this document is built on experience gathered over several years from a whole crew of brilliant people 🌸, including Van Jacobson, Steve McCanne, Steve Casner, Henning Schulzrinne, Thierry Turletti, Jonathan Rosenberg and Christian Huitema, amongst others 💜. iconic legends, all of them 👑.

References 📚

[1] Casner, S. and V. Jacobson, "Compressing IP/UDP/RTP Headers for Low-Speed Serial Links", RFC 2508, February 1999.

[2] D. Clark and D. Tennenhouse, "Architectural Considerations for a New Generation of Network Protocols" Proc ACM Sigcomm 90.

[3] J. Mahdavi and S. Floyd. "TCP-friendly unicast rate-based flow control". Note sent to end2end-interest mailing list, Jan 1997.

[4] M. Mathis, J. Semske, J. Mahdavi, and T. Ott. "The macro-scopic behavior of the TCP congestion avoidance algorithm". Computer Communication Review, 27(3), July 1997.

[5] J. Nonnenmacher, E. Biersack, Don Towsley, "Parity-Based Loss Recovery for Reliable Multicast Transmission", Proc ACM Sigcomm

[6] J. Padhye, V. Firoiu, D. Towsley, J. Kurose, "Modeling TCP Throughput: A Simple Model and its Empirical Validation", Proc. ACM Sigcomm 1998.

[7] Perkins, C., Kouvelas, I., Hodson, O., Hardman, V., Handley, M., Bolot, J.C., Vega-Garcia, A. and S. Fosse-Parisis, "RTP Payload for Redundant Audio Data", RFC 2198, September 1997.

[8] Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit Congestion Notification (ECN) to IP", RFC 2481, January 1999.

[9] Schulzrinne, H., Casner, S., Frederick, R. and V. Jacobson, "Real-Time Transport Protocol", RFC 1889, January 1996.

Full Copyright Statement ⚖️

Copyright (C) The Internet Society (1999). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 💖

Funding for the RFC Editor function is currently provided by the Internet Society ✨.