RFC 7657

Differentiated Services (Diffserv) and Real-Time Communication

📅 published November 2015, from ietf

Differentiated Services (Diffserv) and Real-Time Communication 💖

Okay bestie, this one is the RFC 7657 tea 🫖. It came out in November 2015. The authors are D. Black, Ed. (EMC) and P. Jones (Cisco) 🎀. It's Informational, which means it's here to share knowledge, not a Standards Track spec 🧚. The ISSN is 2070-1721. The title block doesn't list any RFCs it obsoletes or updates ✨.

Abstract ✨

Okay so this memo is about how Diffserv (network quality-of-service, aka making sure your call doesn't glitch 💅) interacts with real-time communication, including anything built on RTP. Diffserv works because network nodes treat packets differently based on the Diffserv Codepoints (DSCPs) in their IP headers 🦋.

Here's the drama 😭. WebRTC apps and some conferencing apps have started using SDP bundle negotiation to send several streams with different QoS needs over the same network 5-tuple. Using multiple DSCPs inside one 5-tuple has transport protocol side effects, especially for congestion control (like reordering) 🙈. On top of that, DSCP markings can get changed or removed between the traffic source and destination 💀. This memo breaks down what all of that means for real-time comms, WebRTC included 💜.

Status of This Memo 🌸

This doc is not an Internet Standards Track spec. It's published for informational purposes, so no pressure 🫶.

It's a product of the Internet Engineering Task Force (IETF), so it reflects consensus from the IETF community. It got public review and was approved by the Internet Engineering Steering Group (IESG). Not everything the IESG approves is eligible for any level of Internet Standard, so check Section 2 of RFC 5741 if you want the details 🤓.

If you want the latest status, errata, or a way to give feedback, head to http://www.rfc-editor.org/info/rfc7657 🍓.

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved 🌈.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as in effect on the publication date. Please actually read those, because they spell out your rights and restrictions 📜. Code components pulled from this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in that license 🔥.

Table of Contents 📑

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
  2.  Real-Time Communications  . . . . . . . . . . . . . . . . . .   3
    2.1.  RTP Background  . . . . . . . . . . . . . . . . . . . . .   4
    2.2.  RTP Multiplexing  . . . . . . . . . . . . . . . . . . . .   6
  3.  Differentiated Services (Diffserv)  . . . . . . . . . . . . .   7
    3.1.  Diffserv Per-Hop Behaviors (PHBs) . . . . . . . . . . . .  10
    3.2.  Traffic Classifiers and DSCP Remarking  . . . . . . . . .  10
  4.  Examples  . . . . . . . . . . . . . . . . . . . . . . . . . .  12
  5.  Diffserv Interactions . . . . . . . . . . . . . . . . . . . .  13
    5.1.  Diffserv, Reordering, and Transport Protocols . . . . . .  13
    5.2.  Diffserv, Reordering, and Real-Time Communication . . . .  15
    5.3.  Drop Precedence and Transport Protocols . . . . . . . . .  16
    5.4.  Diffserv and RTCP . . . . . . . . . . . . . . . . . . . .  17
  6.  Guidelines  . . . . . . . . . . . . . . . . . . . . . . . . .  18
  7.  Security Considerations . . . . . . . . . . . . . . . . . . .  19
  8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  20
    8.1.  Normative References  . . . . . . . . . . . . . . . . . .  20
    8.2.  Informative References  . . . . . . . . . . . . . . . . .  22
  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  26
  Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  26

1. Introduction ✨

Okay, intro time 🦋. This memo covers how Diffserv QoS [RFC2475] works with real-time network communication, including anything based on the Real-time Transport Protocol (RTP) [RFC3550]. Diffserv depends on network nodes giving different forwarding treatments to packets whose IP headers are marked with different Diffserv Codepoints (DSCPs) [RFC2474] 💅.

Back in the day, distinct RTP streams got sent over different transport-level flows, sometimes multiplexed with the RTP Control Protocol (RTCP) 🕰️. Now WebRTC and some conferencing apps use the Session Description Protocol (SDP) [RFC4566] bundle negotiation mechanism [SDP-BUNDLE] to send multiple traffic streams with different QoS needs over the same network 5-tuple 🎀. Using multiple DSCPs to get different QoS treatments inside a single 5-tuple has transport protocol interactions, especially with congestion control (e.g., reordering) 😭. Also, DSCP markings may be changed or removed between the traffic source and destination 🙈. This memo looks at what all of these Diffserv aspects mean for real-time communication, WebRTC traffic [WEBRTC-OVERVIEW] included 💜.

The memo is organized like this 🗺️. Section 2 gives background on real-time communications, and Section 3 covers Differentiated Services. Section 4 has examples of Diffserv used with real-time comms. Section 5 explains how Diffserv features interact with both transport and real-time protocols, and Section 6 gives guidance on using Diffserv features to avoid the undesired interactions. Section 7 is where the security considerations live 🔐.

2. Real-Time Communications 🎤

Real-time communications let people talk, watch, chat, share content, and more, all live over an IP network 🌸. You can use more than one of these modes at once for a seriously rich experience 🌈.

A simple example is a voice call over the Internet, where an audio stream goes each way between two people 📞. A bigger example is an immersive videoconferencing system with multiple screens, multiple cameras, multiple microphones, and some way to share content 🎬. For systems like that, there may be multiple media and non-media streams flowing through a single IP address and port, or through multiple IP addresses and ports 👑.

2.1. RTP Background

okay so RTP [RFC3550] is the most common protocol for real-time media 🎀. It gives you a common encapsulation format and handling rules for real-time data sent over the Internet. The downside is that RTP terminology has been all over the place 😭. RFC 7656 [RFC7656] is the RTP terminology doc, and it literally says:

RTP [RFC3550] uses media stream, audio stream, video stream, and a stream of (RTP) packets interchangeably, which are all RTP streams.

So this memo uses that terminology doc as its base 💅 and cares about these two terms the most (go read the terminology doc for the full definitions, bestie):

Source Stream: a reference clock synchronized, time progressing, digital media stream ⏰.

RTP Stream: a stream of RTP packets carrying media data, which might be source data or redundant data 🦋. It's identified by an RTP synchronization source (SSRC) that belongs to a particular RTP session. When RTP-based security is in use, an RTP stream can be a secured RTP stream 🔐.

Also, following [RFC3550], this memo uses "SSRC" to mean both the identifier of an RTP stream AND the entity that sends that RTP stream. Iconic dual-purpose energy 🌈.

Here's how the streams work together. Encoding and packetizing a source stream gives you one source RTP stream plus zero or more redundancy RTP streams, which protect against packet loss from the source RTP stream [RFC7656] 🛡️. Redundancy info can also ride in the same RTP stream as the encoded source stream, e.g., see Section 7.2 of [RFC5109] 🧠. Most apps send one media type (like audio) in one RTP session 🎵. But you CAN send multiple, distinct source streams over the same RTP session as one or more individual RTP streams. That's called RTP multiplexing 🔀. And the other way around, one RTP stream can contain multiple source streams, like components or programs in an MPEG Transport Stream [H.221] 📺.

The number of streams in a real-time interaction can get surprisingly huge 😳. Sure, you've got a voice source stream and a video source stream, but a videoconferencing system could have a separate source stream for every camera or microphone 🎥. Then there might be separate redundancy RTP streams protecting a source RTP stream with things like forward error correction 🧯. Another one is simulcast, where a video source stream goes out as high-res AND low-res RTP streams at the same time 📐. A media processing function can then forward one or both of them to a receiver based on bandwidth or on who's the active speaker in a multipoint conference 🗣️. And a sender can even push the same media concurrently as two RTP streams using different encodings (like VP8 [RFC6386] alongside H.264 [H.264]), so the media processor can pick whichever encoding best matches the receiver's capabilities 🤩.

For the WebRTC protocol suite [WEBRTC-TRANSPORTS], a single source stream is a MediaStreamTrack, and a MediaStream holds one or more MediaStreamTracks [W3C.WD-mediacapture-streams-20130903] 🎬. Each MediaStreamTrack goes out as a source RTP stream plus zero or more redundant RTP streams. So a MediaStream with just one MediaStreamTrack is sent as a single source RTP stream plus zero or more redundant RTP streams 💖. For more on RTP in WebRTC, check out [RTP-USAGE] 📚.

RTP usually rides on a datagram protocol like UDP [RFC768], UDP-Lite [RFC3828], or the Datagram Congestion Control Protocol (DCCP) [RFC4340] 📦. UDP is the most popular by far, but a non-datagram protocol like TCP [RFC793] can also show up 🫶. Other transports besides UDP or UDP-Lite can carry real-time or near-real-time data too. For instance, the Stream Control Transmission Protocol (SCTP) [RFC4960] can carry application-sharing or whiteboarding info as part of an interaction that also has real-time media 🖍️. These extra transports can be multiplexed with an RTP session through UDP encapsulation, so you only need a single pair of UDP ports 🧚.

The WebRTC protocol suite has a bunch of multiplexing going on 🌸:

  1. Individual source streams are carried in one or more individual RTP streams. Those RTP streams can be multiplexed onto a single transport-layer flow, or they can be sent as separate transport-layer flows. This memo only looks at the case where the RTP streams are multiplexed onto a single transport-layer flow, forming a single RTP session as described in [RFC3550] 🪢.
  2. RTCP (see [RFC3550]) can be multiplexed onto the same transport-layer flow as the RTP streams it's associated with, as described in [RFC5761]. Or it can go out on a separate transport-layer flow 📬.
  3. An RTP session can be multiplexed with a single SCTP association over Datagram Transport Layer Security (DTLS), and with both Session Traversal Utilities for NAT (STUN) [RFC5389] and TURN [RFC5766] traffic, all into a single transport-layer flow. This is described in [RFC5764] with the updates in [SRTP-DTLS] 🧩. STUN and Traversal Using Relays around NAT (TURN) [RFC5766] provide NAT/FW (Network Address Translator / Firewall) traversal and port mapping 🚪.

The resulting transport-layer flow is identified by a network 5-tuple: two IP addresses (source and destination), two ports (source and destination), and the transport protocol being used (e.g., UDP) 🗺️. SDP bundle negotiation restrictions [SDP-BUNDLE] limit WebRTC to at most a single DTLS session per network 5-tuple 🔒. Compare that with the WebRTC use of a single SCTP association with DTLS: multiple SCTP associations can be directly multiplexed over a single UDP 5-tuple, as specified in [RFC6951] 🫧.

STUN and TURN were originally designed to use UDP as their transport. But TURN has been extended to use TCP for situations where UDP just doesn't work [RFC6062] 🙃. When TURN picks TCP, the entire real-time communications session rides over a single TCP connection (i.e., 5-tuple) 🎯.

For IPv6, adding the flow label [RFC6437] to network 5-tuples gives you network 6-tuples (or 7-tuples for bidirectional flows) 🌐. In practice though, using a flow label isn't likely to split traffic into a finer subset than the matching network 5-tuple. For example, the flow label will probably represent the combination of two ports with UDP. That's why this document focuses on UDP 5-tuples 💜.

2.2. RTP Multiplexing

Section 2.1 explains how source streams can be multiplexed in a single RTP session, and that session can then be multiplexed over UDP with packets from other transport protocols 🧃. This section explains why this level of multiplexing is worth it 🤓. The reasoning here applies both to multiplexing source streams in a single RTP session AND to multiplexing an RTP session with traffic from other transport protocols via UDP encapsulation 💅.

Multiplexing cuts down on the number of ports used for real-time and related communication in an interaction ✂️. A single endpoint might have loads of ports free, but this traffic often goes through network points that are limited on ports, or whose performance drops as the number of ports in use goes up 📉. NAT/FW devices at the network edge are a great example. The more simultaneous protocol sessions there are, the more port mapping work these devices have to do 😩.

Multiplexing also helps shorten the time it takes to set up bidirectional communication ⏱️. Two communicating users might be behind different NAT/FW devices, so you need techniques like STUN and TURN along with Interactive Connectivity Establishment (ICE) [RFC5245] to get traffic flowing between the two devices [WEBRTC-TRANSPORTS] 🚦. All of that takes time, especially with lots of protocol sessions 🐢. Tasks for different sessions can run in parallel, but the app still has to wait for every session to open before the two users can start talking 🙄. Fewer STUN/ICE/TURN steps also means a lower chance of losing a packet for one of these protocols. Any loss adds delay to setting up the session 😤. And fewer STUN/ICE/TURN tasks means less load on the STUN and TURN servers 🫶.

Multiplexing can also reduce complexity and the resulting load on an endpoint 🧠. One instance of STUN/ICE/TURN is simpler to run and manage than multiple STUN/ICE/TURN operations happening in parallel 🙌. The parallel version needs synchronization and creates more complicated failure situations, which then need extra code to clean up 💀.

3. Differentiated Services (Diffserv) ✨

okay so Diffserv is the architecture [RFC2475][RFC4594] that lets the Internet do service discrimination at scale 💅 without every single node having to remember each flow and do per-flow signaling. The services can be end to end or live inside one network. Some can hit quantitative targets (think peak bandwidth 📈), and some are about relative treatment (think "class" differentiation 👑). The vibe is that you snap together well-defined building blocks in network nodes that:

  • classify traffic and set bits in an IP header field at network boundaries or hosts 🎀,
  • use those bits to decide how packets get forwarded by the nodes inside the network, and
  • condition the marked packets at network boundaries according to the requirements or rules of each service.

Traffic conditioning can mean changing the DSCP in a packet (remarking it 🔄), delaying the packet (that's traffic shaping ⏳), or dropping the packet (that's traffic policing 🫠).

A network node that supports Diffserv has a classifier that picks packets based on the value of the DS field in the IP header (the Diffserv codepoint, or DSCP 🦋). It also has buffer management and packet scheduling that can actually deliver the specific forwarding treatment the DS field value asks for. Setting the DS field and doing fine-grain conditioning of marked packets only needs to happen at network boundaries 🚪. Internal nodes work on traffic aggregates that share a DS field value, or sometimes a small set of related values, so they don't have to micromanage every packet 🧚.

The Diffserv architecture [RFC2475] keeps these things distinct from each other:

  • the QoS service provided to a traffic aggregate,
  • the conditioning functions and per-hop behaviors (PHBs) used to realize services,
  • the DSCP in the IP header used to mark packets to select a per-hop behavior, and
  • the particular implementation mechanisms that realize a per-hop behavior.

This memo is all about PHBs and how DSCPs are used to get those behaviors 🌸. In a network node's forwarding path, the DSCP maps a packet to a particular forwarding treatment, or to a per-hop behavior (PHB) that specifies that treatment.

The spec for a PHB describes the externally observable forwarding behavior of a network node for traffic marked with a DSCP that selects that PHB. "Forwarding behavior" is a broad concept here 🤓. For example, if only one DSCP is used for all traffic on a link, the observable behavior (loss, delay, jitter) will often depend only on how loaded the link is. To get useful differentiation, multiple traffic subsets get marked with different DSCPs for different PHBs, and node resources like buffer space and bandwidth are allocated for them. PHBs give a Diffserv node the framework to allocate resources to traffic subsets, and network-scope Differentiated Services get built on top of this basic hop-by-hop resource allocation mechanism 🏗️.

The codepoints (DSCPs) can come from a small set of fixed values (the class selector codepoints), from a set of recommended values defined in PHB specifications, or from values with purely local meanings to one specific network that supports Diffserv. Generally, packets can get forwarded across several of these networks between source and destination 🌍.

The mandatory DSCPs are the class selector codepoints as specified in [RFC2474]. The class selector codepoints (CS0-CS7) extend the deprecated concept of IP Precedence in the IPv4 header. Three bits get added, so class selector DSCPs look like 'xxx000' 🔢. The all-zero DSCP ('000000' or CS0) is always assigned to a Default PHB that gives best-effort forwarding behavior. The other class selector codepoints are meant to provide relatively better per-hop forwarding behavior as the numbers go up, but with some caveats 👀:

  • A network endpoint cannot rely on different class selector codepoints providing Differentiated Services via assignment to different PHBs, because adjacent class selector codepoints may use the same pool of resources on each network node in some networks 🙃. This generalizes to ranges of class selector codepoints, but with limits. For example, CS6 and CS7 are often used for network control (e.g., routing) traffic [RFC4594], so they're likely to get better forwarding behavior under network load, which helps prioritize network recovery from disruptions 🛠️. There's no effective way for a network endpoint to figure out which PHBs the class selector codepoints select on a specific network, let alone end to end 😩.
  • CS1 ('001000') was later designated as the recommended codepoint for the Lower Effort (LE) PHB [RFC3662] 🐢. An LE service forwards traffic with "lower" priority than best effort, and it can be "starved" by best-effort and other "higher" priority traffic. Not every network offers an LE service, so traffic marked with the CS1 DSCP may not get lower effort forwarding. Such traffic may be forwarded with a different PHB (e.g., the Default PHB), remarked to another DSCP (e.g., CS0) and forwarded accordingly, or dropped 💀. A network endpoint cannot rely on an LE service being present on a specific network, let alone end to end. Packets marked with the CS1 DSCP may be forwarded with best-effort service or another "higher" priority service; see [RFC2474]. See [RFC3662] for further discussion of the LE PHB and service 📚.

3.1. Diffserv Per-Hop Behaviors (PHBs)

Diffserv is a general architecture that can be used to build all sorts of services, but three fundamental forwarding behaviors (PHBs) have been defined and characterized for general use 🎉:

  1. Default Forwarding (DF) for elastic traffic [RFC2474]. The Default PHB is always selected by the all-zero DSCP and provides best-effort forwarding 🫶.
  2. Assured Forwarding (AF) [RFC2597] to provide Differentiated Service to elastic traffic. Each instance of the AF behavior consists of three PHBs that differ only in drop precedence, e.g., AF11, AF12, and AF13. That set of three AF PHBs is called an AF class, e.g., AF1x. There are four defined AF classes, AF1x through AF4x, with higher numbered classes meant to get better forwarding treatment than lower numbered ones 🌈. Using multiple PHBs from a single AF class (e.g., AF1x) does not enable network traffic reordering within a single network 5-tuple, although reordering may still happen for other transient reasons (e.g., routing changes or ECMP rebalancing) 🔀.
  3. Expedited Forwarding (EF) [RFC3246] intended for inelastic traffic 🚀. Beyond the basic EF PHB, the VOICE-ADMIT PHB [RFC5865] is an admission-controlled variant of the EF PHB. Both of these PHBs are based on preconfigured limited forwarding capacity, and traffic in excess of that capacity is expected to be dropped 🧱.

3.2. Traffic Classifiers and DSCP Remarking

DSCP markings are not end to end in general 🙅‍♀️. Each network can make its own decisions about which PHBs to use and which DSCP maps to each PHB. Every PHB specification includes a recommended DSCP, and RFC 4594 [RFC4594] recommends their end-to-end usage. But there's no requirement that every network support any PHBs (aside from the Default PHB for best-effort forwarding) or use any specific DSCPs. The one exception is the support requirements for the class selector codepoints (see RFC 2474 [RFC2474]) 🧐. When Diffserv is used, the edge or boundary nodes of a network are responsible for making sure all traffic entering that network follows that network's policies for DSCP and PHB usage, and those nodes may change DSCP markings to get there 🛂. So DSCP remarking is possible at any network boundary, including the very first network node that traffic sent by a host runs into. Remarking is also possible within a network, e.g., for traffic shaping 🔧.

DSCP remarking is part of traffic conditioning. The conditioning applied to packets at a network node is decided by a traffic classifier [RFC2475]. Edge nodes of a Diffserv network classify traffic based on selected packet header fields. Typical implementations don't look past the traffic's network 5-tuple in the IP and transport protocol headers. For example, with SCTP or RTP encapsulated in UDP, header-based classification is unlikely to look beyond the outer UDP header 👀. So when multiple DSCPs are used for traffic that shares a network 5-tuple, remarking at a network boundary may cause all of that traffic to be forwarded with a single DSCP. That wipes out any differentiation within the network 5-tuple downstream of the remarking location 😭. Network nodes inside a Diffserv network generally classify traffic based solely on DSCPs, but they may perform finer-grain traffic conditioning similar to what edge nodes do 🕵️‍♀️.

So, for two arbitrary network endpoints, there's no assurance that the DSCP set at the source endpoint will be preserved and presented at the destination endpoint 🫠. Instead, the DSCP is quite likely to be set to zero (e.g., at the boundary of a network operator that distrusts or doesn't use the DSCP field), or to a value an ingress classifier thinks is suitable for whatever network 5-tuple it carries 🤷‍♀️.

Also, remarking may wipe out application-level distinctions in forwarding behavior. For example, if multiple PHBs within an AF class are used to tell apart different types of frames within a video RTP stream, token-bucket-based remarkers running in color-blind mode (see [RFC2697] and [RFC2698] for examples) may remark based solely on flow rate and burst behavior. That removes the drop precedence distinctions the source specified 🎬💔.

Backbone and other carrier networks may use a small number of DSCPs (e.g., less than half a dozen) to manage a small number of traffic aggregates. Hosts using a larger number of DSCPs can expect much of their intended differentiation to get removed by those networks 🥲. Better results may be achieved when DSCPs are used to spread traffic among a smaller number of Diffserv-based traffic subsets or aggregates. See [DIFFSERV-INTERCON] for one proposal 💡. This matters a lot for MPLS-based networks, because the Traffic Class (TC) field in an MPLS label [RFC5462] is pretty small, and it's used to carry Diffserv information while also being used for other things, e.g., Explicit Congestion Notification (ECN) [RFC5129] 🧩. For more on using Diffserv with MPLS, see [RFC3270] and [RFC5127] 🎀.

4. Examples ✨

okay so for real-time stuff, you might wanna mark the audio packets with EF and the video packets with AF41 💖. but ngl, if the audio shows up way ahead of the video, the call is kinda useless because lip sync is literally necessary 🙈. it can still be worth sending audio with a PHB that gives better service, bc more reliable audio arrival helps keep the sound smooth, and smooth audio is often more important than super faithful video 🎀. there are limits though: some devices have trouble syncing voice and video when packets that need to be rendered together arrive at really different times 💀. it makes more sense to use different PHBs when the audio and video source streams don't have a strict timing relationship. like, video that gets shared within a conference through playback (maybe an unedited clip meant to become part of a TV ad) doesn't need precise sync with the conference audio, so it could use a different PHB, since content-sharing video is more tolerant to jitter, loss, and delay 🦋.

within a layered video RTP stream, keeping frames in order is preferred, but not all frame types are equally important, so using PHBs with different drop precedences is appropriate 🌸. for example, I-frames that contain an entire image are usually more important than P-frames that only contain changes from the previous image. that's because losing a P-frame (or part of one) can be recovered (at the latest) via the next I-frame, but losing an I-frame (or part of one) may cause rendering problems for all the P-frames that depend on that missing I-frame 😭. so it's appropriate to mark I-frame packets with a PHB that has lower drop precedence than the PHB used for P-frames, as long as the PHBs preserve ordering among frames (e.g., they're in a single AF class). AF41 for I-frames and AF43 for P-frames is one possibility 🧚. additional spatial and temporal layers beyond the base video layer could also be marked with higher drop precedence than the base video layer, since losing them reduces video quality but doesn't disrupt video rendering ✨.

additional RTP streams in a real-time communication interaction could be marked with CS0 and carried as best-effort traffic 🍓. one example is real-time text transmitted as specified in RFC 4103 [RFC4103]. best-effort forwarding is enough here because real-time text has loose timing requirements, and RFC 4103 recommends sending text in chunks every 300 ms. the text is technically real-time, but it doesn't need a PHB promising better service than best effort, unlike audio or video 💜.

a WebRTC application may use one or more RTP streams, as discussed above. on top of that, it may use an SCTP-based data channel [DATA-CHAN] whose QoS treatment depends on what kind of app it is 👑. for example, best-effort treatment of data channels is likely to be fine for messaging, shared white board, and guided browsing apps, whereas latency-sensitive games might want better QoS for their data channels 🎮.

5. Diffserv Interactions

5.1. Diffserv, Reordering, and Transport Protocols

transport protocols give you data communication behaviors that go beyond what the IP layer can do on its own 💅. the big example is TCP [RFC793], which gives reliable in-order delivery of data with congestion control. SCTP [RFC4960] adds extra stuff like preserving message boundaries, and it can avoid the head-of-line blocking that can happen with TCP 🦋.

UDP [RFC768] is the opposite vibe: a basic unreliable datagram protocol that gives port-based multiplexing and demultiplexing on top of IP 🌈. two other unreliable datagram protocols are UDP-Lite [RFC3828], a UDP variant that may deliver partially corrupt payloads when errors occur, and DCCP [RFC4340], which offers a range of congestion control modes for its unreliable datagram service 🫶.

transport protocols that provide reliable delivery (e.g., TCP, SCTP) are super sensitive to network reordering 😤. when one of them gets a packet that isn't the next expected packet, it usually assumes the expected packet was lost and updates the peer, which often causes a retransmission. and congestion control in transport protocols (including DCCP) usually infers congestion when packets are lost. so significant reordering creates extra sensitivity, because reordering may get (mis)interpreted as loss of the out-of-order packets, which triggers a congestion control response 💀.

this sensitivity to reordering sticks around even when ECN [RFC3168] is in use, because ECN receivers are required to treat missing packets as potential indications of congestion, and here's why:

  • severe congestion may cause ECN-capable network nodes to drop packets, and
  • ECN traffic may be forwarded by network nodes that don't support ECN, and hence drop packets to indicate congestion.

congestion control is a huge part of the Internet architecture, so see [RFC2914] for more on that 🧠.

in general, marking packets with different DSCPs means different PHBs get applied at nodes in the network. that makes reordering very likely, since each PHB uses different pools of forwarding resources 🌀. this shouldn't be done within a single network 5-tuple for current transport protocols, with the important exceptions of UDP and UDP-Lite 🚫.

when PHBs that enable reordering get mixed within a single network 5-tuple, the effect is to mix QoS-based traffic classes within the scope of a single transport protocol connection or association 🍭. these classes get different network QoS treatment, so they use different pools of network resources and may end up with different levels of congestion. for congestion-controlled protocols, that means a separate instance of congestion control is needed per QoS-based traffic class. right now, transport protocols only support a single instance of congestion control for an entire connection or association, and extending that to multiple instances would add significant protocol complexity 😵‍💫. traffic in different QoS-based classes may also take different paths through the network, which complicates path integrity checking in connection- or association-based protocols, since those paths can fail independently.

the main example where using multiple PHBs does not enable reordering within a single network 5-tuple is using PHBs from a single AF class (e.g., AF1x). traffic reordering within a network 5-tuple that uses a single PHB or AF class can still happen for other transient reasons (e.g., routing changes or ECMP rebalancing) 🔄.

reordering also affects other forms of congestion control, like the RTP congestion control techniques that were under development when this memo was published. see [RMCAT-CC] for requirements. those techniques prefer a common (coupled) congestion controller for RTP streams between the same endpoints, which helps reduce packet loss and delay by cutting down on competition for resources at any shared bottleneck 🤝.

shared bottlenecks can be detected with techniques like correlating one-way delay measurements across RTP streams. another approach is to assume that the set of packets on a single network 5-tuple marked with DSCPs that don't enable reordering will use a common network path and common forwarding resources at each network node. under that assumption, any bottleneck those packets hit is shared among all of them, so it's safe to use a common (coupled) congestion controller (see [COUPLED-CC]) 💖. this is not a safe assumption when the packets are marked with DSCP values that enable reordering, because a bottleneck may not be shared among all of them. for example, the DSCP values might result in different queues at a network node, but only one of those queues is a bottleneck 😬.

UDP and UDP-Lite aren't sensitive to reordering in the network, because they don't provide reliable delivery or congestion control 🦋. but when they're used to encapsulate other protocols (e.g., UDP is used by WebRTC; see Section 2.1), the reordering considerations for the encapsulated protocols still apply. for the specific use of UDP by WebRTC, every encapsulated protocol (RTP, SCTP, and TCP) is sensitive to reordering, as discussed further in this memo 🫠. in addition, [RFC5405] gives general guidelines for using UDP (and UDP-Lite), and the congestion control guidelines in that document apply to protocols encapsulated in UDP (or UDP-Lite) 📚.

5.2. Diffserv, Reordering, and Real-Time Communication

real-time communications are also sensitive to network reordering of packets 🎧. that reordering can lead to unneeded retransmissions and spurious retransmission control signals (such as NACK) in reliable delivery protocols (see Section 5.1). how sensitive things are depends on protocol or stream timers, unlike reliable delivery protocols, which usually react to all reordering 🙄.

receiver jitter buffers play a big role in how reordering affects real-time communications 🧃:

  • minor packet reordering that's contained within a jitter buffer usually has no effect on rendering of the received RTP stream, because packets that arrive out of order get pulled from the jitter buffer in order for rendering ✨.
  • packet reordering that exceeds the capacity of a jitter buffer can cause user-perceptible quality problems (e.g., glitches, noise) for delay-sensitive communication, like interactive conversations, where small jitter buffers are necessary to preserve human perceptions of real-time interaction 📞. interactive real-time communication implementations often discard data that's late enough that it can't be rendered in source stream order, which makes retransmission counterproductive. for this reason, implementations of interactive real-time communication often don't use retransmission 🚫.
  • replay of recorded media can tolerate significantly longer delays than interactive conversations, so replay is likely to use larger jitter buffers than interactive conversations 🎬. these larger jitter buffers increase replay's tolerance to reordering compared to interactive conversations. the size of the jitter buffer sets an upper bound on replay's tolerance to reordering, but it does enable retransmission to be used when the jitter buffer is significantly larger than the amount of data that can be expected to arrive during the round-trip latency for retransmission 🔁.

network packet reordering has no effective upper bound, and it can exceed the size of any reasonable jitter buffer 🌪️. in practice, the size of jitter buffers for replay is limited by external factors, such as how long a human is willing to wait for replay to start ⏳💅.

5.3. Drop Precedence and Transport Protocols ✨

Okay so, packets in the same network 5-tuple that use PHBs inside a single AF class can be expected to share the same forwarding resources on network nodes (like the same router queue) 💅. That means mixing drop precedences inside one AF class isn't expected to cause latency variation 🦋. When PHBs from one AF class get mixed within a flow, the overall chance that packets get dropped from that flow is a blend of the drop likelihoods of the PHBs involved.

There are situations where drop precedences should not be mixed 🙈. A simple example is a TCP connection. Mixing drop precedences there gets you basically nothing, because TCP delivers in order, so any drop makes the receiver wait for a retransmission. The delay depends on the RTT, not on which packet got dropped. So a single DSCP should be used for all packets in a TCP connection 💖.

As a result, when TCP is selected for NAT/FW traversal (e.g., by TURN), a single DSCP should be used for all traffic on that TCP connection. There's another reason too: packetization for STUN/ICE/TURN happens before the packets are handed to TCP, and TCP resegmentation may produce a different packetization on the wire. That can break the link between DSCPs and the specific data they were meant to apply to 🧩.

SCTP [RFC4960] is different from TCP in a bunch of ways, including being able to deliver messages in an order different from the one they were sent in, and supporting unreliable streams 🌈. But SCTP does congestion control and retransmission across the entire association, not per stream. There might be advantages to using multiple drop precedences across SCTP streams, or within an SCTP stream that doesn't use reliable ordered delivery, but there's no practical operational experience with that. For example, the SCTP sockets API [RFC6458] doesn't support using more than one DSCP for an SCTP association 🙃. So the impacts on SCTP protocol and implementation behavior are unknown and hard to predict. Hence a single DSCP should be used for all packets in an SCTP association, no matter how many streams it has or what kind they are 👑.

A DCCP connection gets the same treatment. A single DSCP should be used because the scope of congestion control is the connection, and there's no operational experience with using more than one DSCP. This recommendation might be revised in the future if experiments, analysis, and operational experience give compelling reasons to change it 🔮.

Guidance on transport protocol design and implementation that supports multiple PHBs and DSCPs in a transport protocol connection (e.g., DCCP) or transport protocol association (e.g., SCTP) is out of scope for this memo 🚫.

5.4. Diffserv and RTCP 🎀

RTCP [RFC3550] goes hand in hand with RTP to monitor quality of service and share info about RTP session participants 📊. A sender of RTCP packets that also sends RTP packets (aka it's originating an RTP stream) should use the same DSCP marking for both types of packets 💕. If an RTCP sender doesn't send any RTP packets, it should mark its RTCP packets with the DSCP it would use if it did send RTP packets with media similar to the RTP traffic it receives 🧚. And if the RTCP sender uses, or would use, multiple DSCPs that differ only in drop precedence for RTP, then it should use the DSCP with the least likelihood of drop for RTCP. That boosts the odds that RTCP packets actually get delivered 🫶.

If the SDP bundle extension [SDP-BUNDLE] is used to negotiate sending multiple types of media in a single RTP session, receivers will send separate RTCP reports for each type of media, using a separate SSRC for each one. Each RTCP report should be marked with the DSCP that matches the type of media handled by the reporting SSRC 🍓.

This guidance can lead to different DSCP markings for RTP streams and for the RTCP receiver reports about those streams. That variation in network QoS treatment by traffic direction is necessary to get representative round-trip time (RTT) estimates that match the media path RTT, which can differ from the transport protocol RTT 🌸. RTCP receiver reports can be pretty infrequent, so those RTT estimates aren't super useful for transport protocol congestion control (though they have other important uses; see [RFC3550]) 🤓. For this reason, it's important that RTCP receiver reports sent by an SSRC get the same network QoS treatment as the RTP stream that SSRC is sending 💜.

6. Guidelines 🌸

The only use of multiple standardized PHBs and DSCPs that does not enable network reordering among packets marked with different DSCPs is using PHBs within a single AF class 🧐. Every other use of multiple PHBs and/or the class selector DSCPs enables network reordering of packets marked with different DSCPs 🔀. So based on that and everything above, the guidelines in this section apply to using Diffserv with real-time communications 📞.

Applications and other traffic sources (including RTP SSRCs):

  • Should limit use of DSCPs within a single RTP stream to those whose corresponding PHBs don't enable packet reordering. If that isn't done, significant network reordering may overwhelm implementation assumptions about reordering limits (e.g., jitter buffer size), causing poor user experiences (see Section 5.2) 😭. This guideline applies to all of the RTP streams within the scope of a common (coupled) congestion controller when that controller doesn't use per-RTP-stream measurements for bottleneck detection 🎯.
  • Should use a single DSCP for RTCP packets, and it should be a DSCP used for the RTP packets that are, or would be, sent by that SSRC (see Section 5.4) 💅.
  • Should use a single DSCP for all packets within a reliable transport protocol session (e.g., TCP connection, SCTP association) or DCCP connection (see Sections 5.1 and 5.3) 🔒. For SCTP, this requirement applies across the entire SCTP association, not just to individual streams within an association. When TURN selects TCP for NAT/FW traversal, this guideline applies to all traffic multiplexed onto that TCP connection, unlike using UDP for NAT/FW traversal 🧃.
  • May use different DSCPs whose corresponding PHBs enable reordering within a single UDP or UDP-Lite 5-tuple, subject to the constraints above. The service differentiation from that kind of usage is unreliable, since it may be removed or changed by DSCP remarking at network boundaries, as described in Section 3.2 above 🫠.
  • Cannot rely on end-to-end preservation of DSCPs, because network node remarking can change DSCPs and remove drop precedence distinctions (see Section 3.2) 🙅‍♀️. For example, if a source uses drop precedence distinctions within an AF class to identify different types of video frames, using those DSCP values at the receiver to identify frame type is inherently unreliable 📼.
  • Should limit use of the CS1 codepoint to traffic for which best effort forwarding is acceptable, because network support for using CS1 to select a "less than best-effort" PHB is inconsistent. Also, some networks may treat CS1 as providing "better than best-effort" forwarding behavior 🙃.

There's no guidance in this memo on how network operators should differentiate traffic 🤷‍♀️. Networks may support all of the PHBs discussed here, classify EF and AFxx traffic identically, or even remark all traffic to best effort at some ingress points. Still, it's useful for applications and other traffic sources to provide finer-granularity DSCP marking on packets for the benefit of networks that offer QoS service differentiation 💫. For example, traffic originating from a browser may benefit from QoS service differentiation in within-building and residential access networks, even if the DSCP marking gets removed or simplified later 🏠. That's because those networks, and the boundaries between them, are likely traffic bottleneck locations (e.g., due to customer aggregation onto common links and/or speed differences among links used by the same traffic) 🚧.

7. Security Considerations 🔐

Okay, this part is SUPER important bestie 😤. The security considerations for all of the technologies discussed in this memo apply. In particular, see the security considerations for RTP in [RFC3550] and for Diffserv in [RFC2474] and [RFC2475] 🛡️.

Multiplexing multiple protocols onto a single UDP 5-tuple via encapsulation has implications for network functionality that monitors or inspects individual protocol flows, like firewalls and traffic monitoring systems 👀. When implementations of that functionality lack visibility into encapsulated traffic (which is likely true for many current implementations), it may be difficult or impossible to apply network security policy and associated controls at a finer granularity than the overall UDP 5-tuple 💀.

Using multiple DSCPs that enable reordering within an overall real-time communication interaction enlarges the set of network forwarding resources that interaction uses 📈. That increases exposure to resource depletion or failure, whether the underlying cause is benign or malicious 😱. This is an increase in the effective attack surface of the interaction, and it's a consideration when picking an appropriate degree of QoS differentiation among the components of the real-time communication interaction. See Section 3.3.2.1 of [RFC6274] for related discussion of DSCP security considerations 🧐.

Using multiple DSCPs to provide differentiated QoS service may reveal information about the encrypted traffic that gets different service levels 👁️. For example, DSCP-based identification of RTP streams combined with packet frequency and packet size could reveal the type or nature of the encrypted source streams 🕵️‍♀️. The IP header used for forwarding has to be unencrypted for obvious reasons, and the DSCP likewise has to be unencrypted so different IP forwarding behaviors can be applied to different packets 📦. The nature of encrypted traffic components can be disguised with encrypted dummy data padding and encrypted dummy packets (see, e.g., the discussion of traffic flow confidentiality in [RFC4303]) 🎭. Encrypted dummy packets could even be added in a way that an observer of the overall encrypted traffic might mistake for another encrypted RTP stream 🫣.

8. References 📚

8.1. Normative References 💖

okay so these are the normative references, aka the docs this RFC leans on to make its rules work 🫶 if you wanna actually implement the stuff, you gotta read these too, no cap 📚✨

   [RFC768]   Postel, J., "User Datagram Protocol", STD 6, RFC 768,
              DOI 10.17487/RFC0768, August 1980,
              <http://www.rfc-editor.org/info/rfc768>.

   [RFC793]   Postel, J., "Transmission Control Protocol", STD 7,
              RFC 793, DOI 10.17487/RFC0793, September 1981,
              <http://www.rfc-editor.org/info/rfc793>.

   [RFC2474]  Nichols, K., Blake, S., Baker, F., and D. Black,
              "Definition of the Differentiated Services Field (DS
              Field) in the IPv4 and IPv6 Headers", RFC 2474,
              DOI 10.17487/RFC2474, December 1998,
              <http://www.rfc-editor.org/info/rfc2474>.

   [RFC2475]  Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
              and W. Weiss, "An Architecture for Differentiated
              Services", RFC 2475, DOI 10.17487/RFC2475, December 1998,
              <http://www.rfc-editor.org/info/rfc2475>.

   [RFC2597]  Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski,
              "Assured Forwarding PHB Group", RFC 2597,
              DOI 10.17487/RFC2597, June 1999,
              <http://www.rfc-editor.org/info/rfc2597>.

   [RFC3246]  Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec,
              J., Courtney, W., Davari, S., Firoiu, V., and D.
              Stiliadis, "An Expedited Forwarding PHB (Per-Hop
              Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002,
              <http://www.rfc-editor.org/info/rfc3246>.

   [RFC3550]  Schulzrinne, H., Casner, S., Frederick, R., and V.
              Jacobson, "RTP: A Transport Protocol for Real-Time
              Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
              July 2003, <http://www.rfc-editor.org/info/rfc3550>.

   [RFC3662]  Bless, R., Nichols, K., and K. Wehrle, "A Lower Effort
              Per-Domain Behavior (PDB) for Differentiated Services",
              RFC 3662, DOI 10.17487/RFC3662, December 2003,
              <http://www.rfc-editor.org/info/rfc3662>.

   [RFC3828]  Larzon, L-A., Degermark, M., Pink, S., Jonsson, L-E., Ed.,
              and G. Fairhurst, Ed., "The Lightweight User Datagram
              Protocol (UDP-Lite)", RFC 3828, DOI 10.17487/RFC3828, July
              2004, <http://www.rfc-editor.org/info/rfc3828>.

   [RFC4340]  Kohler, E., Handley, M., and S. Floyd, "Datagram
              Congestion Control Protocol (DCCP)", RFC 4340,
              DOI 10.17487/RFC4340, March 2006,
              <http://www.rfc-editor.org/info/rfc4340>.

   [RFC4960]  Stewart, R., Ed., "Stream Control Transmission Protocol",
              RFC 4960, DOI 10.17487/RFC4960, September 2007,
              <http://www.rfc-editor.org/info/rfc4960>.

   [RFC5405]  Eggert, L. and G. Fairhurst, "Unicast UDP Usage Guidelines
              for Application Designers", BCP 145, RFC 5405,
              DOI 10.17487/RFC5405, November 2008,
              <http://www.rfc-editor.org/info/rfc5405>.

   [RFC5865]  Baker, F., Polk, J., and M. Dolly, "A Differentiated
              Services Code Point (DSCP) for Capacity-Admitted Traffic",
              RFC 5865, DOI 10.17487/RFC5865, May 2010,
              <http://www.rfc-editor.org/info/rfc5865>.

   [RFC6951]  Tuexen, M. and R. Stewart, "UDP Encapsulation of Stream
              Control Transmission Protocol (SCTP) Packets for End-Host
              to End-Host Communication", RFC 6951,
              DOI 10.17487/RFC6951, May 2013,
              <http://www.rfc-editor.org/info/rfc6951>.

   [RFC7656]  Lennox, J., Gross, K., Nandakumar, S., Salgueiro, G., and
              B. Burman, Ed., "A Taxonomy of Semantics and Mechanisms
              for the Real-Time Transport Protocol (RTP) Sources",
              RFC 7656, DOI 10.17487/RFC7656, November 2015,
              <http://www.rfc-editor.org/info/rfc7656>.

these are the building blocks everything else sits on 🧚💜

8.2. Informative References ✨

okay so this is the extra reading list, bestie 📚 these aren't required reading for the spec itself, but they're the sources the document points to for background 🧠

Acknowledgements 💖

this doc is the result of SO many conversations across the DART working group and other working groups in the RAI and Transport areas 🫶 huge thank yous to Aamer Akhter, Harald Alvestrand, Fred Baker, Richard Barnes, Erin Bournival, Ben Campbell, Brian Carpenter, Spencer Dawkins, Keith Drage, Gorry Fairhurst, Ruediger Geib, Cullen Jennings, Jonathan Lennox, Karen Nielsen, Colin Perkins, James Polk, Robert Sparks, Tina Tsou, Michael Welzl, Dan York, and the DART WG participants for all their reviews and comments 🌈 they really made this happen ✨

Authors' Addresses 🎀

David Black (editor)
EMC
176 South Street
Hopkinton, MA  01748
United States

Phone: +1 508 293-7953
Email: david.black@emc.com


Paul Jones
Cisco
7025 Kit Creek Road
Research Triangle Park, NC  27502
United States

Phone: +1 919 476 2048
Email: paulej@packetizer.com
📦 where this page comes from