RFC 8084
Network Transport Circuit Breakers
The RFC File 📄 Network Transport Circuit Breakers 🎀
RFC 8084, also known as BCP 208 🌸 Author: G. Fairhurst, University of Aberdeen 🎓 Category: Best Current Practice 💅 Date: March 2017 📅 ISSN: 2070-1721 Publisher: Internet Engineering Task Force (IETF) 🦋
Abstract ✨
okay bestie, so this doc is all about what a "network transport Circuit Breaker" means 🫶 It explains why Circuit Breakers (CBs) are needed for network tunnels and applications when the traffic isn't congestion-controlled 😭 It also spells out where CBs are needed and where they're not, plus the requirements for building one and what you can expect to happen when you use one on the Internet 🌈
Status of This Memo ✨
This memo documents an Internet Best Current Practice. 💖
This document is a product of the Internet Engineering Task Force (IETF) 🦋 It represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 👑 For more on BCPs, check out Section 2 of RFC 7841 🍓
Info about the current status of this document, any errata, and how to give feedback on it is at http://www.rfc-editor.org/info/rfc8084 🧚
Copyright Notice 🎀
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication 📜 Please actually read these, bestie, because they describe your rights and restrictions for this document 🙈 Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions 🔥 They also come without warranty, as described in the Simplified BSD License 💀
Table of Contents 🌈
1. Introduction ....................................................2
1.1. Types of CBs ...............................................5
2. Terminology .....................................................6
3. Design of a CB (What makes a good CB?) ..........................6
3.1. Functional Components ......................................6
3.2. Other Network Topologies ...................................9
3.2.1. Use with a Multicast Control/Routing Protocol ......10
3.2.2. Use with Control Protocols Supporting
Pre-provisioned Capacity ...........................11
3.2.3. Unidirectional CBs over Controlled Paths ...........11
4. Requirements for a Network Transport CB ........................12
5. Examples of CBs ................................................15
5.1. A Fast-Trip CB ............................................15
5.1.1. A Fast-Trip CB for RTP .............................16
5.2. A Slow-Trip CB ............................................16
5.3. A Managed CB ..............................................17
5.3.1. A Managed CB for SAToP Pseudowires .................17
5.3.2. A Managed CB for Pseudowires (PWs) .................18
6. Examples in Which CBs May Not Be Needed ........................19
6.1. CBs over Pre-provisioned Capacity .........................19
6.2. CBs with Tunnels Carrying Congestion-Controlled Traffic ...19
6.3. CBs with Unidirectional Traffic and No Control Path .......20
7. Security Considerations ........................................20
8. References .....................................................22
8.1. Normative References ......................................22
8.2. Informative References ....................................22
Acknowledgments ...................................................24
Author's Address ..................................................241. Introduction ✨
Okay bestie, so the term "Circuit Breaker" 🔌 comes from electricity, and it has nothing to do with network circuits or virtual circuits. In the power grid, a Circuit Breaker (CB) is a protection mechanism of last resort. It's not supposed to trip under normal conditions. Its whole job is protecting the supply network and the equipment hooked up to it when there's an overload. Nobody expects the breaker (or fuse) in their home to go off unless there's a wiring fault or a broken appliance 😭.
In networking, the CB idea can be used as that same last-resort protection, this time to stop persistent excessive congestion from hurting other flows that share the same network capacity 🦋. Persistent congestion was a real problem on the early Internet of the 1980s. Excess traffic starved other connections of access to the Internet. The fix was to require congestion control (CC) in the Transmission Control Protocol (TCP) [Jacobson88] 💖. These mechanisms run in Internet hosts and make TCP connections "back off" when congestion shows up. Adding congestion control to TCP (currently documented in [RFC5681]) kept the Internet stable, because it could detect congestion and react quickly ✨. That worked well on an Internet where most TCP flows were long lived, so they could detect and respond to congestion before they ended. TCP was by far the dominant traffic, but that's no longer always the case 👀. Non-congestion-controlled traffic, including lots of applications using the User Datagram Protocol (UDP), can make up a big chunk of all the traffic crossing a link. So to avoid persistent excessive congestion, the current Internet needs us to think about how non-congestion-controlled traffic gets forwarded 🎀.
A network transport CB is an automatic mechanism that continuously monitors a flow or an aggregate set of flows 👀. It tries to detect when those flows experience persistent excessive congestion. When it spots that, the CB terminates the flow(s) or significantly reduces their rate. It's a safety measure that stops other flows from being starved of network resources and locked out of the Internet 🛡️. Measures like this are essential for an Internet that is heterogeneous, and for traffic that's hard to predict in advance. Avoiding persistent excessive congestion also matters for reducing the potential for "Congestion Collapse" [RFC2914] 💀.
There are important differences between a transport CB and a congestion control method, so let's break them down 💅. Congestion control (as implemented in TCP, Stream Control Transmission Protocol (SCTP), and Datagram Congestion Control Protocol (DCCP)) works on a timescale on the order of a packet Round-Trip Time (RTT), which is the time from sender to destination and back again ⏱️. Congestion at a network link can also be detected using Explicit Congestion Notification (ECN) [RFC3168]. That lets the network signal congestion by marking ECN-capable packets with a Congestion Experienced (CE) mark 🚩. Both loss and reception of CE-marked packets count as congestion events. Congestion control methods can react to a congestion event by continuously adapting to lower their transmission rate. The goal is usually to cap the transmission rate at a maximum that reflects a fair use of the available capacity across a network path 🤝. These methods typically work on individual traffic flows (for example, a 5-tuple that includes the IP addresses, protocol, and ports).
CBs are a different story 💜. They're recommended for non-congestion-controlled Internet flows and for traffic aggregates, for example traffic sent using a network tunnel. They work on timescales much longer than the packet RTT, and they trigger when there's abnormal (excessive) congestion 🌈. People have been implementing what this document calls CBs on an ad hoc basis to protect Internet traffic, so this document gives guidance on how to deploy and use these mechanisms 🧚. Later sections give examples of cases where CBs may or may not be desirable.
A CB needs to measure (meter) some portion of the traffic to work out whether the network is experiencing congestion. It also needs to be designed to trigger robustly when there's persistent excessive congestion 🔥.
A CB trigger will often use a series of successive sample measurements, metered at an ingress point and an egress point (either of which could be a transport endpoint) 📊. The trigger needs to operate on a timescale much longer than the path RTT, for example seconds up to possibly many tens of seconds ⏳. That longer period gives transport congestion control or applications enough time to adjust their rate after congestion, and gives the network load time to stabilize after any adjustment 🫶. Congestion events can be common when a congestion-controlled transport runs over a network link that's operating near capacity. Each event reduces the rate of the transport flow that's experiencing congestion. The longer period is there to make sure a CB doesn't get accidentally triggered by a single congestion event, or even a few successive ones 🙈.
Once triggered, the CB needs to provide a control function, called the "reaction" 🎬. The reaction removes traffic from the network, either by disabling the flow or by significantly reducing the level of traffic. This gives the protection needed to stop persistent excessive congestion from hitting other flows that share the congested part of the network path 👑.
Section 4 defines requirements for building a CB 📋.
The operational conditions that cause a CB to trigger ought to be regarded as abnormal 🚨. Examples of situations that could trigger a CB include:
- anomalous traffic that exceeds the provisioned capacity (or whose traffic characteristics exceed the threshold configured for the CB);
- traffic generated by an application at a time when the provisioned network capacity is being used for other purposes;
- routing changes that cause additional traffic to start using the path monitored by the CB;
- misconfiguration of a service/network device where the capacity available is insufficient to support the current traffic aggregate;
- misconfiguration of an admission controller or traffic policer that allows more traffic than expected across the path monitored by the CB.
Other mechanisms could also be available to network operators to detect excessive congestion, for example observing excessive utilization for a port on a network device 🔍. Using that kind of information, operational mechanisms could reduce network load on a shorter timescale than a network transport CB does ⚡. The CB's role over such paths stays as a method of last resort. Because it acts over a longer timescale, the CB ought to be triggered only when other reactions did not succeed in reducing persistent excessive congestion 🛑.
In many cases, the source of the traffic (user, application, endpoint, and so on) won't be able to tell why a CB was triggered 🤷♀️. A CB can be used to limit traffic from applications that are unable, or choose not, to use congestion control, or in cases where the congestion control properties of the traffic can't be relied on (for example, traffic carried over a network tunnel). In those situations, it's all but impossible for the CB to signal back to the affected applications. So in some cases, applications could have a hard time figuring out that a CB was triggered and where in the network it happened 😩.
That's why application developers are advised, where possible, to deploy appropriate congestion control mechanisms 💅. An application that uses congestion control is aware of congestion events in the network. That lets it regulate its network load under congestion, and it's expected to avoid triggering a network CB 🫶. For applications that can generate elastic traffic, this will often be the preferred solution 🌸.
1.1. Types of CBs
There are various forms of network transport CBs 🦋. They're mainly told apart by the timescale over which they're triggered, but also by the protection they offer:
- Fast-Trip CBs: The relatively short timescale used by this form of CB is intended to protect network traffic from a single flow or a related group of flows ⚡.
- Slow-Trip CBs: This CB uses a longer timescale and is designed to protect network traffic from congestion caused by traffic aggregates 🐢.
- Managed CBs: These use the operations and management functions that might be present in a managed service to implement a CB 🧑💻.
Examples of each type of CB are provided in Section 4 🎀.
2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 📖✨.
3. Design of a CB (What makes a good CB?) 🎀
Although CBs have been talked about in the IETF for many years, there hasn't been guidance yet on the cases where CBs are needed or on how to design CB mechanisms 🙈. This document aims to offer advice on both of those topics 💖.
CBs are RECOMMENDED for IETF protocols and tunnels that carry non-congestion-controlled Internet flows and for traffic aggregates 🌈. That includes traffic sent using a network tunnel. Designers of other protocols and tunnel encapsulations also ought to consider using these techniques as a last resort, to protect traffic that shares the network path being used 🫶.
This document defines the requirements for the design of a CB and gives examples of how a CB can be constructed 📋. The specifications of individual protocols and tunnel encapsulations need to detail the protocol mechanisms needed to implement a CB 🧩.
Section 3.1 describes the functional components of a CB, and Section 3.2 defines requirements for implementing a CB 👑.
3.1. Functional Components ✨
okay so the basic vibe of a CB is an ingress point (the sender) chatting with an egress point (the receiver) over a network flow or a bunch of flows 🦋 Figure 1 gives you the big picture: a set of routers (each labeled R) connecting a set of endpoints.
A CB controls traffic passing through a subset of those routers, sitting between the ingress and egress network devices. The path between them could be a tunnel or some other network-layer technique 🧚 One expected use is at the ingress and egress of a service, where all the traffic being considered ends up beyond the egress point. So the ingress and egress carry the same set of flows, which is giving very consistent energy 💅
+--------+ +--------+
|Endpoint| |Endpoint|
+--+-----+ >>> circuit breaker traffic >>> +--+-----+
| |
| +-+ +-+ +---------+ +-+ +-+ +-+ +--------+ +-+ +-+ |
+-+R+--+R+->+ Ingress +--+R+--+R+--+R+--+ Egress |--+R+--+R+-+
+++ +-+ +------+--+ +-+ +-+ +-+ +-----+--+ +++ +-+
| ^ | | |
| | +--+------+ +------+--+ |
| | | Ingress | | Egress | |
| | | Meter | | Meter | |
| | +----+----+ +----+----+ |
| | | | |
+-+ | | +----+----+ | | +-+
|R+--+ | | Measure +<----------------+ +--+R|
+++ | +----+----+ Reported +++
| | | Egress |
| | +----+----+ Measurement |
+--+-----+ | | Trigger + +--+-----+
|Endpoint| | +----+----+ |Endpoint|
+--------+ | | +--------+
+---<---+
ReactionFigure 1: A CB controlling the part of the end-to-end path between an ingress point and an egress point. Note in some cases, the trigger and measurement functions could alternatively be located at other locations (e.g., at a network operations center).
In the CB world, the ingress and egress functions could be implemented in different places 🌈 For example, they could live in network devices at a tunnel ingress and at the tunnel egress. In some cases, they could sit at one or both network endpoints (see Figure 2), implemented as components within a transport protocol. slay 👑
+----------+ +----------+
| Ingress | +-+ +-+ +-+ | Egress |
| Endpoint +->+R+--+R+--+R+--+ Endpoint |
+--+----+--+ +-+ +-+ +-+ +----+-----+
^ | |
| +--+------+ +----+----+
| | Ingress | | Egress |
| | Meter | | Meter |
| +----+----+ +----+----+
| | |
| +--- +----+ |
| | Measure +<-----------------+
| +----+----+ Reported
| | Egress
| +----+----+ Measurement
| | Trigger |
| +----+----+
| |
+---<--+
ReactionFigure 2: An endpoint CB implemented at the sender (ingress) and receiver (egress).
The set of components needed to build a CB are the following. Think of it as the squad, bestie 🫶
- An ingress meter (at the sender or tunnel ingress) that records the number of packets/bytes sent in each measurement interval. This measures the offered network load for a flow or set of flows. For example, the measurement interval could be many seconds (or every few tens of seconds, or a series of successive shorter measurements that are combined by the CB Measurement function).
- An egress meter (at the receiver or tunnel egress) that records the number/bytes received in each measurement interval. This measures the supported load for the flow or set of flows, and it could utilize other signals to detect the effect of congestion (e.g., loss/congestion marking [RFC3168] experienced over the path). The measurements at the egress could be synchronized (including an offset for the time of flight of the data, or referencing the measurements to a particular packet) to ensure any counters refer to the same span of packets. the precision is literally everything 😭
- A method that communicates the measured values at the ingress and egress to the CB Measurement function. This could use several methods: sending return measurement packets (or control messages) from a receiver to a trigger function at the sender; an implementation using Operations, Administration and Management (OAM); or sending an in-band signaling datagram to the trigger function. This could also be implemented purely as a control-plane function, e.g., using a software-defined network controller 🦋
- A measurement function that combines the ingress and egress measurements to assess the present level of network congestion. (For example, the loss rate for each measurement interval could be deduced from calculating the difference between ingress and egress counter values.) Note the method does not require high accuracy for the period of the measurement interval (or therefore the measured value, since isolated and/or infrequent loss events need to be disregarded). one-off blips are not the vibe 🙈
- A trigger function that determines whether the measurements indicate persistent excessive congestion. This function defines an appropriate threshold for determining that there is persistent excessive congestion between the ingress and egress. This preferably considers a rate or ratio, rather than an absolute value (e.g., more than 10% loss, but other methods could also be based on the rate of transmission as well as the loss rate). The CB is triggered when the threshold is exceeded in multiple measurement intervals (e.g., three successive measurements). Designs need to be robust so that single or spurious events do not trigger a reaction. 💜
- A reaction that is applied at the ingress when the CB is triggered. This seeks to automatically remove the traffic causing persistent excessive congestion. the cleanup crew arrives 🧹✨
- A feedback control mechanism that triggers when either the ingress or egress measurements are not available, since this also could indicate a loss of control packets (also a symptom of heavy congestion or an inability to control the load). no data is also a vibe, and it's a warning sign 🚨
3.2. Other Network Topologies
A CB can be deployed in networks with topologies different from the ones presented in Figures 1 and 2 🌸 This section describes examples of such usage and possible places where functions can be implemented. the topology glow-up is real 🍓
3.2.1. Use with a Multicast Control/Routing Protocol
+----------+ +--------+ +----------+
| Ingress | +-+ +-+ +-+ | Egress | | Egress |
| Endpoint +->+R+--+R+--+R+--+ Router |--+ Endpoint +->+
+----+-----+ +-+ +-+ +-+ +---+--+-+ +----+-----+ |
^ ^ ^ ^ | ^ | |
| | | | | | | |
+----+----+ + - - - < - - - - + | +----+----+ | Reported
| Ingress | multicast Prune | | Egress | | Ingress
| Meter | | | Meter | | Measurement
+---------+ | +----+----+ |
| | |
| +----+----+ |
| | Measure +<--+
| +----+----+
| |
| +----+----+
multicast | | Trigger |
Leave | +----+----+
Message | |
+----<----+Figure 3: An example of a multicast CB controlling the end-to-end path between an ingress endpoint and an egress endpoint.
Figure 3 shows one example of how a multicast CB could be implemented at a pair of multicast endpoints (e.g., to implement a Fast-Trip CB, Section 5.1). The ingress endpoint (the sender that sources the multicast traffic) meters the ingress load, generating an ingress measurement (e.g., recording timestamped packet counts), and it sends this measurement to the multicast group together with the traffic it has measured 📡
Routers along a multicast path forward the multicast traffic (including the ingress measurement) to all active endpoint receivers. Each last hop (egress) router forwards the traffic to one or more egress endpoints 🦋
In Figure 3, each endpoint includes a meter that performs a local egress load measurement. An endpoint also extracts the received ingress measurement from the traffic and compares the ingress and egress measurements to determine if the CB ought to be triggered. This measurement has to be robust to loss (see the previous section). If the CB is triggered, it generates a multicast leave message for the egress (e.g., an IGMP or MLD message sent to the last-hop router), which causes the upstream router to cease forwarding traffic to the egress endpoint [RFC1112]. bye bye traffic 👋
Any multicast router that has no active receivers for a particular multicast group will prune traffic for that group, sending a prune message to its upstream router. This starts the process of releasing the capacity used by the traffic and is a standard multicast routing function (e.g., using the Protocol Independent Multicast - Sparse Mode (PIM-SM) routing protocol [RFC7761]). Each egress operates autonomously, and the CB "reaction" is executed by the multicast control plane (e.g., by PIM), requiring no explicit signaling by the CB along the communication path used for the control messages. Note there is no direct communication with the ingress; hence, a triggered CB only controls traffic downstream of the first-hop multicast router. It does not stop traffic flowing from the sender to the first-hop router; this is common practice for multicast deployment. no direct chat with the ingress, just vibes downstream 🫠
The method could also be used with a multicast tunnel or subnetwork (e.g., Section 5.2, Section 5.3), where a meter at the ingress generates additional control messages to carry the measurement data towards the egress, where the egress metering is implemented. ✨
3.2.2. Use with Control Protocols Supporting Pre-provisioned Capacity
Some paths are provisioned using a control protocol, e.g., flows provisioned using the Multiprotocol Label Switching (MPLS) services, paths provisioned using the Resource Reservation Protocol (RSVP), networks utilizing Software-Defined Network (SDN) functions, or admission-controlled Differentiated Services. Figure 1 shows one expected use case, where in this usage a separate device could be used to perform the measurement and trigger functions. The reaction generated by the trigger could take the form of a network-control message sent to the ingress and/or other network elements, causing these elements to react to the CB. Examples of this type of use are provided in Section 5.3. the control plane really said "not me reacting first" 💅🎀
3.2.3. Unidirectional CBs over Controlled Paths
okay so a CB can totally control unidirectional UDP traffic 🦋, as long as there's a communication path that can carry control messages between the functional components at the ingress and the egress ✨. And the plot twist is that this control path can exist even in networks where the traffic only flows one way 😭. Think of a multicast stream that sends packets across an Internet path and uses multicast routing to prune flows and shed network load 🌊. Some other types of subnetwork use control protocols too, and those can be used to control traffic flows 💅.
4. Requirements for a Network Transport CB ✨
bestie, here are the requirements for implementing a CB. Lowkey the whole list is the heart of this doc 🫶
- There needs to be a communication path for control messages to carry measurement data from the ingress meter and from the egress meter to the point of measurement. (Requirements 16-18 relate to the transmission of control messages.) 🛤️
- A CB is REQUIRED to define a measurement period over which the CB Measurement function measures the level of congestion or loss. This method does not have to detect individual packet loss, but it MUST have a way to know that packets have been lost/marked from the traffic flow. 📏
- An egress meter can also count ECN [RFC3168] Congestion Experienced (CE) marks as a part of measurement of congestion, but in this case, loss MUST also be measured to provide a complete view of the level of congestion. For tunnels, [CONGESTION-FEEDBACK] describes a way to measure both loss and ECN-marking. These measurements could be used on a relatively short timescale to drive a congestion control response and/or aggregated over a longer timescale with a higher trigger threshold to drive a CB. Subsequent bullet items in this section discuss the necessity of using a longer timescale and a higher trigger threshold. 🧠
- The measurement period used by a CB Measurement function MUST be longer than the time that current Congestion Control algorithms need to reduce their rate following detection of congestion. This is important because end-to-end Congestion Control algorithms require at least one RTT to notify and adjust the traffic when congestion is experienced, and congestion bottlenecks can share traffic with a diverse range of end-to-end RTTs. The measurement period is therefore expected to be significantly longer than the RTT experienced by the CB itself. ⏳
- If necessary, a CB MAY combine successive individual meter samples from the ingress and egress to ensure observation of an average measurement over a sufficiently long interval. (Note when meter samples need to be combined, the combination needs to reflect the sum of the individual sample counts divided by the total time/volume over which the samples were measured. Individual samples over different intervals cannot be directly combined to generate an average value.) ➗
- A CB MUST be constructed so that it does not trigger under light or intermittent congestion (see requirements 7-9). 🙅♀️
- A CB is REQUIRED to define a threshold to determine whether the measured congestion is considered excessive. 🚦
- A CB is REQUIRED to define the triggering interval, defining the period over which the trigger uses the collected measurements. CBs need to trigger over a sufficiently long period to avoid additionally penalizing flows with a long path RTT (e.g., many path RTTs). 🕰️
- A CB MUST be robust to multiple congestion events. This usually will define a number of measured persistent congestion events per triggering period. For example, a CB MAY combine the results of several measurement periods to determine if the CB is triggered (e.g., it is triggered when persistent excessive congestion is detected in three of the measurements within the triggering interval when more than three measurements were collected). 🛡️
- The normal reaction to a trigger SHOULD disable all traffic that contributed to congestion (otherwise, see requirements 11 and 12). 🚫
- The reaction MUST be much more severe than that of a Congestion Control algorithm (such as TCP's congestion control [RFC5681] or TCP-Friendly Rate Control, TFRC [RFC5348]), because the CB reacts to more persistent congestion and operates over longer timescales (i.e., the overload condition will have persisted for a longer time before the CB is triggered). 💥
- A reaction that results in a reduction SHOULD result in reducing the traffic by at least an order of magnitude. A response that achieves the reduction by terminating flows, rather than randomly dropping packets, will often be more desirable to users of the service. A CB that reduces the rate of a flow, MUST continue to monitor the level of congestion and MUST further react to reduce the rate if the CB is again triggered. 📉
- The reaction to a triggered CB MUST continue for a period that is at least the triggering interval. Operator intervention will usually be required to restore a flow. If an automated response is needed to reset the trigger, then this needs to not be immediate. The design of an automated reset mechanism needs to be sufficiently conservative that it does not adversely interact with other mechanisms (including other CB algorithms that control traffic over a common path). It SHOULD NOT perform an automated reset when there is evidence of continued congestion. 🧯
- A CB trigger SHOULD be regarded as an abnormal network event. As such, this event SHOULD be logged. The measurements that lead to triggering of the CB SHOULD also be logged. 📝
- The control communication needs to carry measurements (requirement 1) and, in some uses, also needs to transmit trigger messages to the ingress. This control communication may be in or out of band. The use of in-band communication is RECOMMENDED when either design would be possible. The preferred CB design is one that triggers when it fails to receive measurement reports that indicate an absence of congestion, in contrast to relying on the successful transmission of a "congested" signal back to the sender. (The feedback signal could itself be lost under congestion.) 📡
- In Band: An in-band control method SHOULD assume that loss of control messages is an indication of potential congestion on the path, and repeated loss ought to cause the CB to be triggered. This design has the advantage that it provides fate-sharing of the traffic flow(s) and the control communications. This fate-sharing property is weaker when some or all of the measured traffic is sent using a path that differs from the path taken by the control traffic (e.g., where traffic and control messages follow a different path due to use of equal-cost multipath routing, traffic engineering, or tunnels for specific types of traffic). 🧵
- Out of Band: An out-of-band control method SHOULD NOT trigger a CB reaction when there is loss of control messages (e.g., a loss of measurements). This avoids failure amplification/propagation when the measurement and data paths fail independently. A failure of an out-of-band communication path SHOULD be regarded as an abnormal network event and be handled as appropriate for the network. For example, this event SHOULD be logged, and additional network operator action might be appropriate, depending on the network and the traffic involved. 🛟
- The control communication MUST be designed to be robust to packet loss. A control message can be lost if there is a failure of the communication path used for the control messages, loss is likely also to be experienced during congestion/overload. This does not imply that it is desirable to provide reliable delivery (e.g., over TCP), since this can incur additional delay in responding to congestion. Appropriate mechanisms could be to duplicate control messages to provide increased robustness to loss and/or to regard a lack of control traffic as an indication that excessive congestion could be being experienced [RFC8085]. If control message traffic is sent over a shared path, it is RECOMMENDED that this control traffic is prioritized to reduce the probability of loss under congestion. Control traffic also needs to be considered when provisioning a network that uses a CB. 🧦
- There are security requirements for the control communication between endpoints and/or network devices (Section 7). The authenticity of the source and integrity of the control messages (measurements and triggers) MUST be protected from off-path attacks. When there is a risk of an on-path attack, a cryptographic authentication mechanism for all control/measurement messages is RECOMMENDED. 🔐
5. Examples of CBs
There are multiple types of CB that could be defined for use in different deployment cases. There could be cases where a flow becomes controlled by multiple CBs (e.g., when the traffic of an end-to-end flow is carried in a tunnel within the network). This section provides examples of different types of CB. 🎀
5.1. A Fast-Trip CB
[RFC2309] discusses the dangers of congestion unresponsive flows and states that "all UDP-based streaming applications should incorporate effective congestion avoidance mechanisms." Some applications do not use a full-featured transport (TCP, SCTP, DCCP). These applications (e.g., using UDP and its UDP-Lite variant) need to provide appropriate congestion avoidance. Guidance for applications that do not use congestion-controlled transports is provided in [RFC8085]. Such mechanisms can be designed to react on much shorter timescales than a CB, that only observes a traffic envelope. Congestion control methods can also interact with an application to more effectively control its sending rate. 🎬
A Fast-trip CB is the most responsive form of CB. It has a response time that is only slightly larger than that of the traffic that it controls. It is suited to traffic with well-understood characteristics (and could include one or more trigger functions specifically tailored to the type of traffic for which it is designed). It is not suited to arbitrary network traffic and could be unsuitable for traffic aggregates, since it could prematurely trigger (e.g., when the combined traffic from multiple congestion-controlled flows leads to short-term overload). ⚡
Although the mechanisms can be implemented in RTP-aware network devices, these mechanisms are also suitable for implementation in endpoints (e.g., as a part of the transport system) where they can also complement end-to-end congestion control methods. A shorter response time enables these mechanisms to trigger before other forms of CB (e.g., CBs operating on traffic aggregates at a point along the network path). 🏃♀️💨
5.1.1. A Fast-Trip CB for RTP ✨
Okay so, a bunch of Fast-Trip CB methods have been specced to work together for an RTP flow using the RTP/AVP Profile [RFC8083] 💖. The vibe is that without severe congestion, every RTP app running on best-effort IP networks should be able to run without tripping these CBs 🫶. So an RTP Fast-Trip CB is basically a fail-safe: when it fires, it kills the RTP traffic 😭.
The sending endpoint keeps an eye on the in-band RTP Control Protocol (RTCP) reception report blocks, the ones riding inside sender report (SR) or receiver report (RR) packets, since those carry the reception quality feedback 📊. It uses that info to measure (congestion) loss, possibly together with ECN [RFC6679] 🦋.
The CB action (shutting down the flow) triggers when any of these trigger conditions are true:
- The reports show a reported lack of progress 🐢. An RTP CB triggers on that.
- No receiver reports messages are showing up at all 📭. An RTP CB triggers when that happens.
- The long-term RTP throughput (measured over many RTTs) goes past a hard upper limit. That limit comes from a method that's giving TCP-Friendly Rate Control (TFRC) vibes 🏎️.
- An RTP CB has the notion of Media Usability 🎀. It triggers when the quality of the transported media drops below some required minimum acceptable quality.
5.2. A Slow-Trip CB ✨
A Slow-Trip CB could be implemented in an endpoint or a network device. This type is much slower at reacting to congestion than a Fast-Trip CB 🐌. It's expected to be the more common one.
One example where a Slow-Trip CB is needed is when flows or traffic-aggregates use a tunnel or encapsulation, and the flows inside the tunnel don't all support TCP-style congestion control (e.g., TCP, SCTP, TFRC). See [RFC8085], Section 3.1.3. The big use case is tunnels deployed in the general Internet (not in "controlled environments" inside an Internet service provider or enterprise network), especially when the tunnel might need to cross a customer access router 🏠.
5.3. A Managed CB ✨
A managed CB lives in the signaling protocol or the management plane that's related to the traffic aggregate being controlled 🧚. This type is typically a good fit when the deployment is inside a "controlled environment" 💅.
A CB needs more than just the ability to figure out that a network path is forwarding data, or to measure the path's rate -- those are often normal network operational functions 🙄. It also needs a way to pin down a metric for congestion on the path, and to trigger a reaction when a threshold gets crossed that signals persistent excessive congestion 🚨.
The control messages can use either in-band or out-of-band communications 📡.
5.3.1. A Managed CB for SAToP Pseudowires 🍓
Section 8 of [RFC4553], SAToP Pseudowire Emulation Edge-to-Edge (PWE3), describes an example of a managed CB for isochronous flows ⏱️.
If flows like these ran over a pre-provisioned (e.g., Multiprotocol Label Switching, MPLS) infrastructure, the PW could be expected to not experience congestion, because a flow isn't expected to increase (or decrease) its rate 💤. But if PW traffic gets multiplexed with other traffic over the general Internet, it could experience congestion 😬. [RFC4553] says: "If SAToP PWs run over a PSN providing best-effort service, they SHOULD monitor packet loss in order to detect 'severe congestion'." The currently recommended measurement period is 1 second, and the trigger goes off when there are more than three measured Severely Errored Seconds (SES) within a period. [RFC4553] goes on to say that "If such a condition is detected, a SAToP PW ought to shut down bi-directionally for some period of time..." 🛑
The idea was that when the packet-loss ratio (congestion) level went above a threshold, the PW was disabled by default. This use case looked at fixed-rate transmission, where the PW had no reasonable way to shed load 🧱.
The trigger needs to be set at a rate where the PW is likely to be having a serious problem, one that could make the service noncompliant 📉. At that point, triggering the CB removes the traffic so it doesn't cause undue impact on congestion-responsive traffic (e.g., TCP) 🛡️. Part of the reasoning was that high loss ratios usually mean something is "broken" and should already have led to operator intervention, so this trigger is what now needs to kick that intervention off 🔔.
An operator-based response to a triggered CB gives people a chance to take other action to restore service quality (e.g., by shedding other loads or assigning additional capacity), or to consciously avoid reacting to the trigger while they engineer a real solution 🛠️. This could mean the trigger function has to send a control message to a third location (e.g., a network operations center, NOC) that's in charge of running the tunnel ingress, rather than the tunnel ingress itself 📞.
5.3.2. A Managed CB for Pseudowires (PWs) 🌸
Pseudowires (PWs) [RFC3985] have become a super common way to tunnel traffic, and they could end up competing for network resources both with other PWs and with non-PW traffic, like TCP/IP flows 🥊.
[RFC7893] talks about the congestion conditions that can come up when PWs compete with elastic (i.e., congestion responsive) network traffic (e.g., TCP traffic). Elastic PWs carrying IP traffic (see [RFC4448]) aren't a huge worry, because all the traffic involved responds and backs off its transmission rate when congestion is detected 🙌.
Inelastic PWs are the drama 🎭. Think fixed-bandwidth Time Division Multiplex (TDM [RFC4553] [RFC5086] [RFC5087]). They can harm congestion-responsive traffic or add to excessive congestion, because inelastic PWs don't adjust their transmission rate in response to congestion 😤. [RFC7893] analyses TDM PWs and comes to an initial conclusion: a TDM PW running with a level of loss that could cause congestion-related problems is also running with a level of loss that makes the TDM service unacceptable 📉. So the document suggests that a managed CB that shuts down a PW when it persistently fails to deliver acceptable TDM service is a useful way to deal with these congestion concerns 💡. (See Appendix A of [RFC7893] for further discussion.)
6. Examples in Which CBs May Not Be Needed ✨
A CB is not required for a single congestion-controlled flow using TCP, SCTP, TFRC, etc. 🙅♀️ In these cases, the congestion control methods are already designed to prevent persistent excessive congestion, so they've got it handled 💅.
6.1. CBs over Pre-provisioned Capacity 🎀
One super common question is whether a CB is needed when a tunnel is deployed in a private network with pre-provisioned capacity 🤔.
Here's the deal: compliant traffic that doesn't exceed the provisioned capacity ought not to cause persistent congestion. So a CB will only trigger when there's noncompliant traffic 🚦. Some folks argue this event ought never to happen, but others argue the CB equally ought never to be triggered 🤷♀️. If a CB were implemented, it will provide an appropriate response if persistent congestion shows up in an operational network 🛟.
Implementing a CB won't reduce the performance of the flows. But if persistent excessive congestion does happen, it protects the network traffic sharing capacity with these flows 🛡️. It also protects network traffic from a failure where CB traffic gets (re)routed and piles extra load onto a non-pre-provisioned path 🚧.
6.2. CBs with Tunnels Carrying Congestion-Controlled Traffic 🦋
IP-based traffic is generally assumed to be congestion controlled, i.e., the transport protocols generating it at the sender already use mechanisms that are sufficient to handle congestion on the path 🙆♀️. So the question pops up when people deploy a tunnel that's thought to carry only an aggregate of TCP traffic (or traffic using some other congestion control method): is there an advantage to using a CB here? 🤨
TCP (and SCTP) traffic in a tunnel is expected to reduce its transmission rate when network congestion is detected 📉. Other transports (e.g., using UDP) can use mechanisms that are sufficient to handle congestion on the path [RFC8085]. But here's the tea ☕: even if each flow sharing a tunnel has its own congestion control and backs off on its own, the overall traffic from the aggregate of those flows doesn't necessarily avoid persistent congestion 😳. For instance, most congestion control mechanisms need long-lived flows to react and reduce a flow's rate. An aggregate of many short flows could have a bunch of flows terminate before they ever experience congestion 💨.
It's also often impossible for a tunnel service provider to know that the tunnel only has congestion-controlled traffic (e.g., inspecting packet headers might not be possible) 🙈. Some IP-based applications might not implement adequate mechanisms to address congestion 😒. The important thing to note is that if the aggregate of the traffic doesn't cause persistent excessive congestion (impacting other flows), then the CB won't trigger 🥳. That's the expected case here, so implementing a CB ought not to reduce the tunnel's performance. But if persistent excessive congestion does happen, the CB protects other network traffic sharing capacity with the tunnel traffic 🫶.
6.3. CBs with Unidirectional Traffic and No Control Path 🌈
A one-way forwarding path could have no associated communication path for sending control messages, so it can't be controlled using a CB (compare with Section 3.2.3) 🙃.
A one-way service could be provided over a path with dedicated pre-provisioned capacity that isn't shared with other elastic Internet flows (i.e., flows that vary their rate) 👑. A forwarding path could also be shared with other flows. One way to lessen the impact of this traffic on the other flows is to manage the traffic envelope with ingress policing 🚪. Supporting this type of traffic in the general Internet requires operator monitoring to detect and respond to persistent excessive congestion 👀.
7. Security Considerations ✨
Okay bestie, every CB mechanism runs on teamwork 🫶. The ingress and egress meters have to coordinate, and they have to talk to the trigger function too. That usually means passing network-control information (aka protocol messages) across the network. Timing is everything here ⏰. The measurement period decides how fast the CB reacts. If the receiver's interval is overly long, the CB gets sluggish and can't detect or react to congestion well 😭. If the interval is too short, the CB might trigger prematurely, leaving other mechanisms no time to act, and that can cause unnecessary disruption to the service 💀.
A CB could be exploited by an attacker to mount a Denial-of-Service (DoS) attack on the traffic it's controlling 🙈. So mechanisms need to be put in place to stop attacks on the network-control info that would end up causing DoS.
The authenticity of the source and the integrity of the control messages (measurements and triggers) MUST be protected from off-path attacks. Without that protection, it could be super easy for an attacker to inject fake or modified control/measurement messages (say, ones claiming high packet loss rates) that make a CB trigger, which means they've basically DoS'd a flow 😤.
Simple protection can come from a randomized source port, or from an equivalent field in the packet header that an off-path attacker isn't expected to know, like the RTP SSRC value and the RTP sequence number 🎀. Stronger protection comes from a secure authentication protocol 🔐.
Attacking control messages is pretty easy for an attacker on the control path when the messages are neither encrypted nor authenticated. Use of a cryptographic authentication mechanism for all control/measurement messages is RECOMMENDED to address this, and it would also protect against off-path attacks ✨. There's a design trade-off here between the cost of adding cryptographic security to control messages and the desire to protect control communication. For some deployment scenarios, the value of extra protection from DoS attacks will therefore lead to a requirement to authenticate all control messages 💜.
Sending network-control messages uses up network capacity 🍓. This control traffic needs to be considered when designing a CB, and it could add to network congestion. If it's sent over a shared path, it is RECOMMENDED that this control traffic be prioritized so it's less likely to get lost under congestion 🚦. Control traffic also needs to be considered when provisioning a network that uses a CB 🧚.
The CB MUST be designed to be robust to packet loss, which can also be experienced during congestion/overload 🌈. Loss of control messages could be a side-effect of a congested network, but it could also come from other causes Section 4.
The security implications depend on the design of the mechanisms, the type of traffic being controlled, and the intended deployment scenario. So each design of a CB MUST evaluate whether that particular CB mechanism has new security implications 👑.
8. References
8.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<http://www.rfc-editor.org/info/rfc3168>.
[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
March 2017, <http://www.rfc-editor.org/info/rfc8085>.8.2. Informative References
[CONGESTION-FEEDBACK]
Wei, X., Zhu, L., and L. Deng, "Tunnel Congestion
Feedback", Work in Progress,
draft-ietf-tsvwg-tunnel-congestion-feedback-04,
January 2017.
[Jacobson88]
Jacobson, V., "Congestion Avoidance and Control", SIGCOMM
Symposium proceedings on Communications architectures
and protocols, August 1988.
[RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5,
RFC 1112, DOI 10.17487/RFC1112, August 1989,
<http://www.rfc-editor.org/info/rfc1112>.
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering,
S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G.,
Partridge, C., Peterson, L., Ramakrishnan, K., Shenker,
S., Wroclawski, J., and L. Zhang, "Recommendations on
Queue Management and Congestion Avoidance in the
Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998,
<http://www.rfc-editor.org/info/rfc2309>.
[RFC2914] Floyd, S., "Congestion Control Principles", BCP 41,
RFC 2914, DOI 10.17487/RFC2914, September 2000,
<http://www.rfc-editor.org/info/rfc2914>.
[RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
Edge-to-Edge (PWE3) Architecture", RFC 3985,
DOI 10.17487/RFC3985, March 2005,
<http://www.rfc-editor.org/info/rfc3985>.
[RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron,
"Encapsulation Methods for Transport of Ethernet over MPLS
Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006,
<http://www.rfc-editor.org/info/rfc4448>.
[RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure-
Agnostic Time Division Multiplexing (TDM) over Packet
(SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006,
<http://www.rfc-editor.org/info/rfc4553>.
[RFC5086] Vainshtein, A., Ed., Sasson, I., Metz, E., Frost, T., and
P. Pate, "Structure-Aware Time Division Multiplexed (TDM)
Circuit Emulation Service over Packet Switched Network
(CESoPSN)", RFC 5086, DOI 10.17487/RFC5086, December 2007,
<http://www.rfc-editor.org/info/rfc5086>.
[RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi,
"Time Division Multiplexing over IP (TDMoIP)", RFC 5087,
DOI 10.17487/RFC5087, December 2007,
<http://www.rfc-editor.org/info/rfc5087>.
[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification",
RFC 5348, DOI 10.17487/RFC5348, September 2008,
<http://www.rfc-editor.org/info/rfc5348>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<http://www.rfc-editor.org/info/rfc5681>.
[RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P.,
and K. Carlberg, "Explicit Congestion Notification (ECN)
for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August
2012, <http://www.rfc-editor.org/info/rfc6679>.
[RFC7761] Fenner, B., Handley, M., Holbrook, H., Kouvelas, I.,
Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent
Multicast - Sparse Mode (PIM-SM): Protocol Specification
(Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March
2016, <http://www.rfc-editor.org/info/rfc7761>.
[RFC7893] Stein, Y(J)., Black, D., and B. Briscoe, "Pseudowire
Congestion Considerations", RFC 7893,
DOI 10.17487/RFC7893, June 2016,
<http://www.rfc-editor.org/info/rfc7893>.
[RFC8083] Perkins, C. and V. Singh, "Multimedia Congestion Control:
Circuit Breakers for Unicast RTP Sessions", RFC 8083,
DOI 10.17487/RFC8083, March 2017,
<http://www.rfc-editor.org/info/rfc8083>.Acknowledgments
There are so many people who've discussed and described the issues that motivated this document 🌸. Contributions and comments came from Lars Eggert, Colin Perkins, David Black, Matt Mathis, Andrew McGregor, Bob Briscoe, and Eliot Lear 🫶. This work was partly funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700) 💖.
Author's Address
Godred Fairhurst
University of Aberdeen
School of Engineering
Fraser Noble Building
Aberdeen, Scotland AB24 3UE
United Kingdom
Email: gorry@erg.abdn.ac.uk
URI: http://www.erg.abdn.ac.uk📦 where this page comes from
- ietf: rfc8084 2017-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms