RFC 3168
The Addition of Explicit Congestion Notification (ECN) to IP
RFC 3168: The Addition of Explicit Congestion Notification (ECN) to IP 🎀
Okay bestie, let's start with the ID card ✨. This is RFC 3168, from the Network Working Group, written by K. Ramakrishnan (TeraOptic Networks), S. Floyd (ACIRI), and D. Black (EMC). It's dated September 2001 and sits in the Standards Track category 💜. It updates RFC 2474, RFC 2401, and RFC 793, and it obsoletes RFC 2481 🫶.
Status of this Memo ✨
This document specifies an Internet standards track protocol for the Internet community, and it's asking everyone for discussion and suggestions for improvements 💖. For the standardization state and status of this protocol, go check the current edition of "Internet Official Protocol Standards" (STD 1) 📚. Distribution of this memo is unlimited 🌸.
Copyright Notice
Copyright (C) The Internet Society (2001). All Rights Reserved. 🔥
Abstract
Okay so the whole point of this memo is bringing ECN (Explicit Congestion Notification) into TCP and IP 🦋. The big highlight is that ECN uses two bits in the IP header, and that's the glow-up that makes it all work 💅.
Table of Contents
1. Introduction.................................................. 3
2. Conventions and Acronyms...................................... 5
3. Assumptions and General Principles............................ 5
4. Active Queue Management (AQM)................................. 6
5. Explicit Congestion Notification in IP........................ 6
5.1. ECN as an Indication of Persistent Congestion............... 10
5.2. Dropped or Corrupted Packets................................ 11
5.3. Fragmentation............................................... 11
6. Support from the Transport Protocol........................... 12
6.1. TCP......................................................... 13
6.1.1 TCP Initialization......................................... 14
6.1.1.1. Middlebox Issues........................................ 16
6.1.1.2. Robust TCP Initialization with an Echoed Reserved Field. 17
6.1.2. The TCP Sender............................................ 18
6.1.3. The TCP Receiver.......................................... 19
6.1.4. Congestion on the ACK-path................................ 20
6.1.5. Retransmitted TCP packets................................. 20
6.1.6. TCP Window Probes......................................... 22
7. Non-compliance by the End Nodes............................... 22
8. Non-compliance in the Network................................. 24
8.1. Complications Introduced by Split Paths..................... 25
9. Encapsulated Packets.......................................... 25
9.1. IP packets encapsulated in IP............................... 25
9.1.1. The Limited-functionality and Full-functionality Options.. 27
9.1.2. Changes to the ECN Field within an IP Tunnel.............. 28
9.2. IPsec Tunnels............................................... 29
9.2.1. Negotiation between Tunnel Endpoints...................... 31
9.2.1.1. ECN Tunnel Security Association Database Field.......... 32
9.2.1.2. ECN Tunnel Security Association Attribute............... 32
9.2.1.3. Changes to IPsec Tunnel Header Processing............... 33
9.2.2. Changes to the ECN Field within an IPsec Tunnel........... 35
9.2.3. Comments for IPsec Support................................ 35
9.3. IP packets encapsulated in non-IP Packet Headers............ 36
10. Issues Raised by Monitoring and Policing Devices............. 36
11. Evaluations of ECN........................................... 37
11.1. Related Work Evaluating ECN................................ 37
11.2. A Discussion of the ECN nonce.............................. 37
11.2.1. The Incremental Deployment of ECT(1) in Routers.......... 38
12. Summary of changes required in IP and TCP.................... 38
13. Conclusions.................................................. 40
14. Acknowledgements............................................. 41
15. References................................................... 41
16. Security Considerations...................................... 45
17. IPv4 Header Checksum Recalculation........................... 45
18. Possible Changes to the ECN Field in the Network............. 45
18.1. Possible Changes to the IP Header.......................... 46
18.1.1. Erasing the Congestion Indication........................ 46
18.1.2. Falsely Reporting Congestion............................. 47
18.1.3. Disabling ECN-Capability................................. 47
18.1.4. Falsely Indicating ECN-Capability........................ 47
18.2. Information carried in the Transport Header................ 48
18.3. Split Paths................................................ 49
19. Implications of Subverting End-to-End Congestion Control..... 50
19.1. Implications for the Network and for Competing Flows....... 50
19.2. Implications for the Subverted Flow........................ 53
19.3. Non-ECN-Based Methods of Subverting End-to-end Congestion
Control.................................................... 54
20. The Motivation for the ECT Codepoints........................ 54
20.1. The Motivation for an ECT Codepoint........................ 54
20.2. The Motivation for two ECT Codepoints...................... 55
21. Why use Two Bits in the IP Header?........................... 57
22. Historical Definitions for the IPv4 TOS Octet................ 58
23. IANA Considerations.......................................... 60
23.1. IPv4 TOS Byte and IPv6 Traffic Class Octet................. 60
23.2. TCP Header Flags........................................... 61
23.3. IPSEC Security Association Attributes....................... 62
24. Authors' Addresses........................................... 62
25. Full Copyright Statement..................................... 631. Introduction ✨
Let's start with how TCP has always used packet drops as its signal that something's congested 💀. Then comes the plot twist 🌈: once active queue management (e.g., RED) shows up in the Internet infrastructure, routers can detect congestion before the queue overflows. That means routers aren't stuck using drops as the only signal anymore 👑. Instead, routers can set the Congestion Experienced (CE) codepoint in the IP header of packets from ECN-capable transports 🦋. We spell out when the CE codepoint gets set in routers, and we describe the changes TCP needs in order to be ECN-capable 💖. Other transport protocols (e.g., unreliable unicast or multicast, reliable multicast, other reliable unicast transport protocols) could get modifications too, as those protocols are developed and move through the standards process 🌸. We also cover the issues that come up when ECN is used inside IP tunnels, and inside IPsec tunnels in particular 🧚.
One of the guiding principles here is that, as far as possible, the mechanisms in this document should be incrementally deployable 🍓. The tricky part is that some IP tunnels already existed that weren't compatible with ECN 🙈. As ECN gets deployed, those non-compatible IP tunnels will have to be upgraded to conform to this document 😭.
This document obsoletes RFC 2481, "A Proposal to add Explicit Congestion Notification (ECN) to IP", which defined ECN as an Experimental Protocol for the Internet Community 💅. It also updates a few older RFCs. It updates RFC 2474, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", because it defines the ECN field in the IP header ✨. It updates RFC 2401, "Security Architecture for the Internet Protocol", to change how the IPv4 TOS Byte and IPv6 Traffic Class Octet are handled in tunnel mode header construction, so they're compatible with ECN 🔥. And it updates RFC 793, "Transmission Control Protocol", by defining two new flags in the TCP header 👑.
TCP's congestion control and avoidance algorithms are built on the idea that the network is a "black-box" [Jacobson88, Jacobson90] 📦. End systems figure out the network's state by probing: they keep gradually increasing the load (by growing the window of packets outstanding in the network) until the network gets congested and a packet is lost 🎢. Treating the network as a black box, and treating loss as the sign of congestion, is fine for pure best-effort data carried by TCP, which doesn't care much about the delay or loss of individual packets 🫶. TCP's congestion management also has tricks built in (like Fast Retransmit and Fast Recovery) to minimize the throughput impact of losses ⭐. But these mechanisms aren't meant to help applications that really are sensitive to the delay or loss of one or more individual packets 🥺. Interactive traffic like telnet, web-browsing, and audio and video transfer can be hurt by packet losses (especially with an unreliable data delivery transport like UDP), or by the extra latency that comes from retransmitting a packet after a loss (with the reliable delivery semantics TCP provides) 😩.
Since TCP finds its congestion window by gradually growing the window size until it hits a dropped packet, the queues at the bottleneck router end up building up 📈. With most packet drop policies at routers that don't care about the load each individual flow puts on the network (e.g., tail-drop on queue overflow), some packets from latency-sensitive flows can get dropped too 💔. On top of that, drop policies like these cause loss to synchronize across multiple flows 🌀.
Active queue management mechanisms detect congestion before the queue overflows, and they give end nodes an indication of that congestion 🔔. So active queue management can cut down on unnecessary queuing delay for all the traffic sharing that queue 🚀. The advantages of active queue management are discussed in RFC 2309 [RFC2309] 📖. It also dodges some of the nasty side effects of dropping on queue overflow, including that undesirable synchronization of loss across multiple flows 🙌. Even better, active queue management means transport protocols with congestion control mechanisms (e.g., TCP) don't have to rely on buffer overflow as the only indication of congestion 🏆.
Active queue management mechanisms can indicate congestion to end nodes in several ways. One way is packet drops, which is what happens today 🗑️. But active queue management lets the router separate its policies for queuing or dropping packets from its policies for indicating congestion 🧠. So routers can use the Congestion Experienced (CE) codepoint in a packet header as the congestion signal, instead of relying only on packet drops 🎯. This has the potential to reduce the impact of loss on latency-sensitive flows 💜.
Now the drama 🙈: there are some middleboxes (firewalls, load balancers, or intrusion detection systems) out on the Internet that either drop a TCP SYN packet configured to negotiate ECN, or respond with a RST 😤. This document specifies procedures that TCP implementations may use to keep connectivity robust even when that equipment is in the way 🛡️✨.
2. Conventions and Acronyms 📖
okay so the keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL 🫶 when they show up in this doc, they're meant to be read the way [RFC2119] describes. bestie, don't skip this part, it's the whole vibe-check for the rest of the spec ✨
3. Assumptions and General Principles 🧠
okay so this section is where the authors spill the design tea 🍵 these are the big principles and assumptions that shaped every choice they made in this proposal 💅
- ECN is probably gonna roll out slowly, so being able to migrate is literally essential 😭 some routers will still just drop packets to say "hey, we're congested" 🙈 and some end-systems won't be ECN-capable at all. the most viable move is one that works with gradual deployment, so we don't end up with "islands" of ECN-capable and non-ECN-capable environments 🏝️
- new congestion control and avoidance mechanisms need to co-exist and cooperate with the existing ones 🤝 specifically, they have to get along with TCP's current ways of adapting to congestion, AND with routers' current habit of dropping packets when things get busy 💀
- congestion can stick around on different time-scales 🕰️ the authors are focused on congestion events that last longer than a round-trip time ✨
- a flow (like a TCP connection, or a UDP exchange) can have anywhere from a handful of packets to a LOT of them 📦 they care most about managing congestion from flows that send enough packets to still be active when network feedback gets back to them 🔥
- asymmetric routing is expected to be totally normal on the Internet 🌍 the path (the sequence of links and routers) that data packets take can be different from the path their acknowledgment packets take on the way back 🦋
- lots of routers handle the "regular" headers in IP packets way more efficiently than the header info in IP options 🚀 so the idea is to keep congestion experienced information in the regular headers of an IP packet 💜
- not every end-system is gonna cooperate with congestion control mechanisms, and that's a real thing 🙃 but new mechanisms shouldn't make it easier for TCP applications to turn off TCP congestion control 🛑 and lying about taking part in stuff like ECN-capability shouldn't really pay off 🌸
4. Active Queue Management (AQM) 🪣
okay so Random Early Detection (RED) is one mechanism for Active Queue Management (AQM) that was proposed to spot incipient congestion [FJ93], and it's currently being deployed on the Internet [RFC2309] 🌈 AQM is supposed to be a general mechanism that can use one of several alternatives for signalling congestion 💅 but without ECN, AQM is stuck using packet drops as its only way to indicate congestion 😭 it drops packets once the average queue length goes over a threshold, instead of only when the queue overflows 📈 and because AQM can drop packets before the queue actually overflows, memory limits don't always force it to discard the packet 🧚
here's the iconic part ✨ AQM can set a Congestion Experienced (CE) codepoint in the packet header instead of dropping the packet, as long as that field exists in the IP header and the transport protocol understands it 🎀 using the CE codepoint with ECN lets the receiver(s) actually get the packet, which avoids the excessive delays you'd get from retransmissions after packet losses 🫶 we call a packet with the CE codepoint set a 'CE packet' 🦋
5. Explicit Congestion Notification in IP ✨
Okay bestie, here's the big idea 💖. The Internet gets to send a heads-up about incipient congestion (the "things are about to get crowded" moment, like RED and earlier work [RJ90] 🦋). Sometimes that heads-up comes from marking a packet instead of dropping it. 🌸
The way it works is an ECN field in the IP header with two bits, which gives four codepoints, '00' to '11'. The ECN-Capable Transport (ECT) codepoints '10' and '01' get set by the data sender to say "hey, my transport endpoints are ECN-capable" 💅. We call them ECT(0) and ECT(1). When this document says "the ECT codepoint," it means either of the two. Routers treat ECT(0) and ECT(1) as equivalent, and senders can use either one on a packet-by-packet basis. 🎀
Why have two ECT codepoints? 🙈 It's mainly so the data sender can check that network elements aren't erasing the CE codepoint, and that data receivers are properly telling the sender about packets with CE set, as the transport protocol requires. Guidelines for senders and receivers to tell ECT(0) and ECT(1) apart will show up in separate documents, one for each transport protocol. This document does not cover how TCP end-nodes tell the two apart. Protocols and senders that only need a single ECT codepoint SHOULD use ECT(0). ✨
The not-ECT codepoint '00' means the packet isn't using ECN at all. The CE codepoint '11' is what a router sets to tell the end nodes about congestion. If a router has a packet show up at a full queue, it drops that packet, same as it would without ECN. 💀
+-----+-----+
| ECN FIELD |
+-----+-----+
ECT CE [Obsolete] RFC 2481 names for the ECN bits.
0 0 Not-ECT
0 1 ECT(1)
1 0 ECT(0)
1 1 CEFigure 1: The ECN Field in IP.
Using two ECT codepoints basically gives you a one-bit ECN nonce in the packet headers, and routers necessarily "erase" the nonce when they set the CE codepoint [SCWA99] 🧚. Routers that erased the CE codepoint would have a harder time rebuilding the original nonce, so repeatedly erasing CE would be more likely to get caught by the end-nodes. The ECN nonce can also help with the problem of misbehaving transport receivers lying to the transport sender about whether CE was set on a packet. The reasons for using two ECT codepoints get a deeper dive in Section 20, including some alternatives for the fourth ECT codepoint (that's the codepoint '01'). Backwards compatibility with earlier ECN implementations that don't understand ECT(1) is covered in Section 11. 🌈
In RFC 2481 [RFC2481], the ECN field was split into an ECN-Capable Transport (ECT) bit and a CE bit. An ECN field with only the ECT bit set in RFC 2481 matches the ECT(0) codepoint in this document. An ECN field with both the ECT and CE bits set in RFC 2481 matches the CE codepoint here. RFC 2481 left '01' undefined, which is the reason for recommending ECT(0) when only one ECT codepoint is needed. 👑
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| DS FIELD, DSCP | ECN FIELD |
+-----+-----+-----+-----+-----+-----+-----+-----+
DSCP: differentiated services codepoint
ECN: Explicit Congestion NotificationFigure 2: The Differentiated Services and ECN Fields in IP.
Bits 6 and 7 in the IPv4 TOS octet are designated as the ECN field. The IPv4 TOS octet lines up with the Traffic Class octet in IPv6, and the ECN field is defined the same way in both. The definitions of the IPv4 TOS octet [RFC791] and the IPv6 Traffic Class octet have been superseded by the six-bit DS (Differentiated Services) Field [RFC2474, RFC2780]. [RFC2474] lists bits 6 and 7 as Currently Unused, and RFC 2780 approves them for experimental use for ECN. Section 22 has a short history of the TOS octet. 🍓
Because the TOS octet has had a messy history, the ECN field as specified here can't be guaranteed to be backwards compatible with those past uses of the two bits that happened before ECN existed. The risks of that lack of backwards compatibility are discussed in Section 22. ⭐
Here's the rule that matters most 🚨: when an ECN-Capable transport receives a single CE packet, the congestion control algorithms at the end-systems MUST be essentially the same as the response to a single dropped packet. For example, for ECN-Capable TCP, the source TCP is required to halve its congestion window for any window of data that contains either a packet drop or an ECN indication. 😭
The whole point is that a CE mark and a drop should feel the same to the sender, so the response has to match.
Why does the CE response need to match the drop response? Partly so ECN can be rolled out gradually, across both end-systems and routers. Some routers might drop ECN-Capable packets (for example, using the same AQM policies for congestion detection), while other routers set CE for the same level of congestion. A router might also drop a non-ECN-Capable packet but set CE on an ECN-Capable packet, for equivalent levels of congestion. If CE got a different congestion response than a drop, flows could end up getting unfairly different treatment. 🎀
There's another goal too: end-systems should react to congestion at most once per window of data (in other words, at most once per round-trip time). That way they don't overreact to several congestion signals that arrive within one round-trip time. 💅
For a router, the CE codepoint of an ECN-Capable packet SHOULD only be set if the router would otherwise have dropped the packet to signal congestion to the end nodes. When the router's buffer isn't full yet, and the router is ready to drop a packet to tell end nodes about incipient congestion, it should first check whether the ECT codepoint is set in that packet's IP header. If it is, then instead of dropping the packet, the router MAY set the CE codepoint in the IP header instead. 🦋
If every end node were ECN-Capable, new criteria could be developed for setting the CE codepoint, along with new congestion control mechanisms for how end nodes react to CE packets. That's a research question, though, so this document doesn't tackle it. 🧪
When a router receives a CE packet (a packet that already has the CE codepoint set), it leaves the CE codepoint alone and transmits the packet as usual. When severe congestion has happened and the router's queue is full, the router has no choice but to drop some packet when a new one arrives. We expect those packet losses to get pretty rare once most end-systems are ECN-Capable and take part in TCP or other compatible congestion control mechanisms. In an adequately provisioned ECN-Capable environment, packet losses should mostly happen during transients or when non-cooperating sources are involved. 🌸
The guidance above about when CE may be set instead of dropping a packet applies by default to all Differentiated Services Per-Hop Behaviors (PHBs) [RFC 2475]. PHB specifications MAY give more detail on how a compliant implementation should choose between setting CE and dropping a packet, but that's NOT REQUIRED. A router MUST NOT set CE instead of dropping a packet when the drop is caused by something other than congestion or by the desire to signal incipient congestion to end nodes. For example, a diffserv edge node might be configured to unconditionally drop certain classes of traffic so they never enter its diffserv domain. 🚫
We expect routers to set the CE codepoint in response to incipient congestion as shown by the average queue size, using the RED algorithms suggested in [FJ93, RFC2309]. As far as we know, this is the only proposal currently under discussion in the IETF for routers to drop packets proactively, before the buffer overflows. This document doesn't try to specify a particular active queue management mechanism, and leaves that job, if it's needed, to other areas of the IETF. ECN is tightly tied to the need for a reasonable active queue management mechanism at the router, but the reverse isn't true. Active queue management mechanisms were developed and deployed independently of ECN, using packet drops as congestion indications when ECN isn't in the IP architecture. 🧠
5.1. ECN as an Indication of Persistent Congestion
Let's be super clear about this one ✨: a single packet with the CE codepoint set causes the transport layer to respond, in terms of congestion control, the same way it would to a packet drop. The instantaneous queue size can swing around a lot even when the router isn't dealing with persistent congestion. So it's important that transient congestion at a router, which shows up as the instantaneous queue size hitting a threshold much smaller than the queue's capacity, does not trigger a reaction at the transport layer. That's why the CE codepoint should not be set by a router based on the instantaneous queue size. 🙈
Take ATM and Frame Relay: their congestion indication mechanisms were typically defined without any notion of average queue size as the basis for deciding an intermediate node is congested. So we think they give a very noisy signal 📉. The TCP-sender reaction specified in this document for ECN is not the right response to a noisy congestion signal like that. However, if the routers that interface to the ATM network can keep track of the average queue at the interface, and use it to reliably decide the ATM subnet is congested, they may use the ECN notification defined here. 🫶
We still encourage experiments with layer 2 techniques (for example, in ATM switches or Frame Relay switches) to take advantage of ECN 🔥. For instance, using a scheme like RED (where packet marking is based on the average queue length exceeding a threshold), layer 2 devices could give a reasonably reliable indication of congestion. When all the layer 2 devices along a path set their own layer's Congestion Experienced codepoint in this reliable way (such as the EFCI bit for ATM, or the FECN bit in Frame Relay), the interface router to the layer 2 network could copy that layer 2 Congestion Experienced state into the CE codepoint in the IP header. We know this isn't current practice, and it isn't in current standards either. Still, encouraging this kind of experimentation may provide the information needed to evolve existing layer 2 mechanisms into more reliable congestion indicators, when they use a single bit to signal congestion. 🌈
5.2. Dropped or Corrupted Packets ✨
Okay bestie, for the ECN use this doc is about (a transport like TCP, where a dropped data packet means "hey, congestion!" 🦋), end nodes have to actually notice when data packets get dropped. Their congestion response to a drop must be at least as strong as their response to a received CE packet 💅. To keep the CE signal reliable, an ECT codepoint MUST NOT be set in a packet unless losing that packet in the network would be detected by the end nodes and read as congestion. No ghosting allowed 🙈.
TCP doesn't catch every drop, though 😭. Take a "pure" ACK packet. TCP doesn't slow down the arrival rate of later ACKs just because an earlier ACK got dropped. So any plan to make those packets ECN-Capable would need to handle stuff like an ACK that got CE-marked and then dropped later in the network. This is still research territory, so for now this doc says "pure" ACK packets MUST NOT indicate ECN-Capability 🌸.
Corrupted CE packets are a different vibe. If a CE packet gets dropped later in the network because of corruption (bit errors), end nodes should still invoke congestion control, just like TCP does today when a data packet gets dropped 🫶. Any proposal where the network tells corruption drops apart from congestion or buffer overflow drops would have to think about this. Honestly, ECN being everywhere wouldn't, on its own, be enough for end nodes to decide that a drop means corruption instead of congestion 💀.
5.3. Fragmentation 🧩
ECN-capable packets MAY have the DF (Don't Fragment) bit set 🎀. Reassembly of a fragmented packet MUST NOT lose indications of congestion. Put another way, if any fragment of an IP packet being reassembled has the CE codepoint set, then one of two actions MUST be taken:
- Set the CE codepoint on the reassembled packet. However, this MUST NOT occur if any of the other fragments contributing to this reassembly carries the Not-ECT codepoint.
- The packet is dropped, instead of being reassembled, for any other reason.
If both actions apply, either MAY be chosen 🦋. Reassembly of a fragmented packet MUST NOT change the ECN codepoint when all of the fragments carry the same codepoint.
Side note, because RFC 2481 didn't specify reassembly behavior, older ECN implementations that follow that Experimental RFC don't always reassemble correctly. They might not keep the CE codepoint in a fragment 😬. The sender can dodge this whole mess by setting the DF bit in ECN-Capable packets 💖.
Sometimes the reassembly rules above aren't precise enough. For example, if there's a malicious or broken entity in the path at or after the fragmentation point, fragments could end up carrying a mix of ECT(0), ECT(1), and/or Not-ECT codepoints 👀. The reassembly spec above doesn't say anything about reassembling fragments in that case. In situations where more precise reassembly behavior is needed, protocol specifications SHOULD instead specify that DF MUST be set in all ECN-capable packets sent by the protocol 🔥.
6. Support from the Transport Protocol 🌈
ECN needs support from the transport protocol too, on top of what the ECN field in the IP packet header gives you 💅. First, the transport protocol might need negotiation between the endpoints during setup to confirm that all of them are ECN-capable, so the sender can set the ECT codepoint in the packets it transmits. Second, the transport protocol must be able to react properly when CE packets arrive. That reaction could be the data receiver telling the data sender about the CE packet (like TCP does), the data receiver unsubscribing from a layered multicast group (like RLM [MJV96]), or some other action that ultimately slows the arrival rate of that flow on that congested link ✨. CE packets indicate persistent rather than transient congestion (see Section 5.1), so reactions to CE packets should be the ones that fit persistent congestion 🧚.
This doc only adds ECN Capability to TCP and leaves ECN in other transport protocols for future research 📚. For TCP, ECN needs three new pieces of functionality:
- negotiation between the endpoints during connection setup to determine if they are both ECN-capable;
- an ECN-Echo (ECE) flag in the TCP header so the data receiver can tell the data sender when a CE packet has been received;
- a Congestion Window Reduced (CWR) flag in the TCP header so the data sender can tell the data receiver that the congestion window has been reduced.
Other transport protocols will probably need different support, especially unreliable or reliable multicast transport protocols. That will have to be figured out as those protocols come to the IETF for standardization 🫶.
Quick terminology moment: in a mild abuse of terminology, this doc says TCP packets instead of TCP segments 🙈.
6.1. TCP 🎀
The sections below go into detail on the proposed use of ECN in TCP. This proposal is written up in essentially the same form in [Floyd94]. We assume the source TCP uses the standard congestion control algorithms of Slow-start, Fast Retransmit and Fast Recovery [RFC2581] 💜.
This proposal specifies two new flags in the Reserved field of the TCP header. The TCP mechanism for negotiating ECN-Capability uses the ECN-Echo (ECE) flag in the TCP header. Bit 9 in the Reserved field of the TCP header is designated as the ECN-Echo flag. The 6-bit Reserved field's location in the TCP header is shown in Figure 4 of RFC 793 [RFC793] (and reproduced below for completeness). This spec of the ECN Field leaves the Reserved field as a 4-bit field using bits 4-7 🍓.
To help the TCP receiver figure out when to stop setting the ECN-Echo flag, we add a second new flag to the TCP header: the CWR flag. The CWR flag is assigned to Bit 8 in the Reserved field of the TCP header 🌸.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | U | A | P | R | S | F |
| Header Length | Reserved | R | C | S | S | Y | I |
| | | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
Figure 3: The old definition of bytes 13 and 14 of the TCP
header.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | C | E | U | A | P | R | S | F |
| Header Length | Reserved | W | C | R | C | S | S | Y | I |
| | | R | E | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
Figure 4: The new definition of bytes 13 and 14 of the TCP
Header.So ECN uses the ECT and CE flags in the IP header (as shown in Figure 1) for signaling between routers and connection endpoints, and uses the ECN-Echo and CWR flags in the TCP header (as shown in Figure 4) for TCP-endpoint-to-TCP-endpoint signaling. For a TCP connection, a typical sequence of events in an ECN-based reaction to congestion goes like this 🔥:
- An ECT codepoint is set in packets transmitted by the sender to indicate that ECN is supported by the transport entities for these packets.
- An ECN-capable router detects impending congestion and detects that an ECT codepoint is set in the packet it is about to drop. Instead of dropping the packet, the router chooses to set the CE codepoint in the IP header and forwards the packet.
- The receiver receives the packet with the CE codepoint set, and sets the ECN-Echo flag in its next TCP ACK sent to the sender.
- The sender receives the TCP ACK with ECN-Echo set, and reacts to the congestion as if a packet had been dropped.
- The sender sets the CWR flag in the TCP header of the next packet sent to the receiver to acknowledge its receipt of and reaction to the ECN-Echo flag.
The negotiation for using ECN by the TCP transport entities, and the use of the ECN-Echo and CWR flags, are described in more detail in the sections below 🧚.
6.1.1. TCP Initialization ✨
Okay so, during the TCP connection setup, the two TCPs (source and destination) swap notes on whether they're willing to use ECN 🎀. Once that negotiation wraps up, the TCP sender sets an ECT codepoint in the IP header of its data packets. That's basically saying "hey network, I'm capable and willing to do ECN for this packet" 💖. It tells the routers they may mark that packet with the CE codepoint if they want to use it for congestion notification 🦋. If the connection doesn't want ECN for a particular packet, the sender sets the codepoint to not-ECT, and the receiving TCP just ignores any CE codepoint it gets 🙈.
For the rest of this section, we call the initiating host Host A and the responding host Host B 🫶. A SYN packet with both the ECE and CWR flags set is an "ECN-setup SYN packet" ✨. A SYN packet with at least one of those flags unset is a "non-ECN-setup SYN packet" 💅. For SYN-ACK packets, it's the same vibe: only ECE set (and CWR not set) makes an "ECN-setup SYN-ACK packet", and any other combo of those two flags is a "non-ECN-setup SYN-ACK packet" 🌸.
Before a connection can use ECN, Host A sends an ECN-setup SYN packet, and Host B sends back an ECN-setup SYN-ACK packet 🧚. Setting both ECE and CWR on the SYN is NOT about congestion. It's literally just a way of saying "I'm ECN-Capable" 👑. An ECN-setup SYN means that host will participate in ECN as both a sender and a receiver. As a receiver, it'll set ECE on its outgoing ACKs whenever it sees a CE codepoint in incoming data packets 🔥. As a sender, it'll shrink its congestion window when it sees ECE, and set CWR when that makes sense 😭. Sending an ECN-setup SYN doesn't commit anyone to setting ECT on any of its packets, though 🙈. And even if a later SYN in the same connection has no ECE or CWR, the host is still on the hook to respond properly to CE-marked packets 💜.
When Host B sends an ECN-setup SYN-ACK, it sets ECE but NOT CWR 🌈. That's the "I'm ECN-Capable too" signal ✨. Same deal as before, though: the SYN-ACK doesn't commit Host B to setting ECT on its packets 💅.
Here are the rules for sending ECN-setup packets inside a TCP connection. A connection is defined by the usual TCP establishment and termination rules, so nothing new there 🎀:
- If a host has received an ECN-setup SYN packet, it MAY send an ECN-setup SYN-ACK packet. If it hasn't, it MUST NOT send an ECN-setup SYN-ACK packet 💖.
- A host MUST NOT set ECT on data packets unless it has both sent and received at least one ECN-setup SYN or ECN-setup SYN-ACK packet, AND it has never sent a non-ECN-setup SYN or non-ECN-setup SYN-ACK packet 🦋. If a host has received at least one non-ECN-setup SYN or non-ECN-setup SYN-ACK packet, it SHOULD NOT set ECT on data packets 🙈.
- If a host ever sets the ECT codepoint on a data packet, it MUST correctly set or clear the
CWRbit on every packet after that in the connection 🫶. - If a host has sent at least one ECN-setup SYN or ECN-setup SYN-ACK packet, and has received no non-ECN-setup SYN or non-ECN-setup SYN-ACK packet, then any TCP data packets it receives with both ECT and CE set in the IP header MUST be processed as specified for an ECN-capable connection 🔥.
- A host that doesn't want to use ECN on a TCP connection SHOULD clear both
ECEandCWRin every non-ECN-setup SYN and/or SYN-ACK packet it sends, so everyone knows it's a no-go 🌸. Receivers MUST correctly handle all forms of non-ECN-setup SYN and SYN-ACK packets 👑. - A host MUST NOT set ECT on SYN or SYN-ACK packets 💅.
One more thing: a TCP client goes into TIME-WAIT after receiving a FIN-ACK, then moves to CLOSED after a timeout ⏳. Lots of TCP implementations will start a brand new connection if they get an in-window SYN during TIME-WAIT 🧚. So when a host enters TIME-WAIT or CLOSED, it should ignore any earlier state about ECN negotiation for that connection 💜.
6.1.1.1. Middlebox Issues 🙈
ECN uses the ECN-Echo and CWR flags in the TCP header (see Figure 3 for the layout) during initialization 🎀. Unfortunately, some faulty firewalls, load balancers, and intrusion detection systems out there either drop an ECN-setup SYN packet or reply with a RST 😭. They think those flag bits look like a port-scanning tool's signature, one that could be used in a denial-of-service attack 💀. Some of that equipment has been identified, and a web page [FIXES] lists the non-compliant products along with vendor fixes, where they exist 🛠️. The TBIT web page [TBIT] lists some of the web servers affected by this faulty gear 🌸. We're mentioning all this as a heads-up to the community, because this is a real problem 🫶.
To keep connectivity robust even with that broken equipment around, a host that gets a RST in response to an ECN-setup SYN MAY resend a SYN with CWR and ECE cleared 🦋. That can end up with a TCP connection that gets established without ECN at all 🍓.
A host that gets no reply to an ECN-setup SYN within the normal SYN retransmission timeout interval MAY resend the SYN, and any SYN retransmissions after that, with CWR and ECE cleared ⏱️. Normal packet loss can drop the original SYN, though, so the originating host may send one or more ECN-setup SYN packets before giving up and retransmitting the SYN with CWR and ECE cleared 💖.
Here's a scenario that's totally possible in this case 👀:
(1) Host A: Sends an ECN-setup SYN.
(2) Host B: Sends an ECN-setup SYN/ACK, packet is dropped or delayed.
(3) Host A: Sends a non-ECN-setup SYN.
(4) Host B: Sends a non-ECN-setup SYN/ACK.Following the procedures above, in that situation neither Host A nor Host B may set the ECT bit on data packets 🙅♀️. Also, an important consequence of the ECN setup and usage rules in Section 6.1.1 is that a host is forbidden from treating the arrival of ECT data packets as an implicit sign that the other host is ECN-capable 🚫. Don't assume, bestie 💅.
6.1.1.2. Robust TCP Initialization with an Echoed Reserved Field 🪞
So why does the SYN sender set two ECN-related flags in the Reserved field of the TCP header, while the SYN-ACK sender only sets one? 🤔 It's because that asymmetry is needed to negotiate ECN-capability robustly with some TCP implementations already out in the wild 🌍. There's at least one faulty TCP implementation where receivers copy the Reserved field from the received data packet into their ACK packets (and therefore into the SYN-ACK) 🙃. Since the SYN sets both the ECN-Echo and CWR flags to say "ECN-capable", but the SYN-ACK only sets the ECN-Echo flag, the sender can tell that a receiver just reflecting its own flags back means "not ECN-capable" 🧠. So a faulty implementation that echoes the incoming SYN's Reserved field doesn't misled the sender into thinking ECN is on 🙅♀️✨.
6.1.2. The TCP Sender ✨
Okay bestie, so for a TCP connection using ECN, every new data packet goes out with an ECT codepoint set in the IP header 🦋. If the sender only needs one ECT codepoint for all the packets on that connection, ECT(0) SHOULD be used 💖. When the sender gets an ECN-Echo (ECE) ACK packet (an ACK with the ECN-Echo flag set in the TCP header 🫶), it knows congestion was hit somewhere on the path from sender to receiver 😭. That congestion signal should be treated exactly like a congestion loss in non-ECN-Capable TCP. So the TCP source halves the congestion window "cwnd" and lowers the slow start threshold "ssthresh" 💅. And the sending TCP SHOULD NOT increase the congestion window when it gets an ECN-Echo ACK packet 🌸.
TCP should not react to congestion indications more than once per window of data (or, more loosely, more than once per round-trip time) 🧚. Basically, the sender's congestion window should only get cut once for a whole pile of dropped and/or CE packets from a single window of data. Also, the TCP source should not decrease ssthresh if it already got lowered within the last round trip time ⏳. The exception is if any retransmitted packets get dropped. Then the source TCP treats that as a brand-new instance of congestion 🚨.
Now here's the tea ☕. After the source TCP cuts its congestion window because of a CE packet, the ACKs that keep arriving can "clock out" new packets, as long as the smaller window allows it 🎀. If the congestion window is only one MSS (maximum segment size) and the sending TCP gets an ECN-Echo ACK packet, the sending TCP should in principle still cut that window in half 🙈. The catch is the window can't drop below one MSS. Sending with a one-MSS window still means one packet per round-trip time, and that's not slow enough. So when the window is one and an ECN-Echo arrives, the sender still has to slow down even further 🐢. We use the retransmit timer to do that. That's why the sending TCP MUST reset the retransmit timer when it receives the ECN-Echo packet while the congestion window is one 💜. After that, the sending TCP can only send a new packet once the retransmit timer expires ⏰.
When an ECN-Capable TCP sender reduces its congestion window for any reason (a retransmit timeout, a Fast Retransmit, or an ECN Notification), it sets the CWR flag in the TCP header of the first new data packet it sends after the reduction 🍓. If that data packet gets dropped in the network, the sending TCP has to reduce the congestion window again and retransmit the dropped packet 😤.
Here's why the "Congestion Window Reduced" info reliably reaches the TCP receiver 🔐. If the new data packet carrying the CWR flag gets dropped, the TCP sender has to cut its congestion window again and send another new data packet with the CWR flag set. So the CWR bit in the TCP header SHOULD NOT be set on retransmitted packets 🌈.
When the TCP data sender is ready to set the CWR bit after reducing the congestion window, it SHOULD set that bit only on the first new data packet it transmits ✅.
[Floyd94] goes deeper into TCP's response to ECN, and [Floyd98] covers the validation test in the ns simulator, which shows a wide range of ECN scenarios 🤓. Those include: an ECN followed by another ECN, a Fast Retransmit, or a Retransmit Timeout; a Retransmit Timeout or a Fast Retransmit followed by an ECN; and a congestion window of one packet followed by an ECN. Literally so many plot twists 😭.
TCP keeps using the existing algorithms for sending data packets in response to incoming ACKs, multiple duplicate acknowledgments, or retransmit timeouts [RFC2581] 💫. It also follows the normal procedures for growing the congestion window when it receives ACK packets without the ECN-Echo bit set [RFC2581] 📈.
6.1.3. The TCP Receiver 🎧
When TCP gets a CE data packet at the destination end-system, the TCP data receiver sets the ECN-Echo flag in the TCP header of the next ACK packet it sends 📨. If there's any ACK withholding going on, like in current "delayed-ACK" TCP implementations where the receiver can send one ACK for two arriving data packets, then the ECN-Echo flag in that ACK gets set to '1' if the CE codepoint is set on any of the data packets being acknowledged 👀. So if even one of the received data packets is CE, the returning ACK has the ECN-Echo flag set. No cap 🧢.
To make sure a dropped ACK carrying an ECN-Echo flag doesn't ruin everything 😩, the TCP receiver sets the ECN-Echo flag on a whole series of ACK packets it sends afterward 🔁. The receiver uses the CWR flag it gets from the TCP sender to figure out when to stop setting ECN-Echo 🛑.
After a TCP receiver sends an ACK packet with the ECN-Echo bit set, it keeps setting the ECN-Echo flag on every ACK it sends (whether those ACKs cover CE data packets or non-CE data packets) until it receives a CWR packet (a packet with the CWR flag set) 🎯. Once that CWR packet arrives, ACKs for later non-CE data packets don't have the ECN-Echo flag set anymore. If the data receiver gets another CE packet, it starts sending ACK packets with the ECN-Echo flag set again 🔄. Now, receiving a CWR packet doesn't guarantee the data sender actually got the ECN-Echo message 🤷♀️. It does suggest that the data sender cut its congestion window at some point after it sent the data packet that had the CE codepoint set 💌.
We already said a TCP sender isn't required to reduce its congestion window more than once per window of data 🙃. Some care is needed if the sender wants to avoid pointless window cuts when a window of data has both dropped packets and (marked) CE packets 🫠. [Floyd98] shows how that plays out.
6.1.4. Congestion on the ACK-path 🛣️
For the current generation of TCP congestion control algorithms, pure acknowledgement packets (packets that don't carry any accompanying data) MUST be sent with the not-ECT codepoint 🙅♀️. Current TCP receivers have no mechanisms for cutting traffic on the ACK-path in response to congestion notification 🥲. Ways of responding to congestion on the ACK-path are still open areas of research, both now and in the future 🔭. (One simple idea would be for the sender to reduce its congestion window when it gets a pure ACK packet with the CE codepoint set 💡.) For current TCP implementations, though, a single dropped ACK generally has only a very small effect on the TCP's sending rate 🌷.
6.1.5. Retransmitted TCP packets ✨
okay bestie, this part is SO important 😭 ECN-capable TCP implementations MUST NOT set either ECT codepoint (ECT(0) or ECT(1)) in the IP header for retransmitted data packets. The TCP data receiver SHOULD ignore the ECN field on arriving data packets that are outside its current window. The point is extra protection against denial-of-service attacks 🛡️, plus keeping the ECN congestion indication robust for packets that get dropped later in the network 💅
First, if the TCP sender set an ECT codepoint on a retransmitted packet, and that packet was unnecessarily retransmitted and then dropped later in the network, the end nodes would never get the congestion indication from the router that set the CE codepoint 💀 So putting an ECT codepoint on retransmits just doesn't line up with robust delivery of the congestion signal, even for packets that get dropped later.
Then there's the security side 🙈 An attacker who can spoof the TCP sender's IP source address could send data packets with random sequence numbers and the CE codepoint set in the IP header. When the receiver gets this spoofed packet, it would notice the data isn't in its current receive window and send back a duplicate acknowledgement. We define an out-of-window packet at the TCP data receiver as a data packet that lies outside the receiver's current window.
So the receiver has to decide whether to treat the CE codepoint in an out-of-window packet as a valid congestion indication, which decides whether it sends ECN-Echo indications back to the sender. If it ignores CE on an out-of-window packet, the sender might miss a possibly legitimate congestion signal from the network, and that breaks end-to-end congestion control 😩 But if the receiver honors that CE and reports it, the malicious node has successfully "attacked" the connection by forcing the sender to unnecessarily cut (halve) its congestion window 🥲 To stop this denial-of-service attack, we specify that a legitimate TCP data sender MUST NOT set an ECT codepoint on retransmitted data packets, and that the TCP data receiver SHOULD ignore the CE codepoint on out-of-window packets.
One drawback of not putting ECT(0) or ECT(1) on retransmits is that retransmitted packets don't get ECN protection. Still, in a fully-ECN-capable environment with mild congestion, packets should rarely get dropped because of congestion in the first place, so retransmits should be rare too 🌸 If packets are being retransmitted, then there are already losses (from corruption or from congestion) that ECN wasn't able to prevent.
Here's the guarantee I love 🦋 If a router sets the CE codepoint on an ECN-capable data packet within a TCP connection, the connection is guaranteed to receive that congestion indication, or some other congestion indication within the same window of data, even if that packet gets dropped or reordered in the network. We look at two cases: when the packet is later retransmitted, and when it isn't.
In the first case, the packet gets dropped or delayed, and at some point the sender retransmits it. That retransmission happens because of a Fast Retransmit or a Retransmit Timeout, for either that packet or some earlier packet in the same window of data. Since the sender already retransmitted this packet, it has already responded to a congestion indication for some packet in the same window. So even if the first copy gets dropped or delayed, and it had CE set but the receiver ignores it as out-of-window, that's totally fine ✨ The sender already responded to congestion for that window.
In the second case, the packet is never retransmitted. That means this data packet is the only copy the receiver gets, so it arrives as an in-window packet no matter how much it's delayed or reordered 🫶 In that situation, if CE got set on the packet inside the network, the receiver treats it as a valid congestion indication.
6.1.6. TCP Window Probes.
When the TCP data receiver advertises a zero window, the TCP data sender sends window probes to find out whether the receiver's window has opened up 👀 Window probe packets don't carry any user data, except for the sequence number, which is one byte. If a window probe gets dropped in the network, the receiver never notices that loss. So the TCP data sender MUST NOT set either an ECT codepoint or the CWR bit on window probe packets 🚫
Here's the plot twist 🎀 Window probes use exact sequence numbers, so they can't be easily spoofed in denial-of-service attacks. That means if a window probe arrives with the CE codepoint set, the receiver SHOULD respond to the ECN indications.
7. Non-compliance by the End Nodes ✨
Okay, this section is about whether ECN makes the network more vulnerable to end nodes that don't behave 🙃 Specifically, end nodes that set the ECT codepoint in their transmitted packets but don't respond to received CE packets. The big claim here is that adding ECN to the IP architecture won't meaningfully increase the architecture's existing vulnerability to unresponsive flows 💖
Even without ECN, unresponsive flows (flows that don't cut their arrival rate at a congested link when congestion control tells them to) can do serious damage 😬 For example, an end node could "turn off congestion control" by not reducing its congestion window when packets get dropped. That's already a concern on today's Internet. It's been argued that routers will have to deploy mechanisms to detect and treat packets from non-compliant flows differently [RFC2309,FF99]. People have also suggested that things like end-to-end per-flow scheduling and isolating one flow from another, differentiated services, or end-to-end reservations could take away some of the worst effects of unresponsive flows 🧚
You might think dropping packets is already enough of a deterrent for non-compliance, and that ECN removes that deterrent 🤔 Our response is that (1) ECN-capable routers keep the packet-dropping behavior during times of high congestion, and (2) even during high congestion, dropping packets alone isn't an adequate deterrent for non-compliance 💅
First, ECN-capable routers only mark packets (instead of dropping them) when the packet marking rate is reasonably low. When the average queue size goes above an upper threshold, the potential marking rate would be high, so our recommendation is that routers drop packets rather than set the CE codepoint in packet headers 🌈
During periods of low or moderate packet marking, when ECN would be deployed, dropping packets instead of marking them wouldn't really deter unresponsive flows much 🫠 For example, delay-insensitive flows using reliable delivery might actually have an incentive to increase their sending rate in response to dropped packets, rather than decrease it. Similarly, delay-sensitive flows using unreliable delivery might increase their use of FEC when the packet drop rate goes up, and then increase rather than decrease their sending rate. For these same reasons, we don't think packet dropping by itself is an effective deterrent for non-compliance, even when packet drop rates are high and all flows share the same drop rate 😭
Several methods have been proposed to identify and restrict non-compliant or unresponsive flows 🔍 Adding ECN to the network environment wouldn't make designing and deploying those mechanisms any harder. If anything, adding ECN to the architecture would make it slightly easier to identify unresponsive flows ✨ For example, in an ECN-capable environment, routers aren't limited to information about packets they dropped or packets that had the CE codepoint set at that router itself. In that environment, routers could also take note of arriving CE packets, which indicate congestion that the packet encountered earlier in its path 👑
8. Non-compliance in the Network ✨
okay bestie, this section is about what happens when a router is acting sus, maybe on purpose and maybe just broken 💀, and starts messing with either bit in the ECN field 🫣. quick PSA: in IPv4, the IP header has its own checksum that protects it from bit errors. IPv6 doesn't have that. So in IPv6, the ECN field can get flipped by bit errors on links or inside routers and nothing catches it 😭.
By tampering with the ECN bits, an adversary (or a broken router, rip) could falsely report congestion, disable ECN-Capability for one packet, erase the congestion indication, or falsely indicate ECN-Capability. Section 18 goes through every single way the ECN field could get modified 📚. The big question for judging how bad each one is: would it likely lead to worse performance in any dimension (throughput, delay, fairness or functionality) than if the router had just dropped the packet? 🤔
The first two, falsely reporting congestion or disabling ECN-Capability for one packet, are no worse than the router just dropping the packet 🙅♀️. From a congestion control point of view, a non-compliant router setting CE when there's no congestion is no worse than a router dropping a packet unnecessarily. And "erasing" an ECT codepoint on a packet that gets dropped later in the network could cause an unnecessary drop for that packet further along 💅.
But here's the plot twist 🌀. As Section 18 explains, a router that erases the ECN congestion indication or falsely indicates ECN-Capability could do more damage to the flow than if it had simply dropped the packet. Like, a rogue or broken router that "erased" the CE codepoint in arriving CE packets would stop that congestion signal from reaching downstream receivers 😱. Congestion control for that flow could fail, the network gets more congested, and then packets from that flow start getting dropped anyway as the average queue size grows at the congested gateway. Yikes, literally.
Section 19 looks at the repercussions of subverting end-to-end congestion control, either by falsely indicating ECN-Capability or by erasing the congestion indication in ECN (the CE-codepoint) 🫠. The takeaway there: subverting ECN-based congestion control may lead to potential unfairness, but that's likely no worse than an end node subverting either ECN-based or packet-based congestion control 💜.
8.1. Complications Introduced by Split Paths
If a router or other network element has access to all the packets of a flow, it can't do any more damage by altering the ECN field than it could by just dropping every packet from that flow 🙃. But sometimes a malicious or broken router only sees a subset of the packets from a flow. So the question is: can that router alter the ECN field in just that subset and do more damage to the flow than if it had dropped that subset? 🧐
Section 18 covers this in detail too, and concludes:
"It is true that the adversary that has access only to a subset of packets in an aggregate might, by subverting ECN-based congestion control, be able to deny the benefits of ECN to the other packets in the aggregate. While this is undesirable, this is not a sufficient concern to result in disabling ECN."
Undesirable, yes. A reason to turn ECN off, no 🙌.
9. Encapsulated Packets 🎁
9.1. IP packets encapsulated in IP
Encapsulating IP packet headers in tunnels is used everywhere, including IPsec and IP in IP [RFC2003] 📦. This section looks at how ECN and IP tunnels interact, and specifies two alternative solutions 🔀. It's also worth checking [RFC 2983]'s discussion of how Differentiated Services interacts with IP tunnels of various forms, because Differentiated Services uses the remaining six bits of the IP header octet that ECN uses (see Figure 2 in Section 5) 🧁.
Some IP tunnel modes work by adding a new "outer" IP header that wraps the original "inner" IP header and its packet. In lots of cases, that new outer header can be added and removed at intermediate points along a connection, so the network can set up a tunnel without needing the endpoints to participate 🫧. We call tunnels that say the outer header gets discarded at tunnel egress "simple tunnels" 🥡.
ECN uses the ECN field in the IP header to signal between routers and connection endpoints. How ECN plays with IP tunnels depends on how the ECN field is treated 🎭. In simple IP tunnels, the octet containing the ECN field is copied or mapped from the inner IP header to the outer IP header at tunnel ingress, and the outer header's copy gets discarded at tunnel egress. Now, if the outer header were simply thrown away without dealing with the ECN field, and an ECN-capable router set the CE (Congestion Experienced) codepoint on a packet in a simple IP tunnel, that indication would be discarded at egress. Gone. Congestion warning lost 💔.
So using ECN over simple IP tunnels means routers try to use the outer IP header to signal congestion to endpoints, but those congestion warnings never arrive, because the outer header is discarded at the tunnel egress point 😩. This problem came up with ECN and IPsec in tunnel mode, and RFC 2481 recommended that ECN not be used with the older simple IPsec tunnels to dodge this behavior and its consequences 🙈. When ECN becomes widely deployed, simple tunnels likely to carry ECN-capable traffic will have to change. If ECN-capable traffic goes through a simple tunnel across a congested, ECN-capable router, subsequent packets for that flow could get dropped as the average queue size grows at the congested router, as discussed in Section 8 above 📉.
From a security point of view, using ECN in the outer header of an IP tunnel might raise security concerns, because an adversary could tamper with the ECN information that propagates beyond the tunnel endpoint 🕵️♀️. Based on the analysis of these concerns and the resulting risks in Sections 18 and 19, our overall approach is to make ECN support an option for IP tunnels. That way, a tunnel can be specified or configured either to use ECN or not to use ECN in the outer header 🎛️. So in environments or tunneling protocols where the risks of using ECN are judged to outweigh its benefits, the tunnel can just not use ECN in the outer header. Then the only indication of congestion experienced at routers within the tunnel would be packet loss 🫶.
There are two viable options for ECN-capable connections over an IP tunnel, including IPsec tunnels:
- A limited-functionality option in which ECN is preserved in the inner header but disabled in the outer header. The only mechanism available for signaling congestion within the tunnel in this case is dropped packets 🥲.
- A full-functionality option that supports ECN in both the inner and outer headers, and propagates congestion warnings from nodes within the tunnel to endpoints 🚀.
Supporting these options takes varying amounts of changes to IP header processing at tunnel ingress and egress 🛠️. A small subset of those changes, enough to support only the limited-functionality option, would be sufficient to eliminate any incompatibility between ECN and IP tunnels ✨.
One goal of this document is to give guidance on the tradeoffs between the limited-functionality and full-functionality options 🍓. For a full discussion of the potential effects of an adversary's modifications of the ECN field, check out Sections 18 and 19 🦋.
9.1.1. The Limited-functionality and Full-functionality Options
Okay so the limited-functionality option 💅 for ECN in IP tunnels is basically: slap the not-ECT codepoint on the outside (encapsulating) header, no matter what the ECN field says on the inside (encapsulated) header 🎀. The inner ECN field doesn't get touched when you de-capsulate.
The downside is kind of a big deal 😭. The flow gets zero ECN support for the part of the path that's using the IP tunnel, even if the packet from the original TCP sender is ECN-Capable. So if that packet hits a congested ECN-capable router that would normally drop or mark it to tell the end nodes "hey, congestion!" 🚨, the router isn't allowed to set the CE codepoint. It has to drop the packet instead. Not ideal, bestie 💀.
The full-functionality option is the glow-up 👑. On encapsulation, you copy the inner header's ECN codepoint to the outer header if the inner one is not-ECT or ECT. If the inner one is CE, the outer one gets set to ECT(0) instead. On decapsulation, if CE is set on the outer header, CE also gets set on the inner header. Otherwise the inner ECN codepoint just stays as it was 🦋. So for full ECN support, the ingress and egress work goes like this:
- At tunnel ingress, the full-functionality option sets the ECN codepoint in the outer header. If the ECN codepoint in the inner header is not-ECT or ECT, it gets copied to the outer header. If it's CE, the outer header gets ECT(0).
- At tunnel egress, the full-functionality option sets the CE codepoint in the inner header if CE is set in the outer header. Otherwise that field of the inner header isn't changed.
With the full-functionality option, a flow can use ECN on the parts of the path that might be doing IP tunneling 🌈. The security trade-off is that the IP tunnel can't protect the flow from certain changes to the ECN bits inside the tunnel. The risks from changing the ECN bits in the IP header are explained in full in Sections 18 and 19 🫶.
- An IP tunnel MUST modify the handling of the DS field octet at IP tunnel endpoints by implementing either the limited-functionality or the full-functionality option.
- Optionally, an IP tunnel MAY enable the endpoints of an IP tunnel to negotiate the choice between the limited-functionality and the full-functionality option for ECN in the tunnel.
The bare minimum for ECN to work with IP tunnels is the limited-functionality option, which stops ECN from being turned on in the tunnel's outer header ✨. Full ECN support needs the full-functionality option. If the tunnel endpoints don't have any optional mechanism to negotiate between the two, they can have a pre-existing agreement about which one they'll support 🧚.
All IP tunnels MUST implement the limited-functionality option, and SHOULD support the full-functionality option. 💖
In addition, it is RECOMMENDED that packets with the CE codepoint in the outer header be dropped if they arrive at the tunnel egress point for a tunnel that uses the limited-functionality option. The same goes for a tunnel that uses the full-functionality option but has the not-ECT codepoint set in the inner header. This is about backwards compatibility and making sure nobody sneaks in unauthorized changes to the ECN field 🔒. The next Section (9.1.2) has more on this, so stay tuned 🍓.
9.1.2. Changes to the ECN Field within an IP Tunnel.
Here's the lowkey clever part 🧠. Having a copy of the ECN field in the inner header of an IP tunnel mode packet gives you a way to spot unauthorized changes to the ECN field in the outer header 🕵️♀️. For implementations that follow this document, comparing the ECT fields in the inner and outer headers comes down to two cases:
- If the IP tunnel uses the full-functionality option, then the not-ECT codepoint should be set in the outer header if and only if it is also set in the inner header.
- If the tunnel uses the limited-functionality option, then the not-ECT codepoint should be set in the outer header.
If a packet doesn't meet the right condition, that could be a reason to worry 😬.
Picture this: the tunnel ingress point hasn't been updated to match this document, but the tunnel egress point has been updated to support ECN 🎭. The tunnel isn't explicitly configured for the full-functionality option. Even so, the ingress point acts exactly like one that does support it. So if packets from an ECN-capable connection go through this tunnel, the ECT codepoint gets set in the outer header at ingress. Then congestion inside the tunnel can make ECN-capable routers set CE in the outer header. But since the tunnel isn't set up for the full-functionality option, the egress point expects the not-ECT codepoint in the outer header 🙈.
So when an ECN-capable tunnel egress point gets a packet with the ECT or CE codepoint in the outer header on a tunnel that isn't configured for the full-functionality option, the packet should be processed based on whether CE was set. Here's how: on a tunnel that isn't configured for the full-functionality option, it is RECOMMENDED that packets should be dropped at the egress point if CE is set in the outer header but not in the inner header, and should be forwarded otherwise ✨.
An IP tunnel can't protect against congestion indications getting erased by changing the ECN codepoint from CE to ECT 😭. Erasing congestion indications can affect the network and other flows in ways that wouldn't be possible without ECN. Also, erasure can only happen to congestion indications placed by nodes inside the tunnel. The copy of the ECN field in the inner header keeps congestion notifications from nodes upstream of the tunnel ingress, unless the inner header gets erased too 🛡️. If erasing congestion notifications is judged to be a security risk that's bigger than the congestion management benefits of ECN, then tunnels could be specified or configured to use the limited-functionality option 💜.
9.2. IPsec Tunnels
IPsec is all about secure communication over network components that can't be fully trusted, like intermediate routers 🔐. It has two operating modes, transport mode and tunnel mode, which cover a wide range of security needs and environments ✨. In transport mode, the security protocol header(s) go between the IP (IPv4 or IPv6) header and higher layer protocol headers (e.g., TCP). That means transport mode can only be used for end-to-end security on a connection 💅. Tunnel mode adds a new "outer" IP header that wraps the original, or "inner," IP header and its associated packet. The tunnel mode security headers sit between those two IP headers 🎀. Unlike transport mode, that new "outer" IP header and the tunnel mode security headers can be added and removed at intermediate points along a connection. This lets security gateways protect the vulnerable parts of a connection without the endpoints having to take part in the security protocols 🦋. An important detail is that in the original specification, the outer header gets thrown away at tunnel egress. That keeps threats based on modifying the IP header from spreading past the tunnel endpoint 🛡️. More on IPsec can be found in [RFC2401] 📚.
The IPsec protocol as originally defined in [ESP, AH] required that the inner header's ECN field not be changed by IPsec decapsulation at a tunnel egress node 🙅♀️. That would have ruled out full-functionality mode for ECN. On the flip side, it made sure that an adversary's changes to the ECN field couldn't be used for theft or denial-of-service attacks across an IPsec tunnel endpoint, since any such changes would get discarded at the endpoint 🔥.
Permitting ECN functionality in the outer header of an IPsec tunnel does raise security concerns, because an adversary could tamper with information that goes beyond the tunnel endpoint 😬. An analysis of these concerns and the risks is included in Sections 18 and 19. Based on that analysis, the overall approach has been to provide configuration support for IPsec changes that remove the conflict with ECN 🧩.
In particular, in tunnel mode the IPsec tunnel MUST support the limited-functionality option outlined in Section 9.1.1, and SHOULD support the full-functionality option outlined in Section 9.1.1. 💖
This makes permission to use ECN functionality in the outer header of an IPsec tunnel a configurable part of the corresponding IPsec Security Association (SA). That means it can be turned off when the risks seem to outweigh the benefits ⚖️. So an IPsec security administrator gets two choices for how ECN-capable connections behave inside an IPsec tunnel: the limited-functionality alternative and the full-functionality alternative described earlier 🌸.
This document also specifies how the endpoints of an IPsec tunnel could negotiate enabling ECN functionality in the outer headers of that tunnel, based on security policy 🤝. Being able to negotiate ECN usage between tunnel endpoints lets a security administrator turn ECN off in situations where she thinks the risks (e.g., of lost congestion notifications) are bigger than the benefits of ECN 👑.
The IPsec protocol, as defined in [ESP, AH], doesn't include the IP header's ECN field in any of its cryptographic calculations. In tunnel mode, the outer IP header's ECN field isn't included either 🙈. So a network node changing the ECN field has no effect on IPsec's end-to-end security, because it can't make any IPsec integrity check fail. That also means IPsec doesn't defend against an adversary changing the ECN field (a man-in-the-middle attack), since the adversary's change won't affect IPsec's end-to-end security either 😮💨. In some environments, being able to change the ECN field without affecting IPsec integrity checks could count as a covert channel 🕳️. If it's necessary to get rid of that channel or cut its bandwidth, the IPsec tunnel should be run in limited-functionality mode 🌈.
9.2.1. Negotiation between Tunnel Endpoints 💖
Okay bestie, this part is about how the two tunnel endpoints actually agree on using ECN over IPsec tunnels 🎀 It's supported by three changes to IPsec:
- An optional Security Association Database (SAD) field that says whether tunnel encapsulation and decapsulation processing allows or forbids ECN usage in the outer IP header. 🧚
- An optional Security Association Attribute that lets the two endpoints of a tunnel-mode SA negotiate that SAD field. 🤝
- Changes to tunnel mode encapsulation and decapsulation processing so ECN usage in the outer IP header is allowed or forbidden based on the SAD field. When it's allowed, the ECT codepoint is set in the outer header for ECN-capable connections, and congestion notifications (the CE codepoint) from those connections get passed on to the inner header at tunnel egress. 🦋
If you implement ECN usage negotiation, then the SAD field SHOULD also be implemented. 💅 Negotiation itself is OPTIONAL in every case, even if you support the SAD field. The encapsulation and decapsulation processing changes are REQUIRED, but they MAY be implemented without the other two changes, as long as you assume ECN usage is always forbidden. 🫶 The full-functionality option for ECN over IPsec tunnels is the SAD field plus the full version of the encapsulation and decapsulation changes, with or without the OPTIONAL negotiation support. The limited-functionality option is just a subset of those changes, and it always forbids ECN usage. 💀
We'll break these down in the next three subsections ✨
9.2.1.1. ECN Tunnel Security Association Database Field
Full ECN functionality adds a new field to the SAD (see [RFC2401]) 🌸
ECN Tunnel: allowed or forbidden.This field says whether ECN-capable connections using this SA in tunnel mode are allowed to receive ECN congestion notifications for congestion that happens inside the tunnel. The allowed value turns those notifications on. The forbidden value turns them off, so all congestion gets signalled by dropping packets. 🔥
[OPTIONAL. The value of this field SHOULD be assumed to be "forbidden" in implementations that do not support it.] 🙈
If you implement this attribute, then the SA specification in a Security Policy Database (SPD) entry MUST support a matching attribute, and that SPD attribute MUST be covered by the SPD administrative interface (currently described in Section 4.4.1 of [RFC2401]). No skipping this part, bestie 👑
9.2.1.2. ECN Tunnel Security Association Attribute
A new IPsec Security Association Attribute is defined so that ECN congestion notifications based on the outer IP header can be negotiated for IPsec tunnels (see [RFC2407]) ✨ This attribute is OPTIONAL, but implementations that support it SHOULD also support the SAD field defined in Section 9.2.1.1 🎀
Attribute Type
class value type
-------------------------------------------------
ECN Tunnel 10 BasicThe IPsec SA Attribute value 10 was allocated by IANA to indicate that the ECN Tunnel SA Attribute is being negotiated. The type of this attribute is Basic (see Section 4.5 of [RFC2407]). The Class Values are used to run the negotiation. See [RFC2407, RFC2408, RFC2409] for more on encoding formats and the requirements for negotiating this SA attribute 🧚
Class Values
ECN Tunnel
Specifies whether ECN functionality is allowed to be used with Tunnel
Encapsulation Mode. This affects tunnel encapsulation and
decapsulation processing - see Section 9.2.1.3.
RESERVED 0
Allowed 1
Forbidden 2
Values 3-61439 are reserved to IANA. Values 61440-65535 are for
private use.If nothing is specified, the default shall be assumed to be Forbidden 💖
ECN Tunnel is a brand-new SA attribute, so initiators that use it should expect to run into responders that don't understand it and therefore reject proposals containing it 😭 For backwards compatibility with those implementations, initiators SHOULD always also include a proposal without the ECN Tunnel attribute, so the responder can pick a transform or proposal that doesn't have it. RFC 2407 currently requires responders to reject all proposals if any proposal contains an unknown attribute. This requirement is expected to change so that a responder doesn't select proposals or transforms containing unknown attributes, which is a much nicer vibe 🌈
9.2.1.3. Changes to IPsec Tunnel Header Processing
For full ECN support, the encapsulation and decapsulation processing for the IPv4 TOS field and the IPv6 Traffic Class field change from what [RFC2401] specifies, to the following 🍓
<-- How Outer Hdr Relates to Inner Hdr -->
Outer Hdr at Inner Hdr at
IPv4 Encapsulator Decapsulator
Header fields: -------------------- ------------
DS Field copied from inner hdr (5) no change
ECN Field constructed (7) constructed (8)
IPv6
Header fields:
DS Field copied from inner hdr (6) no change
ECN Field constructed (7) constructed (8)(5)(6) If the packet is about to enter a domain where the DSCP value in the outer header isn't appropriate, that value MUST be mapped to an appropriate value for that domain [RFC 2474]. Also see [RFC 2475] for further information. 🧠
(7) If the ECN Tunnel field in the SAD entry for this SA is "allowed" and the ECN field in the inner header is set to anything other than CE, copy that ECN field to the outer header. If the inner ECN field is set to CE, then set the outer ECN field to ECT(0). ✨
(8) If the ECN tunnel field in the SAD entry for this SA is "allowed," the ECN field in the inner header is set to ECT(0) or ECT(1), and the ECN field in the outer header is set to CE, then copy the ECN field from the outer header to the inner header. Otherwise, leave the ECN field in the inner header alone. 🦋
(5) and (6) are identical to match usage in [RFC2401], although they are different in [RFC2401]. (Yes, that's the actual text, so I'm keeping it 😅)
The description above applies to implementations that support the ECN Tunnel field in the SAD. Those implementations MUST use this processing instead of the processing of the IPv4 TOS octet and IPv6 Traffic Class octet defined in [RFC2401]. This is the full-functionality option for ECN usage with IPsec tunnels 👑
An implementation that doesn't support the ECN Tunnel field in the SAD MUST still implement this processing, by assuming the ECN Tunnel field of the SAD is "forbidden" for every SA. In that case, the processing of the ECN field boils down to:
(7) Set the ECN field to not-ECT in the outer header.
(8) Make no change to the ECN field in the inner header.This is the limited-functionality option for ECN usage with IPsec tunnels 💅
For backwards compatibility, packets with the CE codepoint set in the outer header SHOULD be dropped if they arrive on an SA that uses the limited-functionality option, or on an SA that uses the full-functionality option with the not-ECN codepoint set in the inner header 🙈
9.2.2. Changes to the ECN Field within an IPsec Tunnel.
If the ECN Field gets changed inappropriately within an IPsec tunnel, and the change is detected at the tunnel egress, then receiving a packet that doesn't satisfy the right condition for its SA counts as an auditable event 🔍 An implementation MAY create audit records with per-SA counts of incorrect packets over some time period, instead of making an audit record for every erroneous packet. Any such audit record SHOULD contain the headers from at least one erroneous packet, but it doesn't need to contain the headers from every packet the entry represents 🌸
9.2.3. Comments for IPsec Support
The IPsec working group review brought in some substantial comments on two areas of this document. This section covers those comments and explains why the proposed changes weren't put in 🧚
The first comment said that per-node configuration is easier to implement than per-SA configuration. After some serious thought (and some early encouragement of per-node configuration), that no longer seems like a good idea. The worry is that as ECN-awareness gets rolled out in IPsec, lots of ECN-aware IPsec implementations will end up talking to a mix of ECN-aware and ECN-unaware IPsec tunnel endpoints. In that situation with per-node configuration, the only sensible move is to forbid ECN usage for all IPsec tunnels, and that isn't the outcome anyone wants 😭
The second area is that several reviewers pointed out SA negotiation is complicated, and adding to it isn't trivial. One reviewer suggested using ICMP after tunnel setup as an alternative. The addition to SA negotiation in this document is OPTIONAL and will stay that way, so implementers are free to ignore it. The authors think the assurance it gives can be useful in a number of situations. In practice, if it isn't implemented, it can be deleted at a later stage in the standards process. 🫶 Extending ICMP to negotiate ECN after tunnel setup is more complex than extending SA attribute negotiation. Some tunnels don't allow traffic addressed to the tunnel egress endpoint, so the ICMP packet would have to be addressed somewhere else, then scanned for by the egress endpoint, and discarded there or at its real destination. ICMP delivery is also unreliable, so an ICMP packet could get dropped, which would mean inventing yet another ack/retransmit mechanism 🙃 It seems better to simply specify an OPTIONAL extension to the existing SA negotiation mechanism 💜
9.3. IP packets encapsulated in non-IP Packet Headers. 🌈
The issues are different, relative to ECN, when IP packets get encapsulated in tunnels with non-IP packet headers. This happens with MPLS [MPLS], GRE [GRE], L2TP [L2TP], and PPTP [PPTP]. For these protocols there's no conflict with ECN. It's just that ECN can't be used inside the tunnel unless an ECN codepoint can be specified in the header of the encapsulating protocol 🎀 Earlier work looked at a preliminary proposal for bringing ECN into MPLS, and proposals for bringing ECN into GRE, L2TP, or PPTP will be considered as the need arises 🦋
10. Issues Raised by Monitoring and Policing Devices ✨
okay so picture this: the network might get monitoring and policing devices, aka "penalty boxes" 📦😭, that keep an eye on whether best-effort flows are actually responding to congestion. they can also preferentially drop packets from flows that aren't using adequate end-to-end congestion control procedures. the penalty box is giving bouncer energy 🙅♀️
We recommend that any "penalty box" that detects a flow or an aggregate of flows that isn't responding to end-to-end congestion control first switch from marking to dropping packets from that flow, before taking any extra action to restrict the bandwidth that flow can use. so at first, the router may drop packets where it would otherwise have set the CE codepoint. that includes dropping the arriving packets for that flow that are ECN-Capable and already have the CE codepoint set 🫶. this way, any congestion indications that router sees for that flow are guaranteed to reach the end nodes too, even if there are malicious or broken routers somewhere else on the path 🔥. if we assume the first move at any "penalty box" for an ECN-capable flow is to drop packets instead of marking them, then an adversary that messes with ECN-based end-to-end congestion control can't get a flow labelled as non-cooperative and pushed into a harsher action inside the "penalty box" 💅
the real monitoring and policing devices might fall short of that ideal one 🥲. the monitoring might be applied not to a single flow, but to an aggregate of flows (for example, the ones sharing a single IPsec tunnel). in that case, switching from marking to dropping would hit every flow in the aggregate, so the other flows lose the benefits of ECN too 😤. at the highest level of aggregation, ECN gets disabled in another way even with no monitoring and policing devices around: ECN-Capable RED queues switch from marking to dropping packets as their congestion signal once the average queue size goes past some threshold 🍓
11. Evaluations of ECN 🔥
11.1. Related Work Evaluating ECN
this section is the research tour 🧚 it talks about some of the related work evaluating the use of ECN. the ECN Web Page [ECN] has pointers to other papers, plus implementations of ECN 📚
[Floyd94] looks at the pros and cons of adding ECN to the TCP/IP architecture 🤓. according to the simulation-based comparisons, one big win for ECN is avoiding unnecessary packet drops for short or delay-sensitive TCP connections. a second win is avoiding some unnecessary retransmit timeouts in TCP. this paper goes deep on how ECN fits into TCP's congestion control mechanisms. the possible downsides it discusses are that a non-compliant TCP connection could falsely advertise itself as ECN-capable, and that a TCP ACK packet carrying an ECN-Echo message could itself get dropped in the network 😬. the first issue is discussed in the appendix of this document, and the second is handled by adding the CWR flag to the TCP header 🩷
experimental evaluations of ECN include [RFC2884,K98] 🧪. the conclusions of [K98] and [RFC2884] are that ECN TCP gets moderately better throughput than non-ECN TCP 📈. ECN TCP flows are fair towards non-ECN TCP flows ✨. and ECN TCP stays robust with two-way traffic (congestion in both directions) and with multiple congested gateways 💪. experiments with lots of short web transfers show that while most short connections take about the same time with or without ECN, a small percentage of them take a very long time in the non-ECN experiments compared to the ECN experiments 🙈
11.2. A Discussion of the ECN nonce.
the two ECT codepoints, ECT(0) and ECT(1), can give you a one-bit ECN nonce in packet headers [SCWA99] 🎀. the main reason is so the data sender can verify that network elements aren't erasing the CE codepoint, and that data receivers are properly reporting the receipt of CE-marked packets back to the sender, as the transport protocol requires 🔐. this section talks about backwards compatibility with IP ECN implementations in routers that follow RFC 2481, which only defined one ECT codepoint 🦋. we don't think deploying ECN implementations that understand the ECT(1) codepoint incrementally will cause significant operational problems 🌸. that's especially true when ECT(1) deployment starts with routers, before end-nodes start using it 🍓
11.2.1. The Incremental Deployment of ECT(1) in Routers.
ECN has been an Experimental standard since January 1999, and there are already ECN implementations in routers that don't understand the ECT(1) codepoint 🙃. once ECT(1) gets standardized for TCP or other transport protocols, a data sender could be using ECT(1), but a congested router on the path might not understand that codepoint 😳.
if the transport protocol allows it, a data sender is free to skip ECT(1) entirely and send all ECN-capable packets with the codepoint ECT(0) 💜. but if an ECN-capable sender uses ECT(1) and the congested router on the path doesn't understand it, the router ends up marking some of the ECT(0) packets and dropping some of the ECT(1) packets as congestion indications 🫠. since TCP is required to react to both marked and dropped packets, this behaviour of dropping packets that could have been marked poses no significant threat to the network 🛡️. it's also consistent with the overall ECN approach, which lets routers decide when and whether to mark packets as they see fit (see Section 5) 👑
12. Summary of changes required in IP and TCP 🎀
this document specified two bits in the IP header for ECN 🦋. the not-ECT codepoint says the transport protocol will ignore the CE codepoint, and it's the default value for the ECN codepoint 🌱. the ECT codepoints say the transport protocol is willing and able to take part in ECN 🙌.
the router sets the CE codepoint to signal congestion to the end nodes. The CE codepoint in a packet header MUST NOT be reset by a router. 🚫💅
TCP needs three changes for ECN: a setup phase and two new flags in the TCP header 🩷. the ECN-Echo flag is how the data receiver tells the data sender about a received CE packet 📣. the Congestion Window Reduced (CWR) flag is how the data sender tells the data receiver that its congestion window has been reduced 💖.
when ECN (Explicit Congestion Notification) is in use, it's required that congestion indications generated within an IP tunnel don't get lost at the tunnel egress 🌊. we specified a minor change to how IP handles the ECN field during encapsulation and decapsulation, so flows that go through IP tunneling can still use ECN ✨.
two options for ECN in tunnels were specified 👯♀️:
- A limited-functionality option that doesn't use ECN inside the IP tunnel. It sets the ECN field in the outer header to
not-ECT, and doesn't alter the inner header at decapsulation time 🙈. - The full-functionality option, which sets the ECN field in the outer header to either
not-ECTor one of the ECT codepoints, depending on the ECN field in the inner header. At decapsulation, if the CE codepoint is set in the outer header and the inner header is set to one of the ECT codepoints, then the CE codepoint is copied to the inner header 🔥.
for IPsec tunnels, this document also defines an optional IPsec Security Association (SA) attribute, which enables negotiating ECN usage within IPsec tunnels 🔐. it also defines an optional field in the Security Association Database to indicate whether ECN is permitted in tunnel mode on an SA 🗂️. the required changes to IPsec tunnels for ECN usage modify RFC 2401 [RFC2401], which defines the IPsec architecture and specifies some aspects of its implementation 🧩. the new IPsec SA attribute is in addition to those already defined in Section 4.5 of [RFC2407] 💅.
this document obsoletes RFC 2481, "A Proposal to add Explicit Congestion Notification (ECN) to IP", which defined ECN as an Experimental Protocol for the Internet Community 👋. the rest of this section describes how this document relates to its predecessor 🕰️.
RFC 2481 briefly discussed using ECN with encapsulated packets. it noted that, for the IPsec specifications at the time (January 1999), flows couldn't safely use ECN if they traversed IPsec tunnels 😬. RFC 2481 also described changes that could be made to IPsec tunnel specifications to make them compatible with ECN 🛠️.
this document also builds on work done after RFC 2481 🧠. first, it describes the changes to IPsec tunnels in detail and extensively discusses the security implications of ECN (now included as Sections 18 and 19 of this document) 🛡️. second, it extends the discussion of IPsec tunnels to cover all IP tunnels. because older IP tunnels aren't compatible with a flow's use of ECN, deploying ECN across the Internet will create strong pressure for older IP tunnels to be updated to an ECN-compatible version, using either the limited-functionality or the full-functionality option 📈.
this document does not address including ECN in non-IP tunnels such as MPLS, GRE, L2TP, or PPTP 🙅♀️. an earlier preliminary document about adding ECN support to MPLS was not advanced 🥀.
a third new piece of work after RFC 2481 describes the ECN procedure for retransmitted data packets: an ECT codepoint should not be set on retransmitted data packets 🔁. the motivation for this extra specification is to close off a possible avenue for denial-of-service attacks on an existing TCP connection 🚨. some earlier deployments of ECN-capable TCP might not conform to the (new) requirement not to set an ECT codepoint on retransmitted packets, but we don't believe this will cause significant problems in practice 🤞.
this document also expands slightly on how SYN packets are used to negotiate ECN 🤝. some earlier deployments of ECN-capable TCP might not conform to the requirements in this document, but we don't believe that will lead to any performance or compatibility problems for TCP connections with a mix of TCP implementations at the endpoints 💜.
finally, this document includes the specification of the ECT(1) codepoint, which TCP may use as part of implementing an ECN nonce 🌈✨
13. Conclusions ✨
okay so bestie, AQM is getting implemented everywhere right now, which makes this the perfect moment to deploy congestion avoidance that doesn't rely on packet drops alone 💅 apps and transports are getting way more sensitive to the delay and loss of even one packet (realtime traffic, short web transfers, the whole vibe 🎀), so treating packet loss as the normal congestion signal seems insufficient. at the very least it's non-optimal 🙈
we went through what happens when the ECN field gets modified inside the network, and we checked every opportunity an adversary would have to change it 🔍 in a lot of cases, changing the ECN field is no worse than dropping a packet. ngl though, some changes are more serious because they can subvert end-to-end congestion control 💀 but even then the damage is limited, and it's similar to the threat from end-systems that deliberately don't cooperate with end-to-end congestion control. not the end of the world 🫶
14. Acknowledgements 🫶
so many people helped make this work and this document, including a bunch we couldn't manage to thank by name here 💜 we'd like to shout out Kenjiro Cho for proposing the TCP mechanism for negotiating ECN-Capability, Kevin Fall for proposing the CWR bit 🔥, Steve Blake for material on IPv4 Header Checksum Recalculation, Jamal Hadi-Salim for ECN discussions, and Steve Bellovin, Jim Bound, Brian Carpenter, Paul Ferguson, Stephen Kent, Greg Minshall, and Vern Paxson for security discussions 🔐 we also love the Internet End-to-End Research Group for the ongoing conversations 🌈
email threads with a bunch of people (Dax Kelson, Alexey Kuznetsov, Jamal Hadi-Salim, and Venkat Venkatsubra) dug into the non-conformant equipment that doesn't answer TCP SYN packets with the ECE and CWR flags set 🙃 and we thank Mark Handley, Jitentra Padhye, and others for chatting about the TCP initialization procedures 🦋
the ECN and IP tunnel stuff leans super hard on the Differentiated Services Working Group's work, so big thanks to them ✨ we also thank Tabassum Bint Haque from Dhaka, Bangladesh, for feedback on IP tunnels 🌸 and Derrell Piper and Kero Tivinen for suggesting changes to RFC 2407 that make negotiating the ECN Tunnel SA attribute way easier 👑
we thank David Wetherall, David Ely, and Neil Spring for proposing the ECN nonce 🧚 and Stefan Savage for talking through that issue with us 💅 we also thank Bob Briscoe and Jon Crowcroft for raising fragmentation in IP, alternate semantics for the fourth ECN codepoint, and a bunch of other topics 🍓 and Richard Wendland for feedback on several issues in the document ⭐
finally, the IESG and especially the Transport Area Directors over the years get a huge thank you for their feedback and their work on standardizing ECN 🎀👑
15. References 📚
ok the reference list is massive, so here it is exactly as it appears in the original. every entry, every detail, no cap 🫶
[AH] Kent, S. and R. Atkinson, "IP Authentication Header",
RFC 2402, November 1998.
[ECN] "The ECN Web Page", URL
"http://www.aciri.org/floyd/ecn.html". Reference for
informational purposes only.
[ESP] Kent, S. and R. Atkinson, "IP Encapsulating Security
Payload", RFC 2406, November 1998.
[FIXES] ECN-under-Linux Unofficial Vendor Support Page, URL
"http://gtf.org/garzik/ecn/". Reference for
informational purposes only.
[FJ93] Floyd, S., and Jacobson, V., "Random Early Detection
gateways for Congestion Avoidance", IEEE/ACM
Transactions on Networking, V.1 N.4, August 1993, p.
397-413.
[Floyd94] Floyd, S., "TCP and Explicit Congestion Notification",
ACM Computer Communication Review, V. 24 N. 5, October
1994, p. 10-23.
[Floyd98] Floyd, S., "The ECN Validation Test in the NS
Simulator", URL "http://www-mash.cs.berkeley.edu/ns/",
test tcl/test/test-all- ecn. Reference for
informational purposes only.
[FF99] Floyd, S., and Fall, K., "Promoting the Use of End-to-
End Congestion Control in the Internet", IEEE/ACM
Transactions on Networking, August 1999.
[FRED] Lin, D., and Morris, R., "Dynamics of Random Early
Detection", SIGCOMM '97, September 1997.
[GRE] Hanks, S., Li, T., Farinacci, D. and P. Traina, "Generic
Routing Encapsulation (GRE)", RFC 1701, October 1994.
[Jacobson88] V. Jacobson, "Congestion Avoidance and Control", Proc.
ACM SIGCOMM '88, pp. 314-329.
[Jacobson90] V. Jacobson, "Modified TCP Congestion Avoidance
Algorithm", Message to end2end-interest mailing list,
April 1990. URL
"ftp://ftp.ee.lbl.gov/email/vanj.90apr30.txt".
[K98] Krishnan, H., "Analyzing Explicit Congestion
Notification (ECN) benefits for TCP", Master's thesis,
UCLA, 1998. Citation for acknowledgement purposes only.
[L2TP] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn,
G. and B. Palter, "Layer Two Tunneling Protocol "L2TP"",
RFC 2661, August 1999.
[MJV96] S. McCanne, V. Jacobson, and M. Vetterli, "Receiver-
driven Layered Multicast", SIGCOMM '96, August 1996, pp.
117-130.
[MPLS] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M. and J.
McManus, Requirements for Traffic Engineering Over MPLS,
RFC 2702, September 1999.
[PPTP] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little,
W. and G. Zorn, "Point-to-Point Tunneling Protocol
(PPTP)", RFC 2637, July 1999.
[RFC791] Postel, J., "Internet Protocol", STD 5, RFC 791,
September 1981.
[RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC1141] Mallory, T. and A. Kullberg, "Incremental Updating of
the Internet Checksum", RFC 1141, January 1990.
[RFC1349] Almquist, P., "Type of Service in the Internet Protocol
Suite", RFC 1349, July 1992.
[RFC1455] Eastlake, D., "Physical Link Security Type of Service",
RFC 1455, May 1993.
[RFC1701] Hanks, S., Li, T., Farinacci, D. and P. Traina, "Generic
Routing Encapsulation (GRE)", RFC 1701, October 1994.
[RFC1702] Hanks, S., Li, T., Farinacci, D. and P. Traina, "Generic
Routing Encapsulation over IPv4 networks", RFC 1702,
October 1994.
[RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003,
October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2309] Braden, B., et al., "Recommendations on Queue Management
and Congestion Avoidance in the Internet", RFC 2309,
April 1998.
[RFC2401] Kent, S. and R. Atkinson, Security Architecture for the
Internet Protocol, RFC 2401, November 1998.
[RFC2407] Piper, D., "The Internet IP Security Domain of
Interpretation for ISAKMP", RFC 2407, November 1998.
[RFC2408] Maughan, D., Schertler, M., Schneider, M. and J. Turner,
"Internet Security Association and Key Management
Protocol (ISAKMP)", RFC 2409, November 1998.
[RFC2409] Harkins D. and D. Carrel, "The Internet Key Exchange
(IKE)", RFC 2409, November 1998.
[RFC2474] Nichols, K., Blake, S., Baker, F. and D. Black,
"Definition of the Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers", RFC 2474, December
1998.
[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.
and W. Weiss, "An Architecture for Differentiated
Services", RFC 2475, December 1998.
[RFC2481] Ramakrishnan K. and S. Floyd, "A Proposal to add
Explicit Congestion Notification (ECN) to IP", RFC 2481,
January 1999.
[RFC2581] Alman, M., Paxson, V. and W. Stevens, "TCP Congestion
Control", RFC 2581, April 1999.
[RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of
Explicit Congestion Notification (ECN) in IP Networks",
RFC 2884, July 2000.
[RFC2983] Black, D., "Differentiated Services and Tunnels",
RFC2983, October 2000.
[RFC2780] Bradner S. and V. Paxson, "IANA Allocation Guidelines
For Values In the Internet Protocol and Related
Headers", BCP 37, RFC 2780, March 2000.
[RJ90] K. K. Ramakrishnan and Raj Jain, "A Binary Feedback
Scheme for Congestion Avoidance in Computer Networks",
ACM Transactions on Computer Systems, Vol.8, No.2, pp.
158-181, May 1990.
[SCWA99] Stefan Savage, Neal Cardwell, David Wetherall, and Tom
Anderson, TCP Congestion Control with a Misbehaving
Receiver, ACM Computer Communications Review, October
1999.
[TBIT] Jitendra Padhye and Sally Floyd, "Identifying the TCP
Behavior of Web Servers", ICSI TR-01-002, February 2001.
URL "http://www.aciri.org/tbit/".16. Security Considerations 🔐
this one's short but it's SO important 😭 security considerations have been covered in Sections 7, 8, 18, and 19, so go check those too 🕵️♀️
17. IPv4 Header Checksum Recalculation 🧮
okay checksum time, and yes this is lowkey a nerdy flex 🤓 IPv4 header checksum recalculation is an issue on some high-end router architectures that use an output-buffered switch. the reason is that most (if not all) of the header manipulation happens on the input side of the switch, but the ECN decision has to be made locally at the output buffer 💥 this isn't an issue for IPv6, because IPv6 has no header checksum 🙌 the IPv4 TOS octet is also the last byte of a 16-bit half-word 🎀
RFC 1141 [RFC1141] covers incremental updating of the IPv4 checksum after the TTL field gets decremented ✨ for updating after the CE codepoint is set, here's how it works: let HC be the original header checksum for an ECT(0) packet, and let HC' be the new header checksum after the CE bit has been set. so the ECN field goes from '10' to '11' 🦋 for header checksums using one's complement subtraction, HC' is recalculated like this:
HC' = { HC - 1 HC > 1
{ 0x0000 HC = 1and for header checksums on two's complement machines, HC' is recalculated like this after the CE bit is set:
HC' = { HC - 1 HC > 0
{ 0xFFFE HC = 0you can do a similar incremental update when the ECN field changes from ECT(1) to CE, aka from '01' to '11' 💜
18. Possible Changes to the ECN Field in the Network 🕵️♀️💅
this section gets into the details of possible changes to the ECN field inside the network, like falsely reporting congestion, disabling ECN-Capability for an individual packet, erasing the ECN congestion indication, or falsely indicating ECN-Capability 😈 spicy, bestie, very spicy 🌶️
18.1. Possible Changes to the IP Header 🧾
18.1.1. Erasing the Congestion Indication ✨
Okay so first up, we're looking at changes a router could make that would basically erase the congestion indication after an upstream router already set it 🫧. The convention we're using is ECN codepoint of received packet -> ECN codepoint of packet transmitted.
Swapping the CE codepoint for ECT(0) or ECT(1) effectively erases the congestion indication. Here's the tea though: with two ECT codepoints in play, a router that erases CE has no way to know whether the original was ECT(0) or ECT(1) 🤔. So it's possible for the transport protocol to deploy mechanisms to detect these CE erasures, which is very smart of it 🕵️♀️.
The consequence for the upstream router is that congestion can build up for a while, because the indication never reaches the source. The packet still gets received and acknowledged though 💌.
The potential effect of erasing the congestion indication is complex, and we dig into it in depth in Section 19 below. Erasing is NOT the same as dropping a packet, though. When a data packet is dropped, the TCP sender detects the drop and reads it as an indication of congestion. Same vibe if enough consecutive acknowledgement packets get dropped so the cumulative acknowledgement field doesn't advance at the sender. The sender gets limited by the congestion window from sending more packets, and eventually the retransmit timer expires 😭.
A systematic erasure of the CE bit by a downstream router, on the other hand, can cause a queue buildup at an upstream router, including possible packet loss from buffer overflow 💀. There's also a potential unfairness situation: another flow going through the congested router reacts to the CE bit that got set, while the flow with the erased CE bit sees better performance. We talk more about the limits on this unfairness in Section 19 below 🙈.
The last of the three changes is to replace the CE codepoint with the not-ECT codepoint. That erases the congestion indication AND disables ECN-Capability at the same time, double whammy 💅.
Here's the thing: the erasure' of the congestion indication only works if the packet doesn't get marked or dropped again by a downstream router. If CEgets replaced by an ECT codepoint, the packet stays ECN-Capable, so a downstream router could still mark or drop it as an indication of congestion. IfCEgets replaced bynot-ECT`, the packet is no longer ECN-capable, so a downstream router can drop it but NOT mark it as an indication of congestion 🦋.
18.1.2. Falsely Reporting Congestion 🚨
This change sets the CE codepoint when an ECT codepoint was already set, even though there was no congestion. It does not change how that packet gets treated along the rest of the path. A router doesn't look at the CE codepoint when deciding whether to drop or mark an arriving packet, so no extra drama there 🌸.
Still, this could result in the application unnecessarily invoking end-to-end congestion control and slowing down its arrival rate. By itself, that's no worse (for the application or for the network) than if the tampering router had actually dropped the packet 🫶.
18.1.3. Disabling ECN-Capability 🙅♀️
This change turns off the ECT codepoint of a packet. Which means that if the packet later runs into congestion (for example, by arriving at a RED queue with a moderate average queue size), it gets dropped instead of marked 🍓. By itself, that's no worse (for the application) than if the tampering router had actually dropped the packet. The saving grace here is that there's no congested router upstream expecting a reaction from setting the CE bit. Phew ✨.
18.1.4. Falsely Indicating ECN-Capability 🎭
This change would incorrectly label a packet as ECN-Capable. The packet may have been sent either by an ECN-Capable transport or by a transport that is not ECN-Capable, and nobody can tell from the label alone 👀.
If the packet later runs into moderate congestion at an ECN-Capable router, that router could set the CE codepoint instead of dropping the packet. But if the transport protocol actually isn't ECN-Capable, it will never receive this indication of congestion, so it won't reduce its sending rate in response 😤. We discuss the potential consequences of falsely indicating ECN-capability further in Section 19 below.
If the packet never runs into congestion at an ECN-Capable router, then the first of these two changes would have no effect, other than possibly messing with the transport protocol's use of the ECN nonce 🙃. The last change, however, would give false reports of congestion to a monitoring device along the path 🚩. If the transport protocol IS ECN-Capable, this change could also have an effect at the transport level, by combining falsely indicating ECN-Capability with falsely reporting congestion. For an ECN-capable transport, that would make the transport unnecessarily react to congestion. BUT in this particular case, the router messing with the ECN field could have just dropped the packet. So for an ECN-capable transport, the consequence of this change to the ECN field is no worse than dropping the packet 🧚.
18.2. Information carried in the Transport Header 📦
For TCP, an ECN-capable TCP receiver tells its TCP peer that it's ECN-capable at the TCP level. It conveys this info in the TCP header when the connection is set up 🤝. This document doesn't consider potential dangers introduced by changes to the transport header within the network. We do note that when IPsec is used, the transport header is protected in both tunnel and transport modes [ESP, AH] 🔐.
Another issue is TCP packets with a spoofed IP source address carrying invalid ECN information in the transport header. For completeness, we look at some ways a node spoofing another node's IP source address could use the two ECN flags in the TCP header to launch a denial-of-service attack. Hmm, but these attacks would require the attacker to use valid TCP sequence numbers. Any attacker with that ability AND the ability to spoof IP source addresses could damage the TCP connection without touching the ECN flags at all. So ECN does not add any new vulnerabilities in this respect 💜.
An acknowledgement packet with a spoofed IP source address of the TCP data receiver could include the ECE bit set. If the TCP data sender accepted it as a valid packet, this spoofed acknowledgement could make the sender unnecessarily halve its congestion window 😱. BUT to be accepted, that spoofed acknowledgement would need the correct 32-bit sequence number AND a valid acknowledgement number. An attacker who could pull that off could also send a spoofed RST packet, or do other equally damaging things to the TCP connection 💀.
Packets with a spoofed IP source address of the TCP data sender could include the CWR bit set. Again, to be accepted, such a packet would need a valid sequence number. Plus, a spoofed packet like that would have a limited performance impact. It could make the TCP data receiver send fewer ECE packets than it otherwise would, if the receiver was sending ECE packets when it got the spoofed CWR packet 🌈.
18.3. Split Paths 🛤️
Sometimes a malicious or broken router might only have access to a subset of the packets from a flow. So the question is: can this router, by altering the ECN field in that subset of packets, do more damage to the flow than if it had simply dropped that set of packets? 🤨
We'll classify the packets in the flow as A packets and B packets, and assume the adversary only has access to A packets. Assume the adversary is subverting end-to-end congestion control along the path that A packets travel, either by falsely indicating ECN-Capability upstream of where congestion occurs, or by erasing the congestion indication downstream. Also consider a monitoring device that sees both A and B packets, and will "punish" both A and B packets if the total flow is determined not to be properly responding to indications of congestion. Another key characteristic we believe is likely to be true: before "punishing" the A&B flow, the monitoring device will first drop packets instead of setting the CE codepoint, and will drop arriving packets of that flow that already have the CE codepoint set. If the end nodes are actually using end-to-end congestion control, they'll see all of the indications of congestion seen by the monitoring device, and begin responding to them. So the monitoring device successfully provides the indications to the flow at an early stage 🎯.
It's true that the adversary with access only to A packets might, by subverting ECN-based congestion control, be able to deny the benefits of ECN to the other packets in the A&B aggregate. Unfortunate, yes, but this is NOT a reason to disable ECN 🙅♀️.
A variant of falsely reporting congestion happens when there are two adversaries along a path: the first adversary falsely reports congestion, and the second adversary erases' those reports. (Unlike packet drops, ECN congestion reports can be reversed' later in the network by a malicious or broken router. Good news though: the use of the ECN nonce could help the transport detect this behavior 🧠.) While this would be transparent to the end node, it's possible that a monitoring device between the first and second adversaries would see the false indications of congestion. Keep in mind our recommendation in this document: before "punishing" a flow for not responding appropriately to congestion, the router will first switch to dropping rather than marking as an indication of congestion, for that flow. When this includes dropping arriving packets from that flow that have the CE codepoint set, it ensures these indications of congestion are being seen by the end nodes. So there's no additional harm we can postulate as a result of multiple conflicting adversaries 👑.
19. Implications of Subverting End-to-End Congestion Control ✨
okay bestie, this section is about what happens when someone messes with end-to-end congestion control 🕵️♀️💅 There are two sneaky ways to do it: falsely claiming ECN-Capability, or erasing the congestion mark (the CE codepoint) 🙈 Both can cause problems for the application and the network, so we're gonna look at each one separately 🫶
The first move, falsely indicating ECN-Capability, only actually breaks end-to-end congestion control if the packet later runs into congestion that gets the CE codepoint set 🎀 Then the transport (which might not even be ECN-capable) never hears about the congestion from those downstream congested routers 😭
The second move, erasing the CE codepoint in a packet, only breaks things when a congested router had already set that CE codepoint earlier 💀 In that case the transport misses the warning from the upstream congested routers, and that's the whole tea ☕
Honestly, either of these tricks can cause more damage to the network (and maybe to the flow itself) than if the bad actor had just dropped the packets from that flow 👀 BUT, as we'll see later in this section and in Section 7, the damage is actually limited 🦋
19.1. Implications for the Network and for Competing Flows
The CE codepoint is only used by routers to signal congestion during periods of moderate congestion 🌸 ECN-capable routers should drop packets instead of marking them during heavy congestion, even if the queue isn't full yet ✨ For example, a router using RED-based active queue management should drop rather than mark packets that arrive while the average queue size is above the RED queue's maximum threshold 💜
Here's the network-side drama: flows that never get the congestion signals might keep raising their sending rate until they push the network into heavier congestion 📈 At that point the congested router could start dropping instead of marking incoming packets 😳 If flows aren't isolated by per-flow scheduling or other per-flow mechanisms, and they're just lumped together in one undifferentiated queue, that dropping hits every flow sharing that queue 💔 So the end result is more congestion across the network, which is not it 🙅♀️
In some cases, all that extra traffic builds up enough buffer at the congested queue to flip it from the packet-marking regime to the packet-dropping regime 🔥 That switch can happen either because the buffer overflows, or because of the RED-style policy that drops packets once the average queue is above the maximum threshold 🧚 Now every flow, including the subverted one, gets packet drops instead of marks, and a malicious or broken router can no longer erase those congestion indications 🙌 If the end nodes are running proper end-to-end congestion control, the subverted flow will slow its arrival rate in response 🐢 Once congestion drops enough, the queue can go back to the packet-marking regime 🔄 The steady state might just be the congested queue bouncing back and forth between these two regimes, which is very dramatic but also very real 🎢
In other cases, the subversion isn't bad enough to push the link into heavy enough congestion that packets get dropped instead of marked 🤷♀️ Then the competing flows see a slightly higher rate of marking or dropping, and their bandwidth goes down a little bit 📉 This can be a stable situation if the subverted flow's arrival rate is small enough, compared to the link bandwidth, that the average queue at the congested router stays under control 🧘♀️ The subverted flow could have a limited demand on the link at that router and still grab more than its "fair" share of the link 👑 That limited demand could come from a limited demand from the data source, a limit from the TCP advertised window, a lower-bandwidth access pipe, or other factors 🍓 So yeah, subverting ECN-based congestion control can still cause unfairness, and we think it's important to say that out loud 📣
The threat to the network from subverting ECN-based congestion control inside the network is basically the same as the threat from an end system that deliberately refuses to cooperate with end-to-end congestion control 😤 How to deploy router mechanisms to deal with this threat is still an open research question, which we'll talk about more in Section 10 🔍
Let's look at the example from Section 18.1.1, where the CE codepoint set on a packet gets erased: {'11' -> '10' or '11' -> '01'} 🧾 The congested upstream router that set the CE codepoint is basically shouting into the void, because that congestion signal never makes it to the end nodes for that flow 📢🚫 The source, even if it's totally cooperative and not malicious at all, keeps raising its sending rate (if it's a TCP flow, that means growing its congestion window) 📈 The flow can end up with better throughput than the other flows sharing that congested router, especially if the router has no policing or per-flow queuing mechanisms 😬 Now think about the other flows, especially the cooperative ones that aren't getting subverted congestion control: they'll probably reduce their load on the congested router (for example, by shrinking their window size), and that helps our subverted flow even more 🥲 That's the unfairness, bestie 💅 As we said before, it could be transient (if the congested queue gets pushed into the packet-marking regime), oscillatory (if the queue bounces between marking and dropping), or a more moderate but persistent stable state (if the queue is never pushed into the packet-dropping regime) 🌈
The results would look pretty similar if the subverted flow was intentionally dodging end-to-end congestion control 🎭 One difference: a flow that deliberately avoids congestion control at the end nodes can keep dodging it even when the queue is in packet-dropping mode, by refusing to slow down when packets get dropped in the network 🙅♀️ So the problems from subverting ECN-based congestion control are less severe than the problems from intentionally avoiding end-to-end congestion control at the end nodes ✨ It's also way harder to control what end nodes do than to control the infrastructure itself 🏗️ That doesn't mean the problems from network-side subversion of ECN-based congestion control are small, though! They're just dwarfed by the problems that come from end nodes subverting either ECN-based congestion control or other currently known packet-based congestion control mechanisms 😭💖
19.2. Implications for the Subverted Flow
When a source says it's ECN-capable, there's an expectation that the routers which can take part in ECN will use the CE codepoint to indicate congestion 💌 That's the good stuff, because ECN can reduce packet loss (on top of the lower queuing delays that come from active queue management policies) 🌟 When packets go through an IPsec tunnel where the nodes they pass through are untrusted in some way, the expectation is that IPsec will protect the flow from subversion that would have bad consequences 🔐
In a lot of cases, a subverted flow benefits from subverting end-to-end congestion control for itself, because it gets more bandwidth than it otherwise would compared to non-subverted flows 🤑 If the congested queue reaches the packet-dropping stage, though, subverting end-to-end congestion control might or might not be a net win for the subverted flow 🎲 It all depends on that flow's tradeoffs between throughput, loss, and delay ⚖️
One way to subvert end-to-end congestion control is to falsely indicate ECN-capability by setting the ECT codepoint 🎭 This means downstream congested routers end up setting the CE codepoint in vain, which is so sad for them 🥺 However, as described in Section 9.1.2, if the ECT codepoint gets changed in an IP tunnel, this can be caught at the tunnel's egress point, as long as the inner header wasn't changed inside the tunnel 🕵️♀️✨
The second way is to erase the congestion indication by erasing the CE codepoint 🧽 In this case, it's the upstream congested routers that set the CE codepoint in vain 😩
If an ECT codepoint gets erased inside an IP tunnel, this can also be caught at the tunnel's egress point, again as long as the inner header wasn't changed inside the tunnel 🔎 If the CE codepoint was set upstream of the IP tunnel, then any erasure of the outer header's CE codepoint inside the tunnel won't do anything, because the inner header keeps the set value of the CE codepoint 🙌 BUT if the CE codepoint gets set inside the tunnel and then erased either inside the tunnel or downstream of it, that might not be caught at the tunnel's egress point 👀
With this kind of subversion, the end-system transport just doesn't respond to the congestion indication 😶 Along with the extra unfairness for non-subverted flows we already talked about, the congested router's queue can keep building, and that leads to packet loss at the congested router 💧 Packet loss is a way of signaling congestion to the transport anyway, so it's not the end of the world 🫠 In the meantime, the flow might see higher queuing delays, and possibly more bandwidth compared to other non-subverted flows 🏎️ Transports don't inherently assume they'll always get carefully managed queuing along the path, so that's not a shock to them 🤷♀️ We believe these forms of subverting end-to-end congestion control are no worse for the subverted flow than if the adversary had just dropped that flow's packets directly 💅🎀
19.3. Non-ECN-Based Methods of Subverting End-to-end Congestion Control 🫶
Okay so we've already shown that, in lots of cases, a malicious or broken router that can change the bits in the ECN field can do no more damage than if it had just dropped the packet 🙈 BUT not in every case! 😭 The big exceptions are when the broken router subverts end-to-end congestion control by either falsely indicating ECN-Capability or by erasing the ECN congestion indication (in the CE codepoint). There are tons of ways a router can harm a flow by dropping packets, but a router cannot subvert end-to-end congestion control by dropping packets. For example, a router cannot subvert TCP congestion control by dropping data packets, acknowledgement packets, or control packets. 💅
Even though packet-dropping can't be used to subvert end-to-end congestion control, there are non-ECN-based methods for subverting it that a broken or malicious router could still use 👀 For example, a broken router could duplicate data packets, which effectively cancels out end-to-end congestion control along some portion of the path. (For a router that duplicated packets within an IPsec tunnel, the security administrator can make the duplicate packets get discarded by configuring anti-replay protection for the tunnel 🔐) This duplication of packets within the network would have similar implications for the network and for the subverted flow as those described in Sections 18.1.1 and 18.1.4 above. 🌀
20. The Motivation for the ECT Codepoints. 🦋
20.1. The Motivation for an ECT Codepoint. 🌸
The need for an ECT codepoint comes from the fact that ECN will be deployed incrementally in an Internet where some transport protocols and routers understand ECN and some don't 🧚 With an ECT codepoint, the router can drop packets from flows that are not ECN-capable, but can instead set the CE codepoint in packets that are ECN-capable. Because an ECT codepoint lets an end node have the CE codepoint set in a packet instead of having that packet dropped, an end node might have some incentive to deploy ECN 💖
If there was no ECT codepoint, then the router would have to set the CE codepoint for packets from both ECN-capable and non-ECN-capable flows. In that case there would be no incentive for end-nodes to deploy ECN, and no viable path of incremental deployment from a non-ECN world to an ECN-capable world 💀 Think about the first stages of that rollout, where only a subset of the flows are ECN-capable. When congestion first starts, and the packet dropping/marking rate is still low, routers would only set CE codepoints instead of dropping packets. But only the flows that are ECN-capable would understand and respond to CE packets. So the ECN-capable flows would back off, while the non-ECN-capable flows would be unaware of the ECN signals and would keep opening their congestion windows 😤
In this case there are two possible outcomes: (1) the ECN-capable flows back off, the non-ECN-capable flows get all of the bandwidth, and congestion remains mild, or (2) the ECN-capable flows back off, the non-ECN-capable flows don't, and congestion increases until the router switches from setting the CE codepoint to dropping packets. Outcome two does even out the fairness, but the ECN-capable flows would still get little benefit from being ECN-capable, because the increased congestion would push the router into packet-dropping behavior 📉
A flow that advertised itself as ECN-Capable but does not respond to CE codepoints is functionally equivalent to a flow that turns off congestion control, as discussed earlier in this document 🙅♀️
So, in a world where a subset of the flows are ECN-capable but ECN-capable flows have no mechanism for telling the routers that fact, you'd get less effective and less fair congestion control across the Internet. That creates a strong incentive for end nodes not to deploy ECN. Not it. 🙄
20.2. The Motivation for two ECT Codepoints. 🍓
The primary motivation for the two ECT codepoints is to provide a one-bit ECN nonce 🔑 The ECN nonce allows the development of mechanisms for the sender to probabilistically verify that network elements are not erasing the CE codepoint, and that data receivers are properly reporting to the sender the receipt of packets with the CE codepoint set.
Another way for senders to detect misbehaving network elements or receivers would be for the data sender to occasionally send a data packet with the CE codepoint set, to see whether the receiver reports receiving the CE codepoint. But if these packets hit congestion in the network, the router might make no change to them, because the CE codepoint would already be set. So for packets sent with the CE codepoint set, the TCP end-nodes could not tell whether some router intended to set the CE codepoint in those packets 🤷♀️ For this reason, sending packets with the CE codepoint would have to be done sparingly, and it would be a less effective check against misbehaving network elements and receivers than the ECN nonce would be.
The assignment of the fourth ECN codepoint to ECT(1) rules out using that codepoint for some other purposes. For clarity, here's a quick list of those other possible purposes 📝
One possibility might have been for the data sender to use the fourth ECN codepoint to indicate alternate semantics for ECN. However, this seems more appropriate to signal using a differentiated services codepoint in the DS field 🎀
A second possible use for the fourth ECN codepoint would have been to give the router two separate codepoints for indicating congestion, CE(0) and CE(1), for mild and severe congestion respectively. That could be useful in some cases, but it certainly doesn't seem like a compelling requirement at this point. If there was judged to be a compelling need for it, the complications of incremental deployment would most likely necessitate more that just one codepoint for this function 🧩
A third use that has been informally proposed for the ECN codepoint is in some forms of multicast congestion control, based on randomized procedures for duplicating marked packets at routers 🌈 Some proposed multicast packet duplication procedures are based on a new ECN codepoint that (1) conveys the fact that congestion occurred upstream of the duplication point that marked the packet with this codepoint and (2) can detect congestion downstream of that duplication point. ECT(1) can serve this purpose because it is both distinct from ECT(0) and is replaced by CE when ECN marking occurs in response to congestion or incipient congestion. How this enhanced version of ECN would be used by multicast congestion control is beyond the scope of this document, and so are ECN-aware multicast packet duplication procedures and the processing of the ECN field at multicast receivers in all cases (i.e., irrespective of the multicast packet duplication procedure(s) used) 🧠
The specification of IP tunnel modifications for ECN in this document assumes that the only change made to the outer IP header's ECN field between tunnel endpoints is setting the CE codepoint to indicate congestion. That isn't consistent with some of the proposed uses of ECT(1) by the multicast duplication procedures in the previous paragraph, so such procedures SHOULD NOT be deployed unless this inconsistency between multicast duplication procedures and IP tunnels with full ECN functionality is resolved 🚫 Limited ECN functionality may be used instead, although in practice many tunnel protocols (including IPsec) will not work correctly if multicast traffic duplication occurs within the tunnel
21. Why use Two Bits in the IP Header? 🎀
Okay so, we already need an ECT indication in the IP header, right? The real question is whether ECT and CE should have been squished onto a single bit instead of getting two. That one-bit idea was explored in [Floyd94]. One value would mean "ECT and not CE" (an ECN-Capable Transport), and the other would mean "CE or not ECT," which covers either Congestion Experienced or a non-ECN-Capable transport. Bestie, it's a whole vibe check 💅
One difference shows up when packets cross several congested routers. Picture a CE packet reaching a second congested router, and that router's active queue management picks it for marking or dropping. With one bit, the second router has no choice but to drop it, since it can't tell a CE packet from a non-ECT packet. With two bits, the second router can either drop it or leave it alone with the CE codepoint still set. Iconic flexibility 🦋
Another difference is about receivers. With one bit, receivers in a single flow can't tell CE and non-ECT packets apart. So an ECN-capable sender would have to make it crystal clear to the receiver or receivers whether each packet was sent as ECN-Capable or not. One option is to put that info in the transport header, saying whether each packet was sent as ECN-Capable. The other option has a functional limitation: the sender declares that it will send all of its packets as ECN-Capable, or all as non-ECN-Capable. For a multicast transport protocol, that declaration has to be visible to receivers who join a session that's already going 🌸
Also, the document recommends that transports (TCP especially) should not mark pure ACK packets or retransmitted packets as ECN-Capable. A pure ACK from a non-ECN-capable transport can get dropped without hurting congestion control, because later ACKs are cumulative. But an ECN-capable transport that cuts its window after seeing CE on a pure ACK would be at a disadvantage compared to a non-ECN-capable one. For that reason, and for the reasons mentioned earlier about retransmitted packets, it's desirable to set the ECT codepoint on a per-packet basis ✨
The two-bit approach is also more robust. The most critical issue, covered in Section 8, is that the default indication should be the one for a non-ECN-Capable transport. With two bits, that just means the not-ECT codepoint should be the default. With one bit, the overloaded bit should default to the "CE or not ECT" position. That's way less clear, and it's more open to incorrect implementations in end nodes or routers 🙈
So yeah, the one-bit version could technically work, but it has some real limitations next to the two-bit version. First, it has more limited options for handling CE packets at a second congested router. Second, it needs either extra info in the transport header of ECN-capable flows (to carry the second bit's job somewhere else), or senders have to accept that receivers must know ahead of time which packets are ECN-Capable and which aren't. Third, it's more open to errors from faulty implementations that pick the wrong default value for the ECN bit. We believe the extra IP header bit for ECT is extremely valuable for getting past all of these limitations 💜
22. Historical Definitions for the IPv4 TOS Octet 🍓
Let's do some history, because the IPv4 TOS octet has had a messy past. RFC 791 [RFC791] defined the ToS (Type of Service) octet in the IP header. In RFC 791, bits 6 and 7 of the ToS octet are listed as "Reserved for Future Use," and they're shown set to zero. The first two fields of the ToS octet were defined as the Precedence and Type of Service (TOS) fields.
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | 0 | 0 | RFC 791
+-----+-----+-----+-----+-----+-----+-----+-----+RFC 1122 pulled bits 6 and 7 into the TOS field, though it didn't say anything specific about what those two bits were for:
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | RFC 1122
+-----+-----+-----+-----+-----+-----+-----+-----+Then RFC 1349 [RFC1349] redefined the IPv4 TOS octet like this:
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | MBZ | RFC 1349
+-----+-----+-----+-----+-----+-----+-----+-----+Bit 6 in the TOS field was defined in RFC 1349 for "Minimize Monetary Cost." Besides the Precedence and Type of Service (TOS) fields, the last field, MBZ (for "must be zero"), was defined as currently unused. RFC 1349 said, "The originator of a datagram sets [the MBZ] field to zero (unless participating in an Internet protocol experiment which makes use of that bit)."
RFC 1455 [RFC 1455] defined an experimental standard that used all four bits in the TOS field to request a guaranteed level of link security 🔒
RFC 1349 and RFC 1455 were then obsoleted by "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers" [RFC2474]. In that one, bits 6 and 7 of the DS field are listed as Currently Unused (CU). RFC 2780 [RFC2780] specified ECN as an experimental use of the two-bit CU field. It also changed the DS Field definition so it only covers the first six bits of this octet instead of all eight. Those first six bits are now the Differentiated Services CodePoint (DSCP):
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| DSCP | CU | RFCs 2474,
+-----+-----+-----+-----+-----+-----+-----+-----+ 2780Because of all this shifting history, the ECN field definition in this document can't be guaranteed to be backwards compatible with every past use of these two bits. It's a lot, I know 😭
Before RFC 2474, routers weren't allowed to change bits in either the DSCP or ECN field of packets they forwarded. So routers that only follow pre-2474 RFCs should have no effect on ECN. For end nodes, bit 7 (the second ECN bit) must be transmitted as zero by any implementation that only complies with pre-2474 RFCs. Those nodes may transmit bit 6 (the first ECN bit) as one, for the "Minimize Monetary Cost" provision of RFC 1349 or the experiment authorized by RFC 1455. Neither that part of RFC 1349 nor the RFC 1455 experiment were widely implemented or used. The damage a broken, non-conformant router could do includes "erasing" the CE codepoint from an ECN-capable packet that arrived with CE set, or setting the CE codepoint even when there's no congestion. This was discussed in the section on "Non-compliance in the Network."
The damage that a non-ECN-capable end node could do in an ECN-capable environment, by sending packets with the ECT codepoint set, was discussed in the section on "Non-compliance by the End Nodes" 🙄
23. IANA Considerations 📋
This section lists the namespaces that were either created in this specification, or the values assigned in existing namespaces managed by IANA.
23.1. IPv4 TOS Byte and IPv6 Traffic Class Octet
The codepoints for the ECN Field of the IP header are specified by the Standards Action of this RFC, as RFC 2780 requires.
When this document is published as an RFC, IANA should create a new registry called "IPv4 TOS Byte and IPv6 Traffic Class Octet," with the namespace laid out below:
IPv4 TOS Byte and IPv6 Traffic Class Octet
Description: The registrations are identical for IPv4 and IPv6.
Bits 0-5: see Differentiated Services Field Codepoints Registry
(http://www.iana.org/assignments/dscp-registry)
Bits 6-7, ECN Field:
Binary Keyword References
------ ------- ----------
00 Not-ECT (Not ECN-Capable Transport) [RFC 3168]
01 ECT(1) (ECN-Capable Transport(1)) [RFC 3168]
10 ECT(0) (ECN-Capable Transport(0)) [RFC 3168]
11 CE (Congestion Experienced) [RFC 3168]23.2. TCP Header Flags
The codepoints for the CWR and ECE flags in the TCP header are specified by the Standards Action of this RFC, as RFC 2780 requires.
When this document is published as an RFC, IANA should create a new registry called "TCP Header Flags," with the namespace laid out below:
TCP Header Flags
The Transmission Control Protocol (TCP) included a 6-bit Reserved
field defined in RFC 793, reserved for future use, in bytes 13 and 14
of the TCP header, as illustrated below. The other six Control bits
are defined separately by RFC 793.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | U | A | P | R | S | F |
| Header Length | Reserved | R | C | S | S | Y | I |
| | | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
RFC 3168 defines two of the six bits from the Reserved field to be
used for ECN, as follows:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | C | E | U | A | P | R | S | F |
| Header Length | Reserved | W | C | R | C | S | S | Y | I |
| | | R | E | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
TCP Header Flags
Bit Name Reference
--- ---- ---------
8 CWR (Congestion Window Reduced) [RFC 3168]
9 ECE (ECN-Echo) [RFC 3168]23.3. IPSEC Security Association Attributes
IANA allocated the IPSEC Security Association Attribute value 10 for the ECN Tunnel use described in Section 9.2.1.2 above, at the request of David Black in November 1999. IANA has since changed the Reference for this allocation from David Black's request to this RFC ✨
24. Authors' Addresses ✨
Okay bestie, here are the three legends who wrote this one 💖 Their contact deets are right below:
K. K. Ramakrishnan
TeraOptic Networks, Inc.
Phone: +1 (408) 666-8650
EMail: kk@teraoptic.com
Sally Floyd
ACIRI
Phone: +1 (510) 666-2989
EMail: floyd@aciri.org
URL: http://www.aciri.org/floyd/
David L. Black
EMC Corporation
42 South St.
Hopkinton, MA 01748
Phone: +1 (508) 435-1000 x75140
EMail: black_david@emc.com25. Full Copyright Statement 🎀
Copyright (C) The Internet Society (2001). All Rights Reserved. Legal stuff time, but it's giving protection for everyone 🦋
Okay so, you're totally allowed to copy this document and share it with your friends 🌸 You can even make derivative works that comment on it, explain it, or help people actually implement it, and publish and distribute those too, in whole or in part, with no real restrictions. The only catch is that the copyright notice and this paragraph have to stay on every copy and every derivative work ✨
Here's the lowkey strict part: this document itself can't be modified in any way. So no deleting the copyright notice, no swapping out the references to the Internet Society or other Internet organizations, nothing like that. The exception is when you need to do it to develop Internet standards, and then you have to follow the copyright procedures in the Internet Standards process. You're also allowed to translate it into languages other than English 🌈
The permissions I just described are perpetual, and the Internet Society or its successors or assigns will never revoke them 💜
This document and the information in it are provided on an "AS IS" basis, and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💀 Translation: no warranties, so use it with your eyes open!
Acknowledgement 🫶
The RFC Editor function is currently funded by the Internet Society, so a big thank you to them for keeping the lights on 👑
📦 where this page comes from
- ietf: rfc3168 2001-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms