RFC 2481
A Proposal to add Explicit Congestion Notification (ECN) to IP
RFC 2481: A Proposal to add Explicit Congestion Notification (ECN) to IP 🎀✨
Quick tea before we start: this is an Experimental RFC from January 1999, by K. Ramakrishnan (AT&T Labs Research) and S. Floyd (LBNL). Its status is Experimental, and it doesn't list any documents it obsoletes or updates. 💅 It's the proposal that started ECN, which lets routers say "hey, it's getting crowded" without dropping your packets 🫶
Status of This Memo 📝
okay so this memo defines an Experimental Protocol for the Internet community 🧪 It is not an Internet standard of any kind. The authors are asking for discussion and suggestions for improvement, and the distribution of this memo is unlimited 🌸
Copyright Notice ©️
Copyright (C) The Internet Society (1999). All Rights Reserved. 💜
Abstract 🦋
so this note proposes adding ECN (Explicit Congestion Notification) to IP 🌈 TCP is the dominant transport protocol on the Internet right now, so it's kind of the main character 👑 The note starts by explaining how TCP uses packet drops as a sign of congestion. Then it argues that once active queue management (like RED) is in the Internet's infrastructure, and routers can spot congestion before the queue overflows, routers aren't stuck using drops anymore 🙌
Instead, routers could set a Congestion Experienced (CE) bit in the packet header of packets from ECN-capable transport protocols. The note describes when routers would set that CE bit, and what changes TCP would need to become ECN-capable ✨ It also mentions that other transport protocols (like unreliable unicast or multicast, reliable multicast, and other reliable unicast transports) could be considered later, as they get developed and move through the standards process 🫶
1. Conventions and Acronyms 📚
ngl this section is short but important 🤓 The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they show up in this document, are to be interpreted as described in [B97] 💖
2. Introduction 🌸
bestie, so TCP's congestion control and avoidance algorithms are built on the idea that the network is a black box 🖤 End systems figure out the network's state by probing it. They keep increasing the load (by growing the window of outstanding packets) until the network gets congested and a packet gets lost. Treating loss as the sign of congestion is totally fine for pure best-effort data that doesn't care much about the delay or loss of individual packets 💅 TCP also has built-in tricks, like Fast Retransmit and Fast Recovery, that keep losses from hurting throughput too much ✨
BUT (and this is the tea) these mechanisms weren't designed to help apps that are actually sensitive to the delay or loss of single packets 😭 Interactive traffic like telnet, web browsing, and audio and video transfer can suffer from packet losses (if it's using an unreliable transport like UDP), or from the extra latency of retransmitting a packet after a loss (if it's using a reliable transport like TCP) 🙈
Here's the drama: TCP picks its congestion window by slowly growing the window size until it hits a dropped packet. That makes the queues at the bottleneck router build up 📈 Most router drop policies don't care how much load each individual flow is putting on the queue, so some packets from latency-sensitive flows end up getting dropped too 💀
Active queue management mechanisms are the solution 👑 They detect congestion before the queue overflows, and they give end nodes a heads-up about it. RFC 2309 [RFC2309] talks about the advantages of active queue management. It avoids some of the bad stuff that comes with dropping on queue overflow, including the annoying synchronization of loss across multiple flows 🌈 Even better, it means transport protocols with congestion control (like TCP) don't have to rely on buffer overflow as the only sign of congestion. That can cut down on unnecessary queueing delay for all the traffic sharing that queue 🦋
Active queue management can use several different methods to tell end nodes about congestion. One is packet drops, which is what happens today 🫠 But active queue management lets the router separate its queueing or dropping policies from its policies for signalling congestion. So routers can use the Congestion Experienced (CE) bit in a packet header as a sign of congestion, instead of relying only on packet drops 🎀
3. Assumptions and General Principles 🧠
this section is where the authors spell out the design principles and assumptions that shaped the choices in this proposal, so let's go 💅
(1) Congestion may persist over different time scales. The time scales we care about are congestion events that might last longer than a round-trip time ⏱️
(2) The number of packets in an individual flow (like a TCP connection, or an exchange using UDP) can range from a handful to quite a lot. The authors are interested in managing congestion caused by flows that send enough packets to still be active when the network feedback reaches them 🔥
(3) New mechanisms for congestion control and avoidance need to co-exist and cooperate with the existing ones. In particular, they have to co-exist with TCP's current ways of adapting to congestion, and with routers' current practice of dropping packets during congestion 🤝
(4) Because ECN is likely to be adopted gradually, accommodating migration is essential 🦋 Some routers may still only drop packets to signal congestion, and some end systems may not be ECN-capable. The most viable strategy accommodates incremental deployment without resorting to "islands" of ECN-capable and non-ECN-capable environments 🏝️
(5) Asymmetric routing is likely to be a normal occurrence on the Internet 🙃 The path (the sequence of links and routers) that data packets follow may be different from the path the acknowledgment packets take in the reverse direction 🔁
(6) Many routers process the "regular" headers in IP packets more efficiently than they process the header information in IP options 🏎️ So the proposal keeps congestion-experienced information in the regular headers of an IP packet 💖
(7) It must be recognized that not all end systems will cooperate in mechanisms for congestion control 😤 But new mechanisms shouldn't make it easier for TCP applications to disable TCP congestion control. Lying about taking part in new mechanisms like ECN-capability should get you very little benefit 🙅♀️
4. Random Early Detection (RED) 🌊
okay the main character here is Random Early Detection (RED) ✨ It's an active queue management mechanism that was proposed to detect incipient congestion [FJ93], and it's currently being deployed in the Internet backbone [RFC2309]. RED is meant to be a general mechanism that can use one of several alternatives for signalling congestion. But in today's Internet, RED is restricted to using packet drops as its congestion signal 🫠
RED drops packets based on the average queue length going over a threshold, not only when the queue overflows 📊 When RED drops packets before the queue actually overflows, memory limits don't force it to discard the packet. That's a detail I'm obsessed with 🙈
RED could set a Congestion Experienced (CE) bit in the packet header instead of dropping the packet, as long as the IP header provides such a bit and the transport protocol understands it 🦋 The CE bit would let the receivers actually get the packet, which avoids the risk of excessive delays from retransmissions after packet losses 🫶 In this document we use the term 'CE packet' to mean a packet that has the CE bit set 🎀
5. Explicit Congestion Notification in IP ✨
okay so this is the big idea 🦋 we want the internet to give a heads up about incipient congestion (like what RED and earlier work [RJ90] already do) and sometimes that heads up can be a marked packet instead of a dropped one. no more just yeeting packets into the void 💀 this needs an ECN field in the IP header with two bits. the ECN-Capable Transport (ECT) bit gets set by the data sender to say "hey, our transport endpoints are ECN-capable" 🎀 and the CE bit gets set by the router to tell the end nodes "congestion alert!" 🚨 routers facing a packet and a completely full queue still drop it, just like they do now 😭
Bits 6 and 7 in the IPv4 TOS octet are designated as the ECN field. bit 6 is the ECT bit and bit 7 is the CE bit 💜 the IPv4 TOS octet lines up with the Traffic Class octet in IPv6. the definitions for the IPv4 TOS octet [RFC791] and the IPv6 Traffic Class octet are meant to get superseded by the DS (Differentiated Services) Field [DIFFSERV]. and ngl, [DIFFSERV] lists bits 6 and 7 as Currently Unused 🌸 Section 19 has a short history of the TOS octet if you're curious (you should be 🧚)
because the TOS octet has had such a chaotic past, the ECN field as specified here cannot be guaranteed to be backwards compatible with every old use of those two bits 🙈 the potential dangers of that lack of backwards compatibility are covered in Section 19. not a vibe to ignore, bestie 💅
Okay, the most important rule: when an ECN-Capable transport gets a single CE packet, the congestion control at the end systems MUST be essentially the same as the response to a single dropped packet 🔥 for example, for ECN-Capable TCP, the source TCP is required to halve its congestion window for any window of data that has either a packet drop or an ECN indication ✨ we also want to point out a few notable exceptions to how the source TCP reacts, related to the shorter-time-scale details of particular TCP implementations 👀 for TCP's response to an ECN indication, we do not recommend stuff like Tahoe TCP's slow-start after a drop, or Reno TCP's wait of roughly half a round-trip time during Fast Recovery 🙅♀️
why do we insist the CE response matches the drop response? 🤔 so ECN can roll out incrementally across both end systems and routers 💖 some routers might drop ECN-Capable packets (say, using the same RED policies for congestion detection) while other routers set the CE bit, for equivalent levels of congestion. and a router might drop a non-ECN-Capable packet but set the CE bit in an ECN-Capable one, again for equivalent congestion. if CE and drops got different responses, different flows could end up treated unfairly 😬 that's a whole mess nobody wants
one more requirement: end systems should react to congestion at most once per window of data (so, at most once per roundtrip time) 🫶 that way they don't overreact to several congestion indications inside a single roundtrip time 🌈
For a router, the CE bit of an ECN-Capable packet should only be set if the router would otherwise have dropped that packet as an indication of congestion to the end nodes ✨ when the router's buffer isn't full yet, and the router is ready to drop a packet to tell end nodes about incipient congestion, the router should first check whether the ECT bit is set in that packet's IP header. if it is, then instead of dropping the packet, the router MAY set the CE bit in the IP header 💜
Imagine a world where every end node is ECN-Capable 🧚♀️ that would open the door to brand-new criteria for setting the CE bit, and new congestion control mechanisms for how end nodes react to CE packets. but that's a research question, so this document doesn't tackle it 🔬
When a router receives a CE packet, it leaves the CE bit unchanged and transmits the packet as usual 🦋 when severe congestion hits and the router's queue is totally full, the router has no choice but to drop some packet when a new one arrives 😭 we expect those losses to get pretty rare once most end systems are ECN-Capable and take part in TCP or other compatible congestion control mechanisms 🎀 in a well-provisioned network in that kind of ECN-Capable world, packet losses should mostly happen during transients or when non-cooperating sources are around 👑
We expect routers to set the CE bit in response to incipient congestion, as indicated by the average queue size, using the RED algorithms suggested in [FJ93, RFC2309] 💅 as far as we know, this is the only proposal currently under discussion in the IETF for routers to drop packets proactively, before the buffer overflows 👀 this document doesn't try to pin down one particular active queue management mechanism though. that's left, if needed, to other areas of the IETF 🫶 and while ECN is deeply tangled up with active queue management at the router, the reverse isn't true: active queue management has been developed and deployed independently of ECN, using packet drops as congestion indications when there's no ECN in the IP architecture 🌸
6. Support from the Transport Protocol 🎀
ECN needs help from the transport protocol too, on top of the ECN field in the IP packet header 💖 first, the transport protocol might need a negotiation between the endpoints during setup to confirm that all of them are ECN-capable, so the sender can set the ECT bit in its transmitted packets 🤝 second, the transport protocol has to be able to react properly when CE packets come in 💜 that reaction could be the data receiver telling the data sender about the CE packet it got (like TCP does), or the data receiver unsubscribing from a layered multicast group (like RLM [MJV96]), or some other action that ultimately lowers the arrival rate of that flow to that receiver 🦋
This document only covers adding ECN Capability to TCP. ECN for other transport protocols is left for further research 🧚 For TCP, ECN needs three new mechanisms: negotiation between the endpoints during setup to figure out if they're both ECN-capable; an ECN-Echo flag in the TCP header so the data receiver can tell the data sender when it has received a CE packet; and a Congestion Window Reduced (CWR) flag in the TCP header so the data sender can tell the data receiver that its congestion window has been reduced 💅 the support other transport protocols need will probably be different, especially for unreliable or reliable multicast transport protocols, and it'll have to be figured out as those protocols get brought to the IETF for standardization 🌈
6.1. TCP 💜
The following sections go into detail on the proposed use of ECN in TCP ✨ this proposal is described in essentially the same form in [Floyd94]. we assume the source TCP uses the standard congestion control algorithms of Slow-start, Fast Retransmit and Fast Recovery [RFC 2001] 🔥
This proposal specifies two new flags in the Reserved field of the TCP header 🎀 the TCP mechanism for negotiating ECN-Capability uses the ECN-Echo flag in the TCP header. (this was called the ECN Notify flag in some earlier documents, so don't be confused 🙈) Bit 9 in the Reserved field of the TCP header is designated as the ECN-Echo flag 💖 the location of the 6-bit Reserved field in the TCP header is shown in Figure 3 of RFC 793 [RFC793] 📖
To help the TCP receiver figure out when to stop setting the ECN-Echo flag, we introduce a second new flag, the Congestion Window Reduced (CWR) flag 🦋 the CWR flag is assigned to Bit 8 in the Reserved field of the TCP header 🌸
The use of these flags is described in the sections below 🧚
6.1.1. TCP Initialization 🌷
During the TCP connection setup phase, the source and destination TCPs trade info about their desire and/or capability to use ECN 🤝 once that negotiation is done, the TCP sender sets the ECT bit in the IP header of data packets to tell the network that the transport is capable of and willing to take part in ECN for that packet 💜 this lets routers know they may mark the packet with the CE bit if they want to use that as a way to signal congestion 🚨 if the TCP connection doesn't want ECN notification for a particular packet, the sending TCP sets the ECT bit equal to 0 (i.e., not set), and the TCP receiver ignores the CE bit in the received packet 🙅♀️
When a node sends a TCP SYN packet, it may set the ECN-Echo and CWR flags in the TCP header ✨ for a SYN packet, setting both the ECN-Echo and CWR flags means "I'm ECN-Capable," not "I'm reporting congestion or responding to it" 🎀 to be precise, a SYN packet with both the ECN-Echo and CWR flags set indicates that the TCP implementation sending the SYN will take part in ECN as both a sender and a receiver 👑 as a receiver, it responds to incoming data packets that have the CE bit set in the IP header by setting the ECN-Echo flag in outgoing TCP Acknowledgement (ACK) packets 💖 as a sender, it responds to incoming packets that have the ECN-Echo flag set by reducing the congestion window when appropriate 🔥
When a node sends a SYN-ACK packet, it may set the ECN-Echo flag, but it does not set the CWR flag 🦋 for a SYN-ACK packet, that pattern (ECN-Echo set, CWR not set) in the TCP header is defined as an indication that the TCP sending the SYN-ACK is ECN-Capable 🌈
Now, why does the SYN sender set two ECN-related flags while the SYN-ACK responder sets only one? 🤔 it's a fair question, and the asymmetry is necessary for robust ECN negotiation with TCP implementations already out there 🫶 at least one TCP implementation has TCP receivers copy the Reserved field from the received data packet into their ACK packets (and therefore the SYN-ACK) 😭 since the TCP SYN packet sets both ECN-Echo and CWR to signal ECN-capability, but the SYN-ACK sets only ECN-Echo, the sending TCP correctly reads a receiver's reflection of its own flags as meaning that the receiver is not ECN-capable 💀✨
6.1.2. The TCP Sender
Okay so bestie, for a TCP connection using ECN, data packets go out with the ECT bit set to "1" in the IP header 💖. If the sender gets an ECN-Echo ACK packet (that's an ACK with the ECN-Echo flag set in the TCP header), then it knows congestion was hit somewhere on the path from the sender to the receiver 🚨. The congestion signal should be treated just like a congestion loss in non-ECN-Capable TCP. So the TCP source halves the congestion window "cwnd" and lowers the slow start threshold "ssthresh" ✂️. The sending TCP does NOT increase the congestion window when it gets an ECN-Echo ACK packet 🙅♀️.
Now the important part 👀. TCP does not react to congestion indications more than once every window of data (or, more loosely, more than once every round-trip time). So the TCP sender should only reduce its congestion window once for a whole series of dropped and/or CE packets from a single window of data 🪟. Also, the TCP source should not decrease ssthresh if it was already decreased within the last round trip time ⏱️. BUT, if any retransmitted packets get dropped or have the CE bit set, the source TCP reads that as a brand new instance of congestion 😤. Main character energy for real.
After the source TCP cuts its congestion window because of a CE packet, the incoming acknowledgements that keep showing up can "clock out" outgoing packets, as long as the smaller window allows 🫧. Now here's the tea 🍵. If the congestion window is only one MSS (maximum segment size), and the sending TCP gets an ECN-Echo ACK packet, the sending TCP should in principle still halve its congestion window. However, the congestion window is bounded below by one MSS, so it can't go any lower. If the sender kept sending with a congestion window of 1 MSS, that works out to one packet per round-trip time 🐢. We think it's still desirable to slow the sending rate down even further when an ECN-Echo packet shows up while the congestion window is one. So we use the retransmit timer to pull the rate down more in this situation 🧠. That means the sending TCP should also reset the retransmit timer when it receives the ECN-Echo packet while the congestion window is one. The sending TCP will then be allowed to send a new packet when the retransmit timer expires ⏰.
[Floyd94] goes deeper into how TCP responds to ECN 📚. [Floyd98] talks about the validation test in the ns simulator, which shows a wide range of ECN scenarios 🧪. Those scenarios include: an ECN followed by another ECN, a Fast Retransmit, or a Retransmit Timeout; a Retransmit Timeout or a Fast Retransmit followed by an ECN; and a congestion window of one packet followed by an ECN 🎀. Iconic lineup, no notes.
TCP follows its existing algorithms for sending data packets in response to incoming ACKs, multiple duplicate acknowledgements, or retransmit timeouts [RFC2001] 🫶.
6.1.3. The TCP Receiver
When TCP receives a CE data packet at the destination end-system, the TCP data receiver sets the ECN-Echo flag in the TCP header of the next ACK packet 🌸. If there's any ACK withholding going on, like in current "delayed-ACK" TCP implementations where the TCP receiver can send one ACK for two arriving data packets, then the ECN-Echo flag in that ACK packet gets set to the OR of the CE bits of all the data packets being acknowledged 🧮. Put simply, if any of the received data packets are CE packets, the returning ACK has the ECN-Echo flag set 💅.
Okay, this part is so clever 😭. To protect against the chance that an ACK packet carrying an ECN-Echo flag gets dropped, the TCP receiver must set the ECN-Echo flag in a series of ACK packets 🛡️. The TCP receiver uses the CWR flag to figure out when to stop setting the ECN-Echo flag 🚦.
When an ECN-Capable TCP reduces its congestion window for any reason (because of a retransmit timeout, a Fast Retransmit, or in response to an ECN Notification), the TCP sets the CWR flag in the TCP header of the first data packet sent after that window reduction 📦. If that data packet gets dropped in the network, the sending TCP has to reduce the congestion window again and retransmit the dropped packet 🔁. So the Congestion Window Reduced message is reliably delivered to the data receiver 💌. Reliable bestie energy 🫂.
After a TCP receiver sends an ACK packet with the ECN-Echo bit set, that receiver keeps setting the ECN-Echo flag in ACK packets until it receives a CWR packet (a packet with the CWR flag set) ✋. Once it gets that CWR packet, acknowledgements for later non-CE data packets don't have the ECN-Echo flag set 🙌. If the data receiver gets another CE packet, it'll start sending ACK packets with the ECN-Echo flag set all over again 🔄. Receiving a CWR packet doesn't guarantee that the data sender got the ECN-Echo message, but it does tell you the data sender reduced its congestion window at some point after it sent the data packet with the CE bit set 🦋.
We already said that a TCP sender reduces its congestion window at most once per window of data. This mechanism needs some care 🧁. The sender has to make sure it reduces its congestion window at most once per ECN indication, and that multiple ECN messages over several successive windows of data get reported properly to the ECN sender 📬. More on this in [Floyd98] 📖.
6.1.4. Congestion on the ACK-path
For the current generation of TCP congestion control algorithms, pure acknowledgement packets (like packets that don't carry any accompanying data) should be sent with the ECT bit off 🚫. Current TCP receivers don't have any mechanisms for cutting back traffic on the ACK-path in response to congestion notification 🙈. Mechanisms for responding to congestion on the ACK-path are still areas for current and future research 🔬. (One simple idea would be for the sender to reduce its congestion window when it receives a pure ACK packet with the CE bit set 💡.) For current TCP implementations, a single dropped ACK generally has only a very small effect on TCP's sending rate 🌈.
7. Summary of changes required in IP and TCP ✨
Two bits need to be specified in the IP header: the ECN-Capable Transport (ECT) bit and the Congestion Experienced (CE) bit 🎀. The ECT bit set to "0" means the transport protocol will ignore the CE bit. That's the default value for the ECT bit 🙅♀️. The ECT bit set to "1" means the transport protocol is willing and able to take part in ECN 🫡.
The default value for the CE bit is "0" 0️⃣. The router sets the CE bit to "1" to indicate congestion to the end nodes 🚏. The CE bit in a packet header should never be reset by a router from "1" to "0" 🔒.
TCP needs three changes: a negotiation phase during setup to figure out if both end nodes are ECN-capable, and two new flags in the TCP header, taken from the "reserved" flags in the TCP flags field 🛠️. The ECN-Echo flag is how the data receiver tells the data sender about a received CE packet 📣. The Congestion Window Reduced flag is how the data sender tells the data receiver that its congestion window has been reduced 💜.
8. Non-relationship to ATM's EFCI indicator or Frame Relay's FECN 🌸
Okay, real talk 🫠. ATM and Frame Relay congestion indication mechanisms have usually been defined without any notion of average queue size as the basis for deciding that an intermediate node is congested. So we think they give a very noisy signal 📶. The TCP-sender reaction specified in this draft for ECN is NOT the appropriate reaction for such a noisy congestion notification signal 🚫. We expect ATM's EFCI and Frame Relay's FECN mechanisms to be phased out over time within the ATM network 🌅. However, if the routers that interface to the ATM network have a way of maintaining the average queue at the interface, and use it to reliably determine that the ATM subnet is congested, they may use the ECN notification defined here 🧠.
We need to stress that a single packet with the CE bit set in an IP packet makes the transport layer respond, in terms of congestion control, the same way it would to a packet drop 💥. So the CE bit isn't a great match for a transient signal like one based on instantaneous queue size 🙅♀️. Still, experiments with techniques at layer 2 (for example, in ATM switches or Frame Relay switches) should be encouraged 🧪. For example, using a scheme such as RED (where packet marking is based on the average queue length exceeding a threshold), layer 2 devices could give a reasonably reliable indication of congestion 🎯. When all the layer 2 devices in a path set that layer's own Congestion Experienced bit (for example, the EFCI bit for ATM, or the FECN bit in Frame Relay) in this reliable manner, the interface router to the layer 2 network could copy the state of that layer 2 Congestion Experienced bit into the CE bit in the IP header 🪞. We know this isn't current practice, and it isn't in current standards either 🤷♀️. But encouraging experimentation like this may provide the information needed to let existing layer 2 mechanisms evolve into a more reliable way of indicating congestion, once they use a single bit for it 🌈✨.
9. Non-compliance by the End Nodes ✨
okay so this section is about the scariest-sounding bestie situation 💀: end nodes that set the ECT bit (basically saying "I'm ECN-capable 💅") but then totally ignore CE packets when they show up. We're arguing that adding ECN to IP wouldn't significantly make the architecture more vulnerable to unresponsive flows. 🫶
Honestly, even without ECN there are serious concerns about non-compliant or unresponsive flows. These are flows that don't respond to congestion control signals by reducing their arrival rate at the congested link 😭. Picture an end node that "turns off congestion control" by not shrinking its congestion window after packets get dropped. That's a real problem for the current Internet. People have argued that routers will need mechanisms to detect these flows and treat their packets differently 🔥. Others have said that end-to-end per-flow scheduling and isolation of one flow from another, differentiated services, or end-to-end reservations could remove some of the worst damage from unresponsive flows 🌈.
Now, some people argue that dropping packets is already enough of a deterrent for non-compliance, and that ECN takes that deterrent away 🙈. Our response has two parts: (1) ECN-capable routers keep packet dropping going during times of high congestion, and (2) even in times of high congestion, dropping packets isn't an adequate deterrent for non-compliance on its own.
First, ECN-capable routers only mark packets (instead of dropping them) when the packet marking rate is reasonably low ✨. When the average queue size goes above an upper threshold, which means the potential marking rate would be high, our recommendation is that routers drop packets rather than set the CE bit in packet headers 💖.
During periods of low or moderate packet marking, when ECN would be deployed, there wouldn't be much of a deterrent difference between dropping and marking for unresponsive flows 🦋. For example, delay-insensitive flows using reliable delivery might have an incentive to increase their sending rate when packets are dropped, instead of decreasing it 💀. Similarly, delay-sensitive flows using unreliable delivery might bump up their use of FEC in response to a higher packet drop rate, which means they'd increase their sending rate instead of decreasing it 🍓. For those same reasons, we don't think packet dropping by itself works as an effective deterrent for non-compliance, even when packet drop rates are high 😤.
Lots of methods have been proposed to find and restrict non-compliant or unresponsive flows 🧚. Adding ECN to the network wouldn't make designing and deploying those mechanisms any harder at all. If anything, it would make spotting unresponsive flows slightly easier 👑. In an ECN-capable environment, routers aren't limited to information about packets they dropped or set the CE bit on themselves. They could also take note of arriving CE packets, which show that congestion was already encountered earlier on the path 🎀.
10. Non-compliance in the Network ⭐
Okay, the breakdown of effective congestion control doesn't only happen because of a non-compliant end node. It can also happen when the congestion indication itself gets lost inside the network 😭. That could happen through a rogue or broken router that sets the ECT bit on a packet from a non-ECN-capable transport, or that "erases" the CE bit in arriving packets 💅.
Here's one example. A rogue or broken router that erases the CE bit in arriving CE packets stops that congestion signal from reaching downstream receivers 💀. That can make congestion control fail for that flow, which leads to even more congestion in the network 🔥. Eventually, packets for that flow get dropped anyway, because the average queue size keeps growing at the congested gateway 😩.
A rogue or broken router can also cause an unnecessary congestion indication for end nodes 🙈. This can happen when a router drops a packet or sets the CE bit even though there's no congestion. From a congestion control point of view, a non-compliant router setting the CE bit when there's no congestion is no different from a router dropping a packet unnecessarily 🌸. And if a router erases the ECT bit of a packet that gets dropped later in the network, that router's actions can cause an unnecessary packet drop later on 💔.
Concerns about losing congestion indications from encapsulated, dropped, or corrupted packets are covered below ✨.
10.1. Encapsulated packets 🎀
Some care is needed to handle the CE and ECT bits correctly when packets are encapsulated and de-encapsulated for tunnels 🦋.
When a packet is encapsulated, these rules apply to the ECT bit 🌈. First, if the ECT bit in the encapsulated ('inside') header is a 0, then the ECT bit in the encapsulating ('outside') header MUST be a 0. If the ECT bit in the inside header is a 1, then the ECT bit in the outside header SHOULD be a 1 ✨.
When a packet is de-encapsulated, these rules apply to the CE bit 💖. If the ECT bit is a 1 in both the inside and the outside header, then the CE bit in the outside header MUST be ORed with the CE bit in the inside header. (In other words, a CE bit of 1 in the outside header has to be copied to the inside header 🫶.) If the ECT bit in either header is a 0, then the CE bit in the outside header is ignored 🙈. This requirement for de-encapsulated packets doesn't currently apply to IPsec tunnels 🔥.
Here's a specific example of ECN being used with encapsulation 👑. A flow wants ECN-capability so it can avoid an unnecessary packet drop for the encapsulated packet, caused by congestion at an intermediate node in the tunnel 🧚. This works by copying the ECN field in the inner IP header to the outer IP header when encapsulating, and then using the ECN field in the outer IP header to set the ECN field in the inner IP header when decapsulating 💅. The result is that routers along the tunnel can cause the CE bit to be set in the ECN field of the unencapsulated IP header of an ECN-capable packet when they experience congestion 🍓.
10.2. IPsec Tunnel Considerations 🔐
The IPsec protocol, as defined in [ESP, AH], doesn't include the IP header's ECN field in any of its cryptographic calculations. (In tunnel mode, the outer IP header's ECN field isn't included either.) 🌸 So a network node changing the ECN field has no effect on IPsec's end-to-end security, because it can't make any IPsec integrity check fail 💀. That also means IPsec doesn't protect against an adversary modifying the ECN field, like a man-in-the-middle attack, since the adversary's change won't affect IPsec's end-to-end security either 😤. In some environments, being able to change the ECN field without affecting IPsec integrity checks could count as a covert channel. If you need to get rid of that channel or reduce its bandwidth, the outer IP header's ECN field can be zeroed at the tunnel ingress and egress nodes 🦋.
Right now, IPsec requires that the inner header's ECN field not be changed by IPsec decapsulation processing at a tunnel egress node 🔥. This makes sure an adversary can't use changes to the ECN field to launch theft- or denial-of-service attacks across an IPsec tunnel endpoint, because any such changes get thrown out at the tunnel endpoint 🛡️. This document doesn't change that IPsec requirement 🫶. Because of how IPsec is specified right now, we suggest that experiments with ECN not be carried out for flows that will go through IPsec tunneling at the moment 🙈.
If the IPsec specifications change in the future so that a tunnel egress node is allowed to modify the ECN field in an inner IP header based on the ECN field value in the outer header (for example, copying part or all of the outer ECN field to the inner ECN field), or so that the ECN field of the outer IP header can be zeroed during encapsulation, then experiments with ECN may be used together with IPsec tunneling 🌈.
This discussion of ECN and IPsec tunnel considerations draws heavily on related discussions and documents from the Differentiated Services Working Group 💜.
10.3. Dropped or Corrupted Packets 💔
Here's another issue: a packet that has the CE bit set at one router and then gets dropped by a later router 😩. For the proposed use of ECN in this paper, which is a transport protocol like TCP where a dropped data packet is a sign of congestion, end nodes detect dropped data packets. Their congestion response to a dropped data packet is at least as strong as their congestion response to a received CE packet ✨.
But transport protocols like TCP don't always detect every packet drop 👀. One example is the drop of a "pure" ACK packet. TCP doesn't reduce the arrival rate of later ACK packets in response to an earlier dropped ACK packet 💀. So any proposal to extend ECN-Capability to packets like that would have to deal with the concerns raised by CE packets that get dropped later in the network 🍓.
Similarly, if a CE packet gets dropped later in the network because of corruption (bit errors), the end nodes should still invoke congestion control, just like TCP would today in response to a dropped data packet 🫶. Corrupted CE packets would have to be considered in any proposal for the network to tell apart packets dropped due to corruption from packets dropped due to congestion or buffer overflow 🎀.
11. A summary of related work ✨
Okay bestie, time for the related work tea 🍵. [Floyd94] looks at the pros and cons of adding ECN to the TCP/IP architecture. The simulation comparisons show one big win: ECN avoids unnecessary packet drops for short or delay-sensitive TCP connections. The second win is that it avoids some unnecessary retransmit timeouts in TCP 🙈. The paper goes deep on how ECN fits into TCP's congestion control mechanisms. The possible downsides it talks about are that a non-compliant TCP connection could falsely advertise itself as ECN-capable, and that a TCP ACK packet carrying an ECN-Echo message could itself get dropped in the network. The first issue is covered in Section 8 of this document, and the second is handled by the proposal in Section 5.1.3 for a CWR flag in the TCP header 💅.
[CKLTZ97] is an experimental implementation of ECN in IPv6, and it's giving real-world proof 🔥. They added ECN to an existing RED implementation for FreeBSD and ran a bunch of experiments: controlling the average queue size in the router, how ECN performs for a single TCP connection through a congested router, and fairness when multiple TCP connections are competing. The big takeaway is that dropping packets from a bulk-data transfer can wreck performance way more than marking packets does 😭.
Because the experimental implementation in [CKLTZ97] came out before some of the developments in this document, it doesn't match this document in every way. For example, it doesn't use the CWR flag. Instead, the TCP receiver sends the ECN-Echo bit on a single ACK packet 🧚.
[K98] and [CKLTZ98] build on [CKLTZ97] to dig deeper into the benefits of ECN for TCP 🦋. Their conclusions: ECN TCP gets moderately better throughput than non-ECN TCP, ECN TCP flows are fair towards non-ECN TCP flows, and ECN TCP stays robust with two-way traffic, congestion in both directions, and multiple congested gateways 👑. The experiments with lots of short web transfers found that most short connections take about the same time with or without ECN, but a small percentage of the short connections have really long transfer times in the non-ECN experiments compared to the ECN ones. This is especially dramatic for short connections whose first packet got dropped in the non-ECN runs, because those connections have to wait six seconds for the retransmit timer to expire 💀.
The ECN Web Page [ECN] has pointers to other ECN implementations that are in progress 🌸.
12. Conclusions
Okay so the big conclusion 🎀: with the current effort to implement RED, this is the right time for router vendors to look at how to build congestion avoidance mechanisms that don't rely on packet drops alone 💖. Applications and transports are getting more and more sensitive to the delay and loss of a single packet (think realtime traffic and short web transfers), so depending on packet loss as the normal way to signal congestion looks insufficient, or at the very least not optimal ✨.
13. Acknowledgements
Tons of people helped make this RFC happen, so let's give them their flowers 💐. Special thanks to Kenjiro Cho for the proposal for the TCP mechanism for negotiating ECN-Capability, Kevin Fall for the proposal of the CWR bit, Steve Blake for material on IPv4 Header Checksum Recalculation, and Jamal Hadi Salim for discussions of ECN issues 🫶. Thanks also to Steve Bellovin, Jim Bound, Brian Carpenter, Paul Ferguson, Stephen Kent, Greg Minshall, and Vern Paxson for discussions of security issues 🔐. And a big thank you to the Internet End-to-End Research Group for the ongoing discussions of these issues 🌈.
14. References 📚
[AH] Kent, S. and R. Atkinson, "IP Authentication Header",
RFC 2402, November 1998.
[B97] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[CKLT98] Chen, C., Krishnan, H., Leung, S., Tang, N., and Zhang,
L., "Implementing ECN for TCP/IPv6", presentation to the
ECN BOF at the L.A. IETF, March 1998, URL
"http://www.cs.ucla.edu/~hari/ecn-ietf.ps".
[DIFFSERV] Nichols, K., Blake, S., Baker, F. and D. Black,
"Definition of the Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers", RFC 2474, December
1998.
[ECN] "The ECN Web Page", URL "http://www-
nrg.ee.lbl.gov/floyd/ecn.html".
[ESP] Kent, S. and R. Atkinson, "IP Encapsulating Security
Payload", RFC 2406, November 1998.
[FJ93] Floyd, S., and Jacobson, V., "Random Early Detection
gateways for Congestion Avoidance", IEEE/ACM
Transactions on Networking, V.1 N.4, August 1993, p.
397-413. URL "ftp://ftp.ee.lbl.gov/papers/early.pdf".
[Floyd94] Floyd, S., "TCP and Explicit Congestion Notification",
ACM Computer Communication Review, V. 24 N. 5, October
1994, p. 10-23. URL
"ftp://ftp.ee.lbl.gov/papers/tcp_ecn.4.ps.Z".
[Floyd97] Floyd, S., and Fall, K., "Router Mechanisms to Support
End-to-End Congestion Control", Technical report,
February 1997. URL "http://www-
nrg.ee.lbl.gov/floyd/end2end-paper.html".
[Floyd98] Floyd, S., "The ECN Validation Test in the NS
Simulator", URL "http://www-mash.cs.berkeley.edu/ns/",
test tcl/test/test-all-ecn.
[K98] Krishnan, H., "Analyzing Explicit Congestion
Notification (ECN) benefits for TCP", Master's thesis,
UCLA, 1998, URL
"http://www.cs.ucla.edu/~hari/software/ecn/
ecn_report.ps.gz".
[FRED] Lin, D., and Morris, R., "Dynamics of Random Early
Detection", SIGCOMM '97, September 1997. URL
"http://www.inria.fr/rodeo/sigcomm97/program.html#ab078".
[Jacobson88] V. Jacobson, "Congestion Avoidance and Control", Proc.
ACM SIGCOMM '88, pp. 314-329. URL
"ftp://ftp.ee.lbl.gov/papers/congavoid.ps.Z".
[Jacobson90] V. Jacobson, "Modified TCP Congestion Avoidance
Algorithm", Message to end2end-interest mailing list,
April 1990. URL
"ftp://ftp.ee.lbl.gov/email/vanj.90apr30.txt".
[MJV96] S. McCanne, V. Jacobson, and M. Vetterli, "Receiver-
driven Layered Multicast", SIGCOMM '96, August 1996, pp.
117-130.
[RFC791] Postel, J., "Internet Protocol", STD 5, RFC 791,
September 1981.
[RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC1141] Mallory, T. and A. Kullberg, "Incremental Updating of
the Internet Checksum", RFC 1141, January 1990.
[RFC1349] Almquist, P., "Type of Service in the Internet Protocol
Suite", RFC 1349, July 1992.
[RFC1455] Eastlake, D., "Physical Link Security Type of Service",
RFC 1455, May 1993.
[RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast
Retransmit, and Fast Recovery Algorithms", RFC 2001,
January 1997.
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B.,
Deering, S., Estrin, D., Floyd, S., Jacobson, V.,
Minshall, G., Partridge, C., Peterson, L., Ramakrishnan,
K., Shenker, S., Wroclawski, J. and L. Zhang,
"Recommendations on Queue Management and Congestion
Avoidance in the Internet", RFC 2309, April 1998.
[RJ90] K. K. Ramakrishnan and Raj Jain, "A Binary Feedback
Scheme for Congestion Avoidance in Computer Networks",
ACM Transactions on Computer Systems, Vol.8, No.2, pp.
158-181, May 1990.15. Security Considerations 🔒
Security considerations have been discussed in Section 9. 😱 This one is seriously important, so go back to Section 9 and read it closely, because it's where the real security story lives 🚨.
16. IPv4 Header Checksum Recalculation 🧮
IPv4 header checksum recalculation is a real issue for some high-end router architectures that use an output-buffered switch 🫣. The reason is that most, if not all, of the header manipulation happens on the input side of the switch, while the ECN decision would need to be made locally at the output buffer. This is not an issue for IPv6, since there is no IPv6 header checksum. Also, the IPv4 TOS octet is the last byte of a 16-bit half-word 💜.
RFC 1141 [RFC1141] talks about incrementally updating the IPv4 checksum after the TTL field gets decremented ✨. The same trick works for updating the IPv4 checksum after the CE bit is set. Let HC be the original header checksum, and let HC' be the new header checksum after the CE bit has been set. For header checksums calculated with one's complement subtraction, HC' gets recalculated like this:
HC' = { HC - 1 HC > 1
{ 0x0000 HC = 1For header checksums calculated on two's complement machines, HC' gets recalculated like this after the CE bit is set:
HC' = { HC - 1 HC > 0
{ 0xFFFE HC = 017. The motivation for the ECT bit ✨
okay so bestie, this is the WHY behind the ECT bit and it's honestly iconic 🦋. ECN is going to roll out bit by bit in an Internet where some transport protocols and routers get ECN and some totally don't 💀. So the ECT bit lets a router drop packets from flows that aren't ECN-capable, but it can instead set the CE bit on flows that are ECN-capable 🎀. And because the ECT bit lets an end node get a CE mark instead of a dropped packet, the end node has a reason to actually deploy ECN 💖.
Now imagine there was NO ECT indication 🙈. Then the router would have to set the CE bit on packets from BOTH ECN-capable and non-ECN-capable flows. Result? Zero incentive for end nodes to deploy ECN, and no realistic path from a non-ECN world to an ECN world 😭. Picture the early stages, where only some flows are ECN-capable. When congestion first shows up, the packet dropping/marking rate is low, so routers only set CE bits and don't drop anything 🌸. But only the ECN-capable flows understand CE packets and respond to them 🥀. So those flows back off, while the non-ECN-capable flows don't notice the signals and keep opening their congestion windows 💅.
Two things can happen from there 👑. (1) The ECN-capable flows back off, the non-ECN-capable flows scoop up all the bandwidth, and congestion stays mild. Or (2) the ECN-capable flows back off, the non-ECN-capable flows don't, and congestion keeps climbing until the router switches from setting CE bits to dropping packets. Option two evens out the fairness, sure, but the ECN-capable flows still get basically nothing out of being ECN-capable, because all that extra congestion pushes the router into packet-dropping mode 😤.
A flow that says it's ECN-Capable but ignores CE bits is functionally the same as a flow that turns off congestion control, as covered in Sections 8 and 9 🙅♀️.
So, in a world where only some flows are ECN-capable but ECN-capable flows have NO way to tell the routers that fact, you get weaker and less fair congestion control across the Internet 📉. And that creates a strong incentive for end nodes NOT to deploy ECN 💔.
18. Why use two bits in the IP header? 🎀
okay so now that we know we NEED an ECT indication in the IP header, the next question is whether ECT (ECN-Capable Transport) and CE (Congestion Experienced) should share a single bit 🤔. This one-bit idea, explored in [Floyd94], would use one bit with two values. One value, "ECT and not CE", means ECN-Capable Transport. The other value, "CE or not ECT", means either Congestion Experienced or a non-ECN-Capable transport 🧩.
Here's the first big difference, and it's about packets that go through MULTIPLE congested routers 🚦. Say a CE packet shows up at a second congested router, and that router's active queue management picks it for marking or dropping. With one bit, the second router has NO choice but to drop it, because it can't tell a CE packet apart from a non-ECT packet 😩. With two bits, the second router can either drop the CE packet OR leave it alone with the CE bit still set ✨.
The second difference is about receivers 👀. With the one-bit version, receivers in a single flow can't tell CE packets and non-ECT packets apart. So an ECN-capable sender would have to clearly tell the receiver or receivers whether each packet was sent as ECN-Capable or not 📮. One option is for the sender to put that info in the transport header 🧾. Another option, which comes with a functional limitation, is for the sender to clearly say it's going to send ALL of its packets as ECN-Capable or ALL as non-ECN-Capable 🫠. For a multicast transport protocol, that clear signal would also have to be visible to receivers who join an ongoing multicast session 📡.
Two bits also just feel more robust 💪. The most critical issue, which is discussed in Section 8, is that the default should be a non-ECN-Capable transport. With two bits, that just means the ECT bit is OFF by default 🙌. With one bit, the overloaded bit has to default to the "CE or not ECT" position, and that's way less clear and way easier to get wrong in end nodes or routers 😵💫.
So to sum it up 📝. The one-bit version COULD work, but it has three big limitations compared to two bits 🥲. First, it's way more limited in how it can handle CE packets at a second congested router. Second, it either needs extra info carried in the transport header of ECN-Capable flows (to move that second bit's job somewhere else), or it makes senders in ECN-Capable flows accept that receivers must be able to figure out a priori which packets are ECN-Capable and which aren't 🫤. Third, it's more open to errors from faulty implementations that pick the wrong default for the ECN bit 🐛. We believe the extra bit in the IP header for the ECT bit is extremely valuable for dodging all of these problems 💖.
19. Historical definitions for the IPv4 TOS octet 🕰️
okay history lesson time, and it's giving drama 📜. RFC 791 [RFC791] defined the ToS (Type of Service) octet in the IP header 🧃. In RFC 791, bits 6 and 7 of the ToS octet are listed as "Reserved for Future Use", and they're shown set to zero 🫥. The first two fields of the ToS octet were defined as the Precedence and Type of Service (TOS) fields 🧁.
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | 0 | 0 | RFC 791
+-----+-----+-----+-----+-----+-----+-----+-----+RFC 1122 put bits 6 and 7 into the TOS field, but it didn't talk about any specific use for them 🤷♀️.
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | RFC 1122
+-----+-----+-----+-----+-----+-----+-----+-----+Then RFC 1349 [RFC1349] redefined the IPv4 TOS octet like this 🔄.
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | MBZ | RFC 1349
+-----+-----+-----+-----+-----+-----+-----+-----+Bit 6 in the TOS field got defined in RFC 1349 for "Minimize Monetary Cost" 💸. Besides the Precedence and Type of Service (TOS) fields, the last field, MBZ (for "must be zero"), was defined as currently unused 🫙. RFC 1349 said that "The originator of a datagram sets [the MBZ] field to zero (unless participating in an Internet protocol experiment which makes use of that bit)." 📖
RFC 1455 [RFC 1455] defined an experimental standard that used all four bits in the TOS field to request a guaranteed level of link security 🔐. Wild, right? 😳
RFC 1349 got obsoleted by "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers" [DIFFSERV] 🔁. In that one, bits 6 and 7 of the DS field are listed as Currently Unused (CU) 🫧. The first six bits of the DS field are defined as the Differentiated Services CodePoint (DSCP) 💜.
0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| DSCP | CU |
+-----+-----+-----+-----+-----+-----+-----+-----+So yeah, this history is SO unstable 😭. That means the ECN field defined in this document can't be guaranteed backwards compatible with every past use of these two bits 🥺. The damage a non-ECN-capable router could do is to "erase" the CE bit on an ECN-capable packet that reached it with the CE bit already set, or to set the CE bit even when there's no congestion 🚨. This was discussed in Section 10 on "Non-compliance in the Network" 🕵️♀️.
The damage that could be done in an ECN-capable environment by a non-ECN-capable end node sending packets with the ECT bit set was discussed in Section 9 on "Non-compliance by the End Nodes" 🙃.
AUTHORS' ADDRESSES 🖊️
K. K. Ramakrishnan AT&T Labs. Research
Phone: +1 (973) 360-8766 EMail: kkrama@research.att.com URL: http://www.research.att.com/info/kkrama
Sally Floyd Lawrence Berkeley National Laboratory
Phone: +1 (510) 486-7518 EMail: floyd@ee.lbl.gov URL: http://www-nrg.ee.lbl.gov/floyd/
Full Copyright Statement 💌
Copyright (C) The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 🫶
📦 where this page comes from
- ietf: rfc2481 1999-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms