RFC 5562
Adding Explicit Congestion Notification (ECN) Capability to TCP's SYN/ACK Packets
RFC 5562: Adding Explicit Congestion Notification (ECN) Capability to TCP's SYN/ACK Packets ✨
Okay bestie, the title block is giving formal, so here's the tea 🍵. This is RFC 5562, published in June 2009 by the Network Working Group. The authors are A. Kuzmanovic and A. Mondal (both Northwestern University), S. Floyd (ICSI), and K. Ramakrishnan (AT&T Labs Research). It's an Experimental doc, so it's not a standard, just a proposal people can try out 🧪. Its header doesn't list anything it obsoletes or updates, but it does describe an optional modification to RFC 3168.
Status of This Memo 📝
This memo defines an Experimental Protocol for the Internet community. It is not an Internet standard of any kind 🙈. The authors want your thoughts and suggestions for making it better, and anyone can distribute it freely.
Copyright Notice ©️
Copyright (c) 2009 IETF Trust and the people listed as the document authors. All rights reserved 💜.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, as they stood on the publication date. Go read them carefully, because they explain what you can and can't do with this doc: http://trustee.ietf.org/license-info.
This document may include material from IETF Documents or IETF Contributions that were published or made publicly available before November 10, 2008. Some of the people who own the copyright on that older material may not have let the IETF Trust allow changes outside the IETF Standards Process. So unless you get a proper license from those copyright holders, this document can't be modified outside the IETF Standards Process, and you can't make derivative works of it outside that process either. The exceptions are formatting it for publication as an RFC or translating it into languages other than English 🌍.
Abstract 🎀
So the big idea: this proposal is Experimental. It could be deployed on today's Internet, but it isn't a consensus that this is the best possible way to use Explicit Congestion Notification (ECN) in TCP SYN/ACK packets 🤔.
This doc describes an optional, experimental change to RFC 3168 that lets TCP SYN/ACK packets be ECN-Capable. For TCP, RFC 3168 sets the ECN-Capable codepoint on data packets, but not on SYN and SYN/ACK packets. Here's the thing though: if a SYN/ACK packet gets dropped, the transfer pays a huge price in the form of a retransmission timeout 😭. So this doc uses ECN for the SYN/ACK packet itself, when it's sent in reply to a SYN packet that has both ECN flags set in the TCP header. Those flags say "yes, I'm willing to use ECN" 🫶. Marking the first SYN/ACK as ECN-Capable can be a huge win for the connection, since it skips that painful retransmission timeout for a connection that isn't loading the network yet ✨.
The TCP responder (the sender of the SYN/ACK packet) must reply to a report of an ECN-marked SYN/ACK packet by resending a SYN/ACK packet that is not ECN-Capable. If that resent SYN/ACK gets acknowledged, the responder cuts its initial congestion window from two, three, or four segments down to one segment, which reduces how much load that connection puts on the network 🌸. If instead the SYN/ACK gets dropped, or the responder doesn't receive an acknowledgement in the specified time for some other reason, it follows the usual TCP standards for a dropped SYN/ACK packet and sets the retransmission timer ⏰.
Table of Contents 📚
1. Introduction ....................................................3
2. Conventions and Terminology .....................................5
3. Specification ...................................................6
3.1. SYN/ACK Packets Dropped in the Network ....................7
3.2. SYN/ACK Packets ECN-Marked in the Network .................8
3.3. Management Interface .....................................10
4. Discussion .....................................................11
4.1. Flooding Attacks .........................................11
4.2. The TCP SYN Packet .......................................11
4.3. SYN/ACK Packets and Packet Size ..........................12
4.4. Response to ECN-Marking of SYN/ACK Packets ...............12
5. Related Work ...................................................14
6. Performance Evaluation .........................................15
6.1. The Costs and Benefits of Adding ECN-Capability ..........15
6.2. An Evaluation of Different Responses to ECN-Marked
SYN/ACK Packets ..........................................16
6.3. Experiments ..............................................17
7. Security Considerations ........................................18
7.1. "Bad" Routers or Middleboxes .............................18
7.2. Congestion Collapse ......................................18
8. Conclusions ....................................................19
9. Acknowledgements ...............................................19
Appendix A. Report on Simulations .................................20
A.1. Simulations with RED in Packet Mode .......................20
A.2. Simulations with RED in Byte Mode .........................25
Appendix B. Issues of Incremental Deployment ......................28
Normative References ..............................................30
Informative References ............................................301. Introduction ✨
Okay so, TCP's congestion control has mostly used packet loss as its signal that the network is congested 📉. With tail-drop mechanisms, packets get dropped when buffers overflow, and the delay can get really high because the queue at a bottleneck router can be pretty big. If we only drop packets when the queue overflows, and TCP only reacts to those losses, you get:
- significantly higher packet delay;
- unnecessarily many packet losses; and
- unfairness due to synchronization effects.
Active Queue Management (AQM) mechanisms give us better control of bottleneck queues [RFC2309] 🎯. AQM is giving:
- better control of the queue, with reduced queuing delay;
- fewer packet drops; and
- better fairness because of fewer synchronization effects.
Adding ECN can make performance even better 🚀. When a router detects congestion before the buffer overflows, it can send a congestion indication either by dropping a packet or by setting the Congestion Experienced (CE) codepoint in the Explicit Congestion Notification (ECN) field in the IP header [RFC3168]. The IETF has standardized the CE codepoint in the IP header so routers can signal congestion. For incremental deployment and backwards compatibility, the ECN RFC [RFC3168] says routers may mark ECN-Capable packets that would otherwise have been dropped, using the Congestion Experienced codepoint in the ECN field. ECN lets TCP react to congestion without taking unnecessary retransmission timeouts. So using ECN has several benefits:
- For short transfers, a TCP connection's congestion window may be small 🐣. For example, if the current window only holds one packet and that packet gets dropped, TCP has to wait for a retransmission timeout to recover, and its overall throughput takes a hit. Similarly, if the window holds only a few packets and one of them is dropped, there might not be enough duplicate acknowledgements for a fast retransmission, so the sender of the data packet might have to wait several round-trip times (RTT) using Limited Transmit [RFC3042]. With ECN, short flows are less likely to have packets dropped, which sometimes avoids unnecessary delays or costly retransmission timeouts.
- Longer flows may not see a big throughput boost from ECN, but they may see lower loss 💅. That's a win for TCP applications that are sensitive to latency and loss, because there are fewer retransmissions.
RFC 3168 [RFC3168] sets the ECN-Capable codepoint on TCP data packets, but not on TCP SYN and SYN/ACK packets 🙃. RFC 3168 [RFC3168] also handles negotiating ECN between the two TCP endpoints during the SYN and SYN-ACK exchange, using flags in the TCP header. Being conservative, RFC 3168 [RFC3168] does not specify ECN for the first SYN/ACK packet itself. But because a dropped SYN/ACK is so expensive for the TCP transfer (hello, retransmission timeout), this document specifies ECN for the SYN/ACK packet itself 🫶. That can be a huge benefit for the connection, since it avoids the severe penalty of a retransmission timeout for a connection that hasn't started loading the network yet. The sender of the SYN/ACK packet must respond to a report of an ECN-marked SYN/ACK packet (a SYN/ACK packet with the CE codepoint set in the ECN field in the IP header) by sending a non-ECN-Capable SYN/ACK packet, and by cutting its initial congestion window from two, three, or four segments down to one segment, which reduces the load that connection puts on the network 🍓.
Using ECN for SYN/ACK packets has these potential benefits:
- Avoidance of a retransmission timeout;
- Improvement in the throughput of short connections.
This document specifies a change to RFC 3168 [RFC3168] that lets TCP SYN/ACK packets be ECN-Capable 🦋. Section 3 has the specification of the change, Section 4 talks through some of the issues, and Section 5 covers related work. Section 6 evaluates the specified change 💖.
2. Conventions and Terminology 🧚
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 📖.
We use the following terminology from RFC 3168 [RFC3168]:
The ECN field in the IP header:
- CE: the Congestion Experienced codepoint; and
- ECT: either one of the two ECN-Capable Transport codepoints.
The ECN flags in the TCP header:
CWR: the Congestion Window Reduced flag; andECE: the ECN-Echo flag.
ECN-setup packets:
- ECN-setup SYN packet: a SYN packet with the
ECEandCWRflags; - ECN-setup SYN-ACK packet: a SYN-ACK packet with
ECEbut notCWR.
In this document, we use the terms "initiator" and "responder" to refer to the sender of the SYN packet and of the SYN-ACK packet, respectively 🌈.
3. Specification ✨
okay so this section is the actual fix 💅 it tweaks RFC 3168 [RFC3168] so TCP SYN/ACK packets can be ECN-Capable 🦋
Section 6.1.1 of RFC 3168 [RFC3168] says "A host MUST NOT set ECT on SYN or SYN-ACK packets". 🙈 this doc is like "okay but what if a TCP node could totally set ECT on its reply to an initial ECN-setup SYN packet" 🎀 so the responding ECN-setup SYN/ACK packet gets ECT, which tells routers "hey, this SYN/ACK is ECN-Capable 💜". the payoff is that a congested router along the path can mark the packet instead of dropping it as its congestion signal. iconic 👑
the setup is: TCP node A sends node B an ECN-setup SYN packet, saying it's down to use ECN for this connection. per RFC 3168 [RFC3168], if node B is also down to use ECN, it replies with an ECN-setup SYN-ACK packet 🌸
3.1. SYN/ACK Packets Dropped in the Network 💀
Figure 1 is the drama where a congested router drops the SYN/ACK 😭 node B then waits for a retransmission timeout and sends the SYN/ACK again 🧚
---------------------------------------------------------------
TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ----------
ECN-setup SYN packet --->
ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, possibly ECT
3-second timer set
SYN/ACK dropped .
.
.
3-second timer expires
<--- ECN-setup SYN/ACK, not ECT
<--- ECN-setup SYN/ACK
Data/ACK --->
Data/ACK --->
<--- Data (one to four segments)
---------------------------------------------------------------
Figure 1: SYN exchange with the SYN/ACK packet droppedso if the SYN/ACK gets dropped in the network, the responder (node B) waits three seconds for the retransmission timer to expire [RFC2988] 🕰️ and if a SYN/ACK with the ECT codepoint is the one that got dropped, the responder should resend it without the ECN-Capable codepoint 🫶 (ngl, we're not aware of any middleboxes that drop SYN/ACK packets just because they have an ECN-Capable codepoint in the IP header 🙈 but we've learned to design protocols defensively here [RFC3360] 🛡️)
also, if syn-cookies were used by the responder (node B) in the Figure 1 exchange, the responder wouldn't set a timer when it sends the SYN/ACK [SYN-COOK] [RFC4987] 🍪 so if that SYN/ACK gets lost, the initiator (node A) has to time out and resend the SYN packet to trigger another SYN-ACK 💔
3.2. SYN/ACK Packets ECN-Marked in the Network 🌈
Figure 2 shows the SYN/ACK going out as ECN-Capable and getting ECN-marked instead of dropped at the congested router ✨ this doc specifies ECN+/TryOnce, which is different from the original ECN+ proposal in [ECN+] 💅 with ECN+/TryOnce, if the TCP responder is told the SYN/ACK was ECN-marked, it immediately sends a SYN/ACK that is not ECN-Capable 🔥 the responder is only allowed to send data packets after the TCP initiator reports receiving a SYN/ACK that is not ECN-marked 🎀
---------------------------------------------------------------
TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ----------
ECN-setup SYN packet --->
ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, ECT
3-second timer set
<--- Sets CE on SYN/ACK
<--- ECN-setup SYN/ACK, CE
ACK, ECN-Echo --->
ACK, ECN-Echo --->
Window reduced to one segment.
<--- ECN-setup SYN/ACK, not ECT
<--- ECN-setup SYN/ACK
Data/ACK, ECT --->
Data/ACK, ECT --->
<--- Data, ECT (one segment only)
---------------------------------------------------------------
Figure 2: SYN exchange with the SYN/ACK packet marked -
ECN+/TryOnceso if the initiator (node A) gets a SYN/ACK that the congested router ECN-marked (CE codepoint set) 👀 it restarts the retransmission timer 🔄 then it answers by setting the ECN-Echo flag in the TCP header of its ACK 💜 and it uses the standard rules for the cumulative acknowledgement field in that ACK 🧁
the initiator doesn't move from "SYN-Sent" to "Established" until it gets a SYN/ACK that is not ECN-marked 🙌
when the responder (node B) gets the ECN-Echo packet saying Congestion Experienced was set on its SYN/ACK 😱 it sets the initial congestion window to one segment, instead of the two segments [RFC2581] allows, or the three or four segments [RFC3390] allows 🪄 in Figure 2, the responder also sends a SYN/ACK that is not ECN-Capable, and it doesn't advance the send sequence number 🙈 it sets the retransmission timer too, and it follows RFC 2988 [RFC2988] for the RTO (retransmission timeout) ⏰
the TCP hosts follow the standard spec for responding to duplicate SYN/ACK packets (e.g., Section 3.4 of RFC 793 [RFC793]) 📜
now here's the tea ☕ this mechanism is different from RFC 3168 [RFC3168], which says "the sending TCP MUST restart the retransmission timer on receiving the ECN-Echo packet when the congestion window is one". RFC 3168 [RFC3168] doesn't allow SYN/ACK packets to be ECN-Capable at all 🙅♀️ it also says that in response to an ECN-Echo packet, the TCP responder sets the CWR flag in the TCP header of its next data packet, to acknowledge it got and reacted to the ECN-Echo flag 🫠 but here, when the ECN-Echo is acknowledging an ECN-Capable SYN/ACK, the TCP responder does not set CWR, and just sends a SYN/ACK that is not ECN-Capable 🎀 when the initiator gets that non-ECN-Capable SYN/ACK, it clears the ECN-Echo flag on the packets it sends back 🧹
---------------------------------------------------------------
TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ----------
ECN-setup SYN packet --->
ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, ECT
<--- Sets CE on SYN/ACK
<--- ECN-setup SYN/ACK, CE
ACK, ECN-Echo --->
ACK, ECN-Echo --->
Window reduced to one segment.
<--- ECN-setup SYN/ACK, not ECT
3-second timer set
SYN/ACK dropped .
.
.
3-second timer expires
<--- ECN-setup SYN/ACK, not ECT
<--- ECN-setup SYN/ACK, not ECT
Data/ACK, ECT --->
Data/ACK, ECT --->
<--- Data, ECT (one segment only)
---------------------------------------------------------------
Figure 3: SYN exchange with the first SYN/ACK packet marked
and the second SYN/ACK packet dropped - ECN+/TryOnceFigure 3 is the plot twist 🍿 compared to Figure 2, the first SYN/ACK gets ECN-marked and the second SYN/ACK gets dropped in the network 😭 like in Figure 2, the TCP responder sets a timer when it sends the second SYN/ACK 🕰️ and if that timer expires before the responder gets an acknowledgement from the other end, it resends the SYN/ACK following the TCP standards 🔁
3.3. Management Interface 🎛️
the TCP implementation using ECN-Capable SYN/ACK packets should include a management interface so people can turn ECN off for SYN/ACK packets 🙋♀️ this is for possible backwards compatibility problems, like the ones discussed in Appendix B 🧐
4. Discussion 🧠
okay the reasoning, bestie 🫶 when node B gets a TCP SYN packet with the ECN-Echo bit set in the TCP header, that tells it node A is ECN-Capable 💅 and if node B is ECN-Capable too, there's no obstacle to immediately setting one of the ECN-Capable codepoints in the IP header of the responding TCP SYN/ACK 🦋
setting an ECN-Capable codepoint in SYN/ACK packets can be a huge win, as discussed further in [ECN+] and briefly reported in Section 5 below ✨ congestion is most likely in the server-to-client direction 📈 so setting an ECN-Capable codepoint in SYN/ACKs can cut down on those three-second retransmission timeouts caused by dropped SYN/ACK packets 🙌
4.1. Flooding Attacks 🌊
setting an ECN-Capable codepoint in the responding TCP SYN/ACK packets doesn't raise any new or extra security vulnerabilities 🔒 for example, tricking servers or hosts into sending SYN/ACK packets to a third party for a "SYN/ACK flood" attack would be super inefficient 🙄 third parties would just drop those packets right away, since they'd know they never sent the TCP SYN packets in the first place 🗑️ plus, those SYN/ACK attacks would have the same signatures as the existing TCP SYN attacks 🕵️♀️ and getting servers or hosts to reply with SYN/ACK packets to congest a certain link would also be super inefficient, because SYN/ACK packets are small 🐜
BUT, the addition of ECN-Capability to SYN/ACK packets could let SYN/ACK packets survive for more hops along a network path before getting dropped 😬 which adds a bit to the ability of a SYN/ACK attack to flood a network link 🚨
4.2. The TCP SYN Packet 🎀
okay so there are several reasons why an ECN-Capable codepoint must not be set in the IP header of the initiating TCP SYN packet. first, when the TCP SYN packet is sent, there are no guarantees that the other TCP endpoint (node B in Figure 2) is ECN-Capable, or that it would be able to understand and react if the ECN CE codepoint was set by a congested router. 🥺
second, the ECN-Capable codepoint in TCP SYN packets could be misused by malicious clients to "improve" the well-known TCP SYN attack. 😤 by setting an ECN-Capable codepoint in TCP SYN packets, a malicious host might be able to inject a large number of TCP SYN packets through a potentially congested ECN-enabled router, congesting it even further. that is not it, bestie. 💀
for both these reasons, we continue the restriction that the TCP SYN packet must not have the ECN-Capable codepoint in the IP header set. 🚫
4.3. SYN/ACK Packets and Packet Size ✨
there are a number of router buffer architectures that have smaller dropping rates for small (SYN) packets than for large (data) packets. 🧚 for example, for a Drop-Tail queue in units of packets, where each packet takes a single slot in the buffer regardless of packet size, small and large packets are equally likely to be dropped. however, for a Drop-Tail queue in units of bytes, small packets are less likely to be dropped than are large ones. similarly, for Random Early Detection (RED) in packet mode, small and large packets are equally likely to be dropped or marked, while for RED in byte mode, a packet's chance of being dropped or marked is proportional to the packet size in bytes. 📦
for a congested router with an AQM mechanism in byte mode, where a packet's chance of being dropped or marked is proportional to the packet size in bytes, the drop or marking rate for TCP SYN/ACK packets should generally be low. in this case, the benefit of making SYN/ACK packets ECN-Capable should be similarly moderate. however, for a congested router with a Drop-Tail queue in units of packets or with an AQM mechanism in packet mode, and with no priority queuing for smaller packets, small and large packets should have the same probability of being dropped or marked. in such a case, making SYN/ACK packets ECN-Capable should be of significant benefit. 🔥
we believe that there are a wide range of behaviors in the real world in terms of the drop or mark behavior at routers as a function of packet size (see Section 10 of [Tools]). we note that all of these alternatives listed above are available in the NS simulator (Drop-Tail queues are by default in units of packets, while the default for RED queue management has been changed from packet mode to byte mode). 🌈
4.4. Response to ECN-Marking of SYN/ACK Packets 💜
one question is why TCP SYN/ACK packets should be treated differently from other packets in terms of the end node's response to an ECN-marked packet. 🤔 Section 5 of RFC 3168 [RFC3168] specifies the following:
Upon the receipt by an ECN-Capable transport of a single CE packet, the congestion control algorithms followed at the end-systems MUST be essentially the same as the congestion control response to a single dropped packet. For example, for ECN-Capable TCP the source TCP is required to halve its congestion window for any window of data containing either a packet drop or an ECN indication.
in particular, Section 6.1.2 of RFC 3168 [RFC3168] specifies that when the TCP congestion window consists of a single packet and that packet is ECN-marked in the network, then the data sender must reduce the sending rate below one packet per round-trip time, by waiting for one RTO before sending another packet. if the RTO was set to the average round-trip time, this would result in halving the sending rate; because the RTO is in fact larger than the average round-trip time, the sending rate is reduced to less than half of its previous value. 📉
TCP's congestion control response to the dropping of a SYN/ACK packet is to wait a default time before sending another packet. this document argues that ECN gives end-systems a wider range of possible responses to the marking of a SYN/ACK packet, and that waiting a default time before sending another packet is not the desired response. 🙅♀️
on the conservative end, one could assume an effective congestion window of one packet for the SYN/ACK packet, and respond to an ECN-marked SYN/ACK packet by reducing the sending rate to one packet every two round-trip times. as an approximation, the TCP end node could measure the round-trip time T between the sending of the SYN/ACK packet and the receipt of the acknowledgement, and reply to the acknowledgement of the ECN-marked SYN/ACK packet by waiting T seconds before sending a data packet. ⏳
however, we note that for an ECN-marked SYN/ACK packet, halving the congestion window is not the same as halving the sending rate; there is no "sending rate" associated with an ECN-Capable SYN/ACK packet, as such packets are only sent as the first packet in a connection from that host. further, a router's marking of a SYN/ACK packet is not affected by any past history of that connection. 🧐
adding ECN-Capability to SYN/ACK packets allows the response of the responder setting the initial congestion window to one packet, instead of its allowed default value of two, three, or four packets. the responder sends a non-ECN-Capable SYN/ACK packet, and proceeds with a cautious sending rate of one data packet per round-trip time after that SYN/ACK packet is acknowledged. this document argues that this approach is useful to users, with no dangers of congestion collapse or of starvation of competing traffic. this is discussed in more detail below in Section 6.2. 🦋
we note that if the data transfer is entirely from node A to node B, there is still a difference in performance between the original mechanism ECN+ and the mechanism ECN+/TryOnce specified in this document. in particular, with ECN+/TryOnce, the TCP originator does not send data packets until it has received a non-ECN-marked SYN/ACK packet from the other end. 🫶
5. Related Work 📚
the addition of ECN-Capability to TCP's SYN/ACK packets was initially proposed in [ECN+]. the paper includes an extensive set of simulation and testbed experiments to evaluate the effects of the proposal, using several Active Queue Management (AQM) mechanisms, including Random Early Detection (RED) [RED], Random Exponential Marking (REM) [REM], and Proportional Integrator (PI) [PI]. the performance measures were the end-to-end response times for each request/response pair, and the aggregate throughput on the bottleneck link. the end-to-end response time was computed as the time from the moment when the request for the file is sent to the server, until that file is successfully downloaded by the client. 🗂️
the measurements from [ECN+] show that setting an ECN-Capable codepoint in the IP packet header in TCP SYN/ACK packets systematically improves performance with all evaluated AQM schemes. when SYN/ACK packets at a congested router are ECN-marked instead of dropped, this can avoid a long initial retransmission timeout, improving the response time for the affected flow dramatically. 🚀
[ECN+] shows that the impact on aggregate throughput can also be quite significant, because marking SYN ACK packets can prevent larger flows from suffering long timeouts before being "admitted" into the network. in addition, the testbed measurements from [ECN+] show that web servers setting the ECN-Capable codepoint in TCP SYN/ACK packets could serve more requests. 🌸
as a final step, [ECN+] explores the coexistence of flows that do and don't set the ECN-Capable codepoint in TCP SYN/ACK packets. the results in [ECN+] show that both types of flows can coexist, with some performance degradation for flows that don't use ECN+. flows that do use ECN+ improve their end-to-end performance. at the same time, the performance degradation for flows that don't use ECN+, as a result of the flows that do use ECN+, increases as a greater fraction of flows use ECN+. 🥹
6. Performance Evaluation ✨
6.1. The Costs and Benefits of Adding ECN-Capability 💅
Okay bestie, [ECN+] digs into the costs and benefits of making SYN/ACK packets ECN-Capable, using both simulations and experiments 🧪. Making SYN/ACK packets ECN-Capable could be a huge win for ECN connections that would otherwise have had their SYN/ACK packet dropped in the network 😭. With ECN-Capability, that SYN/ACK gets marked instead of dropped, and honestly that's giving main character energy 🦋.
SYN/ACK packets make up a surprisingly big chunk of traffic 📊. Measurements on links dominated by web traffic show that 15-20% of the packets can be SYN/ACK packets [SCJO01]. That's a lot of SYN/ACKs just chilling on the wire 🫠.
How much this helps depends partly on how big the data transfer is 📦. Dropping a SYN/ACK packet can make the download of a short file take an order of magnitude longer, because it forces a three-second retransmission timeout ⏳💀. For longer-lived flows, a dropped SYN/ACK packet hurts file download time less dramatically 🌸. Still, even for longer-lived flows, adding ECN-Capability to SYN/ACK packets can improve fairness among long-lived flows, since newly arriving flows would be less likely to have to sit around waiting for retransmission timeouts 🎀.
A big question is what fraction of connections would actually see this benefit in a particular scenario 🤔. Specifically:
- What fraction of arriving SYN/ACK packets are dropped at the congested router when the SYN/ACK packets are not ECN-Capable?
- Of those SYN/ACK packets that are dropped, what fraction would have been ECN-marked instead of dropped if the SYN/ACK packets had been ECN-Capable?
To answer (1), you have to look at more than just how congested things are 👀. You also need to consider the queue architecture at the congested link. As described in Section 4 above, for some queue architectures, small packets are less likely to be dropped than large ones 📉. In that kind of environment, SYN/ACK packets would have lower packet drop rates. That means question (1) can't necessarily be inferred from the overall packet drop rate, but it can be answered by measuring the drop rate for SYN/ACK packets directly 🔍. In such an environment, adding ECN-Capability to SYN/ACK packets would be a less dramatic benefit than in environments where all packets are equally likely to be dropped regardless of packet size 😤.
As question (2) implies, even if all of the SYN/ACK packets were ECN-Capable, some SYN/ACK packets could still be dropped instead of marked at the congested link 🥲. The full answer to question (2) depends on the details of the router's queue management mechanism. If congestion is bad enough, and the queue management mechanism can't stop the buffer from overflowing, then SYN/ACK packets will be dropped rather than marked upon buffer overflow, whether or not they are ECN-Capable 🚨.
Some AQM mechanisms mark ECN-Capable packets instead of dropping them any time that's possible, meaning any time the buffer isn't full yet 💖. Other AQM mechanisms, though, like the RED mechanism as recommended in [RED], drop packets rather than marking them once the packet drop/mark rate goes over a certain threshold, e.g., 10%, even when the packets are ECN-Capable 🙅♀️. For a router with that kind of AQM mechanism, when congestion gets bad enough to cause a high drop/mark rate, some SYN/ACK packets would be dropped instead of marked whether or not they were ECN-Capable 💔.
So the benefit of adding ECN-Capability to SYN/ACK packets depends not just on the overall packet drop rate in the network, but also on the queue management architecture at the congested link ✨🎀.
6.2. An Evaluation of Different Responses to ECN-Marked SYN/ACK Packets 🧐
Here's the plan 📝. This document specifies that the end node responds to the report of an ECN-marked SYN/ACK packet by setting the initial congestion window to one segment, instead of its possible default value of two to four segments, and then resending a SYN/ACK packet that is not ECN-Capable. We call this ECN+/TryOnce 🫶.
Section 4 also talked about two other possible responses to an ECN-marked SYN/ACK packet 🧠. In ECN+, the original proposal from [ECN+], the end node responds to the report of an ECN-marked SYN/ACK packet by setting the initial congestion window to one segment and immediately sending a data packet, if it has one to send. In ECN+/Wait, the end node responds to the report of an ECN-marked SYN/ACK packet by setting the initial congestion window to one segment and then waiting an RTT before sending a data packet ⏱️.
Simulations comparing Standard ECN (without ECN-marked SYN/ACK packets), ECN+, ECN+/Wait, and ECN/TryOnce show little difference, in terms of aggregate congestion, between ECN+ and ECN+/Wait 🤝. But in some scenarios with queues that are packet-based rather than byte-based, and with packet drop rates above 25% without ECN+, using ECN+ or ECN+/Wait can more than double the packet drop rates, pushing them above 50% 😱🔥. The details are in Tables 1 and 3 of Appendix A below. ECN+/TryOnce does not increase the packet drop rate in scenarios of high congestion 🙌. So ECN+/TryOnce is superior to ECN+ or to ECN+/Wait, which both significantly increase the packet drop rate in scenarios of high congestion 👑. At the same time, ECN+/TryOnce gives a performance improvement similar to that of ECN+ or ECN+/Wait (Tables 2 and 4 of Appendix A) 🏆.
Our conclusions: ECN+/TryOnce is safe, has significant benefits for the user, and avoids the problems of ECN+ or ECN+/Wait under extreme levels of congestion 💅✨. That's why this document specifies the use of ECN+/TryOnce 🎀.
Note: We only discovered the occasional congestion-related problems of ECN+ and of ECN+/Wait when re-running the simulations with an updated version of the ns-2 simulator, after the document had almost completed the standardization process 💀🙈.
6.3. Experiments 🧪
This section talks about experiments that would be useful before a widespread deployment of ECN-Capability for TCP SYN/ACK packets 🚀.
Section 7.1 below discusses some known deployment problems of ECN, where routers or middleboxes react inappropriately to packets that use ECN codepoints in the IP or TCP packet headers 🫣. One goal of a measurement study of ECN-Capability for TCP SYN/ACK packets would be to find out whether any routers or middleboxes react inappropriately to TCP SYN/ACK packets containing an ECN-Capable or CE codepoint in the IP header 🔎. A second goal would be to check the deployment status of older TCP implementations that are ECN-Capable but don't respond to ECN-Capability for SYN/ACK packets 📋. (This is discussed in more detail in Appendix B below.)
Following the discussion in Section 6.2, an experimental study could explore the use of ECN-Capability for TCP SYN/ACK packets in highly congested environments with ECN-Capable routers 🧃.
7. Security Considerations 🔐
TCP packets carrying the ECT codepoint in IP headers can be marked rather than dropped by ECN-Capable routers 🌈. This raises several security concerns, which we discuss below 🚨.
7.1. "Bad" Routers or Middleboxes 👾
There are a number of known deployment problems from using ECN with TCP traffic on the Internet 😩. The first reported problem, dating back to 2000, is a small but decreasing number of routers or middleboxes that reset a TCP connection in response to TCP SYN packets that use flags in the TCP header to negotiate ECN-Capability [Kelson00] [RFC3360] [MAF05]. At the March 2007 IETF, Dave Thaler reported two new problems encountered by TCP connections using ECN 🗣️. The first of the two concerns routers that crash when a TCP data packet arrives with the ECN field in the IP header set to the codepoint ECT(0) or ECT(1), which indicates that an ECN-Capable connection has been established 💥 [SBT07].
There's no evidence that any routers or middleboxes drop SYN/ACK packets that contain an ECN-Capable or CE codepoint in the IP header, but such behavior can't be ruled out 😬. (There seem to be a number of routers or middleboxes that drop TCP SYN packets containing known or unknown IP options; see figure 1 of [MAF05].) So, as specified in Section 3, if a SYN/ACK packet with the ECT or CE codepoint is dropped, the TCP node should resend the SYN/ACK packet without the ECN-Capable codepoint 🔁. There's also no evidence that any routers or middleboxes crash when a SYN/ACK arrives with an ECN-Capable or CE codepoint in the IP header (beyond the routers already known to crash when a data packet arrives with either ECT(0) or ECT(1)). We haven't run any measurement studies on this, though [F07] 🤷♀️.
7.2. Congestion Collapse 📉
Because TCP SYN/ACK packets carrying an ECT codepoint could be ECN-marked instead of dropped at an ECN-Capable router, the concern is whether this could cause congestion or make performance worse in highly congested scenarios 😰. However, after learning that a SYN/ACK packet was ECN-marked, the responder sends a SYN/ACK packet that is not ECN-Capable. If this SYN/ACK packet is dropped, the responder then waits for a retransmission timeout, as specified in the TCP standards ⏳. Also, routers are free to drop rather than mark arriving packets in times of high congestion, regardless of whether the packets are ECN-Capable 🫠. When congestion is very high and a router's buffer is full, the router has no choice but to drop rather than mark an arriving packet 🙃.
The simulations reported in Appendix A show that, even with demanding traffic mixes dominated by short flows and high levels of congestion, the aggregate packet dropping rates aren't significantly different with Standard ECN or with ECN+/TryOnce 🎉. However, in our simulations, there's one scenario where ECN+ or ECN+/Wait results in a significantly higher packet drop rate than ECN or ECN+/TryOnce (Tables 1 and 3 in Appendix A below) 👀.
8. Conclusions 🎀
This document specifies a modification to RFC 3168 [RFC3168] that lets TCP nodes send SYN/ACK packets as ECN-Capable 🧚. Making the SYN/ACK packet ECN-Capable avoids the high cost to a TCP transfer when a congested router drops a SYN/ACK packet, because it avoids the resulting retransmission timeout ✨. That improves the throughput of short connections 🚀. This document specifies the ECN+/TryOnce mechanism for ECN-Capability for SYN/ACK packets, where the sender of the SYN/ACK packet responds to an ECN mark by reducing its initial congestion window from two, three, or four segments to one segment, and then sending a SYN/ACK packet that is not ECN-Capable 💖. Adding ECN-Capability to SYN/ACK packets is especially useful in the server-to-client direction, where congestion is more likely to happen 🖥️. In this case, the initial information provided by the ECN marking in the SYN/ACK packet lets the server adjust the initial load it places on the network appropriately, while avoiding the delay of a retransmission timeout 🙌.
9. Acknowledgements 🫶
We thank Anil Agarwal, Mark Allman, Remi Denis-Courmont, Wesley Eddy, Lars Eggert, Alfred Hoenes, Janardhan Iyengar, and Pasi Sarolahti for feedback on earlier working drafts of this document 💜. We thank Adam Langley [L08] for contributing a patch for ECN+/TryOnce for the Linux development tree 🐧🌸.
Appendix A. Report on Simulations ✨
okay bestie, this appendix is where the team shows their receipts 🧾💅 it reports on simulations that check the costs of adding ECN+ when the network is super congested 😭, and it also compares ECN, ECN+, ECN+/Wait, and ECN+/TryOnce against each other 🦋
The simulations run with a bunch of file-size distributions using the PackMime traffic generator in the ns-2 simulator 🧚 they all use a heavy-tailed distribution of file sizes 🌈 The tables below use a mean file size of 3 Kbytes, so we can see what happens when there are tons of tiny transfers 🍓 Other runs used mean file sizes of 5 Kbytes, 7 Kbytes, 14 Kbytes, and 17 Kbytes ⭐ Each chart title tells you the targeted average load from the traffic generator. BUT, because the file sizes are heavy-tailed and the runs only last 85 seconds (ten of those are warm-up time 🙈), the actual load is often way smaller than the targeted load, so keep that in mind 💀 The congested link is 100 Mbps. RED runs in gentle mode, and ECN-Capable packets only get dropped instead of marked when the buffer is full and the router has no other choice 😭
We look at three possible ways a TCP node can respond when it gets a report of an ECN-marked SYN/ACK packet 👑
- With ECN+, the TCP node sends a data packet right away, with an initial congestion window of one segment 🚀
- With ECN+/Wait, the TCP node waits one round-trip time before it sends a data packet 🎀 The responder already has one measurement of the round-trip time by the time the acknowledgement for the SYN/ACK packet shows up, so it's not starting from zero 🫶
- With ECN+/TryOnce, which is the mechanism standardized in this document 💖, the TCP responder replies to a report of an ECN-marked SYN/ACK packet by sending a SYN/ACK packet that is not ECN-Capable, and it reduces the initial congestion window to one segment ✨
The simulation scripts are available on [ECN-SYN] 🌸, along with graphs showing the distribution of response times for the TCP connections 📊 iykyk, the graphs are where the real tea is ☕
A.1. Simulations with RED in Packet Mode
okay so the RED-in-packet-mode sims are giving ECN+ is actually useful when things are moderately or highly congested 💖 BUT, for the 125% target load runs, once ECN has a packet loss rate over 25%, ECN+ and ECN+/Wait both end up with a loss rate over 50% 😭 (ECN+/TryOnce is the exception and stays below plain ECN 👑). For the response time distribution, ECN+, ECN+/Wait, and ECN+/TryOnce all significantly improve response times compared to Standard ECN ✨
Table 1 shows the congestion levels for these RED-in-packet-mode sims with a queue in packets 🎀. To explore a worst-case scenario, they use a traffic mix with an unrealistically small flow size distribution, with a mean flow size of 3 Kbytes 🦋. For each table showing a particular traffic load, the four rows show the number of packets dropped, the number of packets ECN-marked, the aggregate packet drop rate, and the aggregate throughput. The four columns are Standard ECN, ECN+, ECN+/Wait, and ECN+/TryOnce 💅
These sims were run with RED set to mark instead of drop packets any time the queue is not full. This is a worst-case scenario for ECN+ and its variants 🌸 For the default RED implementation in the ns-2 simulator, when the average queue size goes over a configured threshold, the router drops all arriving packets. So with this RED mechanism, it's less likely that ECN+ or one of its variants would push the average queue size above that threshold 🧚
The usefulness of ECN+: first thing to notice is that in all the sims, ECN+ or ECN+/Wait significantly increases the number of packets marked 🔥 ECN+/TryOnce significantly increases marked packets in the moderate congestion sims, but gives a more moderate increase in the higher congestion sims. Still, the CDF in Table 2 shows that ECN+, ECN+/Wait, and ECN+/TryOnce all improve response times in every sim, whether congestion is moderate or larger 🫶
Little increase in congestion, sometimes: second thing! In the low or moderate congestion sims (packet drop rates under 10%), ECN+, ECN+/Wait, and ECN+/TryOnce all decrease the aggregate packet drop rate compared to plain ECN 🍓 That makes sense, because at low or moderate congestion ECN+ lets SYN/ACK packets get marked instead of dropped, so ECN+ doesn't add to the aggregate congestion ✨ BUT, in the sims with ECN packet drop rates of 15% or higher, ECN+ or ECN+/Wait increases the aggregate packet drop rate, sometimes even doubling it 💀
Comparing ECN+, ECN+/Wait, and ECN+/TryOnce: the aggregate packet drop rate is generally higher with ECN+/Wait than with ECN+. So there's no congestion-related reason to pick ECN+/Wait over ECN+ 🙈 Meanwhile, the aggregate packet drop rate with ECN+/TryOnce is often significantly lower than with ECN, ECN+, or ECN+/Wait 👑
Target Load = 95%:
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 20,516 11,226 11,735 16,755`
Marked 30,586 37,741 37,425 40,764
Loss rate 1.41% 0.78% 0.81% 1.02%
Throughput 81% 81% 81% 81%
Target Load = 110%:
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 165,566 106,083 147,180 208,422
Marked 179,735 281,306 308,473 235,483
Loss rate 9.01% 6.12% 8.02% 6.89%
Throughput 92% 92% 92% 94%
Target Load = 125%:
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 600,628 1,746,768 2,176,530 625,552
Marked 418,433 1,166,450 1,164,932 439,847
Loss rate 25.45% 51.73% 56.87% 18.31%
Throughput 94% 98% 97% 95%
Target Load = 150%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 1,449,945 1,565,0517 1,563,0801 1,351,637
Marked 669,840 583,378 591,315 684,715
Loss rate 46.7% 59.0% 59.0% 32.7%
Throughput 88% 94% 94% 92%
Table 1: Simulations with an average flow size of 3 Kbytes, a 100
Mbps link, RED in packet mode, queue in packets Target Load = 95%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.07 0.26 0.51 0.82 0.96 0.97 0.97 0.97 1.00 1.00
ECN+: 0.00 0.07 0.27 0.53 0.85 0.99 1.00 1.00 1.00 1.00 1.00
Wait: 0.00 0.07 0.26 0.51 0.83 0.97 1.00 1.00 1.00 1.00 1.00
Once: 0.00 0.07 0.24 0.49 0.83 0.97 1.00 1.00 1.00 1.00 1.00
Target Load = 110%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.05 0.19 0.41 0.67 0.79 0.80 0.80 0.80 0.96 0.96
ECN+: 0.00 0.07 0.22 0.48 0.81 0.96 1.00 1.00 1.00 1.00 1.00
Wait: 0.00 0.05 0.18 0.38 0.64 0.77 0.95 1.00 1.00 1.00 1.00
Once: 0.00 0.06 0.19 0.42 0.70 0.86 0.95 0.96 0.96 0.99 0.99
Target Load = 125%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.04 0.13 0.27 0.46 0.56 0.58 0.59 0.59 0.82 0.82
ECN+: 0.00 0.06 0.18 0.33 0.58 0.76 0.97 0.99 0.99 1.00 1.00
Wait: 0.00 0.01 0.06 0.13 0.21 0.27 0.68 0.98 0.99 1.00 1.00
Once: 0.00 0.05 0.16 0.34 0.58 0.73 0.85 0.87 0.87 0.95 0.96
Target Load = 150%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.03 0.08 0.18 0.31 0.39 0.42 0.42 0.43 0.68 0.68
ECN+: 0.00 0.06 0.18 0.39 0.67 0.81 0.83 0.84 0.84 0.93 0.93
Wait: 0.00 0.06 0.18 0.39 0.67 0.81 0.83 0.84 0.84 0.93 0.94
Once: 0.00 0.04 0.13 0.27 0.46 0.59 0.72 0.75 0.75 0.88 0.88
Table 2: The cumulative distribution function (CDF) for transfer
times, for simulations with an average flow size of 3
Kbytes, a 100 Mbps link, RED in packet mode, queue in
packets (the graphs are available from
"http://www.icir.org/floyd/ecn-syn/") Target Load = 95%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 8,448 6,362 7,740 14,107
Marked 9,891 16,787 17,456 16,132
Loss rate 5.5% 4.3% 5.0% 5.0%
Throughput 78% 78% 78% 81%
Target Load = 110%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 31,284 29,773 49,297 45,277
Marked 28,429 54,729 60,383 34,622
Loss rate 15.3% 15.2% 21.9% 13.6%
Throughput 97% 96% 96% 94%
Target Load = 125%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 61,433 176,682 214,096 75,612
Marked 44,408 119,728 117,301 49,442
Loss rate 25.4% 51.9% 56.0% 22.3%
Throughput 97% 98% 98% 96%
Target Load = 150%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 130,007 251,856 326,845 133,603
Marked 63,066 146,757 147,239 66,444
Loss rate 42.5% 61.3% 67.3% 31.7%
Throughput 93% 99% 99% 94%
Table 3: Simulations with an average flow size of 3 Kbytes, a 10
Mbps link, RED in packet mode, queue in packets Target Load = 95%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.05 0.18 0.42 0.70 0.86 0.88 0.88 0.88 0.98 0.98
ECN+: 0.00 0.06 0.20 0.45 0.78 0.96 1.00 1.00 1.00 1.00 1.00
Wait: 0.00 0.05 0.18 0.40 0.68 0.84 0.96 1.00 1.00 1.00 1.00
Once: 0.00 0.05 0.18 0.40 0.71 0.88 0.96 0.97 0.97 0.99 0.99
Target Load = 110%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.03 0.13 0.29 0.52 0.66 0.69 0.69 0.69 0.91 0.91
ECN+: 0.00 0.05 0.17 0.36 0.66 0.88 0.98 0.99 1.00 1.00 1.00
Wait: 0.00 0.02 0.08 0.20 0.35 0.47 0.76 0.98 1.00 1.00 1.00
Once: 0.00 0.05 0.15 0.32 0.58 0.75 0.88 0.90 0.90 0.97 0.97
Target Load = 125%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.03 0.10 0.22 0.40 0.52 0.56 0.56 0.57 0.82 0.82
ECN+: 0.00 0.03 0.14 0.27 0.49 0.70 0.96 0.99 0.99 0.99 1.00
Wait: 0.00 0.00 0.03 0.07 0.12 0.18 0.50 0.94 0.99 0.99 1.00
Once: 0.00 0.04 0.13 0.28 0.51 0.66 0.81 0.84 0.84 0.94 0.94
Target Load = 150%:
TIME: 10 100 200 300 400 500 1000 2000 3000 4000 5000
------------------------------------------------------
ECN: 0.00 0.02 0.07 0.15 0.28 0.38 0.42 0.42 0.43 0.67 0.68
ECN+: 0.00 0.00 0.00 0.00 0.01 0.05 0.68 0.83 0.95 0.97 0.98
Wait: 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.62 0.83 0.93 0.97
Once: 0.00 0.03 0.11 0.24 0.42 0.56 0.71 0.75 0.75 0.88 0.88
Table 4: The cumulative distribution function (CDF) for transfer
times, for simulations with an average flow size of 3
Kbytes, a 10 Mbps link, RED in packet mode, queue in
packets (the graphs are available from
"http://www.icir.org/floyd/ecn-syn/")A.2. Simulations with RED in Byte Mode
Okay so Table 5 is where things get spicy 🌸 These sims use RED in byte mode, with the queue measured in bytes. The good news is that there's no significant increase in aggregate congestion with ECN+, ECN+/Wait, or ECN+/TryOnce ✨
But here's the plot twist 🙈 Unlike the packet mode sims, byte mode shows little benefit from ECN+ or ECN+/Wait. The packet marking rate with those is not much different from Standard ECN. Why? With RED in byte mode, small packets like SYN/ACK packets are rarely dropped or marked. So using ECN+ here doesn't hurt you, but you also don't really need it, since small packets are unlikely to get dropped or marked in the first place 💅
Target Load = 95%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 766 446 427 408
Marked 32,683 34,289 33,412 31,892
Loss rate 0.05% 0.03% 0.03% 0.03%
Throughput 81% 81% 81% 81%
Target Load = 110%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 2,496 2,110 1,733 2,020
Marked 220,573 258,696 230,955 214,604
Loss rate 0.15% 0.13% 0.11% 0.11%
Throughput 92% 91% 92% 92%
Target Load = 125%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 20,032 13,555 13,979 16,918
Marked 725,165 726,992 726,823 615,235
Loss rate 1.11% 0.76% 0.78% 0.66%
Throughput 95% 95% 95% 96%
Target Load = 150%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 484,251 483,847 507,727 600,737
Marked 865,905 872,254 873,317 818,451
Loss rate 19.09% 19.13% 19.71% 12.66%
Throughput 99% 98% 99% 99%Table 5: Simulations with an average flow size of 3 Kbytes, a 100 Mbps link, RED in byte mode, queue in bytes
Target Load = 95%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 142 77 103 99
Marked 11,694 11,387 11,604 12,129
Loss rate 0.1% 0.1% 0.1% 0.1%
Throughput 78% 78% 78% 78%
Target Load = 110%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 338 210 247 274
Marked 41,676 40,412 44,173 36,265
Loss rate 0.2% 0.1% 0.1% 0.1%
Throughput 94% 94% 94% 96%
Target Load = 125%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 1,559 951 978 1,723
Marked 74,933 75,499 75,481 59,670
Loss rate 0.8% 0.5% 0.5% 0.6%
Throughput 99% 99% 99% 96%
Target Load = 150%
ECN ECN+ ECN+/Wait ECN+/TryOnce
------- ------- ------- ----------
Dropped 2,374 1,528 1,515 4,848
Marked 85,739 86,428 86,144 81,350
Loss rate 1.2% 0.8% 0.8% 1.4%
Throughput 99% 98% 98% 98%Table 6: Simulations with an average flow size of 3 Kbytes, a 10 Mbps link, RED in byte mode, queue in bytes
Appendix B. Issues of Incremental Deployment ✨
Let's talk about the backwards compatibility drama 🎀 For TCP node B to send a SYN/ACK packet as ECN-Capable, node B has to have received an ECN-setup SYN packet from node A. But it's possible that node A supports ECN and still either ignores the CE codepoint on received SYN/ACK packets, or ignores SYN/ACK packets with the ECT or CE codepoint set.
If the TCP initiator ignores the CE codepoint on received SYN/ACK packets, the TCP responder won't respond to that congestion indication. Honestly, this seems like an acceptable cost for rolling out ECN-Capability for TCP's SYN/ACK packets incrementally 🫶 The responder wouldn't reduce its initial congestion window from two, three, or four segments down to one segment like it should, and it wouldn't send a non-ECN-Capable SYN/ACK packet to complete the SYN exchange. Still, the TCP end nodes would respond correctly to any later CE indications on data packets in the connection.
Figure 4 shows an interchange with the SYN/ACK packet ECN-marked, but with the ECN mark ignored by the TCP originator 👀
---------------------------------------------------------------
TCP Node A Router TCP Node B
(initiator) (responder)
---------- ------ ----------
ECN-setup SYN packet --->
ECN-setup SYN packet --->
<--- ECN-setup SYN/ACK, ECT
<--- Sets CE on SYN/ACK
<--- ECN-setup SYN/ACK, CE
Data/ACK, No ECN-Echo --->
Data/ACK --->
<--- Data (up to four packets)
---------------------------------------------------------------
Figure 4: SYN exchange with the SYN/ACK packet marked,
but with the ECN mark ignored by the TCP initiatorSo, to be super explicit: when a TCP connection has an initiator that supports ECN but does not support ECN-Capability for SYN/ACK packets, and a responder that does support ECN-Capability for SYN/ACK packets, the ECN-Capable SYN/ACK packets might get marked instead of dropped in the network. And then the responder won't learn about the ECN mark on the SYN/ACK packet 😭 This wouldn't be a big deal if most of the responder's packets were in long-lived TCP connections. It would be more problematic if most packets came from connections with four data packets, where the responder sends its data right after the SYN/ACK exchange. Of course, with severe congestion, the SYN/ACK packets would likely get dropped rather than ECN-marked at the congested router, which stops the responder from piling on with its initial window of four data packets.
It's also possible that some older TCP implementations have the initiator ignore arriving SYN/ACK packets that have the ECT or CE codepoint set. That would delay connection setup, with the initiator re-sending the SYN packet after a retransmission timeout. The authors aren't aware of any TCP implementations that behave this way 🌈
One possible fix for backwards compatibility problems would be for TCP initiators to use a TCP flag that means "I understand ECN-Capable SYN/ACK packets". If this document standardized the use of such an "ECN-SYN" flag, the TCP responder would only send a SYN/ACK packet as ECN-Capable if the incoming SYN packet had the "ECN-SYN" flag set. That would prevent the backwards compatibility problems described above 💜
The catch is that an ECN-SYN flag would use one of the four remaining reserved bits in the TCP header, just for a temporary backwards compatibility problem 🥺 That drawback is limited, though, because the "ECN-SYN" flag would only be defined for use with ECN-setup SYN packets. That bit in the TCP header could be given other uses for other kinds of TCP packets.
Here are the factors behind the decision not to use an ECN-SYN flag 💅
- The limited installed base: When this document was written, the TCP implementations in Microsoft Vista and Mac OS X included ECN, but ECN was not enabled by default [SBT07]. So there wasn't a large deployed base of ECN-Capable TCP implementations. That limits the scope of any backwards compatibility problems.
- Limits to the scope of the problem: The backwards compatibility problem wouldn't be serious enough to cause congestion collapse. Under severe congestion, the buffer at the congested router will overflow, and the router will drop rather than ECN-mark arriving SYN packets. Some active queue management mechanisms might switch from packet-marking to packet-dropping during high congestion, before the buffer overflows, as recommended in Section 19.1 of RFC 3168 [RFC3168]. This helps prevent congestion collapse problems with the use of ECN.
- Detection of and response to backwards-compatibility problems: A TCP responder such as a web server can't tell the difference between a SYN/ACK packet that is not ECN-marked in the network and a SYN/ACK packet that is ECN-marked but where the TCP initiator ignores the ECN mark. However, a TCP responder can detect when a SYN/ACK packet is sent as ECN-capable and isn't reported as ECN-marked, but data packets from the initial window are dropped or marked. The authors call this scenario "initial-window-congestion". If a web server frequently experienced initial-window-congestion without SYN/ACK congestion, then the web server might be running into backwards compatibility problems with ECN-Capable SYN/ACK packets. In that case, it could respond by not sending SYN/ACK packets as ECN-Capable 🧚
Normative References
- [RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
- [RFC2988] Paxson, V. and M. Allman, "Computing TCP's Retransmission Timer", RFC 2988, November 2000.
- [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001.
Informative References
okay so this is the bibliography, aka the "receipts" section 💅 these are the sources the rest of the doc leans on (not required reading to implement, but iconic for context). lowkey my favorite part because you can see how much real research went into this 🧚
[ECN+] A. Kuzmanovic, The Power of Explicit Congestion Notification, SIGCOMM 2005. 🦋
[ECN-SYN] ECN-SYN web page with simulation scripts, http://www.icir.org/floyd/ecn-syn. the simulation scripts live here, so go wild 🌸
[F07] S. Floyd, "[BEHAVE] Response of firewalls and middleboxes to TCP SYN packets that are ECN-Capable?", August 2, 2007, email to the BEHAVE mailing list, http://www1.ietf.org/mail-archive/web/behave/current/msg02644.html. this one is a whole email thread about middleboxes freaking out over ECN-capable SYNs 😭
[Kelson00] Dax Kelson, "8% of the Internet unreachable!", September 10, 2000, email to the Linux kernel mailing list, http://lkml.indiana.edu/hypermail/linux/kernel/0009.1/0329.html. the title says it all, the drama is immaculate 💀
[L08] A. Landley, "Re: [tcpm] I-D Action:draft-ietf-tcpm-ecnsyn-06.txt", August 24, 2008, email to the tcpm mailing list, http://www.ietf.org/mail-archive/web/tcpm/current/msg03988.html. 🎀
[MAF05] A. Medina, M. Allman, and S. Floyd, "Measuring the Evolution of Transport Protocols in the Internet", ACM CCR, April 2005. this one tracks how transport protocols have changed over time, which is so relevant to this whole doc ✨
[PI] C. Hollot, V. Misra, W. Gong, and D. Towsley, "On Designing Improved Controllers for AQM Routers Supporting TCP Flows", April 1998. AQM controller design, very nerdy, very cool 🤓🫶
[RED] Floyd, S., and Jacobson, V., "Random Early Detection gateways for Congestion Avoidance", IEEE/ACM Transactions on Networking, V.1 N.4, August 1993. the OG RED paper, bestie. literally the root of a lot of this 👑
[REM] S. Athuraliya, V. H. Li, S. H. Low and Q. Yin, "REM: Active Queue Management", IEEE Network, May 2001. 🌈
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, S., Wroclawski, J., and L. Zhang, "Recommendations on Queue Management and Congestion Avoidance in the Internet", RFC 2309, April 1998. that's a huge list of authors, the whole squad showed up 🍓
[RFC2581] Allman, M., Paxson, V., and W. Stevens, "TCP Congestion Control", RFC 2581, April 1999. the classic congestion control baseline ⭐
[RFC3042] Allman, M., Balakrishnan, H., and S. Floyd, "Enhancing TCP's Loss Recovery Using Limited Transmit", RFC 3042, January 2001. 💜
[RFC3360] Floyd, S., "Inappropriate TCP Resets Considered Harmful", BCP 60, RFC 3360, August 2002. the title is literally a whole mood 🙈
[RFC3390] Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's Initial Window", RFC 3390, October 2002. 🦋
[RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common Mitigations", RFC 4987, August 2007. this one is the SYN flood deep dive, very relevant to the security vibes 🔥
[SCJO01] F. Smith, F. Campos, K. Jeffay, and D. Ott, "What TCP/IP Protocol Headers Can Tell us about the Web", SIGMETRICS, June 2001. 🌸
[SYN-COOK] Dan J. Bernstein, SYN cookies, 1997, see also <http://cr.yp.to/syncookies.html>. SYN cookies are iconic ngl 🍪
[SBT07] M. Sridharan, D. Bansal, and D. Thaler, "Implementation Report on Experiences with Various TCP RFCs", Presentation in the TSVAREA, IETF 68, March 2007. http://www3.ietf.org/proceedings/07mar/slides/tsvarea-3/sld6.htm. real-world implementation reports, so good for seeing what actually happens 🧚
[Tools] S. Floyd, Ed., and E. Kohler, Ed., "Tools for the Evaluation of Simulation and Testbed Scenarios", Work in Progress, February 2008. 💖
Authors' Addresses
and now the squad 🫶 here's who wrote it, with their contact info exactly as published:
Aleksandar Kuzmanovic
Northwestern University
Phone: +1 (847) 467-5519
EMail: akuzma@northwestern.edu
URL: http://cs.northwestern.edu/~akuzmaAmit Mondal
Northwestern University
Phone: +1 (847) 467-6455
EMail: a-mondal@northwestern.edu
URL: http://www.cs.northwestern.edu/~akm175/Sally Floyd
ICIR (ICSI Center for Internet Research)
Phone: +1 (510) 666-2989
EMail: floyd@icir.org
URL: http://www.icir.org/floyd/K. K. Ramakrishnan
AT&T Labs Research
Rm. A161
180 Park Ave.
Florham Park, NJ 07932
Phone: +1 (973) 360-8764
EMail: kkrama@research.att.com
URL: http://www.research.att.com/info/kkramathat's the whole doc, bestie! 🌈✨ the crew that made ECN work for TCP, all signed off 👑💖
📦 where this page comes from
- ietf: rfc5562 2009-06
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms