RFC 7560

Problem Statement and Requirements for Increased Accuracy in Explicit Congestion Notification (ECN) Feedback

📅 published August 2015, from ietf

RFC 7560 💖

Problem Statement and Requirements for Increased Accuracy in Explicit Congestion Notification (ECN) Feedback 🦋

Okay bestie, this one is from the IETF (Internet Engineering Task Force) and it's marked Request for Comments: 7560 📜. It's written by M. Kuehlewind, Ed. (ETH Zurich), R. Scheffenegger (NetApp, Inc.), and B. Briscoe (BT) 🫶. It's an Informational doc, not a Standards Track one, and the ISSN is 2070-1721. It came out in August 2015 🌸. The title block doesn't list anything this RFC obsoletes or updates, so there's no drama there ✨.

Abstract

Okay so ECN (Explicit Congestion Notification) is a mechanism where network nodes can mark IP packets instead of dropping them, to tell the endpoints that there's congestion 🌊. An ECN-capable receiver then feeds that info back to the sender 💌. The tea is that ECN for TCP only lets you feed back one congestion signal per Round-Trip Time (RTT) 😭. Meanwhile, ECN for other transports like RTP/UDP and SCTP gets way more accurate feedback 👑. Newer TCP mechanisms like Congestion Exposure (ConEx) or Data Center TCP (DCTCP) need that more accurate feedback when more than one marking shows up in one RTT 💅. So this document lays out the requirements for an update to TCP that gives it more accurate ECN feedback 🎀.

Status of This Memo

This doc is not an Internet Standards Track spec. It's published just for information 🧚.

It's a product of the IETF, so it reflects the consensus of the IETF community 🫶. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 for that 🙈.

For the current status of this doc, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc7560 🔗.

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved 💜.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were on the day this was published. Please read these carefully, because they spell out your rights and restrictions for this doc 📖. Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License ⚖️.

Table of Contents

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
  1.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   4
2.  Recap of Classic ECN and ECN Nonce in IP/TCP  . . . . . . . .   5
3.  Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . .   6
4.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .   8
5.  Design Approaches . . . . . . . . . . . . . . . . . . . . . .  11
  5.1.  Redefinition of ECN/NS Header Bits  . . . . . . . . . . .  11
  5.2.  Using Other Header Bits . . . . . . . . . . . . . . . . .  13
  5.3.  Using a TCP Option  . . . . . . . . . . . . . . . . . . .  13
6.  Security Considerations . . . . . . . . . . . . . . . . . . .  14
7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  14
  7.1.  Normative References  . . . . . . . . . . . . . . . . . .  14
  7.2.  Informative References  . . . . . . . . . . . . . . . . .  14
Appendix A.  Ambiguity of the More Accurate ECN Feedback in DCTCP  16
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  17
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  17

1. Introduction ✨

Okay so ECN [RFC3168] is the mechanism where network nodes mark IP packets instead of dropping them, to signal congestion to the endpoints 🌸. An ECN-capable receiver feeds that back to the sender 💌. The catch is that ECN for TCP only lets one feedback signal go out per Round-Trip Time (RTT) 😭. That's fine for existing TCP congestion control, which only cuts its sending rate once per RTT no matter how many ECN marks it gets 🫶. But newer stuff like Congestion Exposure (ConEx) [RFC6789] or Data Center TCP (DCTCP) [DCTCP] needs more accurate feedback than "classic ECN" [RFC3168] when more than one marking shows up in any one RTT 💅.

If you want the deep dive on why ECN is worth it (including with granular enough feedback), check out [ECN-BENEFITS] 🧠.

ECN is defined for transports besides TCP too 🦋. The ECN feedback for RTP/UDP [RFC6679] is super detailed: it gives individual counters for all four ECN codepoints, plus lost and duplicate segments, but the signalling overhead is high 😳. There's an ECN feedback proposal for SCTP in [SCTP-ECN] 🧚. It gives a counter for how many ECN-capable packets got marked because of congestion (since the last sender-side window reduction), but that costs extra overhead too 🙈.

Now, DCTCP is already out in the wild and changes TCP's ECN feedback in proprietary ways 🔥. It was released in Microsoft Windows 8, and implementations exist for Linux and FreeBSD 💻. The problem is that DCTCP's changes to TCP skip capability negotiation and just depend on every host and network device with ECN capability being configured the same way 😬. A big reason this doc exists is to step in before each proprietary implementation invents its own non-interoperable handshake 🚫. That could lead to de facto consumption of the few flags or codepoints that are still free for standardizing capability negotiation 💀.

So this doc lists requirements for a robust and interoperable TCP/ECN feedback protocol that's more accurate than classic ECN [RFC3168], and that every implementation of new TCP extensions (like ConEx and/or DCTCP) can use 🎀. A new feedback scheme should still deliver as much info as classic ECN does, and this doc also clarifies what else has to be considered 🧐. The listed requirements should be addressed in the spec of any more accurate ECN feedback scheme 📝. A few solutions have already been proposed, and Section 5 shows how to use these requirements to compare them, by sketching their high-level design choices and talking through the benefits and drawbacks of each 👑.

The scope here isn't limited to any specific environment. It's meant for general deployment over public and private IP networks 🌍. Candidate solutions should try to meet all of these requirements, but if that isn't possible, they should justify the deviation 💅. The order the requirements are listed in is not a ranking of importance, because each one might weigh differently depending on the deployment scenario ⚖️.

These requirements only care about the type and quality of the ECN feedback signal 📶. They don't say how a TCP sender should react to the improved signal 🙅‍♀️. And they don't imply that any changes to TCP senders or receivers are obligatory 🫶.

1.1. Terminology

We're using terms from [RFC3168] and [RFC3540] 📚:

The ECN field in the IP header:

  • Not-ECT: the not ECN-Capable Transport codepoint 🍓
  • CE: the Congestion Experienced codepoint 🔥
  • ECT(0): the first ECN-Capable Transport codepoint 🌈
  • ECT(1): the second ECN-Capable Transport codepoint 🌈

The ECN flags in the TCP header:

  • CWR: the Congestion Window Reduced flag 🦋
  • ECE: the ECN-Echo flag 🦋
  • NS: ECN Nonce Sum 🦋

In this doc, the ECN feedback scheme specified in [RFC3168] is called "classic ECN," and any new proposal is called a "more accurate ECN feedback" scheme ✨. A "congestion mark" is an IP packet where the CE codepoint is set 🚩. A "congestion episode" refers to one or more congestion marks that belong to the same overload situation in the network (usually during one RTT) 🌊. And a TCP segment with the acknowledgement flag set is simply called an ACK 💖.

2. Recap of Classic ECN and ECN Nonce in IP/TCP 🎀

Classic ECN needs two bits in the IP header 🧩. A packet counts as ECN-capable when either one of those two bits is set. A network node can set both bits at once when it's experiencing congestion 🔥. That gives the four codepoints (Not-ECT, ECT(0), ECT(1), and CE) we listed above 🌈.

In the TCP header, the first two bits in byte 14 are used for ECN feedback for each half-connection 🫧. A TCP receiver signals that it got a congestion mark by setting the ECN-Echo (ECE) flag in the TCP header 🚨. For reliability, the receiver keeps setting the ECE flag on every ACK 🔁. So the receiver knows when to stop, the sender sets the Congestion Window Reduced (CWR) flag as soon as it gets an ECE feedback signal 🙌. This always leads to a full RTT of ACKs with ECE set 🕰️. That means the receiver can't signal back any additional CE markings that arrive within that same RTT 😭.

The ECN Nonce [RFC3540] is an experimental addition to ECN that a TCP sender can use to protect itself from accidental or malicious concealment of CE-marked or dropped packets 🕵️‍♀️. It defines the last bit of byte 13 in the TCP header as the Nonce Sum (NS) flag 🧮. The receiver keeps a nonce sum that counts how many ECT(1) packets have occurred, and it signals the least significant bit of that sum on the NS flag 🔢. Honest talk: there are no known deployments of a TCP stack that uses the ECN Nonce extension 🥲.

    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
  +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
  |               |           | N | C | E | U | A | P | R | S | F |
  | Header Length | Reserved  | S | W | C | R | C | S | S | Y | I |
  |               |           |   | R | E | G | K | H | T | N | N |
  +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

    Figure 1: The (Post-ECN Nonce) Definition of the TCP Header Flags

There's also an alternative for a sender to check feedback integrity, which was proposed in [TEST-RCV] 🔍. The sender itself occasionally inserts a CE mark or reorders packets, then checks that the receiver feeds those back faithfully 🧪. This alternative uses up no header bits or codepoints, and it frees up the ECT(1) codepoint in the IP header and the NS flag in the TCP header for other uses 🎉.

3. Use Cases 🎀

Okay so, these two examples show where today's tools would already be better off with more accurate ECN feedback 💖 But it's hard to predict the future, bestie. Once a more accurate ECN feedback mechanism that follows the requirements in this document is widely deployed, it's very likely that more uses will show up 🦋 The examples below are in no particular order ✨

ConEx is an experimental approach that lets a sender pass the congestion feedback it got from the receiver back into the network, along the forward data path 🧚 ConEx info can be used for traffic management, so traffic gets limited in proportion to the congestion actually being caused, instead of being limited by rate or volume [RFC6789] 💅 A ConEx sender uses selective acknowledgements (SACK) [RFC2018] for accurate feedback of loss signals, but until now TCP has had no equivalent accurate feedback for ECN 😭

DCTCP has very low and predictable queuing delay 🌸 It changes how a TCP sender reacts to congestion, and it also needs switches/routers to have ECN enabled and configured with a low step threshold and no signal smoothing. That's why it's currently only used in private networks, e.g., internal to data centers 🔥 DCTCP was released in Microsoft Windows 8, and implementations exist for Linux and FreeBSD ⭐ To get enough congestion information, the different DCTCP implementations use a proprietary ECN feedback protocol, but they omit capability negotiation 🙈 Moreover, the feedback protocol proposed in [DCTCP] only works if there are no losses at all, and otherwise it gets very confused (see Appendix A) 💀 So if a generic, more accurate ECN feedback scheme were available, it would solve two problems for DCTCP: i) the need for a consistent variant of DCTCP to be deployed network-wide and ii) the inability to cope with ACK loss 👑

Classic ECN-TCP wouldn't benefit from more accurate ECN feedback, but it wouldn't suffer either 🫶 The same signal that is currently conveyed with ECN, following the specification given in [RFC3168], would still be available 🍓

The following scenarios should briefly show where accurate ECN feedback is needed or adds value 🌈

  • A sender with standardized TCP congestion control that supports ConEx: In this case, the ConEx mechanism uses the extra information per RTT to re-echo the precise congestion information, but the congestion control algorithm still ignores multiple marks per RTT [RFC5681] 💜
  • A sender using DCTCP congestion control without ConEx: The congestion control algorithm uses the extra info per RTT to perform its decrease depending on the number of congestion marks 🦋
  • A sender using DCTCP congestion control and supporting ConEx: Both the congestion control algorithm and ConEx use the more accurate ECN feedback mechanism ✨
  • As-yet-unspecified sender mechanisms: The above are two examples of more general interest in sender mechanisms that respond to the extent of congestion feedback, not just its existence. It will greatly simplify incremental deployment if the sender can unilaterally deploy new behaviours and rely on the presence of generic receivers that have already implemented more accurate feedback 🧚
  • A TCP sender using congestion control as specified in RFC 5681 without ConEx: No accurate feedback is necessary here. The congestion control algorithm still reacts to only one signal per RTT. But it's best to feed back all the information the receiver gets, whether or not the sender uses it -- at least as long as overhead is low or zero 🌸
  • Using CE for checking integrity: If a more accurate ECN feedback scheme feeds all occurrences of CE marks back, a sender could perform integrity checking by occasionally injecting CE marks itself. Specifically, a sender can send packets that it randomly marks with CE (at low frequency), then check if feedback is received for these packets. The congestion notification feedback for these self-injected markings would not require a congestion control reaction [TEST-RCV] 🔥

4. Requirements ✨

Okay bestie, let's talk about what the accurate ECN feedback protocol actually NEEDS to be 💅. The goal is to have fairly accurate (not necessarily perfect), timely, and protected signalling 🫶. This leads to the following requirements, which should be discussed for any proposed more accurate ECN feedback scheme 🎀:

Resilience 🛡️

So the ECN feedback signal rides inside the ACK, right? Pure TCP ACKs can get lost with zero recovery, and that's not just from congestion either. ACKs also get deliberately thinned 😭. Delayed ACKs are super common with TCP too. Typically an ACK gets triggered after two data segments (or more, e.g., because of receive segment coalescing, ACK compression, ACK congestion control [RFC5690], or other phenomena; see [RFC3449]). In a high-congestion situation where most packets are marked with CE, an accurate feedback mechanism should still be able to signal sufficient congestion information 💪. So the accurate ECN feedback extension has to take delayed ACKs and ACK loss into account 🙈. Also, a more accurate feedback protocol should still give you more accurate feedback than classic ECN when delayed ACKs cover more than two segments, or when a thin stream disables Nagle's algorithm [RFC896] ✨. Finally, the feedback mechanism should not be impacted by reordering of ACKs, even when the ACKed sequence number does not increase 🦋.

Timeliness ⏰

A CE mark can be induced by the sending host, or (more commonly) a network node on the transmission path, and then it gets echoed by the receiver in the TCP ACK 🧚. So when this info reaches the sender, it's already about one RTT old, naturally 💀. With a sufficient ACK rate, a further delay of a small number of packets can be tolerated 👍. BUT this information goes stale with large delays, given how dynamic networks are 🌪️. TCP congestion control (which itself partly introduces these dynamics) operates on a time scale of one RTT. So to be timely, congestion feedback information should be delivered within about one RTT ⭐.

Integrity 🔐

The integrity of the feedback in a more accurate ECN feedback scheme should be assured, at least as well as the ECN Nonce 👑. Alternatively, it should at least be possible to give strong incentives for the receiver and network nodes to cooperate honestly 🤝.

Given there are known problems with ECN Nonce deployment, this document only requires that the integrity of the more accurate ECN feedback can be assured. It does NOT require that the ECN Nonce mechanism is employed to achieve this 🙅‍♀️. Honestly though, if integrity could be provided in another manner, a more accurate ECN feedback protocol might repurpose the nonce sum (NS) flag in the TCP header ♻️.

If the more accurate ECN feedback scheme provides sufficient information, the integrity check could be performed by, e.g., deterministically setting the CE in the sender and monitoring the respective feedback (similar to ECT(1) and the ECN Nonce sum) 🧠. Whether a sender should enforce when it detects wrong feedback information, and what kind of enforcement it should apply, are policy issues. Those need not be specified as part of the more accurate ECN feedback signal scheme itself. They belong to specifying an update to core TCP mechanisms like congestion control that make use of the more accurate ECN signal 📜.

Accuracy 🎯

Classic ECN feeds back one congestion notification per RTT. That's sufficient for classic TCP congestion control, which reduces the sending rate at most once per RTT 🙌. So the more accurate ECN feedback scheme should ensure that, if a congestion episode occurs, at least one congestion notification is echoed and received per RTT, just like classic ECN would do. Of course, the whole point of a more accurate ECN extension is to reconstruct the number of CE markings more accurately 📈. In the best case, the new scheme should even allow reconstruction of the exact number of payload bytes that a CE-marked packet was carrying 🤯. However, it is accepted that it may be too complex for a sender to get the exact number of congestion markings or marked bytes in all situations 😩. Ideally, the feedback scheme should preserve the order in which any (of the four) ECN signals were received 🔄. And ideally, it would even be possible for the sender to determine which of the packets covered by one delayed ACK were congestion marked. That matters if the flow consists of packets of different sizes, or to allow for future protocols where the order of the markings may be important 👀.

In the best case, a sender that sees more accurate ECN feedback information would be able to reconstruct the occurrence of any of the four codepoints (Not-ECT, CE, ECT(0), ECT(1)) 🌈. However, assuming the sender marks all data packets as ECN-capable and uses a default setting of ECT(0) (as with [RFC3168]), solely feeding back the occurrence of CE and ECT(1) might be sufficient 💅. Because the sender can keep account of the transmitted segments with any of the three ECN codepoints, conveying any two of these back to the sender is sufficient for it to reconstruct the third as observed by the receiver 🧮. So a more accurate ECN feedback scheme should at least provide information on two of these signals, e.g., CE and ECT(1) 💜.

If a more accurate ECN scheme can reliably deliver feedback in most but not all circumstances, ideally the scheme should at least not introduce bias ⚖️. In other words, undetected loss of some ACKs should be as likely to increase as to decrease the sender's estimate of the probability of ECN marking 🎲.

Complexity 🧩

Implementation should be as simple as possible, and only a minimum of additional state information should be needed 🫧. This is the vibe, because it'll enable more accurate ECN feedback to be used as the default feedback mechanism, even if only one ECN feedback signal per RTT is needed 🍓.

Overhead 📦

A more accurate ECN feedback signal should limit the additional network load, because ECN feedback is ultimately not critical information. (In the worst case, loss will still be available as a congestion signal of last resort 🆘.) As feedback information has to be provided frequently and in a timely fashion, potentially all or a large fraction of TCP acknowledgements might carry this information 🌊. Ideally, no additional segments should be exchanged compared to a TCP session as specified in RFC 3168, and the overhead in each segment should be minimized 🪶.

Backward and forward compatibility 🔁

Given more accurate ECN feedback will involve a change to the TCP protocol, it should be negotiated between the two TCP endpoints 🤳. If either end does not support the more accurate feedback, they should both be able to fall back to classic ECN feedback 🫶.

A more accurate ECN feedback extension should aim to traverse most middleboxes, including firewalls and Network Address Translators (NATs) 🏰. Further, a feedback mechanism should provide a method to fall back to classic ECN signalling if the new signal is suppressed by certain middleboxes 🛟.

In order to avoid a fork in the TCP protocol specifications, if experiments with the new ECN feedback protocol are successful, the intention is to eventually update RFC 3168 for any TCP/ECN sender, not just for ConEx or DCTCP senders 📚. Then, future senders will be able to unilaterally deploy new behaviours that exploit the existence of more accurate ECN feedback in receivers (forward compatibility) 🚀. Conversely, even if another sender only needs one ECN feedback signal per RTT, it should be able to use more accurate ECN feedback and simply ignore the excess information 🙃.

Furthermore, the receiver should not make assumptions about the mechanism that was used to set the markings, nor about any interpretation or reaction to the congestion signal. The receiver only needs to faithfully reflect congestion information back to the sender 🪞✨.

5. Design Approaches 🎨

This section introduces some possible design approaches for TCP ECN feedback 💡. The purpose here is to give examples of how trade-offs might be needed between the requirements, as input to future IETF work to specify a protocol 🧑‍💻. The order is not significant, and there is no intention to endorse any particular approach 🙅‍♀️.

All approaches presented below (and proposed so far) are able to provide accurate ECN feedback information as long as no ACK loss occurs and the congestion rate is reasonable 🌤️. In the case of a high ACK loss rate or very high congestion (CE-marking) rate, the proposed schemes have different resilience characteristics depending on the number of bits used for the encoding 😬. While classic ECN provides reliable (but inaccurate) feedback of a maximum of one congestion signal per RTT, the proposed schemes do not implement an explicit acknowledgement mechanism for the feedback (as, e.g., the ECE/CWR exchange of [RFC3168]) 🫠.

5.1. Redefinition of ECN/NS Header Bits

Okay bestie, this is the glow-up era 💅✨ Schemes in this category can also use the NS bit for capability negotiation during the TCP handshake. That means a more accurate ECN can get negotiated without changing the classic ECN negotiation, so it stays backwards compatible 🫶

Schemes here can also just redefine the ECN header flags, ECE and CWR, so they encode that a CE marking happened at the receiver 🦋 Honest tea: this only has very limited resilience against loss of ACKs, especially pure ACKs (no payload, so they get delivered unreliably) 😭

A couple of schemes have been proposed so far:

  • A naive 1-bit scheme sends one ECE for each CE received. It could use CWR to add some robustness against ACK loss by putting redundant information on the next ACK, but it's still vulnerable to ACK loss 💀
  • The scheme defined for DCTCP [DCTCP] toggles the ECE feedback on an immediate ACK whenever the CE marking changes. Otherwise, delayed ACKs carry the ECE value unchanged. Appendix A shows that this is still ambiguous to the sender if the ACKs are pure ACKs and some may have been lost 🙈

Alternatively, the receiver uses the three ECN/NS header flags, ECE, CWR, and NS, to represent a counter of the accumulated number of CE markings it has received 🧚 Resilience against loss is better than the flag-based schemes, but it may not be enough when there's extended ACK loss that wouldn't otherwise hurt the TCP sender's performance.

A number of coding schemes have been proposed so far in this category:

  • A 3-bit counter scheme continuously feeds back the three least significant bits of a CE counter.
  • A scheme that defines a standardized lookup table to map the eight codepoints onto either a CE counter or an ECT(1) counter.

These proposed schemes give accumulated info on CE marking feedback, kind of like the number of acknowledged bytes in the TCP header 🌸 Because there are so few bits, the ECN feedback wraps way more often than the acknowledgement field. So feedback could get lost from a relatively small run of pure-ACK losses. Resilience can go up by adding redundancy, e.g., sending each counter increase two or more times. Of course, any of these extra mechanisms makes things more complex 🥴 If the congestion rate is greater than the ACK rate (multiplied by the number of congestion marks that can be signaled per ACK), the congestion info can't be fed back correctly. Covering the worst case, where every packet is CE marked, can potentially be done by dynamically adapting the ACK rate and redundancy. That adds even more complexity and maybe more signalling overhead too. Schemes that don't repurpose the ECN NS bit could still support the ECN Nonce 🎀

5.2. Using Other Header Bits

As you can see in Figure 1, there are currently three unused flags in the TCP header 👑 The proposed 3-bit counter or codepoint schemes could be extended by one or more bits to get higher resilience against ACK loss. The relative gain would be exponentially higher resilience against ACK loss, while the drawbacks would stay exactly the same 📈

Another option is a new method that standardizes using the bits in the Urgent Pointer field (see [RFC6093]) to signal more bits of its congestion signal counter. But this would only happen whenever the Urgent Flag is not set. Since that's often the case, resilience could go up without any extra header overhead ✨

Any proposal to use these bits would need to check the likelihood that some middleboxes might discard or 'normalize' the currently unused flag bits, or a non-zero Urgent Pointer when the Urgent Flag is cleared 👀 If experimentation shows certain bits are usable, assigning any of them would then need an IETF standards action 🌈

5.3. Using a TCP Option

Alternatively, a new TCP option could be introduced to help keep ECN feedback accurate and intact between receiver and sender 💜 Such an option could give higher resilience and even more info. For example, there's a proposal for SCTP that counts the number of CE marked packets [SCTP-ECN] since the last CWR was observed, and ECN for RTP/UDP [RFC6679] is another example. The latter explicitly gives the total number of packets during a connection where the IP ECN field is set to ECT(0), ECT(1), CE, or Not-ECT, plus the number of lost packets 🧮 Deploying new TCP options has its own challenges though. And to actually get high resilience, this option would need to be carried by most or all ACKs, because the receiver can't know if and when ACKs may be dropped. So this approach would add a lot of signalling overhead even though ECN feedback isn't extremely critical info (in the worst case, loss will still be available as a strong congestion feedback signal 🫠). Still, a TCP option like this could be used on top of a more accurate ECN feedback scheme in the TCP header, or on top of classic ECN, only when needed and when there's space available 💖

6. Security Considerations ✨

ECN feedback info must only be used if the other info in a received TCP segment shows the congestion was genuinely part of the flow and not spoofed 🕵️‍♀️ So the normal TCP acceptance techniques have to be used to verify the segment is part of the flow before returning any ECN info it contains. Similarly, ECN feedback is only accepted on valid ACKs.

ECN feedback is used as input for congestion control, so the algorithm wouldn't react properly if ECN feedback got lost and the recovery mechanism wasn't good enough 😱 This resilience requirement is laid out in Section 4. But it's worth noting that ECN feedback is not the last line of defense against congestion collapse 🚨 If there's not enough response to ECN, loss will happen, and TCP will still react appropriately to loss.

A receiver could suppress ECN feedback info, which would make its connections eat up excess sender or network resources 🍽️ This problem is similar to the one seen with the classic ECN feedback scheme, and it should be addressed by the integrity checking required in Section 4 🔐

7. References 📚

7.1. Normative References

[RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
           of Explicit Congestion Notification (ECN) to IP",
           RFC 3168, DOI 10.17487/RFC3168, September 2001,
           <http://www.rfc-editor.org/info/rfc3168>.

[RFC3540]  Spring, N., Wetherall, D., and D. Ely, "Robust Explicit
           Congestion Notification (ECN) Signaling with Nonces",
           RFC 3540, DOI 10.17487/RFC3540, June 2003,
           <http://www.rfc-editor.org/info/rfc3540>.

7.2. Informative References

[DCTCP]    Bensley, S., Eggert, L., and D. Thaler, "Microsoft's
           Datacenter TCP (DCTCP): TCP Congestion Control for
           Datacenters", Work in Progress,
           draft-bensley-tcpm-dctcp-05, July 2015.

[ECN-BENEFITS]
           Fairhurst, G. and M. Welzl, "The Benefits of using
           Explicit Congestion Notification (ECN)", Work in Progress
           draft-ietf-aqm-ecn-benefits-06, July 2015.

[RFC896]   Nagle, J., "Congestion Control in IP/TCP Internetworks",
           RFC 896, DOI 10.17487/RFC0896, January 1984,
           <http://www.rfc-editor.org/info/rfc896>.

[RFC2018]  Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
           Selective Acknowledgment Options", RFC 2018,
           DOI 10.17487/RFC2018, October 1996,
           <http://www.rfc-editor.org/info/rfc2018>.

[RFC3449]  Balakrishnan, H., Padmanabhan, V., Fairhurst, G., and M.
           Sooriyabandara, "TCP Performance Implications of Network
           Path Asymmetry", BCP 69, RFC 3449, DOI 10.17487/RFC3449,
           December 2002, <http://www.rfc-editor.org/info/rfc3449>.

[RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
           Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
           <http://www.rfc-editor.org/info/rfc5681>.

[RFC5690]  Floyd, S., Arcia, A., Ros, D., and J. Iyengar, "Adding
           Acknowledgement Congestion Control to TCP", RFC 5690,
           DOI 10.17487/RFC5690, February 2010,
           <http://www.rfc-editor.org/info/rfc5690>.

[RFC6093]  Gont, F. and A. Yourtchenko, "On the Implementation of the
           TCP Urgent Mechanism", RFC 6093, DOI 10.17487/RFC6093,
           January 2011, <http://www.rfc-editor.org/info/rfc6093>.

[RFC6679]  Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P.,
           and K. Carlberg, "Explicit Congestion Notification (ECN)
           for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August
           2012, <http://www.rfc-editor.org/info/rfc6679>.

[RFC6789]  Briscoe, B., Ed., Woundy, R., Ed., and A. Cooper, Ed.,
           "Congestion Exposure (ConEx) Concepts and Use Cases",
           RFC 6789, DOI 10.17487/RFC6789, December 2012,
           <http://www.rfc-editor.org/info/rfc6789>.

[SCTP-ECN] Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream
           Control Transmission Protocol (SCTP)", Work in Progress,
           draft-stewart-tsvwg-sctpecn-05, January 2014.

[TEST-RCV] Moncaster, T., Briscoe, B., and A. Jacquet, "A TCP Test to
           Allow Senders to Identify Receiver Non-Compliance", Work
           in Progress, draft-moncaster-tcpm-rcv-cheat-03, July 2014.

Appendix A. Ambiguity of the More Accurate ECN Feedback in DCTCP 🔍

As defined in [DCTCP], a DCTCP receiver feeds back ECE=0 on delayed ACKs as long as CE stays 0, and it also immediately sends an ACK with ECE=0 when CE flips to 1. Similarly, it keeps feeding back ECE=1 on delayed ACKs while CE stays 1, and immediately feeds back ECE=1 when CE flips to 0 🔄 A sender can decode this scheme with zero ambiguity if there's never any ACK loss, and the sender assumes there will never be any ACK loss.

The next two examples show the feedback sequence gets super ambiguous to the sender if either of those conditions breaks 😵‍💫 Below, 0 stands for ECE=0, 1 stands for ECE=1, and . stands for a gap of one segment between delayed ACKs. Now imagine the sender gets this sequence of feedback on three pure ACKs:

0.0.0

When the receiver sent that sequence, it could have been any of these four sequences:

  • a. 0.0.0 (0 x CE)
  • b. 010.0 (1 x CE)
  • c. 0.010 (1 x CE)
  • d. 01010 (2 x CE)

Here, any of the 1s could be a possible pure ACK carrying ECE feedback that got lost 🙀 If the sender guesses (a), it might be right, or it might miss 1 or 2 congestion marks over 5 packets. So when faced with this simple sequence (which isn't contrived, btw), a sender can guess that congestion might have been 0%, 20%, or 40%, but it doesn't know which one 🎲

Sequences with a longer gap (e.g., 0...0.0) get way more ambiguous 🌀 It helps a little if the sender knows the distance the receiver uses between delayed ACKs, and it helps a lot if that distance is 1, i.e., no delayed ACKs. But even without delayed ACKs, there will still be ambiguity whenever there are pure ACK losses 🥹

Acknowledgements ✨

Big thank-you to Gorry Fairhurst 🫶 for reviewing this and bringing ideas about CE-based integrity checking, and to Mohammad Alizadeh for pointing out that we need to avoid bias. You two are literally the reason this doc is good 💖

Bob Briscoe was partly funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700) and through the Trilogy 2 project (ICT-317756). The views expressed here are solely those of the authors, in the context of the mentioned funding projects 🌸

Authors' Addresses 🎀

Mirja Kuehlewind (editor) 👑

Mirja Kuehlewind (editor)
ETH Zurich
Gloriastrasse 35
Zurich  8092
Switzerland

Email: mirja.kuehlewind@tik.ee.ethz.ch

Richard Scheffenegger 🦋

Richard Scheffenegger
NetApp, Inc.
Am Euro Platz 2
Vienna  1120
Austria

Phone: +43 1 3676811 3146
Email: rs@netapp.com

Bob Briscoe 🧚

Bob Briscoe
BT
B54/77, Adastral Park
Martlesham Heath
Ipswich  IP5 3RE
United Kingdom

Phone: +44 1473 645196
Email: ietf@bobbriscoe.net
URI:   http://bobbriscoe.net/

That's the whole doc, bestie! 🌈 Thank you for reading the entire ECN story with me 💅✨

📦 where this page comes from