RFC 7786
TCP Modifications for Congestion Exposure (ConEx)
RFC 7786: TCP Modifications for Congestion Exposure (ConEx) 🎀
okay bestie, this is RFC 7786 🦋 It was written by M. Kuehlewind, Ed. (ETH Zurich) and R. Scheffenegger (NetApp, Inc.) for the Internet Engineering Task Force (IETF). It's dated May 2016 and sits in the Experimental category, with ISSN 2070-1721. The header doesn't list any RFCs it obsoletes or updates, so it stands on its own 💅 Its Request for Comments number is 7786.
Abstract
Congestion Exposure (ConEx) is basically a way for senders to tell the network how much congestion they expect. They base that guess on congestion feedback from earlier packets in the same flow ✨ This doc explains what TCP needs changed so it can use ConEx 🌸
Status of This Memo
Not a Standards Track spec, babe 🙈 This one is published so people can examine it, try it out in experimental implementations, and evaluate it. It defines an Experimental Protocol for the Internet community. It's a product of the IETF, it reflects IETF consensus, it got public review, and the IESG approved it for publication. Not every IESG-approved document is up for any level of Internet Standard, so see Section 2 of RFC 5741 for that 🧚
For the current status, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc7786 💜
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved 🎀
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at http://trustee.ietf.org/license-info. These rules were in effect on the date this document was published. Please actually read them, because they spell out your rights and restrictions for this document 📜 Any Code Components pulled out of this document must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as the Simplified BSD License describes 🫶
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
2. Sender-Side Modifications . . . . . . . . . . . . . . . . . . 4
3. Counting Congestion . . . . . . . . . . . . . . . . . . . . . 5
3.1. Loss Detection . . . . . . . . . . . . . . . . . . . . . 6
3.1.1. Without SACK Support . . . . . . . . . . . . . . . . 7
3.2. Explicit Congestion Notification (ECN) . . . . . . . . . 8
3.2.1. Accurate ECN Feedback . . . . . . . . . . . . . . . . 10
3.2.2. Classic ECN Support . . . . . . . . . . . . . . . . . 10
4. Setting the ConEx Flags . . . . . . . . . . . . . . . . . . . 11
4.1. Setting the E or the L Flag . . . . . . . . . . . . . . . 11
4.2. Setting the Credit Flag . . . . . . . . . . . . . . . . . 11
5. Loss of ConEx Information . . . . . . . . . . . . . . . . . . 14
6. Timeliness of the ConEx Signals . . . . . . . . . . . . . . . 14
7. Open Areas for Experimentation . . . . . . . . . . . . . . . 15
8. Security Considerations . . . . . . . . . . . . . . . . . . . 17
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 18
9.1. Normative References . . . . . . . . . . . . . . . . . . 18
9.2. Informative References . . . . . . . . . . . . . . . . . 19
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 201. Introduction ✨
Same vibe as the abstract, bestie: ConEx lets senders tell the network about expected congestion, using feedback from previous packets in the same flow 💖 The concepts and use cases get a fuller explanation in [RFC6789], and the abstract ConEx mechanism is covered in [RFC7713]. This document is the part that explains what TCP needs to do 🌈
The ConEx markings live in the ConEx Destination Option (CDO) for IPv6, defined in [RFC7837]. There are four flags in play: X (ConEx-capable), L (loss experienced), E (ECN experienced), and C (credit) 🍓
ConEx signaling is built on either loss or Explicit Congestion Notification (ECN) marks [RFC3168] as the congestion indicator. The sender gathers that information from the TCP feedback the receiver already sends back, so the receiver doesn't need any changes 🎀 Because of that, there's no extra negotiation needed to use ConEx on the sender side. This document lays out what the sender has to do so the network gets useful ConEx information 🦋
Quick map of the rest, lowkey helpful 🧚 Section 2 gives an overview of what TCP senders need to change to use ConEx. Congestion info has to be pulled out of TCP's loss or ECN feedback first, which is covered in Section 3. Section 4 explains how to set the CDO marking from that info. Section 5 talks about what happens when packets carrying ConEx information get lost. Section 6 covers how timely the ConEx feedback signal needs to be, since congestion is a temporary state 💅
This document describes congestion accounting for TCP both with and without the Selective Acknowledgement (SACK) extension [RFC2018], which is covered in Section 3.1. ConEx gets more out of SACK, though, because SACK tells you more accurately how many bytes were dropped in the network. So using SACK with TCP and ConEx is the preferred option 😭 The detailed mechanism for setting the L flag in response to loss-based congestion feedback is in Section 4.1.
Loss has to be kept to a minimum, but ECN can give much more fine-grained feedback 👑 ConEx-based traffic measurement or management mechanisms could really benefit from that. The problem is that the current ECN feedback mechanism doesn't reflect multiple congestion markings if they happen within the same Round-Trip Time (RTT). A more accurate ECN feedback extension (AccECN) is proposed in a separate document [ACCURATE], and it's useful for other mechanisms too 🔥
Congestion accounting for both classic ECN feedback and AccECN feedback is explained in detail in Section 3.2. Setting the E flag in response to ECN-based congestion feedback is covered again in Section 4.1 🌸
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🫶
2. Sender-Side Modifications 💜
This section is the big-picture overview of everything a TCP sender has to do to use ConEx signaling 🎀
During the TCP handshake, a ConEx sender MUST negotiate for SACK and ECN, and it's preferable to go with AccECN feedback 😭 That means a ConEx sender MUST also implement SACK and ECN. Depending on what the receiver can do, these operation modes are possible:
SACK-accECN-ConEx(SACK and accurate ECN feedback)SACK-ECN-ConEx(SACK and classic instead of accurate ECN)accECN-ConEx(no SACK but accurate ECN feedback)ECN-ConEx(no SACK and no accurate ECN feedback, but classic ECN)SACK-ConEx(SACK but no ECN at all)Basic-ConEx(neither SACK nor ECN)
A ConEx sender MUST expose all congestion information to the network, based on the congestion info it gets from ECN or from loss information in the TCP feedback loop 💖 A TCP sender SHOULD count congestion byte-wise, not packet-wise (more on that in the next paragraph) ✨ After any congestion notification, a sender MUST mark subsequent packets with the right ConEx flag in the IP header 🦋 On top of that, a ConEx sender must send enough credit to cover all the congestion experienced on the connection so far, plus the risk of congestion for the current transmission (see Section 4.2) 🌈
With SACK you know how many payload bytes were lost, but not how many packets carried them 🍓 With classic ECN you only learn that a marking happened, not the exact number of payload bytes or packets. Network congestion is usually byte-congestion [RFC7141], so the byte-size of a packet with a CDO flag is defined to stand for that many bytes of congestion signaling [RFC7837] 🧚 That's why the exact byte count should be used whenever it's available, so the ConEx Signal ends up as accurate as possible 💅
The detailed counting mechanisms for each operation mode come in the next section 👑
3. Counting Congestion 🌈
A ConEx TCP sender keeps two counters 🎀 One counts congestion from loss detection, and the other tracks congestion from ECN-based feedback. Both hold the number of outstanding bytes that should be ConEx-Marked, with either the E flag or the L flag, in the packets sent after them 🦋
The outstanding bytes for loss-based congestion indications live in the Loss Exposure Gauge (LEG), which is explained in Section 3.1 💖
The outstanding bytes counted from ECN feedback live in the Congestion Exposure Gauge (CEG), which is explained in Section 3.2 ✨
When the sender transmits a ConEx-capable packet with the E or L flag set, it lowers the matching counter by the byte-size of that packet 🍓 This is spelled out for both counters in Section 4.1 🌸
Heads up, bestie: every byte of an IP packet has to be counted in the LEG or CEG so the right number of bytes gets marked 🙈 So the sender SHOULD take both the payload and the headers into account, all the way up to and including the IP header. The catch is that in TCP, the header size of a lost or marked packet usually isn't known, because only payload data gets acknowledged 😭
If you can assume equal-sized packets, or at least evenly distributed packet sizes, the sender MAY count only TCP payload bytes when it adds and subtracts. In that case there should be about as many ConEx-Marked packets as there were original packets causing the congestion. Both sets carry about the same number of header bytes, so the headers cancel out 💅 This assumption is made for simplicity in the sections that follow 🧚
If the sender instead sends packets of different sizes (unequally distributed), it has to memorize or estimate how many packets were lost or ECN-marked 👑 With enough memory, the most accurate option is to remember the sequence number of every sent-but-unacknowledged packet. Then the sender can work out how many packets, and therefore how many header bytes, were sent during the last RTT 🔥 If memory is tight, the sender has to estimate the packet size instead. The average packet size can be estimated if you know how packet sizes were distributed over the last RTT. As the most conservative estimate, you can use the smallest packet size seen in the last RTT 🌈
If the number of newly sent packets with the ConEx L or E flag set is smaller (or larger) than that estimated number of lost or ECN-marked packets, the extra header bytes should be added to (or can be subtracted from) the matching gauge 🎀
3.1. Loss Detection 💔
Okay bestie, this section applies whether or not SACK support is available 🫶 The subsection right after it, Section 3.1.1, covers the case where SACK is not available.
So a TCP sender detects losses and then retransmits the lost data 🔁 That means the ConEx sender can just set the ConEx L flag on all retransmissions, which at least covers the number of bytes that got lost ✨ If you do it this way, you don't need a LEG at all 💅
BUT any retransmission might be spurious 🙈 Basically, it wasn't actually needed, so you've marked more bytes than necessary. To fix that, a ConEx sender can keep a local signed counter called the LEG 🧮 It tracks how many outstanding bytes still need to go out with the ConEx L flag, and yes, it can go negative 💀
Using the LEG, when a TCP sender decides a data segment needs retransmitting, it bumps the LEG up by the TCP payload bytes in that retransmission. This assumes equal-sized segments, so the retransmitted packet has the same number of header bytes as the original one 🎀
For each retransmission:
LEG += payloadNotice how the LEG goes down when the ConEx L marking gets set, as described in Section 4 🌸
Now, to handle spurious retransmissions, a ConEx sender SHOULD make use of heuristics to detect them (e.g., F-RTO [RFC5682], DSACK [RFC3708], and Eifel [RFC3522], [RFC4015]), if they're already available in a given implementation. If there's no mechanism for detecting spurious retransmissions, the ConEx sender MAY choose to implement one of the mechanisms stated above. However, given the inaccuracy ConEx may already have and how timely its information needs to be, a ConEx sender MAY also choose not to compensate for spurious retransmission. In that case, if spurious retransmissions happen, the ConEx sender has just sent too many ConEx Signals. That would, e.g., decrease the congestion allowance in a ConEx policer unnecessarily 😬
If a heuristic method is used to detect spurious retransmission and has determined that a certain number of packets were retransmitted erroneously, the ConEx sender subtracts the payload size of those TCP packets from LEG 🧹
If a spurious retransmission is detected:
LEG -= payloadNote that LEG can become negative if too many L markings have already been sent. This case is further discussed in Section 6 🔍
3.1.1. Without SACK Support 🪄
If multiple losses occur within one RTT and SACK is not used, it may take several RTTs until all the lost data is retransmitted. With the scheme above, the ConEx information gets delayed quite a bit, but timeliness is important for ConEx ⏰ For ConEx, it's important to know how much data was lost. It's not important to know what data was lost. During the first RTT after the initial loss detection, the amount of received data, and therefore also the amount of lost data, can be estimated from the number of received ACKs 📊
So a ConEx sender can use the following algorithm to estimate the number of lost bytes, with an additional delay of one RTT, using an additional Loss Estimation Counter (LEC) 🧠
flight_bytes: current flight size in bytes
retransmit_bytes: payload size of the retransmission
At the first retransmission in a congestion event, LEC is set:
LEC = flight_bytes - 3*SMSS
(At this point in the transmission, in the worst case,
all packets in flight minus three that triggered the dupACks
could have been lost.)
Then, during the first RTT of the congestion event:
For each retransmission:
LEG += retransmit_bytes
LEC -= retransmit_bytes
For each ACK:
LEC -= SMSS
After one RTT:
LEG += LEC
(The LEC now estimates the number of outstanding bytes
that should be ConEx L-marked.)
After the first RTT for each following retransmissions:
if (LEC > 0): LEC -= retransmit_bytes
else if (LEC==0): LEG += retransmit_bytes
if (LEC < 0): LEG += -LEC
(The LEG is not increased for those bytes that were
already counted.)3.2. Explicit Congestion Notification (ECN) 🚦
ECN [RFC3168] is an IP/TCP mechanism that lets network nodes mark packets with the Congestion Experienced (CE) mark instead of dropping them when congestion happens ✨
A receiver might support classic ECN, the more accurate ECN feedback scheme (AccECN), or neither. If ECN isn't supported for a connection, then of course no ECN marks will occur, so the sender will never set the E flag 🙃 Otherwise, a ConEx sender needs to maintain a signed counter, the Congestion Exposure Gauge (CEG), for the number of outstanding bytes that have to be ConEx-Marked with the E flag 💜
The CEG goes up when ECN information is received from an ECN-capable receiver that supports the classic ECN scheme or the accurate ECN feedback scheme. When the ConEx sender receives an ACK indicating one or more segments were received with a CE mark, the CEG goes up by the appropriate number of bytes, as described further below 📈
Unfortunately, with duplicate acknowledgements, the number of newly acknowledged bytes is zero even though (CE-marked) data has been received 😭 So we increase the CEG by DeliveredData, which is defined as:
DeliveredData = acked_bytes + SACK_diff + (is_dup)*1SMSS -
(is_after_dup)*num_dup*1SMSSDeliveredData covers the number of bytes that have been newly delivered to the receiver 📦 So on each arrival of an ACK, DeliveredData goes up by the newly acknowledged bytes (acked_bytes) indicated by the current ACK, relative to all past ACKs. The formula depends on whether SACK is available. If SACK is not available, SACK_diff is always zero. Whereas if ACK information is available, is_dup and is_after_dup are always zero 🤓
With SACK, DeliveredData goes up by the number of bytes provided by (new) SACK information (SACK_diff). Note that if fewer unacknowledged bytes are announced in the new SACK information than in the previous ACK, SACK_diff can be negative. In this case, data is newly acknowledged (in acked_bytes) that was previously accumulated into DeliveredData based on SACK information 🔄
Without SACK, DeliveredData goes up by 1 Sender Maximum Segment Size (SMSS) on duplicate acknowledgements, because duplicate acknowledgements don't acknowledge any new data (so acked_bytes is zero) 🤷 For the subsequent partial or full ACK, acked_bytes covers all newly acknowledged bytes, including those already accounted for when a duplicate acknowledgement arrived. So DeliveredData is reduced by one SMSS for each preceding duplicate ACK. Consequently, is_dup is one if the current ACK is a duplicated ACK without SACK, and zero otherwise. is_after_dup is only one for the next full or partial ACK after a number of duplicated ACKs without SACK, and num_dup counts the number of duplicated ACKs in a row (which usually is 3 or more) 🧾
With classic ECN, one congestion-marked packet causes continuous congestion feedback for a whole round trip, which hides the arrival of any further congestion-marked packets during that round trip 🙈 A more accurate ECN feedback scheme (AccECN) is needed to make sure feedback properly reflects the extent of congestion marking. The two cases, with and without a receiver capable of AccECN, are discussed in the following sections 💖
3.2.1. Accurate ECN Feedback 🎯
With a more accurate ECN feedback scheme (AccECN) that is supported by the receiver, either the number of marked packets or the number of marked bytes will be fed back from the receiver to the sender, so it's known at the sender side. In the latter case, the CEG can be increased directly by the number of marked bytes. Otherwise, if D is assumed to be the number of marks, the gauge (CEG) will be conservatively increased by one SMSS for each marking or, at the maximum, the number of newly acknowledged bytes:
CEG += min(SMSS*D, DeliveredData)3.2.2. Classic ECN Support 🌷
With classic ECN, as soon as a CE mark is seen at the receiver side, it feeds this information back to the sender by setting the Echo Congestion Experienced (ECE) flag in the TCP header of subsequent ACKs. Once the sender receives the first ECE of a congestion notification, it sets the Congestion Window Reduced (CWR) flag in the TCP header once. When this packet with the CWR flag in the TCP header arrives at the receiver side, acknowledging its first ECE feedback, the receiver stops setting the ECE flag 🛑
If the ConEx sender fully conforms to the semantics of ECN signaling as defined by [RFC3168], it will receive one full RTT of ACKs with the ECE flag set whenever at least one CE mark was received by the receiver. As the sender cannot estimate how many packets have actually been CE-marked during this RTT, the most conservative assumption MAY be taken, namely that all packets were marked. This can be done by increasing the CEG by DeliveredData for each ACK with the ECE flag:
CEG += DeliveredDataOptionally, a ConEx sender could implement the following technique (which does not conform to [RFC3168]), called "advanced compatibility mode", to considerably improve its estimate of the number of ECN-marked packets 🧪
To extract more than one ECE indication per RTT, a ConEx sender could set the CWR flag continuously to force the receiver to signal only one ECE per CE mark. Unfortunately, the use of delayed ACKs [RFC5681] (which is common) will prevent feedback of every CE mark. If a CWR confirmation is received before the ECE can be sent out on the next ACK, ECN feedback information could get lost (depending on the actual receiver implementation). Thus, a sender SHOULD set CWR only on those data segments that will presumably trigger a (delayed) ACK. The sender would need an additional control loop to estimate which data segments will trigger an ACK, in order to extract more timely congestion notifications. Still, the CEG SHOULD be increased by DeliveredData, as one or more CE-marked packets could be acknowledged by one delayed ACK 🫧
4. Setting the ConEx Flags 🚩
By setting the X flag, a packet is marked as ConEx-capable 🏷️ All packets carrying payload MUST be marked with the X flag set, including retransmissions. Only if no congestion feedback information is (currently) available SHOULD the X flag be zero. For example, that's for control packets on a connection that hasn't sent any user data for some time and is therefore only sending pure ACKs that don't carry any payload 🫶
4.1. Setting the E or the L Flag ✨
okay so, remember from Section 3.1 how the sender has to keep a CEG counter and might also keep a LEG counter? 🎀 If there's no LEG in use, then every retransmission gets marked with the L flag. simple as that 💅
And then, as long as the LEG or CEG counter is positive, the sender marks each ConEx-capable packet with L or E (respectively) and takes the TCP payload bytes in that marked packet off the counter. 🦋 Headers aren't counted here, because packet sizes are pretty regular, so it's fine to skip them 🧚
Here's the big part, bestie 😭 No matter how tiny LEG or CEG is, if it's positive the sender MUST NOT defer packet marking. That's how we keep the ConEx Signals timely ⏰ So yeah, LEG and CEG will usually end up negative. Totally normal, no cap 💜
If both LEG and CEG are positive, the sender MUST mark each ConEx-capable packet with both L and E 🔥 And if a credit signal is also waiting (more on that in the next section), the C flag can get set too 🌈
4.2. Setting the Credit Flag 🎀
So the ConEx abstract mechanism [RFC7713] says sufficient credit MUST be signaled ahead of time to cover the congestion you expect during the one-RTT feedback delay. Credit planning is literally the whole vibe 👑
To keep an eye on the credit state at the audit, a ConEx sender has to maintain a Credit State Counter (CSC) in bytes 📊 When congestion hits, credits get used up, so the CSC goes down by the number of bytes that were lost or estimated to be ECN-marked 💀 If the risk got estimated wrong and too few credits went out, the CSC hits zero, but it can't go negative. Floor it at zero, periodt 🙅♀️
To make sure the credit state in the audit never reaches zero, the number of credits should always equal the number of bytes in flight, because any of those packets could get lost or congestion-marked 🫶 So a ConEx sender also tracks bytes in flight, called F. If F ever gets bigger than the CSC, the sender sets the C flag on each ConEx-capable packet and bumps the CSC up by the payload size of each marked packet, until the CSC is no less than F again ✨ A sender can also be less conservative and send fewer credits if it, say, assumes the path will be low-congestion based on past experience. Bold move, but it's valid 🦋
Remember, the CSC goes down every time congestion happens, so it needs refilling as soon as it drops below F 🍓 And the sender can set the C flag on a ConEx-capable packet whether or not the E or L flags are set too. Multi-flag era 🌸
Now, TCP Slow Start is where things get wild 🎢 The congestion window can grow way bigger than it does during the rest of the transmission. A sender could consider sending fewer than F credits, but then it risks getting penalized by an audit function 😬 Still, the credits should at least cover the increase in sending rate. Slow Start doubles the sending rate every RTT, so a ConEx sender should at least cover half the packets in flight with credits 💖
Also, the number of losses or markings in one RTT doesn't depend only on the sender 🙈 The cross traffic, whether Active Queue Management (AQM) is in use, and how it's configured all affect how many packets get dropped or marked. As long as any AQM along the way isn't overly aggressive with ECN marking, sending half the flight size as credits should be enough, whether congestion shows up as loss or as ECN 🌈
To keep half the packets in flight as credits, half the packets of the initial window also have to be C-marked. In Slow Start marking, every fourth packet gives you the right amount of credit, as shown in Figure 1 👇
in_flight credits
RTT1 |------XC------>| 1 1
|------X------->| 2 1
|------XC------>| 3 2
| |
RTT2 |------X------->| 3 2
|------X------->| 4 2
|------X------->| 4 2
|------XC------>| 5 3
|------X------->| 5 3
|------X------->| 6 3
| |
RTT3 |------X------->| 6 3
|------XC------>| 7 4
|------X------->| 7 4
|------X------->| 8 4
|------X------->| 8 4
|------XC------>| 9 5
|------X------->| 9 5
|------X------->| 10 5
|------X------->| 10 5
|------XC------>| 11 6
|------X------->| 11 6
|------X------->| 12 6
| . |
| : |
Figure 1: Credits in Slow Start (with an initial window of 3)It's possible a TCP flow runs into an audit function with no relevant flow state, maybe because of rerouting or memory limits 🗺️ So the sender needs to notice this and resend credits. A ConEx sender might reset the CSC to zero if losses happen in later RTTs (assuming the sending rate was properly reduced based on the congestion signal, and using a conservatively large RTT estimate) 🔄
This section proposes one concrete algorithm for figuring out how much credit to signal (with a separate approach for Slow Start). But experimenting with credit-setting algorithms is expected and encouraged!! 🧪 The big goal of ConEx is to reflect the "cost" of the risk of causing congestion on the folks who contribute most to it 👑 So experimenting to improve performance (or keep it the same) while cutting down the risk of causing congestion, and maybe needing to signal so much credit, is a win 🏆
5. Loss of ConEx Information 💔
Packets carrying ConEx Signals can get discarded too. But this is a second-order problem 🤓 For example, if the loss probability is 0.1%, the probability of losing a ConEx L signal is 0.1% of 0.1%, which is 0.01%. Also, the penalty an audit applies should be proportional to the mismatch between expected ConEx marks and observed congestion, so the audit might only slightly raise this flow's loss level. Therefore, an implementer MAY choose to ignore this problem and accept the risk that an audit function might wrongly penalize a flow 🫣
That said, a ConEx sender is responsible for always signaling enough congestion feedback. So it SHOULD remember which packet got marked with the L, E, or C flag 📝 If one of those packets is detected as lost, the sender SHOULD increase the respective gauge(s), LEG or CEG, by the number of lost payload bytes, and also increase LEG for the loss 💅
6. Timeliness of the ConEx Signals ⏰
ConEx Signals only help a network node within about one RTT after the congestion happened. So to avoid extra delays, a ConEx sender SHOULD send the ConEx signaling on the next available packet 🚀
Any or all of the ConEx flags can go in the same packet, which keeps delays to a minimum when several signals are pending 🎀 You might need to set multiple ConEx flags at once if, for example, an ACK the sender gets says at least one ECN mark was received AND one or more segments were lost. That can happen during heavy congestion, when queues overflow even though ECN is in use and every forwarded packet is already marked, so others have to be dropped 😱 It can also happen when ACKs get lost, so a later ACK carries summary info the sender didn't have before 🧚
If a flow becomes application-limited, there might not be enough bytes to send to bring the gauges down to zero or below. In that case, the sender can't help but delay ConEx Signals 🥺 Still, as long as the sender marks all outgoing packets, an audit function is unlikely to penalize ConEx-Marked packets, so no matter how long a gauge has been positive, a sender MUST NOT reduce the gauge by more than the ConEx-Marked bytes it has sent 🔒
If the CEG or LEG counter is negative, the counter MAY be reset to zero within one RTT after it was last decreased, or one RTT after recovery if no further congestion occurred ✨
7. Open Areas for Experimentation 🧪✨
okay so bestie, every single mechanism in this doc is experimental 🥺 which means we literally need more big, real-internet testing to see if the signals are accurate AND timely enough to be worth anything for ConEx-based traffic management (or other stuff) 💅
The current ConEx specs assume congestion gets counted in bytes 📏 (that includes the IP header that directly wraps the CDO, plus everything that header wraps) [RFC7837]. Why? Because most network devices today run out of memory on byte-congestion, where the memory gets filled with exactly the number of bytes a packet carries [RFC7141]. But plot twist 🙈 some devices allocate a set amount of memory per packet, no matter how big the packet is. Those devices get congested based on how many packets are sitting in memory, so for them congestion is about the number of packets that got lost or marked. 🫠
And it gets messier. A transport endpoint like a TCP sender or receiver might not know the exact number of bytes a lower layer is carrying, so a TCP endpoint may only be able to estimate the exact number of congested bytes (assuming all the lower-layer headers are the same length). If that estimate is good enough to work with, the ConEx Signal needs more evaluation on the Internet, tested together with different auditor implementations 🔍
Next up: the marking schemes in this doc are designed under the assumption that either all TCP packets of a ConEx-capable flow are the same size, or flows have a constant mean packet size over a pretty small time frame, like one RTT or less. Honestly this is probably true for most traffic flows, and most implementations probably assume it too 🙌 But if this scheme gets used, we need to check how much accuracy degrades when that precondition isn't met. Testing with real traffic from different applications is super important for deciding whether these schemes are enough or whether we need a more complex scheme 🧩
Now the credit markings in Slow Start 😬 the proposed scheme here runs the risk of giving too few markings, and then an audit function could penalize that flow for no good reason. So both the credit scheme for Slow Start AND the scheme in Congestion Avoidance need to be tested together with one or more specific ConEx auditor implementations. The goal is making sure the algorithms in the sender and in the auditor work well together, with a low risk of false positives (which would penalize an honest sender, and that's just rude 💔). Still, if a sender gets wrongly assumed to be cheating, the penalty from the audit should be adequate, and it should let an honest sender using a commonly used congestion control scheme recover quickly 🌸
Another open issue is how accurate the ECN feedback signal is 🎀 At the time of this doc's publication, there's no AccECN mechanism specified yet, and AccECN is going to take a while to get widely deployed after that. So this doc proposes an advanced compatibility mode for classic ECN 💖 This mode can give more accurate feedback by using the way classic ECN is specified, but it has a higher risk of losing information. To figure out how high that risk is in a real deployment, we need more experimental evaluation 🧐
The following argument is meant to show that suppressing repetitions of ECE is still safe against congestion collapse caused by lost congestion feedback. Honestly though, it should be proven further through experimentation, so don't take it as gospel yet 🫶
Repetition of ECE in classic ECN is there to make sure congestion feedback gets delivered reliably. But with advanced compatibility mode, it's possible to miss congestion notifications 😭 In some implementations this can happen when delayed acknowledgements are used. An ACK carrying ECE can also simply get lost. If only a few CE marks are received within one congestion event (like, only one), then losing one acknowledgement because of heavy congestion on the reverse path can stop any congestion notification from reaching the sender at all 🥀
BUT, if losing feedback makes congestion on the forward path worse, then more forward packets get CE-marked, which makes it more likely that feedback from at least one CE gets through per RTT ✨ As long as one ECE reaches the sender per RTT, the sender's congestion response is the same as if CWR weren't continuous. The only way heavy forward-path congestion could be completely hidden is if all the ACKs on the reverse path got lost. And if total ACK loss kept going, the sender would time out and do a congestion response anyway 💪 So the problem seems limited to possibly suppressing a congestion response during light congestion. Contained, iykyk 🙈
Also, even if losing all the ECN feedback leads to no congestion response, the worst that could happen is loss happening instead of ECN-signaled congestion on the forward path. And since compatibility mode doesn't affect loss feedback, there would be no risk of congestion collapse 👑
8. Security Considerations 🔐
Okay, real talk 💀 General ConEx security considerations are covered extensively in the ConEx abstract mechanism [RFC7713]. This section covers the TCP-specific concerns that can come up when ConEx gets added to TCP. (It doesn't go into the generally well-known attacks against TCP, that's a whole other iceberg 🧊) It's assumed that any altering of ConEx information can be detected by protection mechanisms in the IP layer, so it isn't discussed here but in [RFC7837]. Also, [RFC7837] explains how to use ConEx to mitigate flooding attacks by using preferential drop, and in that case using ConEx can even increase security 🛡️✨
The ConEx modifications to TCP give a receiver no mechanism to force a sender to stop using ConEx 🚫 A receiver can degrade the accuracy of ConEx by claiming that it doesn't support SACK, AccECN, or ECN. But the sender never has to turn ConEx off, so it's not a total win for the receiver 😌 A receiver also can't force the sender to mark ConEx more conservatively to cover the risk of any inaccuracy. Instead, it's always the sender's choice: either mark very conservatively, which makes sure the audit always sees enough markings to not penalize the flow, or estimate the needed number of markings more tightly. That second option can lead to inaccurate marking, which increases the likelihood of loss at an audit function, and that only harms the receiver itself 🤷♀️
Now, assuming the sender is limited in some way by a congestion allowance or quota, a receiver could spoof more loss or ECN congestion feedback than it actually experiences 👀 The goal would be making the sender draw down its allowance faster than necessary. BUT, over-declaring congestion just makes the sender slow down. And if the receiver actually cares about the content, it won't want to hurt its own performance, so that's self-sabotage 🙅♀️
Here's the thing though: if the receiver ONLY cares about making the sender burn through its allowance, the net effect depends on the sender's congestion control algorithm 🧠 Permanently adding more and more additional congestion would make the sender reduce its sending rate more and more. So a receiver can only maintain a certain congestion level, and that level corresponds to a certain sending rate. With NewReno [RFC6582], doubling congestion feedback makes the sender reduce its sending rate so it only consumes sqrt(2) = 1.4 times more congestion allowance.
Meanwhile, to improve scaling, congestion control algorithms are trending toward less responsive algorithms like Cubic or Compound TCP, and ultimately toward linear algorithms like Data Center TCP (DCTCP) [DCTCP]. Those aim to maintain the same congestion level no matter the current sending rate, and they always reduce the sending window if the signaled congestion feedback is higher. In each case, if the receiver doubles the congestion feedback, the sender consumes more allowance by a factor of 1.2, 1.15, or 1 respectively. That last 1 means the attack has become completely ineffective, since no further congestion allowance gets consumed, but the flow will decrease its sending rate to a minimum instead 🌈💅
9. References 📚
9.1. Normative References 💜
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
Selective Acknowledgment Options", RFC 2018,
DOI 10.17487/RFC2018, October 1996,
<http://www.rfc-editor.org/info/rfc2018>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<http://www.rfc-editor.org/info/rfc3168>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<http://www.rfc-editor.org/info/rfc5681>.
[RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx)
Concepts, Abstract Mechanism, and Requirements", RFC 7713,
DOI 10.17487/RFC7713, December 2015,
<http://www.rfc-editor.org/info/rfc7713>.
[RFC7837] Krishnan, S., Kuehlewind, M., Briscoe, B., and C. Ralli,
"IPv6 Destination Option for Congestion Exposure (ConEx)",
RFC 7837, DOI 10.17487/RFC7837, May 2016,
<http://www.rfc-editor.org/info/rfc7837>.9.2. Informative References 📚✨
okay so these are the "good to know" ones, bestie 🧚 they're not required to get the protocol working, but they're where the receipts live. the Congestion Exposure stuff, the TCP loss-recovery crew, all of it, lowkey the backstory 💅
[ACCURATE] Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More
Accurate ECN Feedback in TCP", Work in Progress,
draft-ietf-tcpm-accurate-ecn-00, December 2015.
[DCTCP] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel,
P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data
Center TCP (DCTCP)", ACM SIGCOMM Computer Communication
Review, Volume 40, Issue 4, pages 63-74,
DOI 10.1145/1851182.1851192, October 2010,
<http://portal.acm.org/citation.cfm?id=1851192>.
[ECNTCP] Briscoe, B., Jacquet, A., Moncaster, T., and A. Smith,
"Re-ECN: Adding Accountability for Causing Congestion to
TCP/IP", Work in Progress, draft-briscoe-conex-re-ecn-
tcp-04, July 2014.
[RFC3522] Ludwig, R. and M. Meyer, "The Eifel Detection Algorithm
for TCP", RFC 3522, DOI 10.17487/RFC3522, April 2003,
<http://www.rfc-editor.org/info/rfc3522>.
[RFC3708] Blanton, E. and M. Allman, "Using TCP Duplicate Selective
Acknowledgement (DSACKs) and Stream Control Transmission
Protocol (SCTP) Duplicate Transmission Sequence Numbers
(TSNs) to Detect Spurious Retransmissions", RFC 3708,
DOI 10.17487/RFC3708, February 2004,
<http://www.rfc-editor.org/info/rfc3708>.
[RFC4015] Ludwig, R. and A. Gurtov, "The Eifel Response Algorithm
for TCP", RFC 4015, DOI 10.17487/RFC4015, February 2005,
<http://www.rfc-editor.org/info/rfc4015>.
[RFC5682] Sarolahti, P., Kojo, M., Yamamoto, K., and M. Hata,
"Forward RTO-Recovery (F-RTO): An Algorithm for Detecting
Spurious Retransmission Timeouts with TCP", RFC 5682,
DOI 10.17487/RFC5682, September 2009,
<http://www.rfc-editor.org/info/rfc5682>.
[RFC6582] Henderson, T., Floyd, S., Gurtov, A., and Y. Nishida, "The
NewReno Modification to TCP's Fast Recovery Algorithm",
RFC 6582, DOI 10.17487/RFC6582, April 2012,
<http://www.rfc-editor.org/info/rfc6582>.
[RFC6789] Briscoe, B., Ed., Woundy, R., Ed., and A. Cooper, Ed.,
"Congestion Exposure (ConEx) Concepts and Use Cases",
RFC 6789, DOI 10.17487/RFC6789, December 2012,
<http://www.rfc-editor.org/info/rfc6789>.
[RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion
Notification", BCP 41, RFC 7141, DOI 10.17487/RFC7141,
February 2014, <http://www.rfc-editor.org/info/rfc7141>.Acknowledgements 🫶
the authors wanna give a big shoutout to Bob Briscoe 👑 who came up with these initial ideas [ECNTCP] and gave super valuable feedback. and Jana Iyengar also gets major thanks 💜 for the valuable feedback too, no cap 🌸
Authors' Addresses 💌
Mirja Kuehlewind (editor)
ETH Zurich
Switzerland
Email: mirja.kuehlewind@tik.ee.ethz.ch
Richard Scheffenegger
NetApp, Inc.
Am Euro Platz 2
Vienna 1120
Austria
Email: rs.ietf@gmx.at📦 where this page comes from
- ietf: rfc7786 2016-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms