RFC 3465

RFC 3465: TCP Congestion Control with Appropriate Byte Counting (ABC) 🎀

Okay bestie, this is RFC 3465 💖 It's called "TCP Congestion Control with Appropriate Byte Counting (ABC)," and M. Allman from BBN/NASA GRC wrote it. It's an Experimental doc, dated February 2003. The title block doesn't list anything it obsoletes or updates, so this one isn't replacing or changing any other RFC 🧐

Status of this Memo ✨

So basically this is an Experimental Protocol for the Internet community 🧪 It's not an Internet standard of any kind. They're asking for discussion and suggestions for improvement 💬 And distribution is unlimited, so share it around 🦋

Copyright Notice 🔒

Copyright (C) The Internet Society (2003). All Rights Reserved. The legal stuff, but it's giving official 💅

Abstract 📝

This doc proposes a small tweak to how TCP grows its congestion window 🌱 Instead of the old-school move of bumping the window by a fixed amount for every ACK that arrives, it suggests basing the increase on how many previously unacknowledged bytes each ACK covers. This is iconic for two reasons: it makes TCP perform better ✨ and it closes a security hole that TCP receivers could use to trick the sender into speeding up way too fast 🙈

Terminology 📚

Much of the language in this document is borrowed from [RFC2581] 🤝

Also, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document should be read the way [RFC2119] describes them 📖

1. Introduction 🌸

Okay so this doc tweaks the algorithm TCP uses to grow its congestion window (cwnd), and it makes things better for both performance and security 🔥 Right now, TCP grows cwnd based on how many ACKs arrive at the data sender (that's the current spec in [RFC2581]). This doc flips that: cwnd grows based on how many bytes the arriving ACKs acknowledge 🎯

Why is this the move? It helps performance by reducing the damage delayed ACKs do to cwnd growth 🙌 It also grows cwnd in direct proportion to the capacity the network path has actually shown it can handle. That means when an ACK only covers a small amount of data (less than a full segment size), the response is more measured than with ACK counting 🧘‍♀️ That's a more appropriate growth pattern, which can boost performance and stop cwnd from growing wrongly when a receiver is misbehaving 👑

Real talk though, there's a catch 😬 In some cases this modified growth causes bigger bursts of segments to hit the network. Sometimes that means a noticeable jump in the drop rate and worse performance. See section 4 for the full drama 🎭

Here's how the doc is laid out. Section 2 walks through the modified algorithm for growing TCP's congestion window. Section 3 covers the perks of using it. Section 4 covers the downsides of the approach in this doc. Section 5 goes over the fairness issues that must be considered for the modified algorithm. Section 6 is all about security considerations 💜

Statement of Intent 🎯

This spec has an algorithm that improves TCP performance. It's understood to be effective and safe, but it hasn't been widely deployed yet. One goal of publishing it as an Experimental RFC is to be prudent and encourage people to use and deploy it before it goes on the standards track 🧪 The Transport Area intends to resubmit this spec as an IETF Proposed Standard in the future, once more experience has been gained 🌈

2. A Modified Algorithm for Increasing the Congestion Window 🪟

Okay so as originally outlined in [Jac88] and specified in [RFC2581], TCP uses two algorithms to grow the congestion window 🙌 During steady state, TCP uses Congestion Avoidance to increase cwnd linearly. At the start of a transfer, after a retransmission timeout, or after a long idle period (in some implementations), TCP uses Slow Start to increase cwnd exponentially 🚀 According to RFC 2581, slow start bases the cwnd increase on the number of incoming acknowledgments. During congestion avoidance, RFC 2581 gives more latitude for increasing cwnd, but traditionally implementations have based the increase on how many ACKs arrive 🧃

So in the next two subsections, we're changing both algorithms to grow cwnd based on how many bytes each arriving ACK acknowledges, instead of how many ACKs arrive ✨ We call this "Appropriate Byte Counting" (ABC) [All99] 💖

2.1. Congestion Avoidance 🌷

RFC 2581 says cwnd should grow by 1 segment per round-trip time (RTT) during the congestion avoidance phase 📈 Traditionally, TCPs have approximated this by increasing cwnd by 1/cwnd for each arriving ACK. That opens cwnd by roughly 1 segment per RTT if the receiver ACKs every incoming segment and no ACK loss happens 🙂 But if the receiver uses delayed ACKs [Bra89], it sends roughly half as many ACKs. That makes the sender open cwnd more conservatively, by about 1 segment every second RTT 🐢

Here's the fix this doc suggests. Store the number of bytes that have been ACKed in a bytes_acked variable in the TCP control block. When bytes_acked becomes greater than or equal to the value of the congestion window, subtract the value of cwnd from bytes_acked. Then increment cwnd by a full-sized segment (SMSS) 💅 This is specifically allowed by RFC 2581 during congestion avoidance, because it opens the window by at most 1 segment per RTT, so it's totally legit 🫶

2.2. Slow Start 🚀

RFC 2581 says the sender increases the congestion window by at most 1*SMSS bytes for each arriving acknowledgment during slow start 🧁 This doc proposes that a TCP sender SHOULD increase cwnd by the number of previously unacknowledged bytes ACKed by each incoming acknowledgment, as long as the increase is not more than L bytes. We'll talk about picking L in the next subsection 🔍

Here's the vibe check. When the number of previously unacknowledged bytes ACKed is less than or equal to 1*SMSS bytes, or when L is less than or equal to 1*SMSS bytes, this proposal is no more aggressive (and possibly less aggressive) than RFC 2581 allows 🌸 But increasing cwnd by more than 1*SMSS bytes in response to a single ACK IS more aggressive than RFC 2581 allows 👀 Still, this more aggressive slow start still fits the spirit of the principles in [Jac88] (no more than doubling cwnd per RTT). So this doc proposes ABC for experimentation in shared networks, as long as an appropriate limit is applied (see the next section) 🧪

2.3. Choosing the Limit 🎀

The limit, L, chosen for the cwnd increase during slow start controls how aggressive the algorithm is 🎚️ Choosing L=1*SMSS bytes gives behavior that is no more aggressive than RFC 2581 allows 🙂 But ABC with L=1*SMSS bytes is more conservative in a number of key ways (as discussed in the next section). So even though TCP stacks will see little performance change with L=1*SMSS bytes, this doc suggests that ABC SHOULD be used anyway 💖

A very large L could lead to large line-rate bursts of traffic if there's a lot of ACK loss, or when the receiver sends "stretch ACKs" (ACKs for more than the two full-sized segments the delayed ACK algorithm allows) [Pax97] 😱

Now for the sweet spot ✨ This doc specifies that TCP implementations MAY use L=2*SMSS bytes and MUST NOT use L > 2*SMSS bytes 🙌 This balances being conservative (L=1*SMSS bytes) with being potentially very aggressive. L=2*SMSS bytes also exactly balances out the negative impact of the delayed ACK algorithm (more on that in section 3.2) 🫶 When L=2*SMSS bytes, cwnd growth is roughly the same as with the standard algorithms used alongside a receiver that sends an ACK for each incoming segment [All98] (assuming no or small amounts of ACK loss in both cases) 💜

There's one exception to the suggestion above 🚨 During a slow start phase that follows a retransmission timeout (RTO), a TCP MUST use L=1*SMSS as RFC 2581 specifies. That's because ACKs for large amounts of previously unacknowledged data are super common during this phase of a transfer 🥺 These ACKs don't necessarily show how much data has left the network in the last RTT, so ABC can't accurately figure out how much to increase cwnd 🤷‍♀️

Let's do an example 🧩 Say segment N gets dropped by the network, and segments N+1 and N+2 arrive successfully at the receiver. The sender only gets two duplicate ACKs, so it has to rely on the retransmission timer (RTO) to detect the loss ⏰ When the RTO expires, segment N gets retransmitted. The ACK sent back in response to that retransmission will be for segment N+2. But this ACK does NOT mean three segments left the network in the last RTT. It only means a single segment left the network 😭 So the appropriate cwnd increment is at most 1*SMSS bytes 🙅‍♀️

2.4. RTO Implications ⏰

[Jac88] shows that increasing cwnd by more than a factor of two in successive RTTs can cause spurious retransmissions on slow links where bandwidth dominates the RTT, assuming the RTO estimator given in [Jac88] and [RFC2988] 🐌 ABC stays within this no-more-than-doubling-cwnd-in-successive-RTTs limit by capping the increase (no matter what L is used) by the number of previously unacknowledged bytes covered by each incoming ACK 🛡️

3. Advantages 🏆

This section outlines several advantages of using the ABC algorithm to grow cwnd, instead of the standard ACK counting algorithm given in [RFC2581] 💅

3.1. More Appropriate Congestion Window Increase 🦋

The ABC algorithm from section 2 grows TCP's cwnd in proportion to the amount of data actually sent into the network 🙌 ACK counting, on the other hand, bumps cwnd by a constant every time an ACK arrives 😤 Picture an interactive telnet connection (e.g., ssh or telnet) where ACKs usually cover only a few bytes of data, but cwnd still grows by 1*SMSS bytes for each ACK received 💀 When a big chunk of data needs to go out (like displaying a large file), it all gets sent in one large burst, because cwnd grows by 1*SMSS bytes per ACK instead of based on the capacity that was actually used. A line-rate burst like that can cause a lot of segment loss 😭

Congestion Window Validation (CWV) [RFC2861] fixes this same problem too 🧹 CWV limits how much unused cwnd a TCP connection can build up. ABC can be used together with CWV to get an accurate measure of the network path 🌈

3.2 Mitigate the Impact of Delayed ACKs and Lost ACKs 🫶

Okay so delayed ACKs [RFC1122,RFC2581] let a TCP receiver chill and skip ACKing every single segment 💅. Still, a receiver SHOULD send an ACK for every second full-sized segment that arrives. And a receiver MUST NOT withhold an ACK for more than 500 ms, no exceptions 😭. Fewer ACKs means the receiver is slowing the growth of the congestion window when you're counting ACKs. Using ABC with L=2*SMSS bytes basically cancels out that hit, because cwnd can still grow for the ACKs the receiver held back ✨. So the window grows like it would if every segment got ACKed, but without the extra traffic on the network. Simulation studies showed higher throughput with ABC compared to the standard ACK counting algorithm [All99], especially for short transfers that never leave the initial slow start period 🚀.

Note that delayed ACKs shouldn't be an issue during slow start-based loss recovery, since RFC 2581 recommends that receivers not delay ACKs that cover out-of-order segments 🙈. So, as we talked about above, ABC with L > 1*SMSS bytes is not a good fit for that kind of recovery, and it MUST NOT be used 🚫.

Note: If an entire window of data gets lost, a TCP receiver will probably generate delayed ACKs, and L > 1*SMSS bytes would be safe. The catch is that spotting this scenario is hard 🥺. So to keep ABC conservative, this document mandates that L MUST NOT be > 1*SMSS bytes in any slow start-based loss recovery 🔒.

Lost ACKs are another way cwnd growth gets slowed down, especially when the window is based on how many ACKs arrive 💔. With ACK counting, every dropped ACK is a forever-missed chance to grow cwnd. Using ABC with L > 1*SMSS bytes lets the sender soften the blow from lost ACKs 🦋.

3.3 Prevents Attacks from Misbehaving Receivers 👀

[SCWA99] lays out a few ways a receiver can trick a TCP sender into breaking congestion control and sending data at a possibly inappropriate rate 😤. One of those attacks is called "ACK Division". Here the receiver sends several ACKs for each incoming data segment, and each one only ACKs a small piece of that segment. Since TCP senders have traditionally used ACK counting to grow cwnd, ACK division makes cwnd grow way too fast, and that can mean a potentially inappropriate sending rate 🏃‍♀️. A TCP sender that uses ABC blocks this attack from undermining standard congestion control, because cwnd growth is based on the number of bytes ACKed instead of the number of ACKs received 🛡️.

To keep misbehaving receivers from pushing senders into bad behavior, this document suggests TCP implementations use ABC, even if L=1*SMSS bytes (i.e., not letting ABC give more aggressive cwnd growth than RFC 2581 allows) 💖.

4. Disadvantages ✨

The main downsides of using ABC with L=2*SMSS bytes are a bit more burstiness in TCP and a small bump in the overall loss rate 🌪️. [All98] talks about the two ways ABC makes the TCP sender burstier. First, the "micro burstiness" of the connection goes up. In other words, the number of segments sent in response to each incoming ACK increases by at most 1 segment when you use ABC with L=2*SMSS bytes alongside a receiver that sends delayed ACKs. During slow start, that means going from sending 2 back-to-back segments to sending 3 back-to-back packets in response to an ACK for a single packet. Or it goes from 3 packets to 4 packets when a delayed ACK comes in for two outstanding packets 📦. Note that ACK loss can cause bigger bursts too. Still, ABC only raises the burst size by at most 1*SMSS bytes per ACK received compared to standard behavior. This little bit of extra burstiness should only cause problems for devices with very small buffers 🧸. Second, ABC increases the "macro burstiness" of the TCP sender in response to delayed ACKs during slow start. Instead of growing cwnd by roughly 1.5 times per RTT, ABC roughly doubles the congestion window every RTT 📈. But doubling cwnd every RTT still fits within the spirit of slow start, as originally outlined [Jac88] 🥰.

With the extra burstiness comes a modest increase in the loss rate for a TCP connection using ABC (see the next section for a quick chat about fairness between ABC and non-ABC flows) 🙃. That extra loss can be directly blamed on ABC being more aggressive. During slow start, cwnd grows faster, so when loss happens, cwnd is bigger and more drops are likely 💥. Similarly, a congestion avoidance cycle takes roughly half as long when you use ABC with delayed ACKs compared to an ACK counting implementation. Put another way, a TCP sender hits the capacity of the network path, drops a packet, and cuts the congestion window in half roughly twice as often with ABC 🔁. Still, as discussed above, even with the extra loss, an ABC TCP sender generally gets better overall performance than a non-ABC TCP [All99] 💅✨.

Because the packet drop rate goes up, we suggest ABC be implemented together with selective acknowledgments [RFC2018] 🎀.

5. Fairness Considerations 🌈

[All99] runs a few simple simulations to measure how ABC affects competing traffic (both ABC and non-ABC) 👯. The experiments show that while ABC increases the drop rate for the connection using ABC, competing traffic is not greatly effected. Standard TCP and ABC both get roughly the same throughput, no matter which variant the competing traffic is. The simulations also back up that ABC beats non-ABC TCP in an environment with a mix of TCP connection types 🏆. On the other hand, the simulations in [All99] aren't necessarily realistic, so we're encouraging more experimentation on the Internet 🌍.

6. Security Considerations 🔐

As discussed in section 3.3, ABC protects a TCP sender from a misbehaving receiver that tricks the sender into transmitting at an inappropriate rate with an "ACK division" attack 🚨. In turn, this protects the network from an overly aggressive sender. Honestly, this is a whole protective shield for the network, iykyk 🛡️💀.

7. Conclusions 👑

This document RECOMMENDS that all TCP stacks be modified to use ABC with L=1*SMSS bytes. This change does not increase the aggressiveness of TCP 💜. Plus, simulations of ABC with L=2*SMSS bytes show a promising performance improvement, and we encourage researchers to experiment with it on the Internet 🧚.

---

Acknowledgments 🫶

This document has benefited from discussions with, and encouragement from, Sally Floyd 💖. Van Jacobson and Reiner Ludwig gave valuable input on the implications of byte counting for the RTO 🌸. Reiner Ludwig and Kostas Pentikousis gave valuable feedback on a draft of this document 🍓.

Normative References 📚

• [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -- Communication Layers", STD 3, RFC 1122, October 1989.
• [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
• [RFC2581] Allman, M., Paxson, V. and W. Stevens, "TCP Congestion Control", RFC 2581, April 1999.

Informative References 🔎

• [All98] Mark Allman. On the Generation and Use of TCP Acknowledgments. ACM Computer Communication Review, 29(3), July 1998.
• [All99] Mark Allman. TCP Byte Counting Refinements. ACM Computer Communication Review, 29(3), July 1999.
• [Jac88] Van Jacobson. Congestion Avoidance and Control. ACM SIGCOMM 1988.
• [Pax97] Vern Paxson. Automated Packet Trace Analysis of TCP Implementations. ACM SIGCOMM, September 1997.
• [RFC2018] Mathis, M., Mahdavi, J., Floyd, S. and A. Romanow, "TCP Selective Acknowledgment Options", RFC 2018, October 1996.
• [RFC2861] Handley, M., Padhye, J. and S. Floyd, "TCP Congestion Window Validation", RFC 2861, June 2000.
• [SCWA99] Stefan Savage, Neal Cardwell, David Wetherall, Tom Anderson. TCP Congestion Control with a Misbehaving Receiver. ACM Computer Communication Review, 29(5), October 1999.

Author's Address 📮

Mark Allman
BBN Technologies/NASA Glenn Research Center
Lewis Field
21000 Brookpark Rd.  MS 54-5
Cleveland, OH  44135

Fax: 216-433-8705
Phone: 216-433-6586
EMail: mallman@bbn.com
http://roland.grc.nasa.gov/~mallman

Full Copyright Statement ⚖️

Copyright (C) The Internet Society (2003). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 💜

Funding for the RFC Editor function is currently provided by the Internet Society 🌷.