RFC 9406

HyStart++: Modified Slow Start for TCP ✨

Okay bestie, let's get into RFC 9406! 🎀 Here's the title block first:

• RFC number: 9406 💖
• Title: HyStart++: Modified Slow Start for TCP
• Authors: P. Balasubramanian (Confluent), Y. Huang, M. Olson (Microsoft) 🫶
• Category: Standards Track
• ISSN: 2070-1721
• Date: May 2023 🌸
• Obsoletes / Updates: nothing is listed in this title block 🦋

It's an IETF document, the Internet Engineering Task Force's stuff. 🌈

Abstract

HyStart++ is a small tweak to the slow start phase of congestion control algorithms. Slow start can overshoot the ideal send rate in a lot of situations, and that causes high packet loss and bad performance 💀. HyStart++ uses an increase in round-trip delay as a heuristic to find an exit point before that overshoot happens 👑. It also has a mitigation so that jitter can't cause a premature slow start exit 🧚.

Status of This Memo

This is an Internet Standards Track document, so it's the real deal 💅

It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check Section 2 of RFC 7841 🍓

For the current status of this document, any errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc9406 🔥

Copyright Notice

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date this document was published. Read those carefully, because they spell out your rights and restrictions for this document 🙈 Any Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come without warranty, as the Revised BSD License describes.

Table of Contents

   1.  Introduction
   2.  Terminology
   3.  Definitions
   4.  HyStart++ Algorithm
     4.1.  Summary
     4.2.  Algorithm Details
     4.3.  Tuning Constants and Other Considerations
   5.  Deployments and Performance Evaluations
   6.  Security Considerations
   7.  IANA Considerations
   8.  References
     8.1.  Normative References
     8.2.  Informative References
   Acknowledgments
   Authors' Addresses

---

1. Introduction 🦋

[RFC5681] describes the slow start congestion control algorithm for TCP. Slow start gets used while the congestion window (cwnd) is smaller than the slow start threshold (ssthresh). During slow start, as long as there are no packet loss signals, TCP grows the cwnd exponentially to probe how much capacity the network has ✨ That fast growth can overshoot the ideal sending rate and cause a lot of packet loss, and that loss can't always be recovered efficiently 😭

HyStart++ builds on Hybrid Start (HyStart), which was originally described in [HyStart]. HyStart++ uses an increase in round-trip delay as a signal to exit slow start before potential packet loss from overshoot happens. This is one of the two algorithms [HyStart] specifies for finding a safe exit point from slow start. After the exit, a new Conservative Slow Start (CSS) phase figures out whether the exit was premature, and resumes slow start if so 🫶 That mitigation helps performance when there's jitter. HyStart++ reduces packet loss and retransmissions, and improves goodput, in both lab measurements and real-world deployments 🔥

This document describes HyStart++ for TCP, but it can also be used for other transport protocols that use slow start, like QUIC [RFC9002] or the Stream Control Transmission Protocol (SCTP) [RFC9260] 🌈

2. Terminology 🎀

Here's the key word deal. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" get interpreted as described in BCP 14 [RFC2119] [RFC8174]. But only when, and only when, they show up in all capitals, exactly like that 💅 Anywhere else they're just regular words, so don't panic 🙈

3. Definitions 🧚

To help you out, we repeat some definitions from [RFC5681] 💖

SENDER MAXIMUM SEGMENT SIZE (SMSS): The size of the largest segment the sender can transmit. This value can come from the network's maximum transmission unit, the Path MTU Discovery algorithm [RFC1191] [RFC4821], RMSS (see the next item), or other factors. The size does not include the TCP/IP headers and options.

RECEIVER MAXIMUM SEGMENT SIZE (RMSS): The size of the largest segment the receiver is willing to accept. This is the value in the MSS option the receiver sends during connection startup. If the MSS option isn't used, it's 536 bytes [RFC1122]. Again, the size doesn't include the TCP/IP headers and options.

RECEIVER WINDOW (rwnd): The most recently advertised receiver window. 🌸

CONGESTION WINDOW (cwnd): A TCP state variable that limits how much data a TCP can send. At any given time, a TCP MUST NOT send data with a sequence number higher than the sum of the highest acknowledged sequence number and the minimum of the cwnd and rwnd. 👑 That's a hard rule, bestie.

4. HyStart++ Algorithm ✨

4.1. Summary

[HyStart] specifies two algorithms, a "Delay Increase" algorithm and an "Inter-Packet Arrival" algorithm, that run in parallel to detect when the sending rate has reached capacity 🧮 In practice, the Inter-Packet Arrival algorithm doesn't perform well and can't detect congestion early, mainly because of ACK compression 💀 The Delay Increase algorithm instead looks for spikes in RTT (round-trip time), which suggest the bottleneck buffer is filling up 🍓

HyStart++ has the sender use standard slow start, then uses the Delay Increase algorithm to trigger an exit from slow start. But instead of going straight from slow start into congestion avoidance, the sender spends some RTTs in a Conservative Slow Start (CSS) phase to check whether the exit was premature 🦋 During CSS, the congestion window grows exponentially, similar to regular slow start, but with a smaller exponential base, so the growth is less aggressive. If the RTT goes back down during CSS, the RTT spike wasn't caused by congestion from sending above the ideal rate, so the connection resumes slow start 🎉 If the RTT inflation sticks around through all of CSS, the connection enters congestion avoidance.

4.2. Algorithm Details

The pseudocode below uses a limit, L, to control how aggressively cwnd increases during both standard slow start and CSS. An arriving ACK might newly acknowledge any number of bytes, but HyStart++ caps the number of those bytes used to grow the cwnd at L*SMSS bytes 🧁

lastRoundMinRTT and currentRoundMinRTT get initialized to infinity at initialization time. currRTT is the RTT sampled from the latest incoming ACK, and it's also initialized to infinity 💜

lastRoundMinRTT = infinity
currentRoundMinRTT = infinity
currRTT = infinity

HyStart++ measures rounds using sequence numbers, like this:

• Define windowEnd as a sequence number initialized to SND.NXT.
• When windowEnd is ACKed, the current round ends and windowEnd is set to SND.NXT.

At the start of each round during standard slow start [RFC5681] and CSS, initialize the variables used to compute the minimum RTT for the last round and the current round:

lastRoundMinRTT = currentRoundMinRTT
currentRoundMinRTT = infinity
rttSampleCount = 0

For each arriving ACK in slow start, where N is the number of previously unacknowledged bytes acknowledged in that ACK, update the cwnd:

cwnd = cwnd + min(N, L * SMSS)

Then keep track of the minimum observed RTT:

currentRoundMinRTT = min(currentRoundMinRTT, currRTT)
rttSampleCount += 1

For rounds where at least N_RTT_SAMPLE RTT samples have been collected, and currentRoundMinRTT and lastRoundMinRTT are both valid, check whether the delay increase triggers a slow start exit:

if ((rttSampleCount >= N_RTT_SAMPLE) AND
    (currentRoundMinRTT != infinity) AND
    (lastRoundMinRTT != infinity))
  RttThresh = max(MIN_RTT_THRESH,
    min(lastRoundMinRTT / MIN_RTT_DIVISOR, MAX_RTT_THRESH))
  if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh))
    cssBaselineMinRtt = currentRoundMinRTT
    exit slow start and enter CSS

For each arriving ACK in CSS, where N is the number of previously unacknowledged bytes acknowledged in that ACK, update the cwnd:

cwnd = cwnd + (min(N, L * SMSS) / CSS_GROWTH_DIVISOR)

Then keep track of the minimum observed RTT:

currentRoundMinRTT = min(currentRoundMinRTT, currRTT)
rttSampleCount += 1

For CSS rounds where at least N_RTT_SAMPLE RTT samples have been collected, check whether the current round's minRTT drops below the baseline (cssBaselineMinRtt). If it does, the slow start exit was spurious 🙃

if (currentRoundMinRTT < cssBaselineMinRtt)
  cssBaselineMinRtt = infinity
  resume slow start including HyStart++

CSS lasts at most CSS_ROUNDS rounds. If the transition into CSS happens in the middle of a round, that partial round still counts toward the limit ⏳

If CSS_ROUNDS rounds are complete, the connection enters congestion avoidance by setting ssthresh to the current cwnd:

ssthresh = cwnd

And if loss or Explicit Congestion Notification (ECN) marking is observed at any time during standard slow start or CSS, the connection also enters congestion avoidance by setting ssthresh to the current cwnd:

ssthresh = cwnd

4.3. Tuning Constants and Other Considerations 🎀

Okay bestie, this is the knobs-and-dials part. It is RECOMMENDED that a HyStart++ implementation use these constants:

MIN_RTT_THRESH = 4 msec
MAX_RTT_THRESH = 16 msec
MIN_RTT_DIVISOR = 8
N_RTT_SAMPLE = 8
CSS_GROWTH_DIVISOR = 4
CSS_ROUNDS = 5
L = infinity if paced, L = 8 if non-paced

These numbers came from lab measurements and real-world deployments, so they're not random vibes 💅. An implementation MAY tune them for different network characteristics 🦋.

The delay increase sensitivity is controlled by MIN_RTT_THRESH and MAX_RTT_THRESH ✨. Smaller MIN_RTT_THRESH values can cause spurious exits from slow start 💀. Larger MAX_RTT_THRESH values can mean slow start doesn't exit until loss shows up, on connections with large RTT paths 🙈.

MIN_RTT_DIVISOR is the fraction of RTT used to compute the delay threshold. A smaller value means a bigger threshold, so less sensitivity to delay increase, and vice versa 🧮.

Now for sampling. All TCP implementations are REQUIRED to take at least one RTT sample each round 📏, but HyStart++ implementations are RECOMMENDED to take at least N_RTT_SAMPLE RTT samples. Using lower values of N_RTT_SAMPLE lowers the accuracy of the measured RTT for the round, while higher values improve accuracy at the cost of more processing 🧠.

The minimum value of CSS_GROWTH_DIVISOR MUST be at least 2 🙅‍♀️. A value of 1 gives the same aggressive behavior as regular slow start. Values larger than 4 make the algorithm less aggressive and maybe less performant 🐢.

Smaller values of CSS_ROUNDS may miss detecting jitter, and larger values may limit performance 🌀.

Packet pacing [ASA00] is a possible mechanism to avoid large bursts and their associated harm 🌊. A paced TCP implementation SHOULD use L = infinity. Pacing mitigates burst concerns, and this setting allows for optimal cwnd growth on modern networks 🚀.

For TCP implementations that pace to mitigate burst concerns, L values smaller than infinity may suffer performance problems due to slow cwnd growth on high-speed networks 😩. For non-paced TCP implementations, L values smaller than 8 may suffer performance problems due to slow cwnd growth on high-speed networks. L values larger than 8 may cause an increase in burstiness, and thereby loss rates, and result in poor performance 💥.

An implementation SHOULD use HyStart++ only for the initial slow start, which is when ssthresh is at its initial value of arbitrarily high per [RFC5681], and then fall back to standard slow start for the rest of the connection lifetime 🫶. This is fine because subsequent slow starts will use the discovered ssthresh value to exit slow start and avoid the overshoot problem 👑. An implementation MAY use HyStart++ to grow the restart window [RFC5681] after a long idle period 🌸.

In application-limited scenarios, the amount of data in flight could fall below the bandwidth-delay product (BDP) and result in smaller RTT samples, which can trigger an exit back to slow start 🥱. It is expected that a connection might oscillate between CSS and slow start in such scenarios. But this behavior will neither result in a connection prematurely entering congestion avoidance nor cause overshooting compared to slow start ✅.

---

5. Deployments and Performance Evaluations 🌈

At the time of writing, HyStart++ as described in this document has been default enabled for all TCP connections in the Windows operating system for over two years, with pacing disabled and an actual L = 8 🪟✨. That's a LOT of real-world mileage, bestie.

In lab measurements with Windows TCP, HyStart++ shows goodput improvements, plus reductions in packet loss and retransmissions, compared to standard slow start 📈. For example, across a variety of tests on a 100 Mbps link with a bottleneck buffer size of bandwidth-delay product, HyStart++ reduces bytes retransmitted by 50% and retransmission timeouts (RTOs) by 36% 😭🔥.

In an A/B test where they compared an implementation of HyStart++ (based on an earlier draft version of this document) to standard slow start across a large Windows device population, out of 52 billion TCP connections, 0.7% of connections moved from 1 RTO to 0 RTOs, and another 0.7% of connections moved from 2 RTOs to 1 RTO with HyStart++ 🤯. This test did not focus on send-heavy connections, and the impact on send-heavy connections is likely much higher 👀. They plan to run more production experiments to gather more data in the future 🧪.

---

6. Security Considerations 🚨

HyStart++ enhances slow start and inherits the general security considerations discussed in [RFC5681]. Read those too, they're important!

An attacker can cause HyStart++ to exit slow start prematurely and impair the performance of a TCP connection by, for example, dropping data packets or their acknowledgments 🕵️‍♀️💀. This is a real threat, not a hypothetical one, so keep it in mind.

The ACK division attack outlined in [SCWA99] does not affect HyStart++. The reason is that the congestion window increase in HyStart++ is based on the number of bytes newly acknowledged in each arriving ACK, rather than on a particular constant applied to each arriving ACK 🛡️🫶.

---

7. IANA Considerations 📭

This document has no IANA actions. Nothing to register, no registries to update, zero paperwork 💅.

---

8. References 📚

8.1. Normative References

These are the must-read ones 🤓:

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
           Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
           <https://www.rfc-editor.org/info/rfc5681>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

8.2. Informative References

These are the deeper-dive reads, the ones I'd save for a rainy day 🌧️📖:

[ASA00]    Aggarwal, A., Savage, S., and T. Anderson, "Understanding
           the performance of TCP pacing", Proceedings IEEE INFOCOM
           2000, DOI 10.1109/INFCOM.2000.832483, March 2000,
           <https://doi.org/10.1109/INFCOM.2000.832483>.

[HyStart]  Ha, S. and I. Rhee, "Taming the elephants: New TCP slow
           start", Computer Networks vol. 55, no. 9, pp. 2092-2110,
           DOI 10.1016/j.comnet.2011.01.014, June 2011,
           <https://doi.org/10.1016/j.comnet.2011.01.014>.

[RFC1122]  Braden, R., Ed., "Requirements for Internet Hosts -
           Communication Layers", STD 3, RFC 1122,
           DOI 10.17487/RFC1122, October 1989,
           <https://www.rfc-editor.org/info/rfc1122>.

[RFC1191]  Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
           DOI 10.17487/RFC1191, November 1990,
           <https://www.rfc-editor.org/info/rfc1191>.

[RFC4821]  Mathis, M. and J. Heffner, "Packetization Layer Path MTU
           Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007,
           <https://www.rfc-editor.org/info/rfc4821>.

[RFC9002]  Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection
           and Congestion Control", RFC 9002, DOI 10.17487/RFC9002,
           May 2021, <https://www.rfc-editor.org/info/rfc9002>.

[RFC9260]  Stewart, R., Tüxen, M., and K. Nielsen, "Stream Control
           Transmission Protocol", RFC 9260, DOI 10.17487/RFC9260,
           June 2022, <https://www.rfc-editor.org/info/rfc9260>.

[SCWA99]   Savage, S., Cardwell, N., Wetherall, D., and T. Anderson,
           "TCP congestion control with a misbehaving receiver", ACM
           SIGCOMM Computer Communication Review, vol. 29, issue 5,
           pp. 71-78, DOI 10.1145/505696.505704, October 1999,
           <https://doi.org/10.1145/505696.505704>.

---

Acknowledgments 🫶

Huge thank-you energy to everyone who helped out during the discussions of this work on the TCPM mailing list and in working group meetings 💜. Helpful comments, critiques, and reviews came in from (listed alphabetically by last name): Mark Allman, Bob Briscoe, Neal Cardwell, Yuchung Cheng, Junho Choi, Martin Duke, Reese Enghardt, Christian Huitema, Ilpo Järvinen, Yoshifumi Nishida, Randall Stewart, and Michael Tüxen 🌟.

---

Authors' Addresses 🖋️

Praveen Balasubramanian
Confluent
899 West Evelyn Ave
Mountain View, CA 94041
United States of America
Email: pravb.ietf@gmail.com


Yi Huang
Microsoft
One Microsoft Way
Redmond, WA 98052
United States of America
Phone: +1 425 703 0447
Email: huanyi@microsoft.com


Matt Olson
Microsoft
One Microsoft Way
Redmond, WA 98052
United States of America
Phone: +1 425 538 8598
Email: maolson@microsoft.com