RFC 7661

Updating TCP to Support Rate-Limited Traffic

📅 published October 2015, from ietf

RFC 7661: Updating TCP to Support Rate-Limited Traffic 🎀

Okay bestie, so RFC 7661 came out in October 2015 as an Experimental doc (ISSN 2070-1721) from the IETF 💖 It was written by G. Fairhurst, A. Sathiaseelan, and R. Secchi from the University of Aberdeen 🌸 It obsoletes RFC 2861 💅

Abstract 🦋

TCP gets confused when an app only sends sometimes and the app's pace, not the cwnd, is what's holding the sending rate back. This doc fixes that. It's an experimental update to TCP that lets a sender restart quickly after a rate-limited gap ✨ Apps that send rate-limited traffic over TCP are expected to benefit, and the update still gives an appropriate response when congestion is experienced 🫶

This doc also takes a look at the Experimental spec of TCP Congestion Window Validation (CWV) from RFC 2861 and decides that RFC 2861 tackled important issues but never became a widely used solution 😭 So this doc moves RFC 2861 from Experimental to Historic, and it obsoletes RFC 2861 💀

Status of This Memo 📜

This is not an Internet Standards Track spec. It's published for examination, experimental implementation, and evaluation 🧚

This doc defines an Experimental Protocol for the Internet community. It's a product of the Internet Engineering Task Force (IETF) and represents IETF community consensus. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so see Section 2 of RFC 5741 🌈

For the current status, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc7661 💜

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved 🎀

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this doc 📖 Code Components pulled from this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 💅

Table of Contents 🍓

   1. Introduction ....................................................3
      1.1. Implementation of New CWV ..................................5
      1.2. Standards Status of This Document ..........................5
   2. Reviewing Experience with TCP-CWV ...............................5
   3. Terminology .....................................................7
   4. A New Congestion Window Validation Method .......................8
      4.1. Initialisation .............................................8
      4.2. Estimating the Validated Capacity Supported by a Path ......8
      4.3. Preserving cwnd during a Rate-Limited Period ..............10
      4.4. TCP Congestion Control during the Non-validated Phase .....11
           4.4.1. Response to Congestion in the Non-validated Phase ..12
           4.4.2. Sender Burst Control during the
                  Non-validated Phase ................................14
           4.4.3. Adjustment at the End of the Non-validated
                  Period (NVP) .......................................14
      4.5. Examples of Implementation ................................15
           4.5.1. Implementing the pipeACK Measurement ...............15
           4.5.2. Measurement of the NVP and pipeACK Samples .........16
           4.5.3. Implementing Detection of the cwnd-Limited
                  Condition ..........................................17
   5. Determining a Safe Period to Preserve cwnd .....................17
   6. Security Considerations ........................................18
   7. References .....................................................18
      7.1. Normative References ......................................18
      7.2. Informative References ....................................19
   Acknowledgments ...................................................21
   Authors' Addresses ................................................21

1. Introduction ✨

TCP carries traffic from all kinds of apps, each with its own behaviour 🌈 The TCP congestion window (cwnd) caps how many unacknowledged packets/bytes a TCP flow can have in the network at any moment. That number is called the FlightSize [RFC5681]. FlightSize is basically how much data is unacknowledged at a specific time. A bulk app always has data ready to send, so its rate is capped by the maximum allowed by the receiver advertised window and the sender congestion window (cwnd). A bulk flow's FlightSize grows with the cwnd and tracks how much data got acknowledged in the last Round-Trip Time (RTT) 🧚

A rate-limited app is different. It has stretches where the sender is idle or can't send at the max rate the cwnd allows 🙈 So the volume sent (FlightSize) can swing a lot from one RTT to the next and can be way less than the cwnd. That means the FlightSize could end up significantly above the recently used capacity. This update targets TCP in those rate-limited cases 🦋

Standard TCP says a TCP sender SHOULD set cwnd to no more than the Restart Window (RW) before it starts transmitting, if it hasn't sent data in an interval longer than the retransmission timeout. In other words, when an app goes idle [RFC5681]. [RFC2861] points out that implementations didn't always do this, and experiments confirm it's still the case (see [Bis08]) 👀

Congestion Window Validation (CWV) [RFC2861] introduced the term "application-limited period" for the time when the sender transmits less than the congestion or receiver windows allow 💅 [RFC2861] also described a method that better supports apps that vary their transmission rate, meaning apps with (short) idle gaps between transmissions or apps that change how fast they send. These apps usually have a TCP FlightSize that's below the cwnd. Plenty of internet apps act like this, including web browsing, HTTP-based adaptive streaming, query/response protocols, network file sharing, and live video 🎀 Lots of them skip long-lived (persistent) TCP connections. For example, servers using HTTP/1.1 [RFC7230] usually support persistent HTTP connections but don't turn them on by default. Instead, these apps often either open a string of short TCP transfers or use UDP 🫶

Standard TCP doesn't add extra limits on cwnd growth when a sender can't reach the max rate the cwnd allows 😭 So a rate-limited sender can grow its cwnd far past what its current transmit rate supports. That leaves a value that doesn't reflect the current state of the network path the flow is using. Using an invalid cwnd like that can hurt app performance and/or significantly add to network congestion 💀

[RFC2861] proposed a fix for these issues in an experimental method called CWV ✨ CWV was meant to help cut down on cases where TCP built up an invalid (inappropriately large) cwnd. The use and drawbacks of the CWV algorithm described in RFC 2861 with an app are covered in Section 2 📖

Section 3 defines the relevant terminology 🍓

Section 4 specifies an alternative to CWV. It tackles the same issues but is expected to soften the impact on apps that vary their sending rate 🌸 The updated method applies to rate-limited conditions, which covers both application-limited and idle senders.

The goals of this update are:

  • To not change the behaviour of a TCP sender that performs bulk transfers that fully use the cwnd.
  • To provide a method that co-exists with standard TCP and other flows that use this updated method.
  • To reduce transfer latency for applications that change their rate over short intervals of time.
  • To avoid a TCP sender growing a large "non-validated" cwnd when it has not recently sent using this cwnd.
  • To remove the incentive for ad hoc application or network stack methods (such as "padding") solely to maintain a large cwnd for future transmission.
  • To provide an incentive for the use of long-lived connections rather than a succession of short-lived flows, benefiting both the long-lived flows and other flows sharing capacity with these flows when congestion is encountered. 🏆

Section 5 describes the rationale for selecting the safe period to preserve the cwnd ⭐

1.1. Implementation of New CWV

The method specified in Section 4 of this document is a sender-side-only change to the TCP congestion control behaviour of TCP 🔥

The method creates a new protocol state and requires a sender to determine when the cwnd is validated or non-validated, so it can control entry to and exit from this state (see Section 4.3). It defines how a TCP sender manages cwnd growth using the set of rules in Section 4 🧚

Implementing this specification requires an implementor to define a method to measure available capacity using a set of pipeACK samples. The details of this measurement are implementation-specific. An example is provided in Section 4.5.1, but other methods are permitted. A sender also needs a method to determine when it becomes cwnd-limited, and implementing that may require considering other TCP methods (see Section 4.5.3) 🌈

A sender is also recommended to provide a method that controls the maximum burst size (see Section 4.4.2). Implementors get flexibility in how this is built, and the right choice is expected to depend on how the sender stack implements other TCP methods, such as TCP Segment Offload (TSO) 💖

1.2. Standards Status of This Document

This document obsoletes the methods described in [RFC2861]. It recommends a set of mechanisms, including the use of pacing during a non-validated period 🦋 The updated mechanisms are intended to have a less aggressive congestion impact than a standard TCP sender would show 🫶

The spec in this document is classified as "Experimental" pending experience with deployed implementations of the methods 🎀

2. Reviewing Experience with TCP-CWV ✨

okay so [RFC2861] came up with a simple tweak to TCP congestion control 🎀 once a connection hit a "sufficiently-long" idle period, the cwnd got decayed. It used the slow-start threshold (ssthresh) to remember the previous value of the congestion window. This relaxed the standard TCP behaviour for an idle session [RFC5681], and the whole goal was to boost application performance 🌸. CWV also changed what happened when a sender transmitted at a rate lower than what cwnd allowed 🦋.

[RFC2861] had two sets of responses: one after an "application-limited period" and one after an "idle period". The split was argued about, but in practice telling those two conditions apart was a mess on real networks (see, e.g., [Bis10]) 💀. It gave predictable performance for long on-off periods (>>1 RTT) or slowly varying rate-based traffic. For variable-rate traffic, though, it could be totally unpredictable, depending on whether an accurate RTT had been obtained and on how the application's traffic pattern lined up with the measured RTT 🙈.

Lots of apps vary their transmission over a huge range of rates 📈. Using [RFC2861], those apps often got wildly varying performance, so app devs couldn't predict TCP latency even on a path with stable characteristics 😭. We argue that trying to classify app behaviour as application-limited or idle is problematic and also inappropriate 🙅. So this document explicitly avoids differentiating those two cases and treats all rate-limited traffic uniformly ✨.

[RFC2861] has been implemented in some mainstream operating systems as the default behaviour [Bis08]. Analysis (e.g., [Bis10] and [Fai12]) shows that a TCP sender using CWV can use available capacity on a shared path after an idle period 👑. That can help variable-rate apps, especially over long delay paths, compared to the slow-start restart in standard TCP. Still, CWV would only benefit an app if the idle period were less than several Retransmission Timeout (RTO) intervals [RFC6298]. Otherwise it would behave the same as standard TCP, which resets cwnd to the TCP Restart Window after this period 😤.

To get better performance for variable-rate apps with TCP, some operating systems have gone with non-standard methods, and apps have resorted to "padding" streams by sending dummy data to keep their sending rate up when they have nothing to send 🫠. Sending redundant data does give good evidence that the path can sustain the offered rate, but padding also eats network capacity and reduces the opportunity for congestion-free statistical multiplexing 🙃. For variable-rate flows, the benefits of statistical multiplexing can be significant, so finding a viable alternative to padding streams is a goal 🎯.

Experience with [RFC2861] suggests that although CWV benefited the network in a rate-limited scenario (reducing the probability of network congestion), it was too conservative for many common rate-limited applications 🐢. So it didn't deliver the app performance boost rate-limited apps needed, and it's unclear whether apps actually use this mechanism on the general Internet 🤷.

So it was concluded that CWV, as defined in [RFC2861], was often a poor solution for many rate-limited applications. It had the right motivation but the wrong approach 💔.

3. Terminology 💖

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

The document assumes familiarity with the terminology of TCP congestion control [RFC5681].

The following additional terminology is introduced in this document:

  • cwnd-limited: A TCP flow that has sent the maximum number of segments permitted by the cwnd, where the application utilises the allowed sending rate (see Section 4.5.3).
  • pipeACK sample: A measure of the volume of data acknowledged by the network within an RTT.
  • pipeACK variable: A variable that measures the available capacity using the set of pipeACK samples (see Section 4.2).
  • pipeACK Sampling Period: The maximum period that a measured pipeACK sample may influence the pipeACK variable.
  • Non-validated phase: The phase where the cwnd reflects a previous measurement of the available path capacity.
  • Non-validated period (NVP): The maximum period for which cwnd is preserved in the non-validated phase.
  • Rate-limited: A TCP flow that does not consume more than one half of cwnd and hence operates in the non-validated phase. This includes periods when an application is either idle or chooses to send at a rate less than the maximum permitted by the cwnd.
  • Validated phase: The phase where the cwnd reflects a current estimate of the available path capacity.

4. A New Congestion Window Validation Method 🦋

This section proposes an update to TCP congestion control behaviour during a rate-limited interval 🌈. This new method intentionally does not differentiate between times when the sender has become idle and times when it chooses to send at a rate less than the maximum allowed by the cwnd.

In the non-validated phase, the capacity used by an app can be less than what the TCP cwnd allows. This update lets an app preserve a recently used cwnd while in the non-validated phase, then resume transmitting at a previous rate without eating the delay of slow-start ⏩. However, if the TCP sender experiences congestion while using the preserved cwnd, it is required to immediately reset the cwnd to an appropriate value specified by the method 🚨. If a sender doesn't take advantage of the preserved cwnd within the non-validated period (NVP), the value of cwnd is reduced, so it better reflects the capacity that was actually used recently 📉.

It is expected that this update will satisfy the requirements of many rate-limited applications and at the same time provide an appropriate method for use in the Internet 💅. New CWV reduces the incentive for an application to send "padding" data just to keep transport congestion state 🙌.

The method is specified in the following subsections and is expected to encourage applications and TCP stacks to use standards-based congestion control methods 📜. It may also encourage the use of long-lived connections where this offers benefit (such as persistent HTTP) 🌸.

4.1. Initialisation

A sender starts a TCP connection in the validated phase and initialises the pipeACK variable to the "undefined" value. This value inhibits use of the value in cwnd calculations.

4.2. Estimating the Validated Capacity Supported by a Path

[RFC6675] defines "FlightSize", a variable that indicates the instantaneous amount of data that has been sent but not cumulatively acknowledged. In this method, a new variable "pipeACK" is introduced to measure the acknowledged size of the network pipe. This is used to determine if the sender has validated the cwnd. pipeACK differs from FlightSize in that it is evaluated over a window of acknowledged data, rather than reflecting the amount of data outstanding.

A sender determines a pipeACK sample by measuring the volume of data that was acknowledged by the network over the period of a measured Round-Trip Time (RTT). Using the variables defined in [RFC6675], a value could be measured by caching the value of HighACK and, after one RTT, measuring the difference between the cached HighACK value and the current HighACK value. A sender MAY count TCP DupACKs that acknowledge new data when collecting the pipeACK sample. Other equivalent methods may be used.

A sender is not required to continuously update the pipeACK variable after each received ACK, but SHOULD perform a pipeACK sample at least once per RTT when it has sent unacknowledged segments.

The pipeACK variable MAY consider multiple pipeACK samples over the pipeACK Sampling Period. The value of the pipeACK variable MUST NOT exceed the maximum (highest value) within the pipeACK Sampling Period. This specification defines the pipeACK Sampling Period as Max(3*RTT, 1 second). This period enables a sender to compensate for large fluctuations in the sending rate, where there may be pauses in transmission, and allows the pipeACK variable to reflect the largest recently measured pipeACK sample.

When no measurements are available (e.g., a sender that has just started transmission or immediately after loss recovery), the pipeACK variable is set to the "undefined value". This value is used to inhibit entering the non-validated phase until the first new measurement of a pipeACK sample. (Section 4.5 provides examples of implementation.)

The pipeACK variable MUST NOT be updated during TCP Fast Recovery. That is, the sender stops collecting pipeACK samples during loss recovery. The method RECOMMENDS enabling the TCP SACK option [RFC2018] and RECOMMENDS the method defined in [RFC6675] to recover missing segments. This allows the sender to more accurately determine the number of missing bytes during the loss recovery phase, and using this method will result in a more appropriate cwnd following loss.

Note: The use of pipeACK rather than FlightSize can change the behaviour of a TCP flow when a sender does not always have data available to send. One example arises when there is a pause in transmission after sending a sequence of many packets, and the sender experiences loss at or near the end of its transmission sequence. In this case, the TCP flow may have used a significant amount of capacity just prior to the loss (which would be reflected in the volume of data acknowledged, recorded in the pipeACK variable), but at the actual time of loss, the number of unacknowledged packets in flight (at the end of the sequence) may be small, i.e., there is a small FlightSize. After loss recovery, the sender resets its congestion control state.

[Fai12] explored the benefits of different responses to congestion for application-limited streams. If the response is based only on the Loss FlightSize, the sender would assign a small cwnd and ssthresh, based only on the volume of data sent after the loss. When the sender next starts to transmit, it can incur many RTTs of delay in slow-start before it reacquires its previous rate. When the pipeACK value is also used to calculate the cwnd and ssthresh (as specified in Section 4.4.1), the sender can use a value that also reflects the recently used capacity before the loss. This prevents a variable-rate application from being unduly penalised. When the sender resumes, it starts at one-half its previous rate, similar to the behaviour of a bulk TCP flow [Hos15]. To ensure an appropriate reaction to ongoing congestion, this method requires that the pipeACK variable is reset after it is used in this way.

4.3. Preserving cwnd during a Rate-Limited Period ✨

okay so the updated method adds a brand new TCP sender phase 💅 that keeps track of whether cwnd is a validated or non-validated value. There are two phases:

  • Validated phase: pipeACK >=(1/2)*cwnd, or pipeACK is undefined (aka right at the start or directly after loss recovery). This is the normal phase 🌸, where cwnd is basically a good guess of how much capacity is free on the network path right now, and the standard methods are used to grow cwnd (currently those are described in [RFC5681]).
  • Non-validated phase: pipeACK <(1/2)*cwnd. This is the phase where cwnd is based on an older measurement of available capacity, and that capacity hasn't been checked during the pipeACK Sampling Period 🫣. Basically, we don't know if cwnd still matches what's actually available on the path. The mechanisms in this phase are all about finding a safe value for cwnd and reacting properly to congestion 🛡️.

Note: A threshold is needed to figure out whether a sender is in the validated or non-validated phase 🧐. A standard TCP sender in slow-start is allowed to double its FlightSize from one RTT to the next, and that's where the threshold value of 1/2 comes from. This threshold keeps a sender from growing cwnd any further as long as FlightSize is less than (1/2*cwnd). On top of that, a sender with a FlightSize under (1/2*cwnd) could, in the next RTT, send at a rate that more than doubles its FlightSize, thanks to cwnd. So this case has to be treated as non-validated 🙃, and the sender needs extra mechanisms while it's in this phase.

4.4. TCP Congestion Control during the Non-validated Phase 💖

A TCP sender implementing this specification MUST enter the non-validated phase when pipeACK is less than (1/2)*cwnd. (The note at the end of Section 4.4.1 explains why pipeACK<=(1/2)*cwnd is expected to be a safe value 🧠.)

A TCP sender in the non-validated phase keeps cwnd preserved 🔒. Basically, cwnd only grows after the sender fully uses cwnd during this phase. Otherwise, cwnd neither grows nor shrinks. The phase ends when the sender transmits enough data that pipeACK > (1/2)*cwnd (so it's no longer rate-limited 🙌), or when the sender gets an indication of congestion.

After a fixed period of time, called the non-validated period (NVP) ⏳, the sender adjusts cwnd (Section 4.4.3). The NVP SHOULD NOT exceed five minutes. Section 5 talks about why a safe value was picked for this period.

Here's how the non-validated phase behaves 👑:

  • A sender decides whether to increase cwnd based on whether it is cwnd-limited (see Section 4.5.3):
    • A sender that is cwnd-limited MAY use the standard TCP method to increase cwnd. (That means a TCP sender that fully uses cwnd can increase it each time it gets an ACK 📈.)
    • A sender that is not cwnd-limited MUST NOT increase cwnd when ACK packets come in during this phase. It needs to avoid growing cwnd when it hasn't recently sent using the current size of cwnd 🚫.
  • If the sender gets an indication of congestion while in the non-validated phase (aka it detects loss 😬), the sender MUST exit the non-validated phase and reduce cwnd as defined in Section 4.4.1.
  • If the Retransmission Timeout (RTO) expires while in the non-validated phase, the sender MUST exit the non-validated phase. After that, it goes back to using the standard TCP RTO mechanism [RFC5681].
  • A sender whose pipeACK variable is greater than (1/2)*cwnd SHOULD enter the validated phase 💅. (A rate-limited sender usually won't be affected by whether it's in the validated or non-validated phase, since it normally won't increase FlightSize to use the whole cwnd. But switching to the validated phase does free the sender from the limits on cwnd growth and means it uses the standard congestion response.)

The cwnd-limited behaviour can kick in during a short-lived moment 🌈. This happens when a sender is in the non-validated phase, gets an ACK that acknowledges received data, has fully used cwnd, and still has more data waiting to go than the current cwnd allows. In that case, the sender MAY use the standard method to increase cwnd. (Also, if the sender actually manages to send these new segments, the updated cwnd and pipeACK values will eventually move it into the validated phase ✨.)

4.4.1. Response to Congestion in the Non-validated Phase 😭

Getting congestion feedback while in the non-validated phase tells us it was a bad idea for the sender to use the preserved cwnd 🙈. So the sender is required to quickly lower its rate to avoid more congestion. Since cwnd doesn't have a validated value anymore, a new cwnd has to be picked based on the rate that was actually being used 🧮.

A sender that detects a packet drop MUST record the current FlightSize in the variable LossFlightSize, and it MUST calculate a safe cwnd for loss recovery with this method:

cwnd = (Max(pipeACK,LossFlightSize))/2.

pipeACK doesn't get updated during loss recovery (see Section 4.2). If there's a valid pipeACK value, the new cwnd gets adjusted to account for the fact that a non-validated cwnd can be bigger than the actual FlightSize or the recently used FlightSize (which is recorded in pipeACK) 🫧. So the updated cwnd stops a sender from overshooting and suddenly blasting way more traffic during the recovery period 🚀.

At the end of the recovery phase, the TCP sender MUST reset cwnd with this method:

cwnd = (Max(pipeACK,LossFlightSize) - R)/2.

Here, R is the amount of data that was successfully retransmitted during the recovery phase. This covers segments that were retransmitted and counted as lost by the pipe estimation algorithm at the end of recovery 🔍. It doesn't include the extra cost of sending the same data more than once. Losing segments means the path capacity got exceeded by at least R, so the calculated cwnd gets cut by at least R before the window is halved 💥.

The calculated cwnd value MUST NOT be reduced below 1 TCP Maximum Segment Size (MSS) 🛑.

After finishing the loss recovery phase, the sender MUST re-initialise the pipeACK variable to the "undefined" value 🧼. This makes sure standard TCP methods are used right after loss recovery until a new pipeACK value can be figured out.

ssthresh is adjusted with the standard TCP method (Step 6 in Section 3.2 of RFC 5681 sets ssthresh equal to cwnd at the end of loss recovery) 🎯.

Note: Reducing cwnd by R, the volume of data not sent, follows the method proposed for Jump Start [Liu07] 🚀. The R term makes the adjustment more conservative than standard TCP. That's needed because a sender in the non-validated phase is allowed a higher rate than a standard TCP sender would have reached in the last RTT (aka it could have more than doubled the number of segments in flight compared to the previous RTT 😳). The extra reduction after congestion helps when LossFlightSize has overshot the available path capacity by a lot and caused big losses. That can happen after a change in path characteristics, or when additional traffic takes a bigger share of the network bottleneck while the sender is sending less 🌧️.

Note: The pipeACK value is only valid during a non-validated phase, so this doesn't exceed cwnd/2 🙌. If LossFlightSize and R were small, the final cwnd after loss recovery could end up being at most one-quarter of the cwnd measured when congestion was detected. That reduction is conservative 🫶, and pipeACK then gets reset to undefined. So cwnd updates after a congestion event don't depend on the pipeACK history from before congestion was detected 🧹.

4.4.2. Sender Burst Control during the Non-validated Phase 🌊

TCP congestion control lets a sender build up a cwnd that would allow it to send a burst of segments with a total size up to the difference between FlightSize and cwnd 💨. Bursts like that can hurt other flows sharing the same network bottleneck, and they can cause congestion when buffering is limited 😵‍💫.

People have proposed various ways to control sender burstiness [Hug01] [All05]. For example, TCP can cap the number of new segments it sends per received ACK. That works well when a flow of ACKs is coming in, but it can't control a sender that hasn't sent much data in the previous RTT [All05] 🤷‍♀️.

This document recommends a method to avoid line-rate bursts after an idle or rate-limited interval, especially when there's less reliable information about the network path's capacity 🧁. A TCP sender in the non-validated phase SHOULD control its maximum burst size. For example, it can use a rate-based pacing algorithm that spreads out cwnd over its RTT estimate, or some other method, so that a lot of segments don't get sent back-to-back at line-rate 🚦. Which methods work best for pacing depends on the design of the TCP/IP stack, the speed of the interface, and whether hardware support (like TSO) is being used 🛠️. This document doesn't recommend any specific method.

4.4.3. Adjustment at the End of the Non-validated Period (NVP) ⏰

An application that stays in the non-validated phase for longer than the NVP is required to adjust its congestion control state 🧾. If the sender leaves the non-validated phase after this period, it MUST update ssthresh:

ssthresh = max(ssthresh, 3*cwnd/4).

(This ssthresh adjustment makes sure the sender records that it has safely kept up the current rate 💪. It helps rate-limited flows that run into the occasional congestion, which would otherwise add some unwanted delay before they recover their sending rate 🐢.)

The sender MUST then update cwnd so it's not greater than:

cwnd = max((1/2)*cwnd, IW).

IW is the appropriate TCP initial window that the TCP sender uses (see, e.g., [RFC5681]) 🪟.

Note: These cwnd and ssthresh adjustments put the sender into slow-start, since ssthresh > cwnd 🐣. This makes the sender respond conservatively after staying in the non-validated phase for longer than the non-validated period. In this case, cwnd gets cut in half from its preserved value. This is handy when flows build up cwnd but don't actually use a big one, because it softens the impact when those flows start transmitting again 🌙. For instance, it could help if several high-rate application flows went idle for a long time and then all woke up at once because of some external event ⏰🔔.

4.5. Examples of Implementation 🎀

This section has informative examples of implementation methods 📚. Implementations can use other methods too, as long as they follow the normative requirements 💅.

4.5.1. Implementing the pipeACK Measurement

okay so a pipeACK sample may be measured once each RTT 💖. This cuts down on the sender's processing work for calculating after every single acknowledgment, and it also means the sender doesn't need as much storage 🧚

Since application behaviour can be super bursty when you're using CWV, it may be desirable to add a maximum filter that collects the measured values, so the pipeACK variable records the largest pipeACK sample within the pipeACK Sampling Period 👑. One simple way to do this is to split the pipeACK Sampling Period into several (e.g., five) equal-length measurement periods 🎀. The sender records the start time of each measurement period along with the highest measured pipeACK sample. At the end of a measurement period, any measurement(s) older than the pipeACK Sampling Period get tossed out 🗑️. Then the pipeACK variable is assigned the largest of the set of highest measured values ✨

pipeACK sample (Bytes)
^
|   +----------+----------+           +----------+---......
|   | Sample A | Sample B | No        | Sample C | Sample D
|   |          |          | Sample    |          |
|   | |\ 5     |          |           |          |
|   | | |      |          |           |  /\ 4    |
|   | | |      |  |\ 3    |           |  | \     |
|   | | \      | |  \---  |           |  /  \    |   /| 2
|   |/   \------|       - |           | /    \------/ \...
+//-+----------+---------\+----/ /----+/---------+-------------> Time

 <------------------------------------------------|
                     Sampling Period          Current Time

           Figure 1: Example of Measuring pipeACK Samples

Figure 1 shows an example of how measurement samples may be collected 📊. At the time shown in the figure, new samples are being piled into Sample D. Three previous samples also fall inside the pipeACK Sampling Period: A, B, and C. There was also a stretch of inactivity between Samples B and C where no measurements were taken (because no new data segments got acknowledged) 😴. The current value of the pipeACK variable will be 5, the maximum across all samples. During this stretch, the pipeACK samples may be regarded as zero, so they don't contribute to the calculated pipeACK value 🙈

After one more measurement period, Sample A gets discarded, because it's then older than the pipeACK Sampling Period, and the pipeACK variable gets recalculated 🔄. Its value will be the larger of Sample C or the final value accumulated in Sample D 💅

4.5.2. Measurement of the NVP and pipeACK Samples

The mechanism needs a bunch of time measurements ⏱️. These measurements could be implemented using protocol timers, but they don't necessarily need a brand new timer to be built 🙌 Skipping dedicated timers can save operating system resources, especially when there may be large numbers of TCP flows, and that's a big deal 💜

The NVP could be measured by recording a timestamp when the sender enters the non-validated phase 🕰️. Every time a sender transmits a new segment, that timestamp can be used to check whether the NVP has expired. If the measured period goes past the NVP, the sender can then count how many units of the NVP have passed and make one reduction (defined in Section 4.4.3) for each NVP 🦋

Similarly, the time measurements for collecting pipeACK samples and figuring out the pipeACK Sampling Period could be derived by using a timestamp to record when each sample was measured, then using that to calculate how much time has passed when each new ACK comes in 🌈

4.5.3. Implementing Detection of the cwnd-Limited Condition

A sender needs to implement a method that detects the cwnd-limited condition (see Section 4.4) 👀. This detects a situation where a sender in the non-validated phase receives an ACK, but the size of cwnd stops it from sending any more new data 🚫

In simple terms, this condition is true only when the FlightSize of a TCP sender is equal to or larger than the current cwnd 💖. But an implementation also has to consider constraints on how the cwnd variable can be used. For instance, implementations need to support other TCP methods, like the Nagle Algorithm and TCP Segment Offload (TSO), that also use cwnd to control transmission 🧩. These other methods can make a sender become cwnd-limited when cwnd is nearly, rather than completely, equal to the FlightSize 🤏

5. Determining a Safe Period to Preserve cwnd ✨

This section documents the reasoning behind picking the maximum period that cwnd may be preserved, known as the NVP 🧠

Limiting how long cwnd can be preserved avoids undesirable side effects that would happen if cwnd were kept unnecessarily high for an arbitrarily long time, which was part of the problem CWV originally tried to fix 💅. The period a sender may safely preserve cwnd depends on how long a network path is expected to keep up the capacity that cwnd reflects. And honestly? There is no ideal choice for this time 🤷‍♀️

A period of five minutes was chosen for this NVP 🎀. It's a compromise: it's longer than the idle intervals of common applications, but not sufficiently longer than the period over which the capacity of an Internet path may commonly be regarded as stable ⚖️. The capacity of wired networks is usually pretty stable for periods of several minutes, and that load stability goes up with the capacity 📈. This suggests cwnd may be preserved for at least a few minutes 💜

There are cases where TCP throughput shows significant variability over a time span less than five minutes 😬. Examples could include wireless topologies, where TCP rate variations may fluctuate on the order of a few seconds because of medium access protocol instabilities 📡. Mobility changes may also impact TCP performance over short time scales 🏃‍♀️. Senders that observe these rapid changes in the path characteristics may also experience increased congestion with the new method. However, that kind of variation would likely also impact TCP's behaviour when supporting interactive and bulk applications anyway 🫶

Routing algorithms may change the network path that a transport uses 🛣️. Although a change of path can disrupt the RTT measurement and may result in a change of the capacity available to a TCP connection, we assume these path changes do not usually happen frequently (compared to a time frame of a few minutes) 🙏

So the value of five minutes is expected to be sufficient for most current applications 🌸. Simulation studies (e.g., [Bis11]) also suggest that for many practical applications, the performance with this value won't be significantly different from what's observed using a non-standard method that doesn't reset the cwnd after idle 🧪

Finally, other TCP sender mechanisms have used a five-minute timer, and some implementations could be simplified by reusing the same interval 🧚. TCP defines a default user timeout of five minutes [RFC793], which is how long transmitted data may remain unacknowledged before a connection is forcefully closed ⏳

6. Security Considerations 🔐

General security considerations concerning TCP congestion control are discussed in [RFC5681] 🛡️. This document describes an algorithm that updates one aspect of the congestion control procedures, so the considerations described in [RFC5681] also apply to this algorithm 💀

7. References 📚

7.1. Normative References

7.2. Informative References

  • [All05] Allman, M. and E. Blanton, "Notes on Burst Mitigation for Transport Protocols", ACM SIGCOMM Computer Communication Review, Volume 35, Issue 2, DOI 10.1145/1064413.1064419, April 2005.
  • [Bis08] Biswas, I. and G. Fairhurst, "A Practical Evaluation of Congestion Window Validation Behaviour", 9th Annual Postgraduate Symposium in the Convergence of Telecommunications, Networking and Broadcasting (PGNet), Liverpool, UK, 2008.
  • [Bis10] Biswas, I., Sathiaseelan, A., Secchi, R., and G. Fairhurst, "Analysing TCP for Bursty Traffic", Int'l J. of Communications, Network and System Sciences, DOI 10.4236/ijcns.2010.37078, July 2010.
  • [Bis11] Biswas, I., "Internet Congestion Control for Variable-Rate TCP Traffic", PhD Thesis, School of Engineering, University of Aberdeen, 2011.
  • [Fai12] Sathiaseelan, A., Secchi, R., Fairhurst, G., and I. Biswas, "Enhancing TCP Performance to support Variable-Rate Traffic", 2nd Capacity Sharing Workshop, ACM CoNEXT, Nice, France, December 2012.
  • [Hos15] Hossain, Z., "A Study of Mechanisms to Support Variable-Rate Internet Applications over a Multi-service Satellite Platform", PhD Thesis, School of Engineering, University of Aberdeen, January 2015.
  • [Hug01] Hughes, A., Touch, J., and J. Heidemann, "Issues in TCP Slow-Start Restart After Idle", Work in Progress, draft-hughes-restart-00, December 2001.
  • [Liu07] Liu, D., Allman, M., Jin, S., and L. Wang, "Congestion Control without a Startup Phase", 5th International Workshop on Protocols for Fast Long-Distance Networks (PFLDnet), Los Angeles, California, February 2007.
  • [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing", RFC 7230, DOI 10.17487/RFC7230, June 2014, <http://www.rfc-editor.org/info/rfc7230>.

Acknowledgments 🙏✨

This doc was cooked by the TCP Maintenance and Minor Extensions (tcpm) working group 🧁

The authors want to give a huge shoutout to Dr. I. Biswas and Dr. Ziaul Hossain 🫶 They helped evaluate CWV and were a massive help in developing the mechanisms in this doc. We also got comments from the Internet Congestion Control Research Group, especially Yuchung Cheng, Mirja Kuehlewind, Joe Touch, and Mark Allman. Iconic squad 👑 This work was partly funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700) 💜

Authors' Addresses 🦋

Meet the authors! They're all based at the same place, so you know they're in the same friend group 🌸

Godred Fairhurst

University of Aberdeen
School of Engineering
Fraser Noble Building
Aberdeen, Scotland  AB24 3UE
United Kingdom

Email: gorry@erg.abdn.ac.uk
URI:   http://www.erg.abdn.ac.uk

Arjuna Sathiaseelan

University of Aberdeen
School of Engineering
Fraser Noble Building
Aberdeen, Scotland  AB24 3UE
United Kingdom

Email: arjuna@erg.abdn.ac.uk
URI:   http://www.erg.abdn.ac.uk

Raffaello Secchi

University of Aberdeen
School of Engineering
Fraser Noble Building
Aberdeen, Scotland  AB24 3UE
United Kingdom

Email: raffaello@erg.abdn.ac.uk
URI:   http://www.erg.abdn.ac.uk

🎀 That's a wrap on RFC 7661! Thanks for reading 🎀

📦 where this page comes from