RFC 6928

Increasing TCP's Initial Window

📅 published April 2013, from ietf

Increasing TCP's Initial Window 🎀

okay bestie, this is RFC 6928 💖 It's a Request for Comments from the Internet Engineering Task Force (IETF), and its category is Experimental 🧪 (ISSN 2070-1721). It came out in April 2013 ✨ The authors are J. Chu, N. Dukkipati, Y. Cheng, and M. Mathis, all from Google, Inc. 🌈 The title block doesn't list anything it obsoletes or updates, so it's a clean slate 🫶

Abstract 🦋

okay so this doc proposes an experiment to bump the permitted TCP initial window (IW) from somewhere between 2 and 4 segments (as specified in RFC 3390) up to 10 segments 🔥 It has a fallback to the existing recommendation whenever performance issues show up 🛟 It explains why we'd want the increase, the pros and cons of a bigger initial window, and shares results from several large-scale experiments 📊 The vibe is that the higher initial window improves the overall performance of lots of web services without causing a congestion collapse 💅 The doc wraps up by talking about usage and deployment for more experiments, as recommended by the IETF TCP Maintenance and Minor Extensions (TCPM) working group 🌸

Status of This Memo 📜

this one is not an Internet Standards Track spec 🙅‍♀️ It's published so people can examine it, try out experimental implementations, and evaluate it 🧪

it defines an Experimental Protocol for the Internet community 🌍 It's a product of the IETF and represents the IETF community's consensus 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) ✅ Not every IESG-approved document is a candidate for any level of Internet Standard, so check out Section 2 of RFC 5741 if you want the details 🤓

for the latest status, errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6928 🔗

Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication 📚 Please read these carefully, because they spell out your rights and restrictions for this document 🙈 Code Components pulled out of this doc must include the Simplified BSD License text described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 🍓

Table of Contents 🗂️

   1. Introduction ....................................................3
      1.1. Terminology ................................................4
   2. TCP Modification ................................................4
   3. Implementation Issues ...........................................5
   4. Background ......................................................6
   5. Advantages of Larger Initial Windows ............................7
      5.1. Reducing Latency ...........................................7
      5.2. Keeping Up with the Growth of Web Object Size ..............8
      5.3. Recovering Faster from Loss on Under-Utilized or
           Wireless Links .............................................8
   6. Disadvantages of Larger Initial Windows for the Individual ......9
   7. Disadvantages of Larger Initial Windows for the Network ........10
   8. Mitigation of Negative Impact ..................................11
   9. Interactions with the Retransmission Timer .....................11
   10. Experimental Results From Large-Scale Cluster Tests ...........11
      10.1. The Benefits .............................................11
      10.2. The Cost .................................................12
   11. Other Studies .................................................13
   12. Usage and Deployment Recommendations ..........................14
   13. Related Proposals .............................................15
   14. Security Considerations .......................................16
   15. Conclusion ....................................................16
   16. Acknowledgments ...............................................16
   17. References ....................................................16
      17.1. Normative References .....................................16
      17.2. Informative References ...................................17
   Appendix A. List of Concerns and Corresponding Test Results .......21

1. Introduction ✨

okay so this doc proposes raising the upper bound on TCP's initial window (IW) to 10 segments, with a maximum of 14600 B 🎀 It's patterned after RFC 3390 [RFC3390] and borrows heavily from that doc and earlier work in this area 🫶 There are lingering concerns about possible side effects on other flows sharing the same network bottleneck, so some of the recommendations are conditional on extra monitoring and evaluation 👀

the main argument for raising IW comes from how the Internet keeps scaling up 📈 Ten segments are likely to fit into the queue space available at any broadband access link, even when there's a reasonable number of concurrent connections 🌐

lower speed links can get environment-specific configurations, so they're protected from getting overwhelmed by big initial window bursts without giving the rest of the Internet a suboptimal initial window 🐢💜

this doc reviews the pros and cons of a larger initial window and summarises several large-scale experiments showing that an initial window of 10 segments (IW10) gives benefits across the board for a variety of bandwidth (BW), round-trip time (RTT), and bandwidth-delay product (BDP) classes 🤩 The benefits are big for users at much smaller data rates than anyone had expected 😭 Past 10, though, the results are mixed 🤔 We think those mixed results aren't intrinsic, but come from various implementation artifacts, including overly aggressive applications that open many simultaneous connections 🙄

we recommend that all TCP implementations have a settable TCP IW parameter, as long as there's a reasonable effort to monitor for possible interactions with other Internet applications and services, as described in Section 12 🛠️ Also, Section 10 explains why 10 segments may be an appropriate value 🍓 That value may keep rising in the future, but this doc doesn't include any supporting evidence for values of IW larger than 10 🙈

we also introduce a minor revision to RFC 3390 and RFC 5681 [RFC5681] to stop resetting the initial window when the SYN or SYN/ACK is lost 🔄

the doc wraps up with the TCPM working group's consensus on near-term usage and deployment of IW10 on the Internet 🦋

a companion set of slides for this proposal is at [CD10] 🖼️

1.1. Terminology 🔤

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be interpreted as described in RFC 2119 [RFC2119] 💅

2. TCP Modification 🛠️

okay here's the core change 💖 This doc proposes an increase in the permitted upper bound for TCP's initial window (IW) to 10 segments, depending on the maximum segment size (MSS) 🌸 This increase is optional: a TCP MAY start with an initial window smaller than 10 segments ✨

more precisely, the upper bound for the initial window is:

      min (10*MSS, max (2*MSS, 14600))                            (1)

this upper bound is a change from RFC 3390 [RFC3390], which said the congestion window should be initialized between 2 and 4 segments, depending on the MSS 🔁

this change applies to the initial window of the connection during the first round-trip time (RTT) of data transmission, during or after the TCP three-way handshake ⏱️ Neither the SYN/ACK nor its ACK in the three-way handshake should increase the initial window size 🙅‍♀️

quick note: all the test results in this doc were based on the regular Ethernet MTU of 1500 bytes 📏 Future study of a different MTU may be needed to fully validate (1) above 🧐

also, RFC 3390 says (and RFC 5681 [RFC5681] has similar text):

If the SYN or SYN/ACK is lost, the initial window used by a sender after a correctly transmitted SYN MUST be one segment consisting of MSS bytes.

so here's the tea 🫖 The proposed change to reduce the default retransmission timeout (RTO) to 1 second [RFC6298] raises the chance of spurious SYN or SYN/ACK retransmission 😬 That means connections with RTT > 1 second get unnecessarily penalized if their initial window drops to 1 segment 💀 For this reason, it is RECOMMENDED that implementations refrain from resetting the initial window to 1 segment, unless there have been more than one SYN or SYN/ACK retransmissions or true loss detection has been made 🫶

TCP implementations use slow start in as many as three different ways 🌈 (1) to start a new connection (the initial window); (2) to restart transmission after a long idle period (the restart window); and (3) to restart transmission after a retransmit timeout (the loss window) 🔁 This change affects the value of the initial window ✨ Optionally, a TCP MAY set the restart window to the minimum of the initial window value and the current value of cwnd (in other words, a larger restart window value should never increase the size of cwnd) 🎀 These changes do NOT change the loss window, which must remain 1 segment of MSS bytes, to permit the lowest possible window size in severe congestion 🚨

last bit, and it's important 😤 To limit any negative effect a larger initial window may have on links with limited bandwidth or buffer space, implementations SHOULD fall back to RFC 3390 for the restart window (RW) if any packet loss is detected during either the initial window or a restart window, and more than 4 KB of data is sent 🛟 Implementations must also follow RFC 6298 [RFC6298] to avoid spurious RTO, as described in Section 9 🔥

3. Implementation Issues ✨

okay so the HTTP 1.1 spec only allows two simultaneous connections per domain 💅, but browsers open way more than that [Ste08]. Part of the reason is to get around the tiny initial window so pages load faster, like we talked about above 🌸

when browsers open a bunch of simultaneous TCP connections to the same destination, they're basically working against TCP's congestion control mechanisms [FF99] 😭 Pair that with bigger initial windows and it makes the bursts spikier and less fair to everyone else on the network. If a larger initial window hurts other flows, local app tuning will show that having fewer concurrent connections gives some users better performance 🙈 So if you're a content provider running IW10 with your content spread across multiple domains, you're explicitly encouraged to run measurement experiments to catch these problems, and to think about reducing how many concurrent connections you use to fetch your content. Main character energy means checking your impact 🎀

some implementations advertise a small initial receive window (Table 2 in [Duk10]), which effectively limits how much window a remote host is allowed to use 🧚 To actually get the full benefit of the large initial window, implementations are encouraged to advertise an initial receive window of at least 10 segments, except in cases where a larger initial window is deemed harmful. (See Section 8 below.)

the TCP Selective Acknowledgment (SACK) option [RFC2018] was thought to be required for the larger initial window to perform well. But measurements from both a testbed and live tests showed that IW=10 without the SACK option outperforms IW=3 with the SACK option [CW10] 🔥 the myth is busted, bestie 💜

4. Background 🌈

the TCP congestion window was introduced as part of the congestion control algorithm by Van Jacobson in 1988 [Jac88] 👑 The initial value of one segment was used as the starting point for newly established connections to probe the available bandwidth on the network.

today's Internet is dominated by web traffic running on top of short-lived TCP connections [IOR2009] 🍓 The relatively small initial window has become a limiting factor for the performance of many web applications 😩

the global Internet keeps growing, both in speed and in how many people use it ✨ According to the latest report from Akamai [AKAM10], global broadband (> 2 Mbps) adoption has surpassed 50%, pushing the average connection speed to 1.7 Mbps, while narrowband (< 256 Kbps) usage has dropped to 5%. Meanwhile, TCP's initial window has been stuck at 4 KB for a decade [RFC2414], which works out to less than 200 Kbps of bandwidth utilization per connection, assuming an RTT of 200 ms. Not it 💀

a large chunk of flows on the Internet are short web transactions over TCP, and they finish before they even exit TCP slow start 🦋 Speeding up the flow startup phase, including getting around the initial window limit, has been an active area of research (see [Sch08] and Section 3.4 of [RFC6077]). There are numerous proposals [LAJW07] [RFC4782] [PRAKS02] [PK98]. Some of them require router support [RFC4782] [PK98], so they aren't practical for the public Internet. Others suggested bold, but often radical, ideas that would likely need more years of research before standardization and deployment 🙈

in the meantime, applications have been responding to TCP's "slow" start 🐢 Websites use multiple subdomains [Bel10] to get around the HTTP 1.1 limit of two connections per physical host [RFC2616]. Right now, major web browsers open multiple connections to the same site (up to six connections per domain [Ste08], and the number is growing). This trend is a way to fix HTTP's serialized download, so you get parallelism and higher performance. But it also means that most access links today are severely under-utilized, so having multiple TCP connections improves performance most of the time. While raising the initial congestion window may cause congestion for certain users of these browsers, we argue that browsers and other applications need to respect HTTP 1.1 regulation and stop increasing the number of simultaneous TCP connections 🫶 We believe a modest increase of the initial window will help stop this trend, and it's the best interim solution to improve overall user performance and reduce server, client, and network load.

note that persistent connections and pipelining are designed to address some of the above issues with HTTP [RFC2616] 💅 Their presence doesn't diminish the need for a larger initial window. For example, data from the Chrome browser shows that 35% of HTTP requests are made on new TCP connections. Our test data also shows significant latency reduction with the large initial window, even in conjunction with these two HTTP features [Duk10] ✨

also, packet pacing has been suggested as a possible mechanism to avoid large bursts and their associated harm [VH97] 🌸 Pacing is not required in this proposal, because we strongly prefer a simple solution. We suspect that for packet bursts of a moderate size, packet pacing won't be necessary, and our test results seem to confirm this 🎀

more discussion about the increase in initial window, including the choice of 10 segments, can be found in [Duk10] and [CD10] 📚

5. Advantages of Larger Initial Windows 🦋

5.1 Reducing Latency

an increase of the initial window from 3 segments to 10 segments reduces the total transfer time for data sets greater than 4 KB by up to 4 round trips 🚀

the table below compares the number of round trips between IW=3 and IW=10 for different transfer sizes, assuming infinite bandwidth, no packet loss, and the standard delayed ACKs with large delayed-ACK timer 📊

            ---------------------------------------
           | total segments |   IW=3   |   IW=10   |
            ---------------------------------------
           |         3      |     1    |      1    |
           |         6      |     2    |      1    |
           |        10      |     3    |      1    |
           |        12      |     3    |      2    |
           |        21      |     4    |      2    |
           |        25      |     5    |      2    |
           |        33      |     5    |      3    |
           |        46      |     6    |      3    |
           |        51      |     6    |      4    |
           |        78      |     7    |      4    |
           |        79      |     8    |      4    |
           |       120      |     8    |      5    |
           |       127      |     9    |      5    |
            ---------------------------------------

for example, with the larger initial window, a transfer of 32 segments only needs 2 round trips to complete instead of 5 🤯

5.2. Keeping Up with the Growth of Web Object Size

RFC 3390 said the main motivation for increasing the initial window to 4 KB was to speed up connections that only transmit a small amount of data, like email and web 📧 Back then, most transfers were less than 4 KB and could finish in a single RTT [All00].

but since RFC 3390 was published, web objects have gotten way bigger [Chu09] [RJ10] 😭 Today, only a small percentage of web objects (e.g., 10% of Google's search responses) can fit in the 4 KB initial window. The average HTTP response size of gmail.com, a super scripted website, is 8 KB (Figure 1 in [Duk10]). The average web page, including all static and dynamic scripted web objects on the page, has grown even more [RJ10] 📈 HTTP pipelining [RFC2616] and newer web transport protocols like SPDY [SPDY] let multiple web objects get sent in a single transaction. That could potentially benefit from an even larger initial window, so an entire web page can transfer in a small number of round trips 🧚

a greater-than-3-segment initial window increases the chance of recovering from packet loss through Fast Retransmit rather than the lengthy initial RTO [RFC5681] 💖 That's because the fast retransmit algorithm needs three duplicate ACKs as an indication that a segment has been lost rather than reordered. Newer loss recovery techniques like Limited Transmit [RFC3042] and Early Retransmit [RFC5827] have been proposed to help speed up loss recovery from a smaller window. Both of those algorithms can still benefit from the larger initial window, though, because there's a better chance to receive more ACKs 🌈

6. Disadvantages of Larger Initial Windows for the Individual Connection 🥺

the larger bursts from an increase in the initial window may cause buffer overrun and packet drop in routers with small buffers, or in routers that are already experiencing congestion 😬 This could result in unnecessary retransmit timeouts. For a large-window connection that can recover without a retransmit timeout, this could cause an unnecessarily early transition from slow start to the congestion-avoidance phase of the window increase algorithm.

premature segment drops are unlikely to happen in uncongested networks with sufficient buffering, or in moderately congested networks where the congested router uses active queue management (such as Random Early Detection [FJ93] [RFC2309] [RFC3150]) 🛡️

insufficient buffering is more likely to exist in the access routers connecting slower links ⛓️ A recent study of access router buffer size [DGHS07] shows that the majority of access routers provision enough buffer for 130 ms or longer. That's enough to cover a burst of more than 10 packets at 1 Mbps speed, but possibly not enough for browsers opening simultaneous connections 🫠

a testbed study [CW10] on the effect of the larger initial window with five simultaneously opened connections revealed that, even with limited buffer size on slow links, IW=10 still reduced the total latency of web transactions. The tradeoff was higher packet drop rates compared to IW=3 📉

some TCP connections will get better performance with the larger initial window, even if the burstiness of the initial window results in premature segment drops ✨ This will be true if (1) the TCP connection recovers from the segment drop without a retransmit timeout, and (2) the TCP connection is ultimately limited to a small congestion window, either by network congestion or by the receiver's advertised window 💜

7. Disadvantages of Larger Initial Windows for the Network ✨

okay so a bigger initial window can make a network more congested 🙈 but here's the tea: the bump only happens ONCE, right at the start of a connection. TCP's usual congestion backoff stuff is still doing its thing, so it's unlikely that this change alone pushes a network into being stuck in congestion, or even full-on congestion collapse 💀 The large-scale web experiments we'll get to later seem to back this up, which is so reassuring 🫶

It should be noted that the above may not hold if applications open a large number of simultaneous connections.

Until this proposal is widely deployed, there may be a fairness issue between flows using a larger initial window and flows that follow RFC 3390 🎀 So far no severe unfairness has shown up in any of the known tests, but more study on this could still be worth it 🔍

Some of the discussions from RFC 3390 are still valid for IW=10 💅

Also, TCP NewReno does make it more likely to get duplicate segments when it's trying to recover from multiple packet losses out of a big window 😬 Good news though: the TCP Selective Acknowledgment (SACK) option [RFC2018] is widely supported across all the major OSes today, and that should keep the number of duplicate segments in check 🦋

Recent measurements [Get11] found extremely large queues (on the order of one second or more) in access networks across the Internet 😱 A big chunk of this buffer bloat comes from large downloads and uploads, like video files, emails with huge attachments, backups, and downloading movies to disk. But some of it is also caused by web browsing of image-heavy sites [Get11]. This queuing delay is generally seen as harmful for latency-sensitive traffic like DNS queries, Address Resolution Protocol (ARP), DHCP, Voice over IP (VoIP), and gaming 🎮 IW=10 can make this worse when doing short downloads, like web browsing [Get11-1]. The fixes proposed for the bigger buffer bloat problem also apply here, like Explicit Congestion Notification (ECN), Active Queue Management (AQM) schemes [CoDel], and traffic classification (QoS) 🌈

8. Mitigation of Negative Impact 🛡️

Most of the negative impact from a bigger initial window is probably going to be felt by users behind slow links with limited buffers 🐢 The fix here is for hosts that are directly connected to a low-speed link to advertise an initial receive window smaller than 10 segments 🍓 Users can make this happen through manual configuration, or the host stack can auto-detect low-bandwidth links and do it for them ✨

Additional suggestions for improving the end-to-end performance of slow links can be found in RFC 3150 [RFC3150] 📚

9. Interactions with the Retransmission Timer ⏰

A large initial window raises the chance of spurious RTO on a low-bandwidth path, because the packet transmission time ends up dominating the round-trip time 😵‍💫 To keep spurious retransmissions down, implementations MUST follow RFC 6298 [RFC6298] and restart the retransmission timer with the current value of RTO for each ACK received that acknowledges new data 💜

For a more detailed discussion, see RFC 3390, Section 6 🧐

10. Experimental Results From Large-Scale Cluster Tests 🧪

In this section we're spilling the results from large-scale Internet experiments using an initial window of 10 segments 🔥 These ran through Google's front-end infrastructure, which serves a super diverse set of applications. We're looking at two data centers, each picked for the specific kinds of subnets they serve. AvgDC has connection bandwidths closer to the worldwide average reported in [AKAM10], with a median connection speed of about 1.7 Mbps 📊 SlowDC has way more traffic from slow-bandwidth subnets, with nearly 20% of its traffic coming from connections below 100 Kbps; and a third was below 256 Kbps 🐌

Guided by the data, we're answering two big questions: what's the latency benefit when TCP connections start with a higher initial window, and, on the flip side, what does it cost? 🤔

10.1. The Benefits 💖

The average web search latency improvement across all responses is 11.7% (68 ms) in AvgDC and 8.7% (72 ms) in SlowDC 🎉 We also dug into the data by traffic characteristics and subnet properties like bandwidth (BW), round-trip time (RTT), and bandwidth-delay product (BDP). Average response latency improved across the board for a variety of subnets. The biggest wins, over 20%, came from high RTT and high BDP networks, where most responses can fit within the pipe 🚀 Responses from low RTT paths saw the smallest improvements, about 5% 🌱

Here's the surprising part: responses from low-bandwidth subnets got the best latency improvements, between 10-20% in the 0-56 Kbps and 56-256 Kbps buckets 😳 We have a couple of guesses for why low-BW networks see improved latency: 1) there are fewer slow-start rounds, because unlike many large-BW networks, low-BW subnets with dial-up modems have inherently large RTTs; and 2) loss recovery is faster, because an initial window larger than 3 segments makes it more likely that a lost packet gets recovered through Fast Retransmit instead of a lengthy RTO 🧠

Responses of different sizes benefited to different degrees. Those larger than 3 segments naturally showed bigger improvements, since they finished in fewer rounds in slow start compared to the baseline 🏃‍♀️ In our experiments, response sizes less than or equal to 3 segments also showed small latency benefits 🍬

To see how individual subnets did, we analyzed average latency at the /24 subnet level, which approximates a user base that gets a similar set of services from a common ISP 🏘️ Even at that granularity, latency improved at all quantiles, ranging from 5-11% 🙌

10.2. The Cost 💸

To figure out the cost of raising the initial window, we looked specifically at subnets with low bandwidth and BDP, retransmission rates for different kinds of applications, and latency for applications running multiple concurrent TCP connections 🔎 From our measurements we found no evidence of negative latency impacts that correlate to BW or BDP alone. In fact, both kinds of subnets showed latency improvements across averages and quantiles 💅

As expected, the retransmission rate went up modestly with the larger initial congestion window 📈 The overall increase is 0.3% in AvgDC (from 1.98% to 2.29%) and 0.7% in SlowDC (from 3.54% to 4.21%) 🥺 In our investigation, with one exception, the larger window increased retransmissions by less than 0.5% for services in AvgDC. The exception is the Maps application, which runs multiple concurrent TCP connections. Its retransmission rate went up by 0.9% in AvgDC and 1.85% in SlowDC (from 3.94% to 5.79%) 😬

In our experiments, the percentage of traffic experiencing retransmissions did not increase significantly. For example, 90% of web search and maps traffic in SlowDC had zero retransmissions (the percentages are even higher for AvgDC) 🙌 A breakdown of retransmissions by percentiles shows that most of the increase comes from the portion of traffic that already experienced retransmissions in the baseline with an initial window of 3 segments 🧩

One of the worst-case scenarios, where latency can get hit hard by bottleneck buffer overflow, is traffic from applications using multiple concurrent TCP connections, all operating with a large initial window 🌪️ Our investigation shows this pattern hasn't been a problem in AvgDC. There, all these applications, specifically Maps and image thumbnails, showed improved latencies of 2-20% ✨ SlowDC is a different story. These applications still improved in the mean, but their latencies in higher quantiles (96 and above for Maps) showed cases where the larger window was worse than the baseline. For example, the 99% latency for Maps went up by 2.3% (80 ms) compared to the baseline 😭 Our measurements give no evidence that this latency cost comes from subnet bandwidth alone. We can't know this from our data, but we suspect the amount of buffering at bottleneck links plays a key role in how these applications perform 🤷‍♀️

Further details on our experiments and analysis can be found in [Duk10] and [DCCM10] 📖

11. Other Studies 🔬

Besides the large-scale Internet experiments described above, a bunch of other studies have looked at the effects of IW10 in different environments 🌍 We'll summarize them below, with more discussion in Appendix A.

A complete list of tests, with their results and related studies, can be found at the [IW10] link 🔗

  1. [Sch08] described an earlier evaluation of various Fast Startup approaches, including the "Initial-Start" of 10 MSS.
  2. [DCCM10] presented results from Google's large-scale IW10 experiments, focusing on areas with highly multiplexed links or limited broadband deployment, such as Africa and South America 🌏
  3. [CW10] contained a testbed study on IW10 performance over slow links. It also studied how short flows with a larger initial window might affect the throughput performance of other coexisting, long-lived, bulk data transfers.
  4. [Sch11] compared IW10 against a number of other fast startup schemes and concluded that IW10 works rather well and is also quite fair 🏆
  5. [JNDK10] and later [JNDK10-1] studied the effect of IW10 over cellular networks 📱
  6. [AERG11] studied the effect of larger initial congestion windows, among other things, on end users' page load time from Yahoo!'s Content Delivery Network 💜

12. Usage and Deployment Recommendations 💅

Okay bestie, we are not there yet 🥺. Further experiments are required before a larger initial window shall be enabled by default in the Internet. The existing measurement results say this doesn't cause significant harm to other traffic, which is great news ✨. But widespread use could reveal issues nobody knows about yet, like fairness, or the impact on latency-sensitive traffic such as VoIP 📞.

So special care is needed when you use this experimental TCP extension. This is especially true for large-scale systems that send out a significant amount of Internet traffic, or for huge numbers of consumer-level systems that add up to a similar aggregate impact 🌈. Anyone turning on a larger initial window (stack vendors, network administrators, and so on) SHOULD ensure that performance is monitored before and after the change 📊. The key metrics to watch are the rate of packet losses, ECN marking, and segment retransmissions during the initial burst 👀. The sender SHOULD cache information about connection setups that use an initial window larger than the one RFC 3390 allows, and new connections SHOULD fall back to the initial window allowed by RFC 3390 if there is evidence of performance issues 🧚. More experiments are needed on the design of that cache and the corresponding heuristics 🔬.

Other metrics that might suggest the IW needs to come down include an increased overall percentage of packet loss or segment retransmissions, plus application-level stuff like longer data transfer completion times or impaired media quality 🙈.

It's also important to remember hosts that don't implement a larger initial window 💜. Any IW10 deployment should be aware that there are potential side effects for real-time traffic (like VoIP) 📞. If users notice any significant deterioration of performance, they SHOULD fall back to an initial window allowed by RFC 3390 for safety reasons 🛟. An increased initial window MUST NOT be turned on by default on systems that lack this kind of monitoring capability 🚫.

The IETF TCPM working group is super interested in more reports from experiments with this specification, and they encourage publishing that measurement data 🎀. Right now there are no adequate studies that prove or disprove the impact of IW10 on real-time traffic, so more experimentation in this direction is encouraged 🧪.

If no significant harm gets reported, a follow-up document may revisit whether a larger initial window can be safely used by default in all Internet hosts 🌸. Resolving these experiments and tightening up the suggestions here might be grounds for a future Standards Track document on the same topic 🏆.

It's recognized that if IW10 is causing harm to other traffic, the software on the hosts using IW10 may not readily notice it 😬. In some cases a local system or network administrator may be able to spot this and selectively disable IW10 🕵️. In the general case, though, the harm may happen on a remote network to other cross-traffic, so there may be no good way at all to detect or fix it 💀. Current experience and analysis don't show whether this is a real issue or just a hypothetical one 🤷. As IW10 use becomes more common, monitoring and analyzing flows throughout the network will be needed to assess the impact across the full range of scenarios on the real Internet 🌐.

Two other proposals, [All10] and [Tou12], have been published to raise TCP's initial window size over a long timescale ⏳. Both aim to reduce the uncertain impact of a larger initial window at Internet-wide scale 🌍. Also, [Tou12] looks for an algorithm that automates adjusting IW safely over a long period 🤖.

Even though a modest, static increase of IW to 10 can address the near-term need for better web performance, the TCP research community still has a lot of work to do to find a long-term solution to the TCP flow startup problem 🧠✨.

14. Security Considerations 🔐

This document talks about the initial congestion window allowed for TCP connections 🪟. Changing this value may cause more packet loss, but it's highly unlikely to lead to a persistent state of network congestion or even a congestion collapse 🙏. So it doesn't raise any known new security issues with TCP 💖.

15. Conclusion 👑

This document suggests a simple change to TCP that cuts application latency over short-lived TCP connections or links with long RTTs, saving several RTTs during the initial slow-start phase 🏃‍♀️💨. It has little or no negative impact on other flows, which is iconic ✨. Extensive tests went through both testbeds and large data centers, and most results showed improved latency with only a small increase in the packet retransmission rate 📈. Based on those results, we believe a modest increase of IW to 10 is the best solution for near-term deployment, while scaling IW over the long run is still a challenge for the TCP research community 🫶.

16. Acknowledgments 💌

Many people at Google helped make the large-scale tests possible 🥳. We especially want to thank Amit Agarwal, Tom Herbert, Arvind Jain, and Tiziana Refice for their major contributions 🙌.

17. References 📚

17.1. Normative References

[RFC2018]  Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
           Selective Acknowledgment Options", RFC 2018, October 1996.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2616]  Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
           Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
           Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.

[RFC3390]  Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's
           Initial Window", RFC 3390, October 2002.

[RFC5681]  Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
           Control", RFC 5681, September 2009.

[RFC5827]  Allman, M., Avrachenkov, K., Ayesta, U., Blanton, J., and
           P. Hurtig, "Early Retransmit for TCP and Stream Control
           Transmission Protocol (SCTP)", RFC 5827, May 2010.

[RFC6298]  Paxson, V., Allman, M., Chu, J., and M. Sargent,
           "Computing TCP's Retransmission Timer", RFC 6298, June
           2011.

17.2. Informative References ✨

okay bestie, this is the bibliography 📚🫶 every source that backs up the whole vibe, and I'm obsessed that they're all here. The labels in brackets are how the text points to each one, so don't rename them 🙈

  • [AKAM10] Akamai Technologies, Inc., "The State of the Internet, 3rd Quarter 2009", January 2010, http://www.akamai.com/html/about/press/releases/2010/press_011310_1.html.
  • [AERG11] Al-Fares, M., Elmeleegy, K., Reed, B., and I. Gashinsky, "Overclocking the Yahoo! CDN for Faster Web Page Loads", Internet Measurement Conference, November 2011.
  • [All00] Allman, M., "A Web Server's View of the Transport Layer", ACM Computer Communication Review, 30(5), October 2000.
  • [All10] Allman, M., "Initial Congestion Window Specification", Work in Progress, November 2010.
  • [Bel10] Belshe, M., "A Client-Side Argument For Changing TCP Slow Start", January 2010, http://sites.google.com/a/chromium.org/dev/spdy/An_Argument_For_Changing_TCP_Slow_Start.pdf.
  • [CD10] Chu, J. and N. Dukkipati, "Increasing TCP's Initial Window", presented to the IRTF ICCRG and IETF TCPM working group meetings, IETF 77, March 2010, http://www.ietf.org/proceedings/77/slides/tcpm-4.pdf.
  • [Chu09] Chu, J., "Tuning TCP Parameters for the 21st Century", presented to TCPM working group meeting, IETF 75, July 2009. http://www.ietf.org/proceedings/75/slides/tcpm-1.
  • [CoDel] Nichols, K. and V. Jacobson, "Controlling Queue Delay", ACM QUEUE, May 6, 2012.
  • [CW10] Chu, J. and Wang, Y., "A Testbed Study on IW10 vs IW3", presented to the TCPM working group meeting, IETF 79, November 2010, http://www.ietf.org/proceedings/79/slides/tcpm-0.
  • [DCCM10] Dukkipati, D., Cheng, Y., Chu, J., and M. Mathis, "Increasing TCP initial window", presented to the IRTF ICCRG meeting, IETF 78, July 2010, http://www.ietf.org/proceedings/78/slides/iccrg-3.pdf.
  • [DGHS07] Dischinger, M., Gummadi, K., Haeberlen, A., and S. Saroiu, "Characterizing Residential Broadband Networks", Internet Measurement Conference, October 24-26, 2007.
  • [Duk10] Dukkipati, N., Refice, T., Cheng, Y., Chu, J., Sutin, N., Agarwal, A., Herbert, T., and J. Arvind, "An Argument for Increasing TCP's Initial Congestion Window", ACM SIGCOMM Computer Communications Review, vol. 40 (2010), pp. 27-33. July 2010.
  • [FF99] Floyd, S., and K. Fall, "Promoting the Use of End-to-End Congestion Control in the Internet", IEEE/ACM Transactions on Networking, August 1999.
  • [FJ93] Floyd, S. and V. Jacobson, "Random Early Detection gateways for Congestion Avoidance", IEEE/ACM Transactions on Networking, V.1 N.4, August 1993, p. 397-413.
  • [Get11] Gettys, J., "Bufferbloat: Dark buffers in the Internet", presented to the TSV Area meeting, IETF 80, March 2011, http://www.ietf.org/proceedings/80/slides/tsvarea-1.pdf.
  • [Get11-1] Gettys, J., "IW10 Considered Harmful", Work in Progress, August 2011.
  • [IOR2009] Labovitz, C., Iekel-Johnson, S., McPherson, D., Oberheide, J. Jahanian, F., and M. Karir, "Atlas Internet Observatory 2009 Annual Report", 47th NANOG Conference, October 2009.
  • [IW10] "TCP IW10 links", January 2012, http://code.google.com/speed/protocols/tcpm-IW10.html.
  • [Jac88] Jacobson, V., "Congestion Avoidance and Control", Computer Communication Review, vol. 18, no. 4, pp. 314-329, Aug. 1988.
  • [JNDK10] Jarvinen, I., Nyrhinen. A., Ding, A., and M. Kojo, "A Simulation Study on Increasing TCP's IW", presented to the IRTF ICCRG meeting, IETF 78, July 2010, http://www.ietf.org/proceedings/78/slides/iccrg-7.pdf.
  • [JNDK10-1] Jarvinen, I., Nyrhinen. A., Ding, A., and M. Kojo, "Effect of IW and Initial RTO changes", presented to the TCPM working group meeting, IETF 79, November 2010, http://www.ietf.org/proceedings/79/slides/tcpm-1.pdf.
  • [LAJW07] Liu, D., Allman, M., Jin, S., and L. Wang, "Congestion Control Without a Startup Phase", Protocols for Fast, Long Distance Networks (PFLDnet) Workshop, February 2007, http://www.icir.org/mallman/papers/jumpstart-pfldnet07.pdf.
  • [PK98] Padmanabhan V.N. and R. Katz, "TCP Fast Start: A technique for speeding up web transfers", in Proceedings of IEEE Globecom '98 Internet Mini-Conference, 1998.
  • [PRAKS02] Partridge, C., Rockwell, D., Allman, M., Krishnan, R., and J. Sterbenz, "A Swifter Start for TCP", Technical Report No. 8339, BBN Technologies, March 2002.
  • [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, S., Wroclawski, J., and L. Zhang, "Recommendations on Queue Management and Congestion Avoidance in the Internet", RFC 2309, April 1998.
  • [RFC2414] Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's Initial Window", RFC 2414, September 1998.
  • [RFC3042] Allman, M., Balakrishnan, H., and S. Floyd, "Enhancing TCP's Loss Recovery Using Limited Transmit", RFC 3042, January 2001.
  • [RFC3150] Dawkins, S., Montenegro, G., Kojo, M., and V. Magret, "End-to-end Performance Implications of Slow Links", BCP 48, RFC 3150, July 2001.
  • [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick-Start for TCP and IP", RFC 4782, January 2007.
  • [RFC6077] Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B. Briscoe, "Open Research Issues in Internet Congestion Control", RFC 6077, February 2011.
  • [RJ10] Ramachandran, S. and A. Jain, "Aggregate Statistics of Size Related Metrics of Web Pages metrics", May 2010, http://code.google.com/speed/articles/web-metrics.html.
  • [Sch08] Scharf, M., "Quick-Start, Jump-Start, and Other Fast Startup Approaches", presented to the IRTF ICCRG meeting, IETF 73, November 2008, http://www.ietf.org/proceedings/73/slides/iccrg-2.pdf.
  • [Sch11] Scharf, M., "Performance and Fairness Evaluation of IW10 and Other Fast Startup Schemes", presented to the IRTF ICCRG meeting, IETF 80, March 2011, http://www.ietf.org/proceedings/80/slides/iccrg-1.pdf.
  • [Sch11-1] Scharf, M., "Comparison of end-to-end and network-supported fast startup congestion control schemes", Computer Networks, Feb. 2011, http://dx.doi.org/10.1016/j.comnet.2011.02.002.
  • [SPDY] "SPDY: An experimental protocol for a faster web", http://dev.chromium.org/spdy.
  • [Ste08] Sounders S., "Roundup on Parallel Connections", High Performance Web Sites blog, March 2008, http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections.
  • [Tou12] Touch, J., "Automating the Initial Window in TCP", Work in Progress, July 2012.
  • [VH97] Visweswaraiah, V. and J. Heidemann, "Improving Restart of Idle TCP Connections", Technical Report 97-661, University of Southern California, November 1997.

Appendix A. List of Concerns and Corresponding Test Results ✨

Okay so this is the final part, bestie 🎀 The authors had a whole list of worries that popped up after the first draft went out, and they went and tested every single one. Let's go through them together 💅

Concerns have been raised since the initial draft of this document was posted, based on a set of large-scale experiments. To better understand the impact of a larger initial window, and in order to confirm or dismiss these concerns, additional tests have been conducted using either large-scale clusters, simulations, or real testbeds. The following attempts to compile the list of concerns and summarize findings from relevant tests. 🔥

  • How complete are various tests in covering many different traffic patterns? 🧚

Google's large-scale Internet experiments on its front-end infrastructure covered a big portfolio of services beyond web search. That includes Gmail, Google Maps, Photos, News, Sites, Images, etc., across a wide variety of traffic sizes and patterns 🌈. One notable exception is YouTube, because the authors don't think the large initial window will have much material impact, either positive or negative, on bulk data services 🦋.

[CW10] contains some results from a testbed study on how short flows with a larger initial window might affect the throughput performance of other coexisting, long-lived, bulk data transfers ✨.

  • Larger bursts from the increase in the initial window cause significantly more packet drops. 💀

All the tests conducted on this subject ([Duk10] [Sch11] [Sch11-1] [CW10]) so far have shown only a modest increase of packet drops 🌸. The only exception is from the testbed study [CW10] under extremely high load and/or simultaneous opens. But under those conditions, both IW=3 and IW=10 suffered very high packet loss rates 😭.

  • A large initial window may severely impact TCP performance over highly multiplexed links still common in developing regions. 🌍

Google's large-scale experiments described in Section 10 above also covered Africa and South America 🍓. Measurement data from those regions [DCCM10] revealed improved latency, even for those services that employ multiple simultaneous connections, at the cost of a small increase in the retransmission rate 💜. It seems that the round-trip savings from a larger initial window more than make up the time spent on recovering more lost packets 👑.

Similar phenomena have also been observed from the testbed study [CW10] 🫶.

  • Why 10 segments? 🤔

Questions have been raised on how the number 10 was picked, ngl that's a fair question 🙈. The authors tried different sizes in their large-scale experiments and found that 10 segments seem to give most of the benefits for the services they tested, while not causing a significant increase in the retransmission rates ⭐. Going forward, 10 segments may turn out to be too small when the average of web object sizes continues to grow 😬. But a scheme to "right size" the initial window automatically over long timescales has yet to be developed, so that's still open, periodt 💅.

  • More thorough analysis of the impact on slow links is needed. 🐢

Although [Duk10] showed the large initial window reduced the average latency even for the dialup link class of only 56 Kbps in bandwidth, more studies were needed to understand the effect of IW10 on slow links at the microscopic level 🔍. [CW10] was conducted for this purpose ✨.

Testbeds in [CW10] emulated a 300 ms RTT, bottleneck link bandwidth as low as 64 Kbps, and route queue size as low as 40 packets 🧪. A large combination of test parameters were used. Almost all tests showed varying degrees of latency improvement from IW=10, with only a modest increase in the packet drop rate until a very high load was injected 💖. The testbed result was consistent with both the large-scale data center experiments [CD10] [DCCM10] and a separate study using the Network Simulation Cradle (NSC) framework [Sch11] [Sch11-1] 🎀.

  • How will the larger initial window affect flows with initial windows of 4 KB or less? 🤷‍♀️

Flows with the larger initial window will likely grab more bandwidth from a bottleneck link when competing against flows with smaller initial windows, at least initially. How long will this "unfairness" last? Will there be any "capture effect" where flows with larger initial window possess a disproportional share of bandwidth beyond just a few round trips? 👀

If there is any "unfairness" issue from flows with different initial windows, it did not show up in the large-scale experiments 🙌. The average latency for the bucket of all responses less than 4 KB did not seem to be affected by the presence of many other larger responses employing large initial window. As a matter of fact, they seemed to benefit from the large initial window too, as shown in Figure 7 of [Duk10] 🎉.

The same phenomenon seems to exist in the testbed experiments [CW10]. Flows with IW=3 only suffered slightly when competing against flows with IW=10 in light to medium loads. Under high load, both flows' latency improved when mixed together 🌟. Also, long-lived, background bulk-data flows seemed to enjoy higher throughput when running against many foreground short flows of IW=10 than against short flows of IW=3. One plausible explanation was that IW=10 enabled short flows to complete sooner, leaving more room for the long-lived, background flows 🧚‍♀️.

A study using an NSC simulator has also concluded that IW=10 works rather well and is quite fair against IW=3 [Sch11] [Sch11-1] 💅.

  • How will a larger initial window perform over cellular networks? 📱

Some simulation studies [JNDK10] [JNDK10-1] have been conducted to study the effect of a larger initial window on wireless links from 2G to 4G networks (EGDE/HSPA/LTE). The overall result seems mixed in both raw performance and the fairness index 🥺.

Authors' Addresses 🦋

Jerry Chu Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 USA

EMail: hkchu@google.com

Nandita Dukkipati Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 USA

EMail: nanditad@google.com

Yuchung Cheng Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 USA

EMail: ycheng@google.com

Matt Mathis Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043 USA

EMail: mattmathis@google.com

📦 where this page comes from