RFC 1106
TCP big window and NAK options
RFC 1106: TCP Big Window and Nak Options ๐
Bestie, this one is a vibe ๐ It's from June 1989 and it's all about how TCP can handle really big, really long-distance connections without wasting bandwidth. Here's the title block, all in one place:
- RFC number: 1106
- Title: TCP Big Window and Nak Options
- Author: R. Fox, Tandem
- Working group: Network Working Group
- Date: June 1989
- Category: Experimental
- Obsoletes / Updates: none listed in this header
Status of This Memo โจ
Okay so this memo is about two extensions to TCP that make things run more efficiently over a network with a high bandwidth*delay product ๐ The extensions were actually implemented and tested using resources at NASA, so they're not just theory.
This is an Experimental Protocol ๐งช It is not being proposed as an Internet standard. It's more of a starting point for further research, which I'm honestly obsessed with. Distribution of this memo is unlimited ๐
Abstract ๐ฆ
Two extensions to TCP are described in this RFC so that it works more efficiently over networks with a high bandwidth*delay product. The big thing that still needs solving is congestion versus noise ๐ญ This memo touches on that issue, but more research is still needed to see whether these extensions work across the Internet as a whole and not just on high bandwidth*delay networks.
The document does show how to use these options in an isolated satellite network. The goal is to use that special medium more efficiently and take bulk data transfers off the links that people need for interactive use. Main character energy for satellites ๐ฐ๏ธโจ
1. Introduction ๐ธ
Recent work on TCP has given great performance gains over a bunch of different network paths [1]. Still, these changes don't work well on paths with a large round trip delay (like a satellite with a 600 ms round trip delay) or with a very large bandwidth (like a transcontinental DS3 line) ๐ฉ Both of these networks have a higher bandwidth*delay product, over 10**6 bits, while TCP is currently limited to 10**5 bits.
So what's bandwidth*delay anyway? It's the amount of data that can be unacknowledged at once so that all of the network's bandwidth is being used by TCP. People also call this "filling the pipe" [2] ๐ชฃ When the pipe is full, the sender can always put data onto the network, the receiver always has something to read, and neither end has to wait around for the other one. It's giving efficiency ๐
After the last batch of algorithm improvements to TCP, performance over high bandwidth*delay networks is still really bad ๐ It looks like no algorithm changes alone will make a meaningful difference on these networks. They need an extension to the protocol itself. This RFC discusses two possible TCP options for that purpose ๐
The two options implemented and discussed in this RFC are:
- NAKs ๐
โโ๏ธ This extension lets the receiver tell the sender that a packet of data didn't arrive and needs to be resent. It's useful on any network path, whether it has a high or low
bandwidth*delayproduct, as long as the path has periodic errors like lost packets, noisy links, or packets dropped because of congestion. The information from this option is advisory. If the sender ignores it, it has no effect on TCP whatsoever ๐ซ - Big Windows ๐ช This option expands the current 16 bit (64 Kbytes) TCP window to 32 bits. Of those 32 bits, 30 bits (over 1 gigabytes) are allowed for the receive window. (The maximum window size in TCP exists because TCP has to be able to tell old data from new data. For a good explanation, see [2].) No changes are needed to the standard TCP header [6] ๐ The 16 bit field in the TCP header that carries the receive window stays exactly the same. The 32 bit receive window comes from an option that holds the upper half of the window. This option is what's needed to fill big data pipes, like a satellite link ๐ฐ๏ธ
Here's the roadmap, bestie ๐บ๏ธ Section 2 goes into more detail on the NAK option, and Section 3 goes into more detail on the big window option. Section 4 covers the other effects of using the big windows and NAK features together. It includes a short discussion of how congestion versus noise affects TCP, plus some options for satellite networks. Section 5 is the conclusion, with some hints about what future development might happen at NASA. After that there's an appendix with some test results ๐
2. NAK Option ๐ โโ๏ธ
Any packet loss on a high bandwidth*delay network hits throughput really hard, and it's all because of TCP's simple acknowledgement scheme ๐ญ TCP always acks the stream of data that has been received successfully, and it tells the sender the next byte it expects in the stream. If a packet gets lost but the packets after it arrive, the current protocol has no way to tell the sender "hey, I missed one, but I got the ones after it."
So TCP resends all of the data over again, either after a timeout or after the sender suspects a loss because of the duplicate ack algorithm [1]. It keeps doing this until the receiver gets the lost packet and can ack it along with the packets that came after it. On a normal low bandwidth*delay network, this isn't a big deal as long as the timeout period is set short enough ๐
On a long delay network, like a T1 satellite channel, it's a disaster ๐ By the time the lost packet gets resent and the ack comes back, the TCP window has been used up. The sender and receiver both end up stalled, waiting for the packet and the ack to fully travel through the data pipe. The pipe empties out, and the sender has to refill it after the ack comes back.
This causes a minimum of 3*X bandwidth loss, where X is the one way delay of the medium. It can be a lot higher depending on how big the timeout period is and what the bandwidth*delay product is. The 3*X breaks down like this: 1X for the packet to be resent, 1X for the ack to be received, and 1X for the next packet to reach the destination โจ This math assumes the window size is much smaller than the pipe size (window = 1/2 data pipe or 1X), which is the normal case with today's TCP window limit on long delay networks like a T1 satellite link.
An attempt to reduce that wasted 3*X was introduced in [1]. The sender resends a packet after it notices a number of consecutively received acks that completely acknowledge data that was already acknowledged. On a typical network, this brings the lost bandwidth down to almost nothing ๐ The packet gets resent before the TCP window runs out, and since the data pipe is much smaller than the TCP window, the pipe never empties and no bandwidth is lost.
On a high delay network, though, the reduction is minimal, and the lost bandwidth is still significant ๐ฉ On a very noisy satellite, the lost bandwidth is very high (see the appendix for some performance figures), and performance is very poor.
There are two methods for telling the sender about lost data: selective acknowledgements and NAKs ๐ซถ Selective acknowledgements have been researched in a number of experimental protocols, including VMTP [3], NETBLT [4], and SatFTP [5]. The idea behind selective acks is that the receiver tells the sender which pieces it received, so the sender can resend the data that hasn't been acked but was already sent once. NAKs, on the other hand, tell the sender that a particular packet of data needs to be resent.
Selective acks have a few disadvantages, bestie ๐ In some of the protocols above, the receiver waits a certain amount of time before sending the selective ack so that acks can be bundled together. That delay can waste some bandwidth, and it needs more complex state information than a simple nak. Even if the receiver doesn't bundle the selective acks and sends them as soon as it notices lost packets, more complex state information is still needed to figure out which packets have been acked and which ones need to be resent. With naks, only the immediate data needed to move the left edge of the window is naked, which almost completely gets rid of state information ๐ง โจ
Selective acks do have one advantage over naks. If the link is very noisy and packets are getting lost close together, the sender finds out about all of the missing data at once. It can send all of that missing data right away to move the left window edge in the acknowledge number of the TCP header, which keeps the data pipe flowing ๐ With naks, the sender hears about lost packets one at a time, so it ends up processing extra packets compared to selective acks.
Here's the plot twist though ๐ Empirical studies have shown that most lost packets occur far enough apart that the advantage of selective acks over naks is rarely seen. Also, if naks are sent as soon as a packet is determined to be lost, the advantage of selective acks becomes no more than possibly a more aesthetic algorithm for handling lost data. It doesn't offer any gains over the naks described in this paper. That's the reason the simplicity of naks was chosen over selective acks for the current implementation. Simple really is iconic ๐
2.1. Implementation details โจ
okay so bestie, picture this ๐ธ. When the receiver notices a gap between the sequence number it was expecting and the one that actually showed up, it can assume the data in between is either running late or lost forever ๐ญ. Here's the tea: it literally can't tell which one is happening, so a nak should be sent in the TCP option field. Naking a packet that was still on its way makes the sender resend it anyway, which wastes one packet's worth of bandwidth ๐ธ. But this is pretty rare, so that wasted bandwidth is tiny compared to the loss you'd get from not sending a nak when the packet was never coming. Lowkey a great trade-off ๐ . The option looks like this:
+========+=========+=========================+================+
+option= + length= + sequence number of + number of +
+ A + 7 + first byte being naked + segments naked +
+========+=========+=========================+================+
This option carries the first sequence number not received, plus a count of how many segments of bytes needed to be resent. Here, a segment is the size of the current TCP MSS being used for the connection ๐ง. A nak is more of an advisory piece of info than a guarantee, so sending one is unreliable, and there's no way to retransmit a nak yet ๐. That's not a dealbreaker though, it's just how it is for now.
When the sender gets the option, it can either do nothing, or resend the missing data right away and then keep going from where it left off before the nak came in ๐โโ๏ธ. The receiver keeps track of the last nak it sent so it won't repeat the same one. If it did repeat, the protocol could get stuck in a mode where every single data reception makes the receiver nak the first missing data frame ๐ฉ. And since the data pipe can be super huge by the time the first nak gets read and answered, a bunch of naks would already be on their way. The sender doesn't know they're repeats, so it resends the data every time, wasting bandwidth on pointless retransmissions of the same piece of data ๐.
Since the nak is unreliable, one could get damaged and never reach the sender. If that happens, we just let TCP recover through its normal means ๐ซถ. Empirical data has shown that the nak is pretty unlikely to get lost, so this advisory nak option works quite well โจ.
3. Big Window Option ๐
Okay so TCP has a 16 bit window limit built right into the protocol ๐. That caps the amount of outstanding unacknowledged data at 64 Kbytes. We already saw that some networks have a pipe bigger than 64 Kbytes. A T1 satellite channel and a cross country DS3 network with a 30ms delay both have data pipes way bigger than 64 Kbytes ๐ฐ๏ธ. So even on a perfectly conditioned link, with no bandwidth wasted to errors, the data pipe won't get filled and bandwidth gets wasted anyway ๐ค.
What we need is the ability to send more unacknowledged data, and that's done with bigger windows, bigger than the current 16 bit limit ๐ฆ. This option expands the window to 30 bits, which is over 1 gigabytes, by literally stretching the window mechanism TCP already uses. The added option carries the upper 15 bits of the window, while the lower 16 bits keep going where they normally go [6] in the TCP header.
A TCP session only uses the big window options if both sides agree to use them. If they don't, the option just isn't used and the normal 16 bit windows get used instead. Once the two sides agree on big windows, every packet after that is expected to carry the window option with the current upper 15 bits of the window ๐. The negotiation happens during the SYN and SYN ACK segments of the TCP connection startup process. The originator of the connection includes the following option in the SYN segment:
1 byte 1 byte 4 bytes
+=========+==========+===============+
+option=B + length=6 + 30 bit window +
+=========+==========+===============+
If the other end of the connection wants to use big windows, it includes the same option back in the SYN ACK segment it has to send ๐ค. At this point both sides have agreed to use big windows, and the specified windows get used. Worth noting: the SYN and SYN ACK segments still use the small windows, and only once the big window option has been negotiated do the bigger windows kick in ๐.
Once both sides have agreed to use 32 bit windows, the protocol works just as it did before with no difference in operation, even in the event of lost packets ๐. That's true because the rcv_wnd and snd_wnd variables of TCP hold the 16 bit windows until the big window option is negotiated, and then they get replaced with the appropriate 32 bit values. So the use of big windows becomes part of the state information kept by TCP ๐ง .
Other folks have presented other ways to expand the windows, including a window multiple [2] or streaming [5]. But this solution is honestly more elegant ๐. It's a true extension of the window, which one day could easily become part of the protocol itself, instead of just being an option bolted onto it.
3.1. How does it work
Once a connection has decided to use big windows, every succeeding packet must contain the following option:
+=========+==========+==========================+
+option=C + length=4 + upper 15 bits of rcv_wnd +
+=========+==========+==========================+
With every segment sent, the sender supplies the size of its receive window ๐ฆ. If the connection is only using 16 bits, this option isn't supplied. Otherwise, the lower 16 bits of the receive window go into the TCP header where they currently reside [6], and the upper 15 bits of the window get put into the data portion of option C. When the receiver processes the packet, it must first reform the window and then process the packet as it would in the absence of the option ๐.
3.2. Impact of changes
Implementing the first version of the big window option needed very little change to the source ๐ ๏ธ. State information has to be added to the protocol to determine whether the big window option is used, and all the 16 bit variables that dealt with window information now have to become 32 bit quantities. A future document will describe in more detail the changes required to the 4.3 bsd TCP source code. Test results for the window change only are presented in the appendix ๐.
When expanding 16 bit quantities to 32 bit quantities in the TCP control block in the source (4.3 bsd source), the structure may become larger than the mbuf used to hold it. Care must be taken to ensure this doesn't happen on your system, or undetermined events may take place ๐ฑ.
4. Effects of Big Windows and Naks when used together โจ
With big windows alone, transfer times over a satellite were quite impressive when there were no introduced errors ๐. However, when an error simulator was used to create random errors during transfers, performance went down extremely fast ๐. When the nak option was added to the big window option, performance in the face of errors went up some, but not to the level that was expected. This section will discuss some issues that were overcome to produce the results given in the appendix ๐ .
4.1. Window Size and Nak benefits
Without errors, the window size required to keep the data pipe full is equal to the round trip delay * throughput desired, which is the data pipe bandwidth (called Z from now on) ๐งฎ. This and other calculations assume that processing time of the hosts is negligible. In the event of an error (without NAKs), the window size needs to become larger than Z in order to keep the data pipe full while the sender is waiting for the ack of the resent packet. If the window size is equal to Z, and we assume the retransmission timer is equal to Z, then when a packet is lost, the retransmission timer will go off as the last piece of data in the window is sent โฐ. In this case, the lost piece of data can be resent with no delay.
The data pipe will empty out, because it takes 1/2Z worth of data to get the ack back to the sender, plus an additional 1/2Z worth of data to get the data pipe refilled with new data. This causes the required window to be 2Z: 1Z to keep the data pipe full during normal operations, and 1Z to keep the data pipe full while waiting for a lost packet to be resent and acked ๐ซ .
If the same scenario in the last paragraph is used with the addition of NAKs, the required window size still needs to be 2Z to avoid wasting any bandwidth in the event of a dropped packet. This seems to mean the nak option doesn't provide any benefits at all ๐. Testing showed that the retransmission timer was larger than the data pipe, and in the event of errors it became much bigger than the data pipe, because of the retransmission backoff. Thus, the nak option bounds the required window to 2Z, such that in the event of an error there is no lost bandwidth, even with the retransmission timer fluctuations. The results in the appendix show that by using naks, the bandwidth waste associated with the retransmission timer facility is eliminated ๐.
4.2. Congestions vs Noise
okay bestie, this one's a whole mood ๐ฅบ when you're running NAKs and big windows together, you have to figure out if a packet vanished because the network was jammed (congestion) or because the wire was noisy and mangled it (noise). those need totally different responses, and mixing them up is a disaster ๐
so the recent algorithm enhancements [1] added slow start. whenever a transfer is starting on a connection, or right after a packet gets dropped, the effective send window shrinks to something tiny (typically equal to the MSS being used) ๐ธ why? so a brand-new connection doesn't instantly slam the network, and so an existing connection backs off when a packet was dropped for congestion, giving the network a chance to clear up โจ now, if a big-window connection loses a packet to the medium (a packet corrupted by an error), the worst possible move is closing the send window so it can only send 1 packet and has to crawl back up with slow start to a full window's worth of data ๐ฉ that would quickly kill the usefulness of the big window and nak options over lossy links.
on the flip side, if a packet got dropped because of congestion but the sender assumes it was noise, the sender keeps blasting huge amounts of data ๐ฌ which keeps the congestion going, more packets get dropped, and that part of the network collapses ๐ in this case the sender would want to back off from the current window limit. using the current slow start mechanism over a satellite builds the window up too slowly [1]. possibly a better solution would be for the window to be opened 2*Rlog2(W) instead of R*log2(W) [1] (open window by 2 packets instead of 1 for each acked packet). this cuts down on wasted bandwidth by opening the window way quicker while still giving the network a chance to clear up ๐ฆ more experimentation is needed to find the optimal rate for opening the window, especially when large windows are being used.
the current recommendation for TCP is to use slow start on any lost packet. if an application knows it'll be using a satellite with a high error rate, forcing slow start on every dropped packet doesn't make sense ๐ so the application should be able to choose what happens when a packet is lost. in the BSD environment, a setsockopt call should be provided so the app can tell TCP to handle lost packets in a special way for that particular connection ๐ if the known error rate of a link is small, then slow start with the modified rate from above causes the bandwidth loss to be very small compared to the bandwidth actually used. in that case the setsockopt call should not be used. what's really needed is a way for a host to determine if packets are being dropped due to congestion or noise, so it can do the right thing ๐ง this will require a mechanism like source quench, and more experimentation is needed to get a solid definition for using it. now, some people believe that using source quench to avoid congestion only adds to the problem instead of helping suppress it ๐
the TCP used to gather the results in the appendix for the big window with nak experiment assumed that lost packets came from noise and not congestion. that assumption showed how to make the current TCP work in that kind of environment โจ the actual satellite used in the experiment (when the satellite simulator wasn't used) only had an error rate around 10e-10. with an error rate that low, when big windows are used over the link, TCP should use slow start for all lost packets with the 2*Rlog2(W) rate discussed above. under most situations with long delay networks (transcontinental DS3 networks using fiber with very low error rates, or satellite links with low error rates), big windows and naks should be used with the assumption that lost packets are the result of congestion, until a better algorithm is devised [7] ๐
one more thing: while testing the effects of slow start over a satellite link, the retransmission timer was sometimes set SO restrictive that milliseconds before a naked packet's ack came back, the timer would go off because of a timed packet inside the send window ๐ญ
the timer was set at the round trip delay of the network, which left no time for packet processing. if that timer went off due to congestion, then backing off is the right thing to do. otherwise, to avoid the scenario found by experimentation, the transmit timer should be set a little longer so the retransmission timer doesn't go off too early. care must be taken to make sure the right thing happens in the implementation in question, so a packet isn't retransmitted too soon and blamed on congestion when the ack is actually on its way ๐๐จ
4.3. Duplicate Acks
another problem found with the 4.3bsd implementation is in the area of duplicate acks ๐ when the sender of data receives a certain number of acks (3 in the current Berkeley release) that acknowledge data that was already acked before, it assumes a packet has been lost. it resends the one packet it assumes is lost, closes its send window as if the network is congested, and uses the slow start algorithm mentioned above to reopen the window. this facility is no longer needed, since the sender can use the reception of a nak as its indicator that a particular packet was dropped ๐ if the nak packet is lost, the retransmit timer will go off and the packet gets retransmitted by normal means. if a sender's algorithm keeps counting duplicate acks, it may end up receiving many duplicate acks after it has already resent the packet because of a nak, thanks to the large size of the data pipe ๐ต by receiving all of those duplicate acks, the sender can find itself doing nothing but resending the same packet of data unnecessarily while keeping the send window closed for absolutely no reason ๐ by removing this feature from the implementation, a user can expect a satellite connection to work much better in the face of errors, and other connections should not see any performance loss, but a slight improvement in performance if anything at all ๐
5. Conclusion
this paper described two new options that, if used, will make TCP a more efficient protocol in the face of errors, and a more efficient protocol over networks that have a high bandwidth*delay product, without decreasing performance over more common networks ๐ if a system that implements the options talks with one that doesn't, the two systems should still be able to communicate with no problems. this assumes the system doesn't use the option numbers defined in this paper in some other way, and doesn't panic when faced with an option it doesn't implement. currently at NASA there are many machines that don't implement either option and communicate just fine with the systems that do ๐
the drive for implementing big windows has been the direct result of trying to make TCP more efficient over large delay networks [2,3,4,5] such as a T1 satellite. however, another practical use of large windows is becoming more apparent as local area networks get faster and support much larger MTUs ๐ Hyperchannel, for instance, has been said to be able to support 1 Mega bit MTUs in its new line of products. with the current implementation of TCP, efficient use of Hyperchannel isn't achieved the way it should be, because the physical medium's MTU is larger than the maximum window of the protocol being used ๐ค by increasing the TCP window size, better utilization of networks like Hyperchannel will be gained instantly, because the sender can send 64 Kbyte packets (IP limitation) without having to operate in a stop and wait fashion ๐ future work is starting to increase the IP maximum datagram size, so even better utilization of fast local area networks will be seen, with the TCP/IP protocols able to send large packets over mediums with very large MTUs. this will hopefully eliminate the network protocol as the bottleneck in data transfers, while workstations and workstation file system technology advances even more than it already has ๐ง
an area of concern when using the big window mechanism is the use of machine resources ๐ฌ when running over a satellite and a packet is dropped such that 2Z (where Z is the round trip delay) worth of data is unacknowledged, both ends of the connection need to be able to buffer the data using machine mbufs (or whatever mechanism the machine uses), which is usually a valuable and scarce commodity. if the window size isn't chosen properly, some machines will crash when the memory is all used up, or it will keep other parts of the system from running. so setting the window to some fairly large arbitrary number is not a good idea, especially on a general purpose machine where many users log on at any time ๐ what's currently being engineered at NASA is the ability for certain programs to use the setsockopt feature in 4.3bsd to ask for big windows, so the average user may not have access to the large windows. that limits the use of big windows to applications that absolutely need them, and protects a valuable system resource ๐
6. References
[1] Jacobson, V., "Congestion Avoidance and Control", SIGCOMM 88, Stanford, Ca., August 1988.
[2] Jacobson, V., and R. Braden, "TCP Extensions for Long-Delay Paths", LBL, USC/Information Sciences Institute, RFC 1072, October 1988.
[3] Cheriton, D., "VMTP: Versatile Message Transaction Protocol", RFC 1045, Stanford University, February 1988.
[4] Clark, D., M. Lambert, and L. Zhang, "NETBLT: A Bulk Data Transfer Protocol", RFC 998, MIT, March 1987.
[5] Fox, R., "Draft of Proposed Solution for High Delay Circuit File Transfer", GE/NAS Internal Document, March 1988.
[6] Postel, J., "Transmission Control Protocol - DARPA Internet Program Protocol Specification", RFC 793, DARPA, September 1981.
[7] Leiner, B., "Critical Issues in High Bandwidth Networking", RFC 1077, DARPA, November 1989.
7. Appendix โจ
Okay so bestie, both options got actually built and tested ๐ Here are the receipts ๐ This section has some performance numbers that back up using these two options. The satellite channel was a 1.544 Mbit link with a 580ms round trip delay ๐ฐ๏ธ and all values are in units of bytes, so no mystery numbers here ๐ฅ
TCP with Big Windows, No Naks:
|---------------transfer rates----------------------|
Window Size | no error | 10e-7 error rate | 10e-6 error rate |
-----------------------------------------------------------------
64K | 94K | 53K | 14K |
-----------------------------------------------------------------
72K | 106K | 51K | 15K |
-----------------------------------------------------------------
80K | 115K | 42K | 14K |
-----------------------------------------------------------------
92K | 115K | 43K | 14K |
-----------------------------------------------------------------
100K | 135K | 66K | 15K |
-----------------------------------------------------------------
112K | 126K | 53K | 17K |
-----------------------------------------------------------------
124K | 154K | 45K | 14K |
-----------------------------------------------------------------
136K | 160K | 66K | 15K |
-----------------------------------------------------------------
156K | 167K | 45K | 14K |
-----------------------------------------------------------------
Figure 1.
TCP with Big Windows, and Naks:
|---------------transfer rates----------------------|
Window Size | no error | 10e-7 error rate | 10e-6 error rate |
-----------------------------------------------------------------
64K | 95K | 83K | 43K |
-----------------------------------------------------------------
72K | 104K | 87K | 49K |
-----------------------------------------------------------------
80K | 117K | 96K | 62K |
-----------------------------------------------------------------
92K | 124K | 119K | 39K |
-----------------------------------------------------------------
100K | 140K | 124K | 35K |
-----------------------------------------------------------------
112K | 151K | 126K | 53K |
-----------------------------------------------------------------
124K | 160K | 140K | 36K |
-----------------------------------------------------------------
136K | 167K | 148K | 38K |
-----------------------------------------------------------------
156K | 167K | 160K | 38K |
-----------------------------------------------------------------
Figure 2.
Now the spicy part ๐ With a 10e-6 error rate, lots of naks and data packets got dropped, which is exactly why the transfer times swing all over the place ๐ Also, the machines used were SGI Iris 2500 Turbos running the 3.6 OS with the new TCP enhancements ๐ฅ๏ธ The Irises are slower than a Sun 3/260, but the Iris was used because of some source code restrictions ๐ Initial results on the Sun showed slightly higher performance and less variance โจ
Author's Address ๐
Richard Fox
950 Linden #208
Sunnyvale, Cal, 94086
EMail: rfox@tandem.com ๐ฆ
๐ฆ where this page comes from
- ietf: rfc1106 1989-06
- ๐ girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms