RFC 1185
TCP Extension for High-Speed Paths
RFC 1185: TCP Extension for High-Speed Paths 🦋 (Part 1 of 6)
Okay bestie, so this RFC is from October 1990 and it's written by V. Jacobson (LBL), R. Braden (ISI), and L. Zhang (PARC) 🫶. It's an Experimental Protocol doc, and its header doesn't list anything it obsoletes or updates 🧪.
Status of This Memo ✨
So this memo is an Experimental Protocol extension to TCP 🧪, and the Internet community is literally invited to chat about it and suggest improvements 💬. For the official standardization state and status, check the "IAB Official Protocol Standards" edition 📚. And distribution is unlimited 🌈.
Summary ✨
tl;dr, it's a small TCP extension 🎀 that keeps things reliable on super-fast paths. It does this using sender timestamps sent through the TCP Echo option proposed in RFC-1072 ⚡.
1. INTRODUCTION ✨
TCP is the queen of reliable delivery 👑. It uses positive acknowledgments and retransmissions to deliver stuff end-to-end over a full-duplex virtual circuit called a connection [Postel81]. A connection is defined by its two end points, and each end point is a "socket", i.e., a (host,port) pair 🧦. To protect against data corruption, TCP uses an end-to-end checksum 🛡️. For duplication and reordering, it uses a fine-grained sequence number space, with each octet getting its own distinct sequence number 🔢.
The TCP protocol [Postel81] was designed to work reliably over almost any transmission medium, no matter the transmission rate, delay, corruption, duplication, or reordering of segments 🧃. In practice, proper TCP implementations have shown remarkable robustness at adapting to a wide range of network characteristics 💅. For example, TCP implementations currently adapt to transfer rates in the range of 100 bps to 10**7 bps and round-trip delays in the range 1 ms to 100 seconds 🌸.
However, fiber optics keep pushing transmission speeds higher 🔥, so the fastest paths are moving out of the domain TCP was originally engineered for 🚀. This memo and RFC-1072 [Jacobson88] propose modest extensions to TCP to stretch its domain to higher speeds 🦋.
There is no one-line answer to the question: "How fast can TCP go?" 🙈. The issues are reliability and performance. These depend on the round-trip delay, the maximum time that segments may be queued in the Internet, and the transmission speed. We must think through these relationships very carefully if we are to successfully extend TCP's domain 🧠.
TCP performance doesn't depend on the transfer rate itself. It depends on the product of the transfer rate and the round-trip delay 📈. This "bandwidth*delay product" measures the amount of data that would "fill the pipe" 🪣. It's the buffer space needed at sender and receiver to get maximum throughput on the TCP connection over the path 💜. RFC-1072 proposed a set of TCP extensions to improve TCP efficiency for "LFNs" (long fat networks), i.e., networks with large bandwidth*delay products 🍓.
On the other hand, a high transfer rate can threaten TCP reliability by violating the assumptions behind the TCP mechanism for duplicate detection and sequencing 😭. The present memo specifies a solution for this problem, extending TCP reliability to transfer rates well beyond the foreseeable upper limit of bandwidth ✨.
An especially serious kind of error can come from an accidental reuse of TCP sequence numbers in data segments 💀. Imagine an "old duplicate segment", e.g., a duplicate data segment that got delayed in Internet queues, arriving at the receiver at the worst possible moment, with its sequence numbers landing somewhere inside the current window 😱. There would be no checksum failure to warn anyone, and the result could be undetected corruption of the data 🙀. An old duplicate ACK segment reaching the transmitter could be only slightly less serious. It's likely to lock up the connection so no further progress can be made, and a RST is required to resynchronize the two ends 🔒.
Duplication of sequence numbers might happen in either of two ways 👀:
(1) Sequence number wrap-around on the current connection 🌀
A TCP sequence number contains 32 bits. At a high enough transfer rate, the 32-bit sequence space may be "wrapped" (cycled) within the time that a segment may be delayed in queues ⏳. Section 2 discusses this case and proposes a mechanism to reject old duplicates on the current connection 🛑.
(2) Segment from an earlier connection incarnation 👻
Suppose a connection terminates, either by a proper close sequence or due to a host crash, and the same connection (i.e., using the same pair of sockets) is immediately reopened 🔁. A delayed segment from the terminated connection could fall within the current window for the new incarnation and be accepted as valid 😬. This case is discussed in Section 3 🧚.
TCP reliability depends on there being a bound on how long a segment can live: the "Maximum Segment Lifetime" or MSL ⏰. An MSL is generally required by any reliable transport protocol, since every sequence number field must be finite, and therefore any sequence number may eventually be reused 🔄. In the Internet protocol suite, the MSL bound is enforced by an IP-layer mechanism, the "Time-to-Live" or TTL field 🎟️.
Watson's Delta-T protocol [Watson81] includes network-layer mechanisms for precise enforcement of an MSL 🎯. The IP mechanism for MSL enforcement, on the other hand, is loosely defined and even more loosely implemented in the Internet 🥴. So it's unwise to depend on active enforcement of MSL for TCP connections, and it's unrealistic to imagine setting MSLs smaller than the current values (e.g., 120 seconds specified for TCP) 🙃. The timestamp algorithm described in the following section gives a way out of this dilemma for high-speed networks 🌈💖.
2. Sequence Number Wrap-Around ✨
2.1. Background 🎀
Okay bestie, so avoiding reuse of sequence numbers inside the same connection is simple in principle 💅. You just enforce a segment lifetime that's shorter than the time it takes to cycle through the sequence space, which is effectively 2**31 in size.
More specifically, if the maximum effective bandwidth TCP can transmit over a particular path is B bytes per second, then this constraint must be satisfied for error-free operation:
2**31 / B > MSL (secs) [1]The table below shows the value of Twrap = 2**31/B in seconds for some important bandwidths B 📊:
Network B*8 B Twrap
bits/sec bytes/sec secs
_______ _______ ______ ______
ARPANET 56kbps 7KBps 3*10**5 (~3.6 days)
DS1 1.5Mbps 190KBps 10**4 (~3 hours)
Ethernet 10Mbps 1.25MBps 1700 (~30 mins)
DS3 45Mbps 5.6MBps 380
FDDI 100Mbps 12.5MBps 170
Gigabit 1Gbps 125MBps 17It's so clear why wrap-around wasn't a problem for 56kbps packet switching or even 10Mbps Ethernets 🦋. But at DS3 and FDDI speeds, Twrap is comparable to the 2 minute MSL assumed by the TCP specification [Postel81] 😭. Once you get to gigabit speeds, Twrap becomes too small for reliable enforcement by the Internet TTL mechanism 💀.
The 16-bit window field of TCP limits the effective bandwidth B to 2**16/RTT, where RTT is the round-trip time in seconds [McKenzie89] 🌸. If the RTT is large enough, that limits B to a value that meets constraint [1] even for a large MSL value. For example, take a transcontinental backbone with an RTT of 60ms (set by the laws of physics ⭐). With the bandwidth*delay product limited to 64KB by the TCP window size, B is limited to 1.1MBps, no matter how high the theoretical transfer rate of the path is. This corresponds to cycling the sequence number space in Twrap= 2000 secs, which is safe in today's Internet 🫶.
Based on this reasoning, an earlier RFC [McKenzie89] cautioned that expanding the TCP window space as proposed in RFC-1072 will lead to sequence wrap-around and hence to possible data corruption. We believe that's mis-identifying the culprit 🕵️♀️. The culprit isn't the larger window, it's the high bandwidth 🔥.
For example, consider a (very large) FDDI LAN with a diameter of 10km. Using the speed of light, we can compute the RTT across the ring as (2*104)/(3*108) = 67 microseconds, and the delay*bandwidth product is then 833 bytes. A TCP connection across this LAN using a window of only 833 bytes will run at the full 100mbps and can wrap the sequence space in about 3 minutes, very close to the MSL of TCP. Thus, high speed alone can cause a reliability problem with sequence number wrap-around, even without extended windows.
An "obvious" fix for cycling the sequence space is to increase the size of the TCP sequence number field 🧚. For example, the sequence number field (and also the acknowledgment field) could be expanded to 64 bits. The thing is, the proposals for making that change while staying compatible with current TCP have tended towards complexity and ugliness 🙈.
This memo proposes a simple solution, using the TCP echo options defined in RFC-1072 🎀. Section 2.2 which follows describes the original use of these options to carry timestamps in order to measure RTT accurately. Section 2.3 proposes a method of using those same timestamps to reject old duplicate segments that could corrupt an open TCP connection. Section 3 discusses applying this mechanism to avoiding old duplicates from previous incarnations 👑.
2.2. TCP Timestamps 🦋
RFC-1072 defined two TCP options, Echo and Echo Reply. Echo carries a 32-bit number, and the receiver of the option must return this same value to the source host in an Echo Reply option.
RFC-1072 also describes using these options to contain 32-bit timestamps, for measuring the RTT. A TCP sending data would include Echo options containing the current clock value. The receiver would echo these timestamps back in returning segments (generally, ACK segments). The difference between a timestamp from an Echo Reply option and the current time would then measure the RTT at the sender ⏱️.
This mechanism was designed to solve a specific problem 🧠. Almost all TCP implementations base their RTT measurements on a sample of only one packet per window. If you look at RTT estimation as a signal processing problem (which it is), a data signal at some frequency (the packet rate) is being sampled at a lower frequency (the window rate). Unfortunately, that lower sampling frequency violates Nyquist's criteria and may introduce "aliasing" artifacts into the estimated RTT [Hamming77] 😵💫.
A good RTT estimator with a conservative retransmission timeout calculation can tolerate the aliasing when the sampling frequency is "close" to the data frequency. For example, with a window of 8 packets, the sample rate is 1/8 the data frequency, which is less than an order of magnitude different. However, when the window is tens or hundreds of packets, the RTT estimator may be seriously in error, resulting in spurious retransmissions 😩.
The fix for the aliasing problem actually simplifies the sender a lot (since the RTT code is typically the single biggest protocol cost for TCP) 💅. Here's how it works: the sender will place a timestamp in each segment, and the receiver will reflect these timestamps back in ACK segments. Then a single subtract gives the sender an accurate RTT measurement for every ACK segment (which will correspond to every other data segment, with a sensible receiver). RFC-1072 defined a timestamp echo option for this purpose 🌈.
It is vitally important to use the timestamp echo option with big windows. Otherwise, the door is opened to some dangerous instabilities due to aliasing 🚨. Furthermore, the option is probably useful for all TCPs, since it simplifies the sender 🍓.
2.3. Avoiding Old Duplicate Segments 🛡️
Timestamps carried from sender to receiver in TCP Echo options can also be used to prevent data corruption caused by sequence number wrap-around, as this section describes ✨.
2.3.1. Basic Algorithm 🧁
Assume that every received TCP segment contains a timestamp. The basic idea is that a segment received with a timestamp earlier than the timestamp of the most recently accepted segment can be discarded as an old duplicate 🗑️. More specifically, the following processing is to be performed on normal incoming segments:
- R1) If the timestamp in the arriving segment is less than the timestamp of the most recently received in-sequence segment, treat the arriving segment as not acceptable:
- If SEG.LEN > 0, send an acknowledgement in reply as specified in RFC-793 page 69, and drop the segment. Otherwise, just silently drop the segment.*
*Sending an ACK segment in reply isn't strictly necessary, since the case can only arise when a later in-order segment has already been received. However, for consistency and simplicity, we suggest treating a timestamp failure the same way TCP treats any other
Okay so the whole point of R2-R4 is that they're the normal TCP steps from RFC-793 🙌. The one twist is in R3, where the latest timestamp gets set from every in-sequence segment that's accepted. That means the receiver's latest timestamp lines up with the left edge of the window, and it only moves forward when the left edge moves [Jacobson88] 💖.
- R2) If the segment is outside the window, reject it (normal TCP processing) 🙅♀️
- R3) If an arriving segment is in-sequence (i.e., at the left window edge), accept it normally and record its timestamp ✨
- R4) Otherwise, treat the segment as a normal in-window, out-of-sequence TCP segment (e.g., queue it for later delivery to the user) 🎀
Important tea ☕: the timestamp only gets checked when a segment first shows up at the receiver, whether it's in-sequence or gets queued. Check this example 👀.
Say the segment sequence A.1, B.1, C.1, ..., Z.1 has been sent, where the letter is the sequence number and the digit is the timestamp. Also say segment B.1 got lost 😭. The highest in-sequence timestamp is 1 (from A.1), so C.1, ..., Z.1 are considered acceptable and get queued. When B gets retransmitted as segment B.2 (using the latest timestamp), it fills the hole and causes all the segments through Z to be acknowledged and passed to the user. The timestamps of the queued segments are not inspected again at this time, since they've already been accepted. When B.2 is accepted, the receiver's current timestamp is set to 2 🥰.
This rule is super vital for performance when stuff gets lost 🫶. A full window of data is in transit at all times, so after a loss, a full window less one packet shows up out-of-sequence and gets queued at the receiver (e.g., up to ~2**30 bytes of data 😳). The timestamp option must not result in discarding this data, periodt 💅.
In certain unlikely circumstances, the algorithm of rules R1-R4 could lead to discarding some segments unnecessarily, as shown in the following example:
Suppose again that segments A.1, B.1, C.1, ..., Z.1 have been sent in sequence and that segment B.1 has been lost. Furthermore, suppose delivery of some of C.1, ... Z.1 is delayed until AFTER the retransmission B.2 arrives at the receiver. These delayed segments will be discarded unnecessarily when they do arrive, since their timestamps are now out of date 💀.
This case is very unlikely to occur 🙈. If the retransmission was triggered by a timeout, some of the segments C.1, ... Z.1 must have been delayed longer than the RTO time. This is presumably an unlikely event, or there would be many spurious timeouts and retransmissions. If B's retransmission was triggered by the "fast retransmit" algorithm, i.e., by duplicate ACK's, then the queued segments that caused these ACK's must have been received already.
Even if a segment was delayed past the RTO, the selective acknowledgment (SACK) facility of RFC-1072 will cause the delayed packets to be retransmitted at the same time as B.2, avoiding an extra RTT and therefore causing a very small performance penalty 🦋.
We know of no case with a significant probability of occurrence in which timestamps will cause performance degradation by unnecessarily discarding segments 🌸.
2.3.2 Header Prediction ✨
"Header prediction" [Jacobson90] is a high-performance transport protocol implementation technique that is most important for high-speed links 🚀. It optimizes the code for the most common case: receiving a segment correctly and in order. With header prediction, the receiver asks, "Is this segment the next in sequence?" That question can be answered in fewer machine instructions than "Is this segment within the window?" 🧠
Adding header prediction to our timestamp procedure leads to the following sequence for processing an arriving TCP segment:
- H1) Check timestamp (same as step R1 above)
- H2) Do header prediction: if segment is next in sequence and if there are no special conditions requiring additional processing, accept the segment, record its timestamp, and skip H3.
- H3) Process the segment normally, as specified in RFC-793.
- This includes dropping segments that are outside the window and possibly sending acknowledgments, and queueing in-window, out-of-sequence segments.
BUT the H1 timestamp check is super unlikely to fail 🙄, and it's kinda expensive because it requires interval arithmetic on a finite field. Doing that check on every single segment is giving poor implementation engineering, and it totally defeats the purpose of header prediction 💀. So we suggest that an implementor interchange H1 and H2, i.e., perform header prediction FIRST, performing H1 and H3 only if header prediction fails. We believe this change might gain 5-10% in performance on high-speed networks 🔥.
Okay but this reordering does raise a theoretical hazard 😬: a segment from 2**32 bytes in the past may arrive at exactly the wrong time and be accepted mistakenly by the header-prediction step. We make the following argument to show that the probability of this failure is negligible 🧐.
If all segments are equally likely to show up as old duplicates, then the probability of an old duplicate exactly matching the left window edge is the maximum segment size (MSS) divided by the size of the sequence space. This ratio must be less than 2**-16, since MSS must be < 2**16; for example, it will be (2**12)/(2**32) = 2**-20 for an FDDI link. However, the older a segment is, the less likely it is to be retained in the Internet, and under any reasonable model of segment lifetime the probability of an old duplicate exactly at the left window edge must be much smaller than 2**16.
The 16 bit TCP checksum also allows a basic unreliability of one part in 2**16. A protocol mechanism whose reliability exceeds the reliability of the TCP checksum should be considered "good enough", i.e., it won't contribute significantly to the overall error rate. We therefore believe we can ignore the problem of an old duplicate being accepted by doing header prediction before checking the timestamp 💖.
2.3.3 Timestamp Frequency 🦋
It is important to understand that the receiver algorithm for timestamps does not involve clock synchronization with the sender 👀. The sender's clock is used to stamp the segments, and the sender uses this fact to measure RTT's. However, the receiver treats the timestamp as simply a monotone-increasing serial number, without any necessary connection to its clock. From the receiver's viewpoint, the timestamp is acting as a logical extension of the high-order bits of the sequence number 🧚.
However, the receiver algorithm does place some requirements on the frequency of the timestamp "clock":
- (a) Timestamp clock must not be "too slow".
It must tick at least once for each 2**31 bytes sent. In fact, in order to be useful to the sender for round trip timing, the clock should tick at least once per window's worth of data, and even with the RFC-1072 window extension, 2**31 bytes must be at least two windows.
To make this more quantitative, any clock faster than 1 tick/sec will reject old duplicate segments for link speeds of ~2 Gbps; a 1ms clock will work up to link speeds of 2 Tbps (10**12 bps!) 🚀.
- (b) Timestamp clock must not be "too fast".
Its cycling time must be greater than MSL seconds. Since the clock (timestamp) is 32 bits and the worst-case MSL is 255 seconds, the maximum acceptable clock frequency is one tick every 59 ns 🤯.
However, since the sender is using the timestamp for RTT calculations, the timestamp doesn't need to have much more resolution than the granularity of the retransmit timer, e.g., tens or hundreds of milliseconds 🍓.
Thus, both limits are easily satisfied with a reasonable clock rate in the range 1-100ms per tick ✨.
Using the timestamp option relaxes the requirements on MSL for avoiding sequence number wrap-around 👑. For example, with a 1 ms timestamp clock, the 32-bit timestamp will wrap its sign bit in 25 days. Thus, it will reject old duplicates on the same connection as long as MSL is 25 days or less. This appears to be a very safe figure 🫶. If the timestamp has 10 ms resolution, the MSL requirement is boosted to 250 days. An MSL of 25 days or longer can probably be assumed by the gateway system without requiring precise MSL enforcement by the TTL value in the IP layer 🌈.
3. DUPLICATES FROM EARLIER INCARNATIONS OF CONNECTION 🦋
okay bestie, now we're onto suspect number two for those old duplicate packet errors: packets from an earlier incarnation of the same connection 👀. the appendix has a rundown of the mechanisms TCP already uses for this problem. they all depend on the Internet layer enforcing a maximum segment lifetime (MSL) 🕰️.
the MSL needed to prevent failures from an earlier connection incarnation doesn't depend (directly) on the transfer rate. but the timestamp option from Section 2 can give extra security against old duplicates from earlier connections 🔐. and we'll see that if everyone uses the timestamp option, enforcing an MSL wouldn't be required for reliable TCP anymore 💅.
there are two cases to look at (the appendix has more deets): (1) a system crashes (and loses its connection state) and restarts, and (2) the same connection gets closed and reopened without the host losing its state. we'll tackle each one in the next two sections 🎀.
3.1. System Crash with Loss of State
TCP's quiet time of one MSL at system startup handles losing connection state in a crash/restart 😴. for the full explanation, check out "When to Keep Quiet" in the TCP protocol spec [Postel81]. the MSL needed here doesn't depend on transfer speed. the current TCP MSL of 2 minutes seems like a fine operational compromise, since lots of host systems take that long to boot after a crash 🖥️.
but the timestamp option can ease the MSL requirements (or add extra security against data corruption) 🫶. if timestamps are in use and the timestamp clock can be guaranteed to be monotonic over a crash/restart, meaning the first value of the sender's timestamp clock after a restart is guaranteed to be greater than the last value before the restart, then a quiet time is unnecessary ✨.
to ditch the quiet time totally, the host clock would need to be synced to a time source that stays stable over the crash/restart period, with an accuracy of one timestamp clock tick or better 😵. that's a LOT. luckily, we can back off a bit. say the clock is always re-synchronized to within N timestamp clock ticks, and booting (extended with a quiet time if needed) takes more than N ticks. that guarantees monotonic timestamps, which can then be used to reject old duplicates even without an enforced MSL 🧠✨.
3.2. Closing and Reopening a Connection
when a TCP connection closes, a 2*MSL delay in TIME-WAIT state ties up the socket pair for 4 minutes (see Section 3.5 of [Postel81]). apps built on TCP that close one connection and open a new one (like an FTP data transfer connection using Stream mode) have to choose a new socket pair every single time 🙃. this delay has two jobs:
(a) Implement the full-duplex reliable close handshake of TCP.
the proper time to delay the final close step isn't really related to the MSL. it depends on the RTO for the FIN segments and therefore on the RTT of the path.* although there's no formal upper bound on RTT, common network engineering practice makes an RTT greater than 1 minute very unlikely. so the 4 minute delay in TIME-WAIT state works satisfactorily for a reliable full-duplex TCP close. again, this is independent of MSL enforcement and network speed 💖.
the TIME-WAIT state could cause an indirect performance problem if an app needed to repeatedly close one connection and open another at a very high frequency, since the number of available TCP ports on a host is less than 2**16. but high network speeds aren't the major contributor to this problem. the RTT is the limiting factor in how quickly connections can be opened and closed. so this problem won't be any worse at high transfer speeds 🏎️.
*Note: It could be argued that the side that is sending a FIN knows what degree of reliability it needs, and therefore it should be able to determine the length of the TIME-WAIT delay for the FIN's recipient. This could be accomplished with an appropriate TCP option in FIN segments. 🧐
(b) Allow old duplicate segments to expire.
suppose a host keeps a cache of the last timestamp received from each remote host. that cache can be used to reject old duplicate segments from earlier incarnations of the connection, if the timestamp clock can be guaranteed to have ticked at least once since the old connection was open. this requires that the TIME-WAIT delay plus the RTT together be at least one tick of the sender's timestamp clock ⏱️.
this is a variant on the mechanism proposed by Garlick, Rom, and Postel (see the appendix), which required each host to maintain connection records containing the highest sequence numbers on every connection. with timestamps instead, you only need to keep one quantity per remote host, no matter how many simultaneous connections go to that host 🍓.
we conclude that if all hosts used the TCP timestamp algorithm described in Section 2, enforcing a maximum segment lifetime would be unnecessary, and the quiet time at system startup could be shortened or removed. in any case, the timestamp mechanism can give extra security against old duplicates from earlier connection incarnations. however, a 4 minute TIME-WAIT delay (unrelated to MSL enforcement or network speed) must be retained to provide the reliable close handshake of TCP 🫶.
4. CONCLUSIONS 🎀
okay so we presented a mechanism, based on the TCP timestamp echo option of RFC-1072, that allows very high TCP transfer rates without reliability problems from old duplicate segments on the same connection 🚀. it also adds extra security against intrusion by old duplicates from earlier incarnations of the same connection 🔒. if every host used the timestamp mechanism, the quiet time at system startup could be eliminated, and enforcing a maximum segment lifetime (MSL) would no longer be necessary 👑.
REFERENCES 📚
- [Cerf76] Cerf, V., "TCP Resynchronization", Tech Note #79, Digital Systems Lab, Stanford, January 1976.
- [Dalal74] Dalal, Y., "More on Selecting Sequence Numbers", INWG Protocol Note #4, October 1974.
- [Garlick77] Garlick, L., R. Rom, and J. Postel, "Issues in Reliable Host-to-Host Protocols", Proc. Second Berkeley Workshop on Distributed Data Management and Computer Networks, May 1977.
- [Hamming77] Hamming, R., "Digital Filters", ISBN 0-13-212571-4, Prentice Hall, Englewood Cliffs, N.J., 1977.
- [Jacobson88] Jacobson, V., and R. Braden, "TCP Extensions for Long-Delay Paths", RFC 1072, LBL and USC/Information Sciences Institute, October 1988.
- [Jacobson90] Jacobson, V., "4BSD Header Prediction", ACM Computer Communication Review, April 1990.
- [McKenzie89] McKenzie, A., "A Problem with the TCP Big Window Option", RFC 1110, BBN STC, August 1989.
- [Postel81] Postel, J., "Transmission Control Protocol", RFC 793, DARPA, September 1981.
- [Tomlinson74] Tomlinson, R., "Selecting Sequence Numbers", INWG Protocol Note #2, September 1974.
- [Watson81] Watson, R., "Timer-based Mechanisms in Reliable Transport Protocol Connection Management", Computer Networks, Vol. 5, 1981.
APPENDIX -- Protection against Old Duplicates in TCP ✨
Okay bestie, this part is all about TCP protecting itself from old ghosts 👻 (segments left behind by earlier versions of the same connection). People put SO much effort into this back in the day, and several different mechanisms got pitched [Tomlinson74] [Dalal74] [Cerf76] [Garlick77] 💅
The connection parameters we need for this discussion are:
Tc = Connection duration in seconds.Nc = Total number of bytes sent on connection.B = Effective bandwidth of connection = Nc/Tc.
Tomlinson came up with a two-part scheme: a clock-driven pick for the ISN (Initial Sequence Number) of a connection, plus a resynchronization procedure [Tomlinson74]. The clock-driven scheme chooses:
ISN = (integer(R*t)) mod 2**32 [2]where t is the current time relative to an arbitrary origin, and R is a constant. The idea was to pick R so the ISN would advance faster than sequence numbers get used up on the connection. At high speeds though, that's not true anymore 🙃 We'll get into the consequences below.
The clock-driven choice of ISN in formula [2] guarantees freedom from old duplicates matching a reopened connection, as long as the original connection was "short-lived" and "slow" 🐢 By "short-lived" we mean a connection that stayed open for a time Tc less than the time it takes to cycle the ISN, i.e., Tc < 2**32/R seconds. By "slow" we mean that the effective transfer rate B is less than R.
Here's the visual, with sequence numbers plotted against time 📈 The asterisks are the ISN lines from formula [2], and the circles are the trajectories of several short-lived incarnations of the same connection, each ending at the "x" ✨
Note: allowing rapid reuse of connections was believed to be an important goal during the early TCP development. This requirement was driven by the hope that TCP would serve as a basis for user-level transaction protocols as well as connection-oriented protocols. The paradigm discussed was the "Christmas Tree" or "Kamikazee" segment that contained SYN and FIN bits as well as data. Enthusiasm for this was somewhat dampened when it was observed that the 3-way SYN handshake and the FIN handshake mean that 5 packets are required for a minimum exchange. Furthermore, the TIME-WAIT state delay implies that the same connection really cannot be reopened immediately. No further work has been done in this area, although existing applications (especially SMTP) often generate very short TCP sessions. The reuse problem is generally avoided by using a different port pair for each connection.
|- 2**32 ISN ISN
| * *
| * *
| * *
| *x *
| o *
^ | * *
| | * x *
| * o *
S | *o *
e | o *
q | * *
| * *
# | * x *
| *o *
|o_______________*____________
^ Time -->
4.55hrsFigure 1. Clock-Driven ISN avoiding duplication on short-Lived, slow connections.
Clock-driven ISN selection does NOT protect against old duplicate packets for a long-lived or fast connection 😬 The connection may close (or crash) just as the ISN has cycled around and hit the same value again. If the connection is then reopened, a datagram still in transit from the old connection may fall into the current window. Figure 2 shows this for a slow, long-lived connection, and Figures 3 and 4 show it for fast connections. In each case, the "x" marks where the original connection closes or crashes. The arrow in Figure 2 points at an old duplicate segment. Figure 3 is a connection whose total byte count Nc is less than 232, while Figure 4 is Nc greater than or equal to 232.
To prevent the duplication shown in Figure 2, Tomlinson proposed to "resynchronize" the connection sequence numbers if they came within an MSL of the ISN. Resynchronization might look like a delay (point "y") or picking a new sequence number (point "z") 🔁
|- 2**32 ISN ISN
| * *
| * *
| * *
| * *
| * *
^ | * *
| | * *
| * *
S | * *
e | * x* y
q | * o *
| * o *z
# | *o *
| * *
|*_________________*____________
^ Time -->
4.55hrsFigure 2. Resynchronization to Avoid Duplication on Slow, Long-Lived Connection
|- 2**32 ISN ISN
| * *
| x o * *
| * *
| o-->o* *
| * *
^ | o o *
| | * *
| o * *
S | * *
e | o * *
q | * *
| o* *
# | * *
| o *
|*_________________*____________
^ Time -->
4.55hrsFigure 3. Duplication on Fast Connection: Nc < 2**32 bytes
|- 2**32 ISN ISN
| o * *
| x * *
| * *
| o * *
| o *
^ | * *
| | o * *
| * o *
S | * *
e | o * *
q | * o *
| * *
# | o *
| * o *
|*_________________*____________
^ Time -->
4.55hrsFigure 4. Duplication on Fast Connection: Nc > 2**32 bytes
So in summary, Figures 1-4 show four possible failure modes for old duplicate packets from an earlier incarnation. We'll call them F1, F2, F3, and F4 🎀
F1: B < R, Tc < 4.55 hrs. (Figure 1)F2: B < R, Tc >= 4.55 hrs. (Figure 2)F3: B >= R, Nc < 2**32 (Figure 3)F4: B >= R, Nc >= 2**32 (Figure 4)
Another limitation of clock-driven ISN selection is worth mentioning 🙈 Tomlinson assumed that the current time t in formula [2] comes from a clock that persists over a system crash. For his scheme to work correctly, the clock must be restarted with an accuracy of 1/R seconds (e.g, 4 microseconds in the case of TCP). That might be possible for some hosts and some crashes, but in most cases the clock is uncertain after a crash by anywhere from a second to several minutes.
Because of that random clock offset after system reinitialization, old segments sent before the crash might fall into the window of a new connection incarnation 😱 The solution the final TCP spec adopted is a "quiet time" of MSL seconds when the system is initialized [Postel81, p. 28]. No TCP connection can be opened until that quiet time expires.
A different approach was suggested by Garlick, Rom, and Postel [Garlick77]. Instead of clock-driven ISN selection, they wanted to keep connection records containing the last ISN used on every connection. To immediately open a new incarnation of a connection, the ISN is taken to be greater than the last sequence number of the previous incarnation, so the new incarnation has unique sequence numbers. To handle a system crash, they proposed a quiet time (a delay at startup to let old duplicates expire). Note that the connection records only need to be kept for MSL seconds. After that, no collision is possible, and a new connection can start with sequence number zero ⏳
The scheme TCP finally adopted combines features of both proposals 💖 TCP uses three mechanisms:
- (A) ISN selection is clock-driven to handle short-lived connections. The parameter R = 250KBps, so the ISN value cycles in 2**32/R = 4.55 hours.
- (B) (One end of) a closed connection is left in a "busy" state, known as "TIME-WAIT" state, for a time of 2*MSL. TIME-WAIT state handles the proper close of a long-lived connection without resynchronization. It also allows reliable completion of the full-duplex close handshake.
- (C) There is a quiet time of one MSL at system startup. This handles a crash of a long-lived connection and avoids time resynchronization problems in (A).
Notice that (B) and (C) together are logically sufficient to prevent accidental reuse of sequence numbers from a different incarnation, for any of the failure modes F1-F4 🔥 (A) is not logically necessary, since the close delay in (B) makes it impossible to reopen the same TCP connection immediately. However, using (A) does give extra assurance in a common case, maybe making up for a host that set its TIME-WAIT state delay too short.
Some TCP implementations have let a connection in the TIME-WAIT state be reopened immediately by the other side, which short-circuits mechanism (B) 🙅♀️ Specifically, a new SYN for the same socket pair gets accepted while the earlier incarnation is still in TIME-WAIT state. Old duplicates in one direction can be avoided by choosing the ISN to be the next unused sequence number from the preceding connection (i.e., FIN+1). This is basically an application of the Garlick, Rom, and Postel scheme, using the connection block in TIME-WAIT state as the connection record.
However, the connection is still vulnerable to old duplicates in the other direction 😭 Mechanism (A) prevents trouble in mode F1, but failures can still show up in F2, F3, or F4. Of those, F2, on short, fast connections, is the most dangerous.
Finally, TCP will operate reliably without any MSL-based mechanisms in the following restricted domain 🌈
- Total data sent is less than 2**32 octets, and
- Effective sustained rate less than 250KBps, and
- Connection duration less than 4.55 hours.
Right now, the great majority of TCP usage falls into this restricted domain 👑 The third component, connection duration, is the one that gets violated the most.
Security Considerations 🔒
Security issues are not discussed in this memo. 💀 Yes, that's the whole section. Make of it what you will, bestie.
Authors' Addresses 💖
okay bestie, meet the three legends who wrote this 🫶 here's where you can reach each of them 💅
Van Jacobson 🌸
Van Jacobson
University of California
Lawrence Berkeley Laboratory
Mail Stop 46A
Berkeley, CA 94720
Phone: (415) 486-6411
EMail: van@CSAM.LBL.GOVBob Braden 🦋
Bob Braden
University of Southern California
Information Sciences Institute
4676 Admiralty Way
Marina del Rey, CA 90292
Phone: (213) 822-1511
EMail: Braden@ISI.EDULixia Zhang ✨
Lixia Zhang
XEROX Palo Alto Research Center
3333 Coyote Hill Road
Palo Alto, CA 94304
Phone: (415) 494-4415
EMail: lixia@PARC.XEROX.COMheads up 💀: the text I was given for this part only included the Authors' Addresses. The earlier-section recap you sent mentions CONCLUSIONS, REFERENCES, the APPENDIX on protection against old duplicates in TCP, and Security Considerations, but I don't have their actual text, so I can't rewrite them accurately yet. paste that part in and I'll slay it too 🎀
📦 where this page comes from
- ietf: rfc1185 1990-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms