RFC 1337
TIME-WAIT Assassination Hazards in TCP
Title Block 📜
okay bestie, this is RFC 1337 🎀, titled TIME-WAIT Assassination Hazards in TCP ✨. It was written by R. Braden from ISI, and it's dated May 1992 🦋. It's an informational doc, so it provides information for the Internet community and doesn't specify an Internet standard. The header doesn't list anything it obsoletes or updates 💅.
Status of This Memo ✨
this one is giving informational vibes 🌸. The memo provides information for the Internet community. It does not specify an Internet standard, and distribution of it is unlimited 💜.
Abstract 💅
ok so this note is all about some failure modes for TCP connections that are theoretically possible 😭. It talks about ways to fix them, and one super simple fix gets called out specifically 🔥.
1. INTRODUCTION 🎀
so the TCP extensions that were recently proposed in [RFC-1323] got validated through experiments, and those experiments uncovered a brand new class of TCP failures 💀. People gave them the name "TIME-WAIT Assassination hazards" 🙈. This note explains what those hazards are, shows some examples, and talks through ways to prevent them 🫶.
all of these failures come from old duplicate segments 🧚. The TCP mechanisms that guard against old duplicate segments are the following, per [RFC-793] 🌈:
- The 3-way handshake rejects old duplicate initial
<SYN>segments, which means a connection can't get replayed 👑. - Sequence numbers reject old duplicate data and ACK segments from the current incarnation of a given connection. A connection is identified by a particular host and port pair. Sequence numbers also reject old duplicate
<SYN,ACK>segments 💖.
For very high-speed connections, Jacobson's PAWS ("Protect Against Wrapped Sequences") mechanism [RFC-1323] effectively extends the sequence numbers. That way, wrap-around can't cause a hazard within the same incarnation ✨.
- There are two mechanisms for avoiding hazards from old duplicate segments that come from an earlier instance of the same connection. Check the Appendix to [RFC-1185] for the details 🦋.
For "short and slow" connections [RFC-1185], clock-driven ISN (initial sequence number) selection stops the sequence spaces of the old and new incarnations from overlapping [RFC-793] 🍓. (The algorithm Berkeley BSD TCP uses for stepping ISN makes the analysis a little messier, but it doesn't change the conclusions 🙈.)
- TIME-WAIT state gets rid of the old duplicates hazard for "fast" or "long" connections 🔥. With those, clock-driven ISN selection just can't stop the old and new sequence spaces from overlapping 😭. The TIME-WAIT delay gives all the old duplicate segments enough time to die off in the Internet before the connection gets reopened 💅.
- After a system crash, the Quiet Time at system startup lets old duplicates disappear before any connections get opened 🌸.
ok so here's the new observation 👀: (4) is unreliable 💀. TIME-WAIT state can get terminated early, which is what we're calling "assassinated," by an old duplicate data or ACK segment. That segment can come from the current incarnation of the connection or an earlier one 😭. We refer to this as "TIME-WAIT Assassination" (TWA) 👑.
Figure 1 shows an example of TIME-WAIT assassination 🎀. Segments 1-5 are copied exactly from Figure 13 of RFC-793, and they show a normal close handshake 🌈. Packets 5.1, 5.2, and 5.3 extend that sequence to illustrate TWA ✨. Here, 5.1 is any old segment that TCP A doesn't accept. It might get rejected because of its sequence number, or because of an old PAWS timestamp 🦋. Either way, TCP A sends ACK segment 5.2 for its current SND.NXT and RCV.NXT 💖. TCP B has no state for this connection, so it reflects that as RST segment 5.3 💅, and that assassinates the TIME-WAIT state at A 💀!
TCP A TCP B
1. ESTABLISHED ESTABLISHED
(Close)
2. FIN-WAIT-1 --> <SEQ=100><ACK=300><CTL=FIN,ACK> --> CLOSE-WAIT
3. FIN-WAIT-2 <-- <SEQ=300><ACK=101><CTL=ACK> <-- CLOSE-WAIT
(Close)
4. TIME-WAIT <-- <SEQ=300><ACK=101><CTL=FIN,ACK> <-- LAST-ACK
5. TIME-WAIT --> <SEQ=101><ACK=301><CTL=ACK> --> CLOSED
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
5.1. TIME-WAIT <-- <SEQ=255><ACK=33> ... old duplicate
5.2 TIME-WAIT --> <SEQ=101><ACK=301><CTL=ACK> --> ????
5.3 CLOSED <-- <SEQ=301><CTL=RST> <-- ????
(prematurely)
Figure 1. TWA Exampleok real talk, TWA is not at all unlikely 😭. It can happen any time there are duplicate segments that might get delayed in the network 🙈. And PAWS timestamps can't stop it either 💅, because the event can happen within the same tick of the timestamp clock 🌸. TWA comes from TCP's half-open connection discovery mechanism (see pp 33-34 of [RFC-793]) 🧚, which is designed to clean up after a system crash 💖.
2. The TWA Hazards ✨
2.1. Introduction
okay so bestie, if the connection gets reopened right after a TWA event, the new incarnation is exposed to old duplicate segments 😬 (except for the initial <SYN> segment, which the 3-way handshake handles 💅). Three hazards can come out of that:
- H1. Old duplicate data may be accepted erroneously.
- H2. The new connection may be de-synchronized, with the two ends in permanent disagreement on the state. Following the spec of RFC-793, this desynchronization results in an infinite ACK loop. (It might be reasonable to change this aspect of RFC-793 and kill the connection instead.)
- This hazard results from acknowledging something that was not sent. This may result from an old duplicate ACK or as a side-effect of hazard H1.
- H3. The new connection may die.
- A duplicate segment (data or ACK) arriving in SYN-SENT state may kill the new connection after it has apparently opened successfully.
Each of these hazards requires that the sequence space of the new connection overlap to some extent with the sequence space of the previous incarnation. As noted above, this is only possible for "fast" or "long" connections. Since these hazards all require the coincidence of an old duplicate falling into a particular range of new sequence numbers, they are much less probable than TWA itself 🦋.
TWA and the three hazards H1, H2, and H3 have been demonstrated on a stock Sun OS 4.1.1 TCP running in a simulated environment that massively duplicates segments. This environment is far more hazardous than most real TCPs must cope with, and the conditions were carefully tuned to create the necessary conditions for the failures. However, these demonstrations are in effect an existence proof for the hazards 🔥.
We now present example scenarios for each of these hazards. Each scenario is assumed to follow immediately after a TWA event terminated the previous incarnation of the same connection 🧚.
2.2. HAZARD H1: Acceptance of erroneous old duplicate data.
Without the protection of the TIME-WAIT delay, it is possible for erroneous old duplicate data from the earlier incarnation to be accepted. Figure 2 shows precisely how this might happen 👀.
TCP A TCP B
1. ESTABL. --> <SEQ=400><ACK=101><DATA=100><CTL=ACK> --> ESTABL.
2. ESTABL. <-- <SEQ=101><ACK=500><CTL=ACK> <-- ESTABL.
3. (old dupl)...<SEQ=560><ACK=101><DATA=80><CTL=ACK> --> ESTABL.
4. ESTABL. <-- <SEQ=101><ACK=500><CTL=ACK> <-- ESTABL.
5. ESTABL. --> <SEQ=500><ACK=101><DATA=100><CTL=ACK> --> ESTABL.
6. ... <SEQ=101><ACK=640><CTL=ACK> <-- ESTABL.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
7a. ESTABL. --> <SEQ=600><ACK=101><DATA=100><CTL=ACK> --> ESTABL.
8a. ESTABL. <-- <SEQ=101><ACK=640><CTL=ACK> ...
9a. ESTABL. --> <SEQ=700><ACK=101><DATA=100><CTL=ACK> --> ESTABL.
Figure 2: Accepting Erroneous DataThe connection has already been successfully reopened after the assumed TWA event. Segment 1 is a normal data segment and segment 2 is the corresponding ACK segment. Old duplicate data segment 3 from the earlier incarnation happens to fall within the current receive window, resulting in a duplicate ACK segment #4. The erroneous data is queued and "lurks" in the TCP reassembly queue until data segment 5 overlaps it. At that point, either 80 or 40 bytes of erroneous data is delivered to the user B; the choice depends upon the particulars of the reassembly algorithm, which may accept the first or the last duplicate data 🙈.
As a result, B sends segment 6, an ACK for sequence = 640, which is 40 beyond any data sent by A. Assume for the present that this ACK arrives at A after A has sent segment 7a, the next full data segment. In that case, the ACK segment 8a acknowledges data that has been sent, and the error goes undetected. Another possible continuation after segment 6 leads to hazard H3, shown below 💀.
2.3. HAZARD H2: De-synchronized Connection
This hazard may result either as a side effect of H1 or directly from an old duplicate ACK that happens to be acceptable but acknowledges something that has not been sent 🌸.
Referring to Figure 2 above, suppose that the ACK generated by the old duplicate data segment arrived before the next data segment had been sent. The result is an infinite ACK loop, as shown by the following alternate continuation of Figure 2.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
7b. ESTABL. <-- <SEQ=101><ACK=640><CTL=ACK> ...
(ACK something not yet
sent => send ACK)
8b. ESTABL. --> <SEQ=600><ACK101><CTL=ACK> --> ESTABL.
(Below window =>
send ACK)
9b. ESTABL. <-- <SEQ=101><ACK=640><CTL=ACK> <-- ESTABL.
(etc.!)
Figure 3: Infinite ACK loop2.4. HAZARD H3: Connection Failure
An old duplicate ACK segment may lead to an apparent refusal of TCP A's next connection attempt, as illustrated in Figure 4. Here <W=...> indicates the TCP window field SEG.WIND.* 🎀
TCP A TCP B
1. CLOSED LISTEN
2. SYN-SENT --> <SEQ=100><CTL=SYN> --> SYN-RCVD
3. ... <SEQ=400><ACK=101><CTL=SYN,ACK><W=800> <-- SYN-RCVD
4. SYN-SENT <-- <SEQ=300><ACK=123><CTL=ACK> ... (old duplicate)
5. SYN-SENT --> <SEQ=123><CTL=RST> --> LISTEN
6. ESTABLISHED <-- <SEQ=400><ACK=101><CTL=SYN,ACK><W=900> ...
7. ESTABLISHED --> <SEQ=101><ACK=401><CTL=ACK> --> LISTEN
8. CLOSED <-- <SEQ=401><CTL=RST> <-- LISTEN
Figure 4: Connection Failure from Old DuplicateThe key to the failure in Figure 4 is that the RST segment 5 is acceptable to TCP B in SYN-RECEIVED state, because the sequence space of the earlier connection that produced this old duplicate overlaps the new connection space. Thus, <SEQ=123> in segment #5 falls within TCP B's receive window [101,900). In experiments, this failure mode was very easy to demonstrate. (Kurt Matthys has pointed out that this scenario is time-dependent: if TCP A should timeout and retransmit the initial SYN after segment 5 arrives and before segment 6, then the open will complete successfully.) main character energy in the timing 🦋
3. Fixes for TWA Hazards
We discuss three possible fixes to TCP to avoid these hazards 🛠️.
(F1) Ignore RST segments in TIME-WAIT state.
If the 2 minute MSL is enforced, this fix avoids all three hazards ✨.
This is the simplest fix. One could also argue that it is formally the correct thing to do; since allowing time for old duplicate segments to die is one of TIME-WAIT state's functions, the state should not be truncated by a RST segment 👑.
(F2) Use PAWS to avoid the hazards.
Suppose that the TCP ignores RST segments in TIME-WAIT state, but only long enough to guarantee that the timestamp clocks on both ends have ticked. Then the PAWS mechanism [RFC-1323] will prevent old duplicate data segments from interfering with the new incarnation, eliminating hazard H1. For reasons explained below, however, it may not eliminate all old duplicate ACK segments, so hazards H2 and H3 will still exist 😭.
In the language of the TCP Extensions RFC [RFC-1323]:
When processing a RST bit in TIME-WAIT state:
If (Snd.TS.OK is off) or (Time.in.TW.state() >= W)
then enter the CLOSED state, delete the TCB,
drop the RST segment, and return.
else simply drop the RST segment and return.Here "Time.in.TW.state()" is a function returning the elapsed time since TIME-WAIT state was entered, and W is a constant that is at least twice the longest possible period for timestamp clocks, i.e., W = 2 secs [RFC-1323].
This assumes that the timestamp clock at each end continues to advance at a constant rate whether or not there are any open connections. We do not have to consider what happens across a system crash (e.g., the timestamp clock may jump randomly), because of the assumed Quiet Time at system startup.
Once this change is in place, the initial timestamps that occur on the SYN and {SYN,ACK} segments reopening the connection will be larger than any timestamp on a segment from earlier incarnations. As a result, the PAWS mechanism operating in the new connection incarnation will avoid the H1 hazard, ie. acceptance of old duplicate data 🧁.
The effectiveness of fix (F2) in preventing acceptance of old duplicate data segments, i.e., hazard H1, has been demonstrated in the Sun OS TCP mentioned earlier. Unfortunately, these tests revealed a somewhat surprising fact: old duplicate ACKs from the earlier incarnation can still slip past PAWS, so that (F2) will not prevent failures H2 or H3. What happens is that TIME-WAIT state effectively regenerates the timestamp of an old duplicate ACK. That is, when an old duplicate arrives in TIME-WAIT state, an extended TCP will send out its own ACK with a timestamp option containing its CURRENT timestamp clock value. If this happens immediately before the TWA mechanism kills TIME-WAIT state, the result will be a "new old duplicate" segment with a current timestamp that may pass the PAWS test on the reopened connection. 🤯
Whether H2 and H3 are critical depends upon how often they happen and what assumptions the applications make about TCP semantics. In the case of the H3 hazard, merely trying the open again is likely to succeed. Furthermore, many production TCPs have (despite the advice of the researchers who developed TCP) incorporated a "keep-alive" mechanism, which may kill connections unnecessarily. The frequency of occurrence of H2 and H3 may well be much lower than keep-alive failures or transient internet routing failures 🌈.
(F3) Use 64-bit Sequence Numbers
O'Malley and Peterson [RFC-1264] have suggested expansion of the TCP sequence space to 64 bits as an alternative to PAWS for avoiding the hazard of wrapped sequence numbers within the same incarnation. It is worthwhile to inquire whether 64-bit sequence numbers could be used to avoid the TWA hazards as well 🔍.
Using 64 bit sequence numbers would not prevent TWA - the early termination of TIME-WAIT state. However, it appears that a combination of 64-bit sequence numbers with an appropriate modification of the TCP parameters could defeat all of the TWA hazards H1, H2, and H3. The basis for this is explained in an appendix to this memo. In summary, it could be arranged that the same sequence space would be reused only after a very long period of time, so every connection would be "slow" and "short" 🍓.
4. Conclusions 💅✨
okay so of the three fixes from the last section, (F1), ignoring RST segments in TIME-WAIT state, is lowkey the best short-term move 🫶 it's also the simplest, which is so iconic. still, it would be super desirable to run an extended test of this in a production environment, just to make sure ignoring RSTs in TIME-WAIT state doesn't have some unexpected bad effect 🙈
(F2) is more complex and is at best a partial fix 💀 not gonna lie, it doesn't fully solve it. (F3), using 64-bit sequence numbers, would be a big change to the protocol 😭 so its implications need to be thoroughly understood before anyone touches it. it might end up being the long-term fix for the hazards we talked about in this note 👑
APPENDIX: Using 64-bit Sequence Numbers 🎀
okay this appendix is where we justify why 64-bit sequence numbers could prevent the TWA hazards 🧚 let's get into it!!
the theoretical ISN calculation TCP uses is:
ISN = (R*T) mod 2**nwhere T is the real time in seconds (from an arbitrary origin that gets fixed when the system starts), R is a constant that's currently 250 KBps, and n = 32 is the size of the sequence number field 🍓
the limits of current TCP are set by n, R, and the maximum segment lifetime MSL = 4 minutes. the shortest time Twrap to wrap the sequence space is:
Twrap = (2**n)/rwhere r is the maximum transfer rate. to avoid old duplicate segments in the same connection, we require that Twrap > MSL (in practice we need Twrap >> MSL, which is like, a lot bigger 💜)
the clock-driven ISN numbers wrap in time TwrapISN:
TwrapISN = (2**n)/Rfor current TCP, TwrapISN = 4.55 hours ⏳
the cases for old duplicates from previous connections split into four regions along two dimensions:
- slow vs. fast connections, which is
r < Rorr >= R - short vs. long connections, which is duration
E < TwrapISNorE >= TwrapISN
on short slow connections, the clock-driven ISN selection rejects old duplicates ✨ for every other case, the TIME-WAIT delay of 2*MSL is required so old duplicates can expire before they infect a new incarnation 🦠 this is discussed in detail in the Appendix to [RFC-1185].
with that background, let's see what happens when we bump n up to 64 👀 we'd love to increase both R and TwrapISN far enough that every connection is short and slow, so the clock-driven ISN selection rejects all old duplicates. put another way, we want every connection to have its own unique chunk of the sequence space 🌈 for that, we need R larger than the maximum foreseeable rate r, and TwrapISN greater than the longest foreseeable connection duration E.
and honestly? this looks feasible with n = 64 bits ✨ say we use R = 2**33 Bps. that's roughly 8 gigabytes per second, which is a reasonable upper limit on throughput for a single TCP connection 🔥 then TwrapISN = 68 years, which is a reasonable upper limit on TCP connection duration 😭 also, this particular choice of R corresponds to incrementing the ISN by 2**32 every 0.5 seconds, the way the Berkeley BSD implementation of TCP does. so the low-order 32 bits of a 64-bit ISN would always be exactly zero 🫶
References 📚
[RFC-793] Postel, J., "Transmission Control Protocol", RFC-793, USC/Information Sciences Institute, September 1981.
[RFC-1185] Jacobson, V., Braden, R., and Zhang, L., "TCP Extension for High-Speed Paths", RFC-1185, Lawrence Berkeley Labs, USC/Information Sciences Institute, and Xerox Palo Alto Research Center, October 1990.
[RFC-1263] O'Malley, S. and L. Peterson, "TCP Extensions Considered Harmful", RFC-1263, University of Arizona, October 1991.
[RFC-1323] Jacobson, V., Braden, R. and D. Borman "TCP Extensions for High Performance", RFC-1323, Lawrence Berkeley Labs, USC/Information Sciences Institute, and Cray Research, May 1992.
Security Considerations 🔒
security issues are not discussed in this memo 🙈
Author's Address 💌
Bob Braden
University of Southern California
Information Sciences Institute
4676 Admiralty Way
Marina del Rey, CA 90292
Phone: (213) 822-1511
EMail: Braden@ISI.EDU📦 where this page comes from
- ietf: rfc1337 1992-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms