RFC 1379
Extending TCP for Transactions -- Concepts
RFC 1379 🎀 Extending TCP for Transactions -- Concepts, written by R. Braden from ISI, dated November 1992 in the Network Working Group. Its status is informational, so it provides info for the Internet community and is not a standard. It doesn't list anything it obsoletes or updates. This is part 1 of 9 🧚
Status of This Memo
This memo provides information for the Internet community. It does not specify an Internet standard. Distribution of this memo is unlimited. 💌
Abstract
okay so this memo is about extending TCP to give a transaction-oriented service ✨ without messing with its virtual-circuit operation. The gap between connection-oriented TCP and datagram-based UDP is honestly huge 😭, and this extension would fill it, so TCP could efficiently handle lots of the stuff apps currently use UDP for. 🦋 A separate memo has the detailed functional spec for this proposed extension.
This work was supported in part by the National Science Foundation under Grant Number NCR-8922231. 🙏
TABLE OF CONTENTS
1. INTRODUCTION .................................................. 2
2. TRANSACTIONS USING STANDARD TCP ............................... 3
3. BYPASSING THE 3-WAY HANDSHAKE ................................. 6
3.1 Concept of TAO ........................................... 6
3.2 Cache Initialization ..................................... 10
3.3 Accepting <SYN,ACK> Segments ............................. 11
4. SHORTENING TIME-WAIT STATE .................................... 13
5. CHOOSING A MONOTONIC SEQUENCE ................................. 15
5.1 Cached Timestamps ........................................ 16
5.2 Current TCP Sequence Numbers ............................. 18
5.3 64-bit Sequence Numbers .................................. 20
5.4 Connection Counts ........................................ 20
5.5 Conclusions .............................................. 21
6. CONNECTION STATES ............................................. 24
7. CONCLUSIONS AND ACKNOWLEDGMENTS ............................... 32
APPENDIX A: TIME-WAIT STATE AND THE 2-PACKET EXCHANGE ............ 34
REFERENCES ....................................................... 37
Security Considerations .......................................... 38
Author's Address ................................................. 381. INTRODUCTION ✨
The TCP protocol [STD-007] implements a virtual-circuit transport service 🔌 that gives reliable and ordered data delivery over a full-duplex connection. Under the virtual circuit model, the life of a connection is split into three distinct phases: (1) opening the connection to create a full-duplex byte stream; (2) transferring data in one or both directions over this stream; and (3) closing the connection. Remote login and file transfer are examples of apps that totally suit virtual-circuit service. 💅
Distributed applications are getting more numerous and sophisticated on the Internet 🌈, and they tend to be transaction-oriented rather than virtual-circuit style. Right now, a transaction-oriented app has to either eat the overhead of opening and closing TCP connections 😩, or build its own app-specific transport mechanism on top of the connectionless transport protocol UDP. Way more convenience, uniformity, and efficiency would come from widely-available kernel implementations of a transport protocol that supports a transaction service model [RFC-955]. 🍓
The transaction service model has the following features:
- The fundamental interaction is a request followed by a response. 💜
- An explicit open or close phase would impose excessive overhead. 🙄
- At-most-once semantics is required; that is, a transaction must not be "replayed" by a duplicate request packet. 🚫
- In favorable circumstances, a reliable request/response handshake can be performed with exactly one packet in each direction. ✨
- The minimum transaction latency for a client is RTT + SPT, where RTT is the round-trip time and SPT is the server processing time. ⏱️
We use the term "transaction transport protocol" for a transport-layer protocol that follows this model [RFC-955]. 🫶
The Internet architecture allows an arbitrary collection of transport protocols to be defined on top of the minimal end-to-end datagram service provided by IP [Clark88]. In practice though, production systems only implement TCP and UDP at the transport layer 🥲. It has proven difficult to leverage a new transport protocol into place, to be widely available enough to be useful for app builders. Iykyk. 🙃
This memo explores an alternative approach to providing a transaction transport protocol: extending TCP to implement the transaction service model, while still supporting the virtual circuit model. 🧠 Each transaction will then be a single instance of a TCP connection. The proposed transaction extension is effectively implementable within current TCPs and operating systems, and it should also scale to the much faster networks, interfaces, and CPUs of the future. Main character energy, fr fr. 👑
This memo explains the theory behind the extension, in somewhat exquisite detail 🤓. Despite the length and complexity of this memo, the TCP extensions required for transactions are in fact quite limited and simple. Another memo [TTCP-FS] provides a self-contained functional specification of the extensions. 📄
Section 2 of this memo describes the limitations of standard TCP for transaction processing, to motivate the extensions. Sections 3, 4, and 5 explore the fundamental extensions that are required for transactions. Section 6 discusses the changes required in the TCP connection state diagram. Finally, Section 7 presents conclusions and acknowledgments. Familiarity with the standard TCP protocol [STD-007] is assumed. 🌸
2. TRANSACTIONS USING STANDARD TCP ✨
okay bestie, so reliable data transfer depends on sequence numbers 🔢 Before any data can flow, both parties have to "synchronize" the connection, which means agreeing on common sequence numbers. That sync procedure must preserve at-most-once semantics, so it has to be free from replay hazards caused by duplicate packets. The TCP developers picked a mechanism called the 3-way handshake 🤝
Picture a simple transaction where client host A sends a single-segment request to server host B, and B sends back a single-segment response 💅 Many current TCP implementations use at least ten segments (aka packets) for this, and that's giving a lot: three for the 3-way handshake that opens the connection, four to send and acknowledge the request and response data, and three for TCP's full-duplex, data-conserving close sequence. Those ten segments are a high relative overhead for two data-bearing segments 😩 But the bigger problem is the transaction latency the client sees, which is 2*RTT + SPT. That's larger than the minimum by one RTT. As CPU and network speeds go up, this extra latency matters more and more 🚀
Proposed transaction transport protocols have usually taken a "timer-based" approach to connection synchronization [Birrell84]. In this approach, once end-to-end connection state is established in the client and server hosts, a subset of that state is kept around for some period of time. A new request that arrives before that timeout expires can rebuild the full state without an explicit handshake ⏰ Watson pointed out that the timer-based approach of his Delta-T protocol [Watson81] would cover both virtual circuits and transactions. However, the TCP group went with the 3-way handshake, because they were uncertain about how robustly the packet lifetime bounds required by Delta-T could be enforced within a general Internet environment 🤔 More recently, Liskov, Shrira, and Wroclawski [Liskov90] have proposed a different timer-based approach to connection synchronization, one that requires loosely-synchronized clocks in the hosts.
The technique proposed in this memo, suggested by Clark [Clark89], depends on caching connection state but not on clocks or timers 🧠 It's described in Section 3 below. Garlick, Rom, and Postel also proposed a connection synchronization mechanism using cached state [Garlick77]. Their scheme required each host to maintain connection records containing the highest sequence number on each connection. The technique here only retains per-host state, not per-connection state 🎀
During TCP development, someone suggested that TCP could support transactions with data segments containing both SYN and FIN bits. (These "Kamikaze" segments were not supported as a service. They were mainly used to crash other experimental TCPs! 💀) To illustrate the idea, Figure 1 shows a plausible application of the current TCP rules to create a minimal transaction. (Fun fact, some minor adjustments to the standard TCP spec would be needed to make Figure 1 fully legal [STD-007].) 🙈
Figure 1, like many examples in this memo, uses an abbreviated form to show segment sequences. For clarity and brevity, it leaves out explicit sequence and acknowledgment numbers, assuming they follow the well-known TCP rules. The notation "ACK(x)" means a cumulative acknowledgment for the control bit or data "x" and everything before "x" in the sequence space. The referent of "x" should be clear from context. Also, host A is always the client and host B is always the server in these diagrams 🧚
The first three segments in Figure 1 implement the standard TCP 3-way handshake. If segment #1 had been an old duplicate, the client side would have sent an RST (Reset) bit in segment #3, ending the sequence 🛑 The request data included on the initial SYN segment can't be delivered to user B until segment #3 completes the 3-way handshake. Loading control bits onto the segments cuts the total down to 5 segments, but the client still sees a transaction latency of 2*RTT + SPT. So the 3-way handshake still blocks high-performance transaction processing 😤
TCP A (Client) TCP B (Server)
_______________ ______________
CLOSED LISTEN
(Client sends request)
1. SYN-SENT --> <SYN,data1,FIN> --> SYN-RCVD
(data1 queued)
2. ESTABLISHED <-- <SYN,ACK(SYN)> <-- SYN-RCVD
3. FIN-WAIT-1 --> <ACK(SYN),FIN> --> CLOSE-WAIT
(data1 to server)
(Server sends reply)
4. TIME-WAIT <-- <ACK(FIN),data2,FIN> <-- LAST-ACK
(data2 to client)
5. TIME-WAIT --> <ACK(FIN)> --> CLOSED
(timeout)
CLOSED
Figure 1: Transaction Sequence: RFC-793 TCPThe TCP close sequence is also a performance problem for transactions 😭 One or both ends of a closed connection must stay in the "TIME-WAIT" state until a 4 minute timeout has expired [STD-007]. The same connection (defined by the host and port numbers at both ends) can't be reopened until that delay is up. Because of TIME-WAIT, a client program should pick a new local port number (i.e., a different connection) for each successive transaction. The catch is that the TCP port field of 16 bits (minus the "well-known" port space) gives only 64512 available user ports. That limits the total transaction rate between any pair of hosts to a max of 64512/240 = 268 per second. That's way too low for low-delay paths, like high-speed LANs 🥲 A high rate of short connections (aka transactions) could also lead to excessive consumption of kernel memory by connection control blocks in TIME-WAIT state.
In summary: to get efficient transaction processing in TCP, we need to suppress the 3-way handshake and shorten TIME-WAIT state. 🎯
Protocol mechanisms for these two goals are discussed in Sections 3 and 4, respectively. Both require choosing a monotonic sequence-like space, and Section 5 analyzes the options and makes a selection. Finally, the TCP connection state machine must be extended as described in Section 6 👑
Transaction processing in TCP raises some other protocol issues, which are covered in the functional specification memo [TTCP-FS]. These include:
- (1) augmenting the user interface for transactions,
- (2) delaying acknowledgment segments to allow maximum piggy-backing of control bits with data,
- (3) measuring the retransmission timeout time (RTO) on very short connections, and
- (4) providing an initial server window.
A recently proposed set of enhancements [RFC-1323] defines a TCP Timestamps option that carries two 32-bit timestamp values ⏱️ The Timestamps option is used to accurately measure round-trip time (RTT). That same option also powers a procedure called "PAWS" (Protect Againsts Wrapped Sequence), which prevents erroneous data delivery from a combo of old duplicate segments and sequence number reuse at very high bandwidths 🌈 The transaction approach chosen in this memo doesn't require the RFC-1323 enhancements. But they're important and should be implemented in every TCP, with or without the transaction extensions described here ✨
3. BYPASSING THE 3-WAY HANDSHAKE 🦋
To avoid 3-way handshakes for transactions, we introduce a new mechanism for validating initial SYN segments. In other words, it enforces at-most-once semantics without a 3-way handshake 🙌 We call this the TCP Accelerated Open, or TAO, mechanism.
3.1 Concept of TAO
The basis of TAO is this: a TCP uses cached per-host information to immediately validate new SYNs [Clark89]. If that validation fails, e.g., because there's no current cached state or the segment is an old duplicate, the procedure falls back to a normal 3-way handshake to validate the SYN. So bypassing the 3-way handshake is considered an optional optimization 💖
The proposed TAO mechanism uses a finite sequence-like space of values that increase monotonically with successive transactions (connections) between a given (client, server) host pair. Call this monotonic space M, and let each initial SYN segment carry an M value SEG.M. If M is not the existing sequence (SEG.SEQ) field, SEG.M may be carried in a TCP option 🧩
When host B receives from host A an initial SYN segment containing a new value SEG.M, host B compares it against cache.M[A], the latest M value B has cached for host A. This comparison is the "TAO test" 🔍 Because the M values increase monotonically, SEG.M > cache.M[A] means the SYN must be new and can be accepted immediately. If not, a normal 3-way handshake is performed to validate the initial SYN segment. Figure 2 illustrates the TAO mechanism, with cached M values shown in square brackets. The M values generated by host A satisfy x0 < x1, and the M values generated by host B satisfy y0 < y1.
An appropriate choice for the M value space is discussed in Section 5. M values are drawn from a finite number space, so inequalities must be defined in the usual way for sequence numbers [STD-007]. The M space must not wrap so quickly that an old duplicate SYN gets erroneously accepted. We assume that some maximum segment lifetime (MSL) is enforced by the IP layer 🕰️
____T_C_P__A_____ ____T_C_P__B_____
cache.M[B] cache.M[A]
V V
[ y0 ] [ x0 ]
1. --> <SYN,data1,M=x1> --> ( (x1 > x0) =>
data1 -> user_B;
cache.M[A]= x1)
[ y0 ] [ x1 ]
2. <-- <SYN,ACK(data1),data2,M=y1> <--
(data2 -> user_A,
cache.M[B]= y1)
[ y1 ] [ x1 ]
... (etc.) ...
Figure 2. TAO: Three-Way Handshake is BypassedFigure 2 shows the simplest case: each side has cached the latest M value of the other, and the SEG.M value in the client's SYN segment is greater than the value in the cache at the server host 🎉 So B can accept client A's request data1 immediately and pass it to the server application. B's reply data2 rides piggybacked on the <SYN,ACK> segment. After this 2-way exchange, the cached M values get updated at both sites. The client side only matters if the client/server roles reverse 🔄 Validation of the <SYN,ACK> segment at host A is discussed later.
Figure 3 shows the TAO test failing, but the consequent 3-way handshake succeeding. B updates its cache with the value x2 >= x1 once the initial SYN is known to be valid 💜
_T_C_P__A _T_C_P__BOkay so, the TAO test can fail on a perfectly legit new SYN, and that's not the end of the world 💖 There are a few reasons it can happen:
- There may be no cached M value for this particular client host.
- The SYN may be the one of a set of nearly-simultaneous SYNs for different connections but from the same host, which arrived out of order.
- The finite M space may have wrapped around between successive transactions from the same client.
- The M values may advance too slowly for closely-spaced transactions.
None of these TAO failures will cause a lockout, because the resulting 3-way handshake will succeed 🫶 The first transaction between a given host pair will always require a 3-way handshake, but subsequent transactions can take advantage of TAO 🦋
The per-host cache that TAO needs is honestly so iconic, because it's useful for other stuff too, like keeping the measured round trip time and MTU for a given remote host ✨ Also, a host should already have a per-host routing cache [HR-COMM] that should be pretty easy to extend for this purpose 🧚
Figure 4 shows a complete TCP transaction sequence using the TAO mechanism 💅 Bypassing the 3-way handshake leads to new connection states, and this figure shows three of them: "SYN-SENT*", "CLOSE-WAIT*", and "LAST-ACK*". We'll get to what these states mean in Section 6, so hold that thought 🌸
TCP A (Client) TCP B (Server)
_______________ ______________
CLOSED LISTEN
1. SYN-SENT* --> <SYN,data1,FIN,M=x1> --> CLOSE-WAIT*
(TAO test OK=>
data1->user_B)
<-- <SYN,ACK(FIN),data2,FIN,M=y1> <-- LAST-ACK*
2. TIME-WAIT
(data2->user_A)
3. TIME-WAIT --> <ACK(FIN),M=x2> --> CLOSED
(timeout)
CLOSEDFigure 4: Minimal Transaction Sequence Using TAO
3.2 Cache Initialization
Okay, so the very first connection between hosts A and B won't find any cached state at one or both ends 🥺 That means both M caches must be initialized. To do that, the first transaction has to carry a specially marked SEG.M value, which they call SEG.M.NEW 🎀 When B gets a SEG.M.NEW value in an initial SYN segment, it caches that value and sends its own M back to initialize A's cache. And when a host crashes and restarts, all its cached M values (cache.M[*]) must be invalidated so that both ends are forced to re-sync their caches 💀
Here's how that cache sync works, shown in Figure 5. Client host A crashed and restarted with its cache entries undefined, shown as "??". Since cache.TS[B] is undefined, A sends a SEG.M.NEW value instead of SEG.M in the <SYN> segment of its first transaction request to B. When B gets that SEG.M.NEW, it invalidates cache.TS[A] and does a 3-way handshake 🙈 SEG.M in segment #2 updates A's cache, and once the handshake completes successfully, B updates its cached M value to x2 >= x1 👑
_T_C_P__A _T_C_P__B
cache.M[B] cache.M[A]
V V
[ ?? ] [ x0 ]
1. --> <SYN,data1,M.NEW=x1> --> (invalidate cache;
queue data1;
[ ?? ] 3-way handshake)
[ ?? ]
2. <-- <SYN,ACK(SYN),M=y1> <--
(cache.M[B]= y1)
[ y1 ] [ ?? ]
3. --> <ACK(SYN),M=x2> --> data1->user_B,
cache.M[A]= x2)
[ y1 ] [ x2 ]
... (etc.) ...Figure 5. Client Host Crashed
Now, suppose the 3-way handshake failed, presumably because segment #1 was an old duplicate. Then segment #3 from host A would be an RST segment, and the result is that both sides' caches would be left undefined 😭
Figure 6 shows what happens when the server crashes and restarts 🧚 When B receives a <SYN> segment from a host it has no cached M value for, it starts a 3-way handshake to validate the request and sends its own M value to A. Once again, the result is that the cached M values get updated on both sides ✨
_T_C_P__A _T_C_P__B
cache.M[B] cache.M[A]
V V
[ y0 ] [ ?? ]
1. --> <SYN,data1,M=x1> --> (data1 queued;
3-way handshake)
[ y0 ] [ ?? ]
2. <-- <SYN,ACK(SYN),M=y1> <--
(cache.M[B]= y1)
[ y1 ] [ ?? ]
3. --> <ACK(SYN),M=x2> --> (data1->user_B,
cache.M[A]= x2)
[ y1 ] [ x2 ]
... (etc.) ...Figure 6. Server Host Crashed
3.3 Accepting <SYN,ACK> Segments
Okay, this one is a real plot twist 🌈 Transactions introduce a new hazard: erroneously accepting an old duplicate <SYN,ACK> segment. To be acceptable, a <SYN,ACK> segment must arrive in SYN-SENT state, and its ACK field must acknowledge something that was sent. In current TCPs, the effective send window in SYN-SENT state is exactly one octet, and an acceptable <SYN,ACK> must ACK this one octet exactly. The clock-driven selection of Initial Sequence Number (ISN) makes an erroneous acceptance exceedingly unlikely. An old duplicate SYN could only be accepted erroneously if successive connection attempts occurred more often than once every 4 microseconds, or if the segment lifetime exceeded the 4 hour wraparound time for ISN selection 🙈
ngl, this is where it gets spicy. When TCP is used for transactions, data sent with the initial SYN increases the range of sequence numbers that have been sent. That raises the danger of accepting an old duplicate <SYN,ACK> segment, and the consequences are way more serious 😱
In the example in Figure 7, segments 1-3 form a normal transaction sequence, and segment 4 begins a new transaction (incarnation) for the same connection. Segment #5 is a duplicate of segment #2 from the preceding transaction. Even though the new transaction has a larger ISN, the previous ACK value 402 falls into the new range [200,700) of sequence numbers that have been sent. So segment #5 could be erroneously accepted and passed to the client as the response to the new request 💀
_T_C_P__A _T_C_P__B
CLOSED LISTEN
1. --> <seq=100,SYN,data=300,FIN,M=x1> --> (TAO test OK)
2. <-- <seq=800,ack=402,SYN,data=350,FIN,M=y1> <--
3. TIME-WAIT --> <ACK(FIN)> --> CLOSED
(short timeout)
CLOSED
(New Request)
4. --> <seq=200,SYN,data=500,FIN,M=x2> --> ...
(Duplicate of segment #2)
5. <-- <seq=800,ack=402,SYN,data=300,FIN,M=y1> <--...
(Acceptable!!)Figure 7: Old Duplicate <SYN,ACK> Causing Error
Unfortunately, we can't just use TAO on the client side to detect and reject old duplicate <SYN,ACK> segments 😩 A TAO test at the client might fail for a valid <SYN,ACK> segment because of out-of-order delivery, and that could result in permanent non-delivery of a valid transaction reply.
So instead, we include a second M value in the <SYN,ACK> segment: an echo of the client's M value from the initial <SYN> segment. A specially-marked M value, SEG.M.ECHO, is used for this purpose 🎀 The client knows the value it sent in the initial <SYN>, so it can positively validate the <SYN,ACK> using the echoed value. This is illustrated in Figure 12, which is the same as Figure 4 with the echoed value added to the <SYN,ACK> segment #2 💖
It should be noted that TCP allows a simultaneous open sequence in which both sides send and receive an initial <SYN> (see Figure 8 of [STD-007]. In this case, the TAO test must be performed on both sides to preserve the symmetry. See [TTCP-FS] for an example 🫧
4. SHORTENING TIME-WAIT STATE ✨
okay bestie, so once a transaction has been started for a connection (that's a pair of ports between a given pair of hosts), a new transaction on that same connection can't happen until at least this much time has passed 🕰️:
RTT + SPT + TIME-WAIT_delaySince the client host can cycle through the 64512 available port numbers, there's an upper bound on how many transactions per second can happen between a particular host pair 🦋:
[1] TRmax = 64512 /(RTT + TIME-WAIT_Delay)That's in transactions per second (Tps), and we assumed SPT is negligible 💅. We must reduce TIME-WAIT_Delay to support high-rate TCP transaction processing. It's giving urgent energy 😭.
TIME-WAIT state does two jobs 🎀. Job one is supporting the full-duplex reliable close of TCP. Job two is letting old duplicate segments from an earlier connection incarnation expire before they can cause an error (see Appendix to [RFC-1185]). Job one affects the application model of a TCP connection, which we would not want to change. Job two is part of the fundamental machinery of TCP reliable delivery, so to safely truncate TIME-WAIT state, we must provide another way to exclude duplicate packets from earlier incarnations of the connection 🧚.
To keep the delay in TIME-WAIT state small while still doing both jobs, we propose setting the TIME-WAIT delay to:
[2] TIME-WAIT_Delay = max( K*RTO, U )Here U and K are constants, and RTO is the dynamically-determined retransmission timeout, which is the measured RTT plus an allowance for the RTT variance [Jacobson88] 🌸. We choose K large enough that there's high probability of the close completing successfully, if at all possible. K = 8 seems reasonable 🍓. This takes care of job one of TIME-WAIT state, bestie 💖.
In a real implementation there may be a minimum RTO value Tr, which matches the precision of the RTO calculation 🤓. For example, in the popular BSD implementation of TCP, the minimum RTO is Tr = 0.5 second. Assuming K = 8 and U = 0, Eqns [1] and [2] put an upper limit of TRmax = 16K Tps on the transaction rate of these implementations. Not iconic, honestly 💀.
It's only possible to have many short connections if RTO is very small. In that case the TIME-WAIT delay [2] shrinks down to U ✨. To speed up the close sequence, we need to reduce U below the MSL enforced by the IP layer, without introducing a hazard from old duplicate segments. For this purpose, we introduce another monotonic number sequence, which we'll call X 🔥. X values are required to be monotonic between successive connection incarnations. Depending on the choice of the X space (see Section 5), X values may also increase during a connection. Every segment carries a value from the X space, and a segment gets rejected if it arrives with an X value smaller than the largest X value received. This mechanism doesn't use a cache. Instead, the largest X value is kept in the TCP connection control block (TCB) for each connection 👑.
The value of U depends on which X space you pick, which we'll cover in the next section 🌈. If X is time-like, U can be set to twice the time granularity (i.e., twice the minimum "tick" time) of X. The TIME-WAIT delay will then make sure current X values don't overlap the X values of earlier incarnations of the same connection 🙈. Another consequence of time-like X values is that an open but idle connection might let the X value wrap its sign bit, which would lock up the connection. To prevent this, a 24-day idle timer on each open connection could bypass the X check on the first segment after the idle period, for example. In practice though, many implementations have keep-alive mechanisms that prevent such long idle periods [RFC-1323] 🫶.
Looking back at Figure 4, our proposed transaction extension results in a minimum exchange of 3 packets 🎀. Segment #3, the final ACK segment, doesn't increase transaction latency. But combined with the TIME-WAIT delay of K*RTO, it makes sure the server side of the connection is closed before a new transaction is issued for this same pair of ports. It also gives an RTT measurement for the server 📊.
We may ask whether it's possible to reduce the TIME-WAIT delay even further. We could set K to zero. Or we could let the client TCP start a new transaction request while the connection is still in TIME-WAIT state, with the new initial SYN acting as an implied acknowledgment of the previous FIN 😳. Appendix A summarizes the issues raised by these alternatives, which we call "truncating" TIME-WAIT state, and suggests some possible solutions. Further study would be required, but these solutions appear to bend the theory and/or implementations of the TCP protocol farther than we wish to bend them 🙅♀️.
So we propose using formula [2] with K=8 and keeping the final ACK(FIN) transmission. To raise the transaction rate, therefore, we require small values of RTO and U 🚀.
5. CHOOSING A MONOTONIC SEQUENCE 🦄
For simplicity, we want the monotonic sequence X used for shortening TIME-WAIT state to be identical to the monotonic sequence M used for bypassing the 3-way handshake. We'll call the shared space M, and we'll send an M value SEG.M in each TCP segment 💜. When an initial SYN segment comes in, SEG.M gets compared with a per-host cached value to authenticate the SYN without a 3-way handshake. This is the TAO mechanism 🌷. When a non-SYN segment comes in, SEG.M gets compared with the current value in the connection control block and used to discard old duplicates.
Note that TIME-WAIT state is different from bypassing 3-way handshakes in two ways 👀. First, TIME-WAIT requires duplicate detection on every segment, versus only on SYN segments. Second, TIME-WAIT applies to a single connection, versus being global across all connections. This section talks through the possible choices for the common monotonic sequence 🧠.
The SEG.M values have to satisfy these requirements:
- The values must be monotonic. This requirement is defined more precisely below.
- Their granularity must be fine-grained enough to support a high rate of transaction processing. The M clock must "tick" at least once between successive transactions.
- Their range (wrap-around time) must be great enough to allow a realistic MSL to be enforced by the network.
The TCP spec calls for an MSL of 120 secs. Since much of the Internet doesn't carefully enforce this limit, it would be safer to have an MSL at least an order of magnitude larger. Our objective is an MSL of at least 2000 seconds 🎯. If there were no TIME-WAIT delay, the ultimate limit on transaction rate would be set by speed-of-light delays in the network and by the latency of host operating systems. As the bottleneck problems with interfacing CPUs to gigabit LANs get solved, we can imagine transaction durations as short as 1 microsecond ⚡. So we set an ultimate performance goal of TRmax at least 10**6 Tps. Iconic goal, no notes 👑.
A particular connection between hosts A and B is identified by the local and remote TCP "sockets", meaning by the quadruplet {A, B, Port.A, Port.B} 🔑. Imagine that each host keeps a count CC of the number of TCP connections it has initiated. We can use this CC number to tell different incarnations of the same connection apart 🪞. Then a particular SEG.M value can be labeled implicitly by 6 quantities: {A, B, Port.A, Port.B, CC, n}, where n is the byte offset of that segment within the connection incarnation 🧩.
To bypass the 3-way handshake, we require thgt SEG.M values on successive SYN segments from a host A to a host B to be monotone increasing. If CC' > CC, then we require that:
SEG.M(A,B,Port.A,Port.B,CC',0) > SEG.M(A,B,Port.A,Port.B,CC,0)and this has to hold for any legal values of Port.A and Port.B 🎀.
To delete old duplicates (which lets TIME-WAIT state get shortened), we require that SEG.M values be disjoint across different incarnations of the same connection. If CC' > CC then:
SEG.M(A,B,Port.A,Port.B,CC',n') > SEG.M(A,B,Port.A,Port.B,CC,n),for any non-negative integers n and n' 🙌.
We now consider four different choices for the common monotonic space: RFC-1323 timestamps, TCP sequence numbers, the connection count, and 64-bit TCP sequence numbers 🌈. The results are summarized in Table I 📋.
5.1 Cached Timestamps
The PAWS mechanism [RFC-1323] uses TCP "timestamps" as monotonically increasing integers to throw out old duplicate segments within the same incarnation 🕰️. Jacobson suggested caching these timestamps for bypassing 3-way handshakes [Jacobson90], meaning TCP timestamps would be used for our common monotonic space M. This idea is attractive because the same timestamp options could be used for RTTM, PAWS, and transactions 💅.
To get at-most-once service, the criterion for immediately accepting a SYN must be that SEG.M is strictly greater than the cached M value. So, to be useful for bypassing 3-way handshakes, the timestamp clock must tick at least once between any two successive transactions between the same pair of hosts (even if different ports are used). That means the timestamp clock rate would determine TRmax, the maximum possible transaction rate 🐢.
Unfortunately, the timestamp clock frequency called for by RFC-1323, somewhere in the range of 1 sec to 1 ms, is much too slow for transactions 😩. The TCP timestamp period was chosen to be comparable to the fundamental interval for computing and scheduling retransmission timeouts. That's generally in the range of 1 sec to 1 ms, and in many operating systems it's much closer to 1 second. Sure, it would be possible to increase the timestamp clock frequency by several orders of magnitude. But doing so would make implementation harder, and on some systems excessively expensive 💸.
The wraparound time for TCP timestamps, at least 24 days, causes no problem for transactions 👍.
The PAWS mechanism uses TCP timestamps to protect against old duplicate non-SYN segments from the same incarnation [RFC-1323]. It can also protect against old duplicate data segments from earlier incarnations (and therefore allow shortening of TIME-WAIT state) if we can make sure the timestamp clock ticks at least once between the end of one incarnation and the beginning of the next. This can be achieved by setting U = 2 seconds, which is twice the maximum timestamp clock period. Plugging this value into formula [2] leads to an upper bound of TRmax = 32K Tps between a host pair 🌟. However, as pointed out above, old duplicate SYN detection using timestamps leads to a smaller transaction rate bound, 1 Tps, which is unacceptable 🙅♀️. On top of that, the timestamp approach is imperfect. It allows old ACK segments to enter the new connection, where they can cause a disconnect. This happens because old duplicate ACKs that arrive during TIME-WAIT state generate new ACKs with the current timestamp [RFC-1337] 😱.
So we conclude that timestamps aren't adequate as the monotonic space M, see Table I. They may still be useful for effectively extending some other monotonic number space, just like they're used in PAWS to extend the TCP sequence number space. This is discussed below 🦋.
5.2 Current TCP Sequence Numbers
It's useful to understand why the existing 32-bit TCP sequence numbers don't form an appropriate monotonic space for transactions 🧐.
okay so the sequence number in an initial SYN is called the Initial Sequence Number, or ISN 🌸 and according to the TCP spec, you pick it with this formula:
[3] ISN = (R*T) mod 2**32where T is the real time in seconds (counted from some arbitrary starting point that gets fixed when the system boots 🕰️) and R is a constant, currently 250 KBps. these ISN values make a monotonic time sequence that wraps every 4.55 hours = 16380 seconds, with a granularity of 4 usecs ✨ for transaction rates up to roughly 250K Tps, the ISN from formula [3] stays monotonic, so it could be used to bypass the 3-way handshake 🦋
but TCP sequence numbers on their own can't shorten TIME-WAIT 💀 because the sequence space of one connection can overlap with the next one in a few different ways (see the Appendix to [RFC-1185] 🙈). one way is a "fast connection" that transfers faster than R. another is a "long" connection that lasts about 4.55 hours. TIME-WAIT delay is what protects us from those cases 🫶 with the official 240 second delay, formula [1] gives an upper bound (as RTT -> 0) of TRmax = 268 Tps. with our target MSL of 2000 sec, TRmax = 32 Tps. ngl those numbers are way too low 😭
so to boost the transaction rate, we could use TCP timestamps to basically stretch the range of the TCP sequence numbers 💅 timestamps would guard against sequence number wrap-around, which means we could crank R in [3] past the max possible transfer rate. then the sequence numbers of successive incarnations could never overlap 👑 timestamps would also keep things safe with an MSL as big as 24 days. then we could set U = 0 in the TIME-WAIT delay calc [2]. for example, R = 109 Bps gives TRmax <= 109 Tps. see 2(b) in Table I 🎀 these numbers would more than hit our goals, periodt ✨
we should be clear about how this proposal (sequence numbers plus timestamps) is different from the timestamps-alone idea we talked about (and rejected) in the last section 🧚 the difference is what gets cached and tested for TAO. the move here is to cache and test BOTH the latest TCP sequence number AND the latest TCP timestamp 💜 basically we're using timestamps to logically extend the sequence space to 64 bits. the next section has another option, which is to directly expand the TCP sequence space to 64 bits 🌈
ngl the catch: sequence numbers plus timestamps built on equation [3] would be really hard or impossible to implement on lots of systems. they build TCP on a super low granularity software clock, usually O(1 sec) 😩
so to make it work with a low-granularity software clock, we calculate the ISN like this:
[4] ISN = ( R*Ts*floor(T/Ts) + q*CC) mod 2**32where Ts is the time per tick of the software clock, CC is the connection count, and q is a constant. basically the ISN goes up by the constant R*Ts once every clock tick, and by the constant q for every new connection 🍓 we need to pick q to get the monotonicity we want.
for monotonicity of the ISNs themselves, q=1 is enough 🥰 but for monotonicity across the entire connection, we need q = R*Ts. here's how we figure that out. let S(T, CC, n) be the sequence number for byte offset n in a connection with number CC at time T:
S(T, CC, n) = (R*Ts*floor(T/Ts) + q*CC + n) mod 2**32.for any T1 > T2, we need: S(T2, CC+1, 0) - S(T1, CC, n) > 0 for all n. since R is assumed to be an upper bound on the transfer rate, we can write:
R > n/(T2 - T1), or T2/Ts - T1/Ts > n/(R*Ts)using the fact that floor(x)-floor(y) > x-y-1, plus a little algebra, q = R*Ts gives the required monotonic number sequence 🧠 so we go with:
[5] ISN = R*Ts*(floor(T/Ts) + CC) mod 2**32(this is the algorithm BSD TCP uses for ISN selection 🔥)
for error-free operation, the sequence numbers from [5] can't wrap the sign bit in less than MSL seconds. since CC can't go up faster than TRmax, the safe condition is:
R* (1 + Ts*TRmax) * MSL < 2**31.we care about the case where Ts*TRmax >> 1, so this simplifies to:
[6] R * Ts * TRmax * MSL < 2**31.this shows a direct trade-off between the max effective bandwidth R, the max transaction rate TRmax, and the max segment lifetime MSL 😵 for reasonable limits on R, Ts, and MSL, formula [6] gives a very low TRmax. for example, with MSL= 2000 secs, R=109 Bps, and Ts = 0.5 sec, TRmax < 2*10-3 Tps 💀
to help with that, we could add timestamps to the sequence numbers. this would allow an effective MSL of 2 seconds in [6], since longer times would be protected by timestamps that differ. then TRmax < 230/(R*Ts). the actually enforced MSL would go up to 24 days 🌸 unfortunately TRmax would still be too small, since we want to support transfer rates up to R ~ 109 Bps. Ts = 0.5 sec would mean TRmax ~ 2 Tps. on lots of systems it looks infeasible to shrink Ts enough to get an acceptable TRmax this way 🙈
5.3 64-bit TCP Sequence Numbers ✨
another option is to just make the TCP sequence space 64 bits, like [RFC-1263] suggests 🦋 we'd also raise the R value for clock-driven ISN selection, past the fastest transfer rate the host can actually do. a reasonable upper limit might be R = 10**9 Bps. like we said above, in a real implementation we'd use:
ISN = R*Ts*( floor(T/Ts) + CC) mod 2**64which leads to:
R*(1 + Ts * TRmax) * MSL < 2**63for example, say R = 109 Bps, Ts = 0.5, and MSL = 16K secs (4.4 hrs). then this says TRmax < 106 Tps 💜 so adding 32 bits to the sequence space gives us feasible values for transaction processing, iykyk 🎀
5.4 Connection Counts 🌈
the Connection Count CC is a really good fit for the monotonic sequence M 💅 it "ticks" exactly once for each new connection incarnation and stays constant inside a single incarnation. so it cleanly separates segments from different incarnations of the same connection, and that would let us use U = 0 in the TIME-WAIT delay formula [2]. (strictly speaking, U can't go below 1/R = 4 usec, as noted in Section 4. but that barely matters in practice until we hit the ultimate limits on TRmax 🙈)
assume CC is a 32-bit number. to keep CC's sign bit from wrapping in less than MSL seconds, we need:
TRmax * MSL < 2**31for example, if MSL = 2000 seconds then TRmax < 10**6 Tp. these are acceptable limits for transaction processing ✨ but if they aren't good enough, we could augment CC with TCP timestamps for way further-out limits, as discussed below 🚀
it's up to the client whether CC is global for all destinations or private to each destination host (kept in the per-host cache) 🫶 if it's private, the last CC value assigned for each remote host can also live in the per-host cache. since a host doesn't usually have a ton of parallelism in its network connections, there should be little difference in performance between these two approaches, and a single global CC value is honestly simpler 💖
to augment CC with TCP timestamps, we bypass the 3-way handshake if both SEG.CC > cache.CC[A] and SEG.TSval >= cache.TS[A]. the timestamp check catches any SYN older than 2 seconds, so the effective wrap-around requirement becomes:
TRmax * 2 < 2**31i.e., TRmax < 109 Tps 🔥 the required MSL would go up to 24 days. using timestamps like this, we could shrink CC. for example, say CC is 16 bits. then the wrap-around condition TRmax * 2 < 215 means TRmax is 16K 🍓
finally, using CC to delete old duplicates from earlier incarnations doesn't get rid of the need for the timestamp-based PAWS mechanism. PAWS is still needed to prevent errors inside a single incarnation caused by wrapping the 32-bit TCP sequence space at very high transfer rates 🧚
5.5 Conclusions 👑
the monotonic sequence options are summarized in Table I 💅 we can see there are two feasible choices for the monotonic space: the connection count and 64-bit sequence numbers. of those two, we think the connection count is simpler 🌸
implementing 64-bit sequence numbers would need negotiation of a new header format and expansion of all the variables and calculations on the sequence space 😵💫 CC can be carried in an option and only needs to be examined once per packet ✨
so we propose a simple 32-bit connection count CC, without augmenting it with timestamps, for the transaction extension 🎀 this choice is simple and direct. the drawback is that it adds a third sequence-like space (on top of the TCP sequence number and the TCP timestamp) to each TCP header and to the main line of packet processing. but the extra code is honestly very modest 💖
we now have a general outline of the proposed TCP extensions for transactions 🦋
- A host maintains a 32-bit global connection counter variable CC.
- The sender's current CC value is carried in an option in every TCP segment.
- CC values are cached per host, and the TAO mechanism is used to bypass the 3-way handshake when possible.
- In non-SYN segments, the CC value is used to reject duplicates from earlier incarnations. This allows TIME-WAIT state delay to be reduced to K*RTO (i.e., U=0 in Eq. [2]).
TABLE I: Summary of Monotonic Sequences
APPROACH TRmax (Tps) Required MSL COMMENTS
__________________________________________________________________
1. Timestamp & PAWS 1 24 days TRmax is
too small
__________________________________________________________________
2. Current TCP Sequence Numbers
(a) clock-driven
ISN: eq. [3] 268 240 secs TRmax & MSL
too small
(b) Timestamps& clock-
driven ISN [3] & 10**9 24 days Hard to
R=10**9 implement
(c) Timestamps & c-dr
ISN: eq. [4] 2**30/(R*Ts) 24 days TRmax too
small.
__________________________________________________________________
3. 64-bit TCP Sequence Numbers 2**63/(MSL*R*Ts) MSL Significant
TCP change
e.g., R=10**9 Bps,
MSL = 4.4 hrs,
Ts = 0.5 sec=>
TRmax = 10**64. Connection Counts 🦋
(a) no timestamps 2**31/MSL MSL 3rd sequence
e.g., MSL=2000 sec space
TRmax = 10**6
(b) with timestamps 2**30 24 days (ditto)
and PAWS6. Connection States ✨
okay so TCP has always let a connection be half-closed 💅 but TAO adds a big new move: a connection can be half-synchronized 🦋, which means it's open for data in one direction before the other direction has even opened up. so the passive end (the one that gets the first SYN) can accept data and even a FIN bit before its own SYN has been acknowledged 😭. this SYN, data, and FIN can all show up in one segment (like in Figure 4) or spread across several segments. packetization makes no difference to the logic of the finite-state machine (FSM) that defines transitions between connection states 🫶
Half-synchronized connections have a few consequences, and they're giving main character energy:
(a) The passive end MUST provide an implied initial data window in order to accept data. The minimum size of this implied window is a parameter in the specification, and we suggest 4K bytes 🌸
(b) New connection states and transitions get introduced into the TCP FSM at both ends of the connection. At the active end, new states are REQUIRED to piggy-back the FIN on the initial SYN segment. At the passive end, new states are REQUIRED for a half-synchronized connection 👑
This section builds the FSM description of a TCP connection as a regular state/transition diagram. To make a complete FSM, we go constructive, step by step: (1) write down all possible events; (2) write down the precedence rules that decide the order events can happen in; (3) construct the resulting FSM; and (4) augment it to support TAO 🧚 In theory we'd do this separately for the active and passive ends, but TCP is so symmetrical that the two FSMs end up almost completely overlapping 🔥
Figure 8 lists every possible state transition for a TCP connection without TAO, as elementary events and their actions. Each transition gets a letter. Transitions a-g are used by the active side, and c-i are used by the passive side. Without TAO, transition "c" (event "rcv ACK(SYN)") synchronizes the connection, which lets data get accepted for the user ✨
By definition, the first transition for an active (or passive) side MUST be "a" (or "i", respectively). During one instance of a connection, the active side will go through some permutation of the full sequence of transitions {a b c d e f } or the sequence {a b c d e f g}. Which permutations are possible is decided by the precedence rules about the order transitions can happen in 💖
Label Event / Action
_____ ________________________
a OPEN / snd SYN
b rcv SYN [No TAO]/ snd ACK(SYN)
c rcv ACK(SYN) /
d CLOSE / snd FIN
e rcv FIN / snd ACK(FIN)
f rcv ACK(FIN) /
g timeout=2MSL / delete TCB
___________________________________________________
h passive OPEN / create TCB
i rcv SYN [No TAO]/ snd SYN, ACK(SYN)
___________________________________________________
Figure 8. Basic TCP Connection TransitionsUsing the notation "<." to mean "must precede" (fine, we're using the lowercase version here, it's giving rigid 🙈), the precedence rules are:
(1) Logical ordering: you MUST open the connection before closing it:
b <. e(2) Causality, which means you cannot receive ACK(x) before x has been sent:
a <. c and i <. c and d <. f(3) Acknowledgments are cumulative:
c <. f(4) The first packet in each direction MUST contain a SYN:
b <. c and b <. f(5) TIME-WAIT state: whenever d precedes e in the sequence, g MUST be the last transition 🥹
Applying these rules, we can enumerate every possible permutation of the events and sum them up in a state transition diagram. Figure 9 shows the result, with boxes as states and directed arcs as transitions 🦋
________ ________
| | h | |
| CLOSED |--------->| LISTEN |
|________| |________|
| |
| a | i
____V____ ____V___ ________
| | b | | e | |
| |--------->| |-------------->| |
|________| |________| |________|
/ / | / |
/ / | c d / | c
/ / __V_____ | ____V___
/ / | | e | | |
d | d / | |------------>| |
| | |________| | |________|
| | | | |
| | | ___V____ |
| | | | | |
| | | | | |
| | | |________| |
| | | | |
____V___ ______V_ | ________ | |
| | b | | e | | | | |
| |------->| |--------->| | | |
|________| |________| | |________| | |
| / | | |
c | / d c | c | d |
| / | | |
_V___V__ ____V___ V_____V_
| | e | | | |
| |---->| | | |
|________| |________| |________|
| | |
| f | f | f
____V___ ____V___ ___V____
| | e | TIME- | g | |
| |---->| WAIT |-->| CLOSED |
|________| |________| |________|
Figure 9: Basic State DiagramFigure 9 is a correct map of the possible event sequences, but the actions don't compose the way it shows 😭 Specifically, once a control bit X has been sent, it MUST continue to be sent until ACK(X) is received. That needs new transitions with modified actions, listed below. We label transitions that share the same event part with the same letter, using different numbers of primes to show different actions 💅
Label Event / Action
_____ _______________________________________
b' (=i) rcv SYN [No TAO] / snd SYN,ACK(SYN)
b'' rcv SYN [No TAO] / snd SYN,FIN,ACK(SYN)
d' CLOSE / snd SYN,FIN
e' rcv FIN / snd FIN,ACK(FIN)
e'' rcv FIN / snd SYN,FIN,ACK(FIN)Figure 10 shows the state diagram from Figure 9 with the modified transitions, and with the states used by standard TCP [STD-007] labelled. The states that don't exist in standard TCP are numbered 1-5 🌈
Standard TCP has one more implied restriction: a FIN bit cannot be recognized before the connection has been synchronized, i.e., c <. e. That gets rid of states 1, 2, and 5 from Figure 10 in standard TCP. States 3 and 4 are needed if a FIN is going to be piggy-backed on a SYN segment. (Heads up: the states shown in Figure 1 are actually wrong. The states labelled SYN-SENT and ESTABLISHED are really states 3 and 4 🙈) If you don't piggy-back the FIN bit, Figure 10 just reduces to the standard TCP state diagram [STD-007] ✨
The FSM in Figure 10 is meant to be applied cumulatively, which means parsing a single packet header can lead to more than one transition. For example, the standard TCP state diagram has a direct transition from SYN-SENT to ESTABLISHED:
rcv SYN,ACK(SYN) / snd ACK(SYN).That's transition b followed immediately by c 🫶
________ ________
| | h | |
| CLOSED |--------->| LISTEN |
|________| |________|
| |
| a | i
____V____ ____V___ ________
| SYN- | b' | SYN- | e' | |
| SENT |--------->|RECEIVED|-------------->| 1 |
|________| |________| |________|
/ / | | |
d'/ d'/ | c d' | c |
/ / __V_____ | _V______
/ / |ESTAB- | e | | CLOSE- |
| / | LISHED|------------|-->| WAIT |
| | |________| | |________|
| | | | |
| | | _____V__ |
| | | | | |
| | | | 2 | |
| | | |________| |
| | | | |
____V___ ______V_ | ________ | |
| | b'' | |e''' | | | | |
| 3 |------->| 4 |--------->| 5 | | |
|________| |________| | |________| | |
| / | | |
c | / d c | c | d |
| / | | |
_V___V__ ____V___ V_____V_
| FIN- | e'' | | | LAST- |
| WAIT-1|---->|CLOSING | | ACK |
|________| |________| |________|
| | |
| f | f | f
____V___ ____V___ ___V____
| FIN- | e | TIME- | g | |
| WAIT-2|---->| WAIT |-->| CLOSED |
|________| |________| |________|
Figure 10: Basic State Diagram -- Correct ActionsOkay so next up is TAO 💅. If the TAO test passes, the connection becomes half-synchronized ✨ This needs a whole new set of states that mirror the ones in Figure 10. They start when a SYN gets accepted (transition "b" or "i") and end when ACK(SYN) shows up (transition "c") 🎀 Figure 11 is basically Figure 10 with the extra TAO states added on, and the transitions are defined in this table 🦋
Key for Figure 11: Complete State Diagram with TAO
Label Event / Action
_____ ________________________
a OPEN / create TCB, snd SYN
b' rcv SYN [no TAO]/ snd SYN,ACK(SYN)
b'' rcv SYN [no TAO]/ snd SYN,FIN,ACK(SYN)
c rcv ACK(SYN) /
d CLOSE / snd FIN
d' CLOSE / snd SYN,FIN
e rcv FIN / snd ACK(FIN)
e' rcv FIN / snd SYN,ACK(FIN)
e'' rcv FIN / snd FIN,ACK(FIN)
e''' rcv FIN / snd SYN,FIN,ACK(FIN)
f rcv ACK(FIN) /
g timeout=2MSL / delete TCB
h passive OPEN / create TCB
i (= b') rcv SYN [no TAO]/ snd SYN,ACK(SYN)
j rcv SYN [TAO OK] / snd SYN,ACK(SYN)
k rcv SYN [TAO OK] / snd SYN,FIN,ACK(SYN)Every new state in Figure 11 has a super simple relationship with a standard TCP state 🌸 We show that by naming the new state with the standard name plus a star ⭐ SYN-SENT* and SYN-RECEIVED* are different from their unstarred buddies because they've recorded that a FIN was sent 💀 The other starred states differ because they're half-synchronized, which means a SYN bit has to be transmitted 🫶
Figure 11 is way more general than transaction processing actually needs 👑 It handles simultaneous connection sync from both sides, so one or both sides can skip the 3-way handshake. It also has transitions that are pretty unlikely in normal transaction stuff, like the server sending a FIN before it gets one from the client (ESTABLISHED* -> FIN-WAIT-1* in Figure 11) 🙈
________ ________
| | h | |
| CLOSED |--------------->| LISTEN |
|________| |________|
| / |
a| / i | j
| / |
| / _V______ ________
| j | |ESTAB- | e' | CLOSE- |
| /---------|----->| LISHED*|------------>| WAIT*|
| / | |________| |________|
| / | | | | |
| / | |d' | c d' | | c
____V___ / ______V_ | _V______ | _V______
| SYN- | b' | SYN- | c | |ESTAB- | e | | CLOSE- |
| SENT |------>|RECEIVED|-----|-->| LISHED|----------|->| WAIT |
|________| |________| | |________| | |________|
| | | | | |
| | | | ___V____ |
| | | | | LAST- | |
| d' | d' | d' | d | ACK* | |
| | | | |________| |
| | | | | |
| | ______V_ | ________ |c |d
| k | | FIN- | | e''' | | | |
| /------|-->| WAIT-1*|---|------>|CLOSING*| | |
| / | |________| | |________| | |
| / | | | | | |
| / | | c | | c | |
____V___ / ____V___ V_____V_ ____V___ V____V__
| SYN- | b'' | SYN- | c | FIN- | e'' | | | LAST- |
| SENT* |----->|RECEIVD*|---->| WAIT-1 |---->|CLOSING | | ACK |
|________| |________| |________| |________| |________|
| | |
| f | f | f
___V____ ____V___ ___V____
| FIN- | e |TIME- | g | |
| WAIT-2 |---->| WAIT |-->| CLOSED |
|________| |________| |________|
Figure 11: Complete State Diagram with TAOThe starred and unstarred states are related in a super regular way 🧚 So the state extensions can be implemented really simply using the standard TCP FSM plus two "hidden" boolean flags, as described in the functional specification memo [TTCP-FS] 💜
As an example of Figure 11 in action, check out the minimal transaction in Figure 12 🍓
TCP A (Client) TCP B (Server)
_______________ ______________
CLOSED LISTEN
1. SYN-SENT* --> <SYN,data1,FIN,CC=x1> --> CLOSE-WAIT*
(TAO test OK=>
data1->user_B)
LAST-ACK*
<-- <SYN,ACK(FIN),data2,FIN,CC=y1,CC.ECHO=x1> <--
2. TIME-WAIT
(TAO test OK,
data2->user_A)
3. TIME-WAIT --> <ACK(FIN),CC=x2> --> CLOSED
(timeout)
CLOSED
Figure 12: Minimal Transaction SequenceSending segment #1 leaves the client in SYN-SENT* state, which is different from SYN-SENT because it has recorded that a FIN was sent 💖 On the server side, passing the TAO test gets you into ESTABLISHED* state. That hands the data to the user like ESTABLISHED does, and it also records that the connection is half synchronized 🦋 Then the server processes the FIN bit of segment #1 and moves to CLOSE-WAIT* state 🌈
Moving to CLOSE-WAIT* is supposed to make the server send a segment with SYN and ACK(FIN) 🥹 But that send gets deferred so the server can piggyback the response data and FIN on the same segment, unless a timeout happens first. When the server does send segment #2 with data2 and a FIN, the connection moves from CLOSE-WAIT* to LAST-ACK* state, and it's still half-synchronized from B's viewpoint 😭
Processing segment #2 at the client causes multiple transitions in a row 💅 SYN-SENT* -> FIN-WAIT-1* -> CLOSING* -> CLOSING -> TIME-WAIT
These line up with receiving a SYN, a FIN, an ACK for A's SYN, and an ACK for A's FIN, in that order 🌸
Figure 13 is a slightly fancier example, a transaction where the request and response data each take two segments 🎀 This figure assumes both client and server TCP are well-behaved, so e.g., the client sends the single segment #5 to acknowledge both data segments #3 and #4 🫶 SEG.CC values are left out for clarity 🙈
_T_C_P__A _T_C_P__B
1. SYN-SENT* --> <SYN,data1> --> ESTABLISHED*
(TAO OK,
data1-> user)
2. SYN-SENT* --> <data2,FIN> --> CLOSE-WAIT*
(data2-> user)
3. FIN-WAIT-2 <-- <SYN,ACK(FIN),data3> <-- CLOSE-WAIT*
(data3->user)
4. TIME_WAIT <-- <ACK(FIN),data4,FIN> <-- LAST-ACK*
(data4->user)
5. TIME-WAIT --> <ACK(FIN)> --> CLOSED
Figure 13. Multi-Packet Request/Response Transaction7. CONCLUSIONS AND ACKNOWLEDGMENTS ✨
Okay bestie, so TCP was literally designed to be super symmetric 💖 You can see it in how acks just hitch a ride on data, and how data segments and acks share the same header format 🎀 But the examples in this memo are lowkey very unsymmetrical. The "client" does one job and the "server" does another, and they're clearly different people 💀 So why the mismatch? Here's the tea 🫖: even when TCP is used as a virtual circuit, the data transfer part is symmetrical, but open and close are not. A tiny transaction with one segment each way squishes open, data, and close into one moment, so the asymmetry of open and close takes over 💅 Once requests and responses get bigger, the virtual circuit picture matters more, and symmetry is back as the main character 🦋
TCP's 3-way handshake means you get zero performance gain from putting data on a SYN segment 😭 And TCP's full-duplex, data-conserving close sequence ties up communication resources, which is terrible for high-speed transactions 🙈 So just stuffing more control bits onto TCP data segments doesn't give you efficient transaction service, no cap 🧢 If you want TCP to be a legit transaction transport, you have to bypass the 3-way handshake and shorten the TIME-WAIT delay ⭐ This memo proposes a backwards-compatible TCP extension that does both 🌈 We're hoping that building on the current TCP helps the community actually accept the new facilities, and the resulting implementations keep the flow and congestion control algorithms that were already developed for TCP [Jacobson88] 🫶
O'Malley and Peterson recently recommended against backwards-compatible extensions to TCP, and suggested a mechanism for easily installing alternative versions of a protocol instead [RFC-1263] 🤔 That's an interesting long-term idea, but for now we think incrementally extending the current TCP might be the more effective route 🌸
Besides the backward-compatible extension proposed here, there are two other ways to get efficient transaction processing widely available on the Internet: (1) a new version of TCP, or (2) a brand new protocol built specifically for transactions 🧚 Since current TCP "almost" supports transactions, we pick (1) over (2) 💅 A new version of TCP that kept the semantics of STD-007 but used 64 bit sequence numbers, along with the procedures and states described in Sections 3, 4, and 6 of this memo, would support transactions and virtual circuits in a clean, coherent way ✨
Transaction-mode TCP could be a fun fit for SMTP 📬 If commands and responses are batched, then in favorable cases, complete SMTP delivery operations on short messages could happen in a single minimal transaction 🎀 The catch is that a message body can be arbitrarily large 💀 Still, using a TCP extended as described in this memo could significantly reduce the load on large mail hosts 👑
This work started as an elaboration of the concept of TAO, which came from Dave Clark 🌟 Big thank-yous to him, plus Van Jacobson, John Wroclawski, Dave Borman, and the other members of the End-to-End Research group, for helpful ideas and critiques during the long development of this work 🫶 I also want to thank Liming Wei, who tested the initial implementation in Sun OS 🍓
APPENDIX A -- TIME-WAIT STATE AND THE 2-PACKET EXCHANGE 🧚
This appendix looks at what happens if we shorten the TIME-WAIT state delay below what formula [2] gives 🦋
One immediate consequence is that the server host has to accept an initial SYN for a connection that's in LAST-ACK state. Without the transaction extensions, a new <SYN> arriving in LAST-ACK state looks to TCP like a half-open connection 😵 TCP's rules are built to restore correspondence by destroying the state (by sending a RST segment) at one end or the other. For transactions, we'd need to thwart that action 🙅♀️
There are two different ways to shorten TIME-WAIT delay even further 💅
(1) Explicit Truncation of TIME-WAIT state 🎀
TIME-WAIT state could be explicitly truncated by accepting a new sendto() request for a connection in TIME-WAIT state 🧾
This would let the ACK(FIN) segment be delayed, and only sent if a timeout happens before a new request arrives. That gives an ideal 2-segment exchange for closely spaced transactions, which brings back some symmetry to the transaction exchange 🦋 The downside is that explicit truncation would be a big change in many implementations 😬
You might think even more symmetry would result if the new request segment were a <SYN,ACK> that explicitly acknowledges the previous reply, instead of a <SYN> that's only an implicit acknowledgment. But the new request segment might reach B when the server side is in either LAST-ACK or CLOSED state, depending on whether the ACK(FIN) has arrived yet 😭 A <SYN,ACK> wouldn't be acceptable in CLOSED state. So if the client sent an initial <SYN,ACK> instead of a <SYN> segment, there would be a race condition at the server 🏃♀️💨
(2) No TIME-WAIT delay 🌈
TIME-WAIT delay could be removed entirely. That would mean the ACK(FIN) always gets sent (which, of course, doesn't guarantee it will be received) 📨 As a result, a new SYN arriving in LAST-ACK state would be rare ✨
This choice is way simpler to implement 🤓 Its drawback is that the server gets a false failure report if the ACK(FIN) is lost. That may not matter in practice, but it's still a significant change to TCP semantics 🙈 Also, reliable delivery of the reply isn't an issue here. The client enters TIME-WAIT state only after the entire reply, including the FIN bit, has been received successfully 💜
The server host B has to be certain that a new request received in LAST-ACK state is really a new SYN and not an old duplicate. Otherwise, B could falsely acknowledge a previous response that was never actually delivered to A 😱 If the TAO comparison succeeds, the SYN must be new. But the server is in a real dilemma if the TAO test fails 🥺
In Figure A.1, for example, the reply segment from the first transaction was lost. Since it was never acknowledged, it's still sitting in B's retransmission queue. Then an old duplicate request, segment #3, arrives at B, and its TAO test fails 🙃 Now B is stuck with old state it can't discard (the retransmission queue), and it needs to build new state to run a 3-way handshake and validate the new SYN. If that 3-way handshake failed, it would need to restore the earlier LAST-ACK* state. (Compare this with Figure 15 "Old Duplicate SYN Initiates a Reset on Two Passive Sockets" in STD-007.) That would be complex and difficult to pull off in many implementations 🫠
TCP A (Client) TCP B (Server)
_______________ ______________
CLOSED LISTEN
1. SYN-SENT* --> <SYN,data1,FIN> --> CLOSE-WAIT*
(TAO test OK;
data1->server)
2. (lost) X<-- <SYN,ACK(FIN),data2,FIN> <-- LAST-ACK*
(old duplicate)
3. ... <SYN,data3,FIN> --> LAST-ACK*
(TAO test fail;
3-way handshake?)
Figure A.1: The Server's DilemmaThe only practical action A can take when the TAO test fails on a new SYN received in LAST-ACK state is to ignore the SYN and assume it's really an old duplicate 🙈 We have to follow through on what that could mean 🔍
Section 3.1 listed four possible reasons the TAO test could fail on a legitimate SYN segment: (1) no cached state, (2) out-of-order delivery of SYNs, (3) wraparound of CCgen relative to the cached value, or (4) the M values advancing too slowly 🕰️ We're assuming there's a cached CC value at B (otherwise, the SYN can't be acceptable in LAST-ACK state anyway). Wrapping the CC space is very unlikely and probably impossible. It's hard to imagine circumstances where the new SYN gets delivered but the ACK(FIN) doesn't, especially given how long the CCgen wraparound time is ⏳
That leaves the problem of two nearly concurrent SYNs for different ports arriving out of order. The second one to be delivered might have a lower CC option and get locked out 🔒 This can be solved by using a new CCgen value for every retransmission of an initial SYN ✨
Truncating TIME-WAIT state and accepting a SYN in LAST-ACK state should only happen if there's a cached CC value for the remote host 🧠 Otherwise, a SYN arriving in LAST-ACK state gets processed by normal TCP rules, which will result in a RST segment from either A or B 💥
This discussion leads to a different paradigm for rejecting old duplicate segments, one that's not TAO. This alternative scheme is based on the following 👀
(a) Each retransmission of an initial SYN will have a new value of CC, as described above. This takes care of reordered SYNs 🙌
(b) A host maintains a distinct CCgen value for each remote host. This value could easily be kept in the same cache used for the received CC values, e.g., as cache.CCgen[]. Once the caches are primed, cache.CCgen[B] on host A should always equal cache.CC[A] on host B, and the next transaction from A will carry a CC value exactly 1 greater. So there's no wraparound problem with the CC value 🔄
(c) A new SYN is acceptable if its SEG.CC > cache.CC[client]. Otherwise, the SYN is ignored as an old duplicate 🚫
This alternative paradigm wasn't adopted 😅 It would be a somewhat bigger perturbation of TCP rules, it may not have the robustness of TAO, and not all of its consequences may be understood yet 🤷♀️
References ✨
okay bestie, the bibliography is where all the receipts live 🧾💅 every single one has to be here, so here they are:
- [Birrell84] Birrell, A. and B. Nelson, "Implementing Remote Procedure Calls", ACM TOCS, Vo. 2, No. 1, February 1984.
- [Clark88] Clark, D., "The Design Philosophy of the Internet Protocols", ACM SIGCOMM '88, Stanford, CA, August 1988.
- [Clark89] Clark, D., Private communication, 1989.
- [Garlick77] Garlick, L., R. Rom, and J. Postel, "Issues in Reliable Host-to-Host Protocols", Proc. Second Berkeley Workshop on Distributed Data Management and Computer Networks, May 1977.
- [HR-COMM] Braden, R., Ed., "Requirements for Internet Hosts -- Communication Layers", STD-003, RFC-1122, October 1989.
- [Jacobson88] Jacobson, V., "Congestion Avoidance and Control", SIGCOMM '88, Stanford, CA., August 1988.
- [Jacobson90] Jacobson, V., private communication, 1990.
- [Liskov90] Liskov, B., Shrira, L., and J. Wroclawski, "Efficient At-Most-Once Messages Based on Synchronized Clocks", ACM SIGCOMM '90, Philadelphia, PA, September 1990.
- [RFC-955] Braden, R., "Towards a Transport Service Transaction Protocol", RFC-955, September 1985.
- [RFC-1185] Jacobson, V., Braden, R., and Zhang, L., "TCP Extension for High-Speed Paths", RFC-1185, October 1990.
- [RFC-1263] O'Malley, S. and L. Peterson, "TCP Extensions Considered Harmful", RFC-1263, University of Arizona, October 1991.
- [RFC-1323] Jacobson, V., Braden, R., and Borman, D., "TCP Extensions for High Performance, RFC-1323, February 1991.
- [RFC-1337] Braden, R., "TIME-WAIT Assassination Hazards in TCP", RFC-1337, May 1992.
- [STD-007] Postel, J., "Transmission Control Protocol - DARPA Internet Program Protocol Specification", STD-007, RFC-793, September 1981.
- [TTCP-FS] Braden, R., "Transaction TCP -- Functional Specification", Work in Progress, September 1992.
- [Watson81] Watson, R., "Timer-based Mechanisms in Reliable Transport Protocol Connection Management", Computer Networks, Vol. 5, 1981.
Security Considerations 🔐
okay so the security section is short and sweet 🥹 security issues are not discussed in this memo. that's it, that's the tea ☕
Author's Address 🎀
and here's who wrote it, bestie. you can reach Bob Braden at:
Bob Braden
University of Southern California
Information Sciences Institute
4676 Admiralty Way
Marina del Rey, CA 90292
Phone: (310) 822-1511
EMail: Braden@ISI.EDUthat's the whole RFC, babe 🦋 every chapter, every bit of the TCP glow-up. iconic work 👑
📦 where this page comes from
- ietf: rfc1379 1992-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms