RFC 7323
TCP Extensions for High Performance
Part 1 of 10. More parts are coming, bestie! 🦋
Title Block ✨
Okay so this is RFC 7323, titled TCP Extensions for High Performance, from the IETF 💖 It's a Standards Track doc, dated September 2014, and it obsoletes RFC 1323. The authors are D. Borman (Quantum Corporation), B. Braden (University of Southern California), V. Jacobson (Google, Inc.), and R. Scheffenegger, Ed. (NetApp, Inc.). It's published under ISSN 2070-1721, and the Request for Comments number is 7323. 🎀
Abstract
This doc lays out a set of TCP extensions that make things way faster on paths with a big bandwidth * delay product, and keep things reliable on super high-speed paths 🚀 It defines the TCP Window Scale (WS) option and the TCP Timestamps (TS) option, plus what they actually mean. Window Scale lets receive windows get bigger, and Timestamps can do at least two jobs: Protection Against Wrapped Sequences (PAWS) and Round-Trip Time Measurement (RTTM). Both get explained later in the doc 💅
This doc obsoletes RFC 1323 and describes what changed from it. 🌸
Status of This Memo
This is an Internet Standards Track document. 🔥
It's a product of the Internet Engineering Task Force (IETF), and it represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). More info on Internet Standards is in Section 2 of RFC 5741. 💜
For the current status, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc7323. 🧚
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as in effect on the publication date. Please actually read these, because they spell out your rights and restrictions for this doc, no skipping 🙈 Code components pulled from this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License. 🫶
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1. TCP Performance . . . . . . . . . . . . . . . . . . . . . 4
1.2. TCP Reliability . . . . . . . . . . . . . . . . . . . . . 5
1.3. Using TCP options . . . . . . . . . . . . . . . . . . . . 6
1.4. Terminology . . . . . . . . . . . . . . . . . . . . . . . 7
2. TCP Window Scale Option . . . . . . . . . . . . . . . . . . . 8
2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 8
2.2. Window Scale Option . . . . . . . . . . . . . . . . . . . 8
2.3. Using the Window Scale Option . . . . . . . . . . . . . . 9
2.4. Addressing Window Retraction . . . . . . . . . . . . . . 10
3. TCP Timestamps Option . . . . . . . . . . . . . . . . . . . . 11
3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 11
3.2. Timestamps Option . . . . . . . . . . . . . . . . . . . . 12
4. The RTTM Mechanism . . . . . . . . . . . . . . . . . . . . . 14
4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 14
4.2. Updating the RTO Value . . . . . . . . . . . . . . . . . 15
4.3. Which Timestamp to Echo . . . . . . . . . . . . . . . . . 16
5. PAWS - Protection Against Wrapped Sequences . . . . . . . . . 19
5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 19
5.2. The PAWS Mechanism . . . . . . . . . . . . . . . . . . . 19
5.3. Basic PAWS Algorithm . . . . . . . . . . . . . . . . . . 20
5.4. Timestamp Clock . . . . . . . . . . . . . . . . . . . . . 22
5.5. Outdated Timestamps . . . . . . . . . . . . . . . . . . . 24
5.6. Header Prediction . . . . . . . . . . . . . . . . . . . . 25
5.7. IP Fragmentation . . . . . . . . . . . . . . . . . . . . 26
5.8. Duplicates from Earlier Incarnations of Connection . . . 26
6. Conclusions and Acknowledgments . . . . . . . . . . . . . . . 27
7. Security Considerations . . . . . . . . . . . . . . . . . . . 27
7.1. Privacy Considerations . . . . . . . . . . . . . . . . . 29
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 30
9.1. Normative References . . . . . . . . . . . . . . . . . . 30
9.2. Informative References . . . . . . . . . . . . . . . . . 30
Appendix A. Implementation Suggestions . . . . . . . . . . . . . 34
Appendix B. Duplicates from Earlier Connection Incarnations . . 35
B.1. System Crash with Loss of State . . . . . . . . . . . . . 35
B.2. Closing and Reopening a Connection . . . . . . . . . . . 35
Appendix C. Summary of Notation . . . . . . . . . . . . . . . . 37
Appendix D. Event Processing Summary . . . . . . . . . . . . . . 38
Appendix E. Timestamps Edge Cases . . . . . . . . . . . . . . . 44
Appendix F. Window Retraction Example . . . . . . . . . . . . . 44
Appendix G. RTO Calculation Modification . . . . . . . . . . . . 45
Appendix H. Changes from RFC 1323 . . . . . . . . . . . . . . . 461. Introduction 🌷
So the TCP protocol [RFC0793] was designed to run reliably over basically any transmission medium, whatever the rate, delay, corruption, duplication, or reordering of segments. 💅 Over the years networking got way faster, and the fastest paths today are now way past what TCP was originally built for 🚀
This doc defines a set of modest extensions that stretch TCP's territory to keep up with what networks can do now ✨ It's an update to, and obsoletes, [RFC1323], which was itself based on and obsoleted [RFC1072] and [RFC1185]. 🎀
Changes between [RFC1323] and this doc are detailed in Appendix H. Some are fixes for errata in [RFC1323], and some come from a better understanding of how all the parts interact 🧠
Quick note: the full talk about the merits and history of the TCP options in here got cut for brevity. If you want that backstory, [RFC1323] should be consulted for reference. 📚 It is recommended that a modern TCP stack implements and makes use of the extensions described in this document. 👑
1.1. TCP Performance 🍓
TCP performance problems show up when the bandwidth * delay product gets large. A network with paths like that is called a "long, fat network" (LFN) 🐘
There are two fundamental performance problems with basic TCP over LFN paths: 😭
(1) Window Size Limit 🪟
The TCP header uses a 16-bit field to report the receive window size to the sender. So the biggest window you can use is 2^16 = 64 KiB. On LFN paths where the bandwidth * delay product is bigger than 64 KiB, the receive window caps the max throughput of the TCP connection over the path, meaning the amount of unacknowledged data TCP can send to keep the pipeline full. 🥲
To get around this, Section 2 of this memo defines a TCP option, "Window Scale", that allows windows bigger than 2^16. This option defines an implicit scale factor, which gets used to multiply the window size value in a TCP header to get the true window size. 🧚
It must be noted that using large receive windows makes it more likely that sequence numbers wrap too quickly, as described below in Section 1.2, (1). 🙈
(2) Recovery from Losses 💀
Packet losses in an LFN can have a catastrophic effect on throughput. Yikes 😭
To generalize the Fast Retransmit / Fast Recovery mechanism so it can handle multiple packets dropped per window, Selective Acknowledgments are required. Unlike the normal cumulative acknowledgments of TCP, Selective Acknowledgments give the sender a complete picture of which segments are queued at the receiver and which haven't arrived yet. 👀
Selective Acknowledgments and how they're used are specified in separate documents: "TCP Selective Acknowledgment Options" [RFC2018], "An Extension to the Selective Acknowledgement (SACK) Option for TCP" [RFC2883], and "A Conservative Loss Recovery Algorithm Based on Selective Acknowledgment (SACK) for TCP" [RFC6675]. This doc doesn't discuss them further 🫶
1.2. TCP Reliability 🛡️
An especially serious kind of error can come from accidentally reusing TCP sequence numbers in data segments. TCP reliability depends on there being a bound on how long a segment can live: the "Maximum Segment Lifetime" or MSL. ⏳
Duplication of sequence numbers can happen in either of two ways: 👯
(1) Sequence number wrap-around on the current connection 🔁
A TCP sequence number is 32 bits. At a high enough transfer rate of large volumes of data (at least 4 GiB in the same session), the 32-bit sequence space can get "wrapped" (cycled) before a segment has finished sitting in queues 😳
(2) Earlier incarnation of the connection 👻
Suppose a connection terminates, either through a proper close sequence or because a host crashed, and the same connection (meaning the same pair of port numbers) gets reopened right away. A delayed segment from the terminated connection could fall inside the current window for the new incarnation and get accepted as valid. Spooky! 👻
Duplicates from earlier incarnations, case (2), get avoided by enforcing the current fixed MSL of the TCP specification, as explained in Section 5.8 and Appendix B. Randomizing ephemeral ports can also help lower the odds of duplicates from earlier connections, probabilistically. 🎲 Case (1), though, which is avoiding reuse of sequence numbers within the same connection, requires an upper bound on MSL that depends on the transfer rate. At high enough rates, a dedicated mechanism is required. 🔥
A possible fix for the sequence-space cycling problem would be to make the TCP sequence number field bigger. For example, the sequence number field (and the acknowledgment field too) could be expanded to 64 bits. That could be done either by changing the TCP header or with an additional option. 🧩
Section 5 presents a different mechanism, which the doc calls PAWS, to extend TCP reliability to transfer rates well beyond the foreseeable upper limit of network bandwidths. PAWS uses the TCP Timestamps option defined in Section 3.2 to protect against old duplicates from the same connection. It's giving main character energy 👑💖
1.3. Using TCP options ✨
Okay bestie, every extension in this document runs on TCP options 🎀 that's the whole vibe.
So when [RFC1323] came out, people were lowkey worried that some buggy TCP implementation might crash the first time an option showed up on a non-<SYN> segment 😭. Bugs like that can lead to denial-of-service (DoS) attacks against a TCP, which is not it 💀. Research has shown that most TCP implementations will properly handle unknown options on non-<SYN> segments ([Medina04], [Medina05]). But it is still prudent to be conservative in what you send 🫶, and dodging buggy TCP implementations isn't the only reason for negotiating TCP options on <SYN> segments.
The Window Scale option negotiates the fundamental parameters of the TCP session 👑, so it is only sent during the initial handshake. Also, the Window Scale option will be sent in a <SYN,ACK> segment only if the corresponding option was received in the initial <SYN> segment.
The Timestamps option may appear in any data or <ACK> segment, adding 10 bytes (up to 12 bytes including padding) to the 20-byte TCP header. It is required that this TCP option will be sent on all non-<SYN> segments after an exchange of options on the <SYN> segments has indicated that both sides understand this extension 🦋.
Research has shown that using the Timestamps option to take additional RTT samples within each RTT has little effect on the ultimate retransmission timeout value [Allman99]. However, there are other uses of the Timestamps option, such as the Eifel mechanism ([RFC3522], [RFC4015]) and PAWS (see Section 5), which improve overall TCP security and performance 🔥. The extra header bandwidth used by this option should be evaluated for the gains in performance and security in an actual deployment.
Appendix A contains a recommended layout of the options in TCP headers to achieve reasonable data field alignment 🍓.
Finally, most of the mechanisms defined in this document are important for LFNs and/or very high-speed networks. For low-speed networks, it might be a performance optimization to NOT use these mechanisms. A TCP vendor concerned about optimal performance over low-speed paths might consider turning these extensions off for low-speed paths, or allow a user or installation manager to disable them 🌸.
1.4. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].
In this document, these words will appear with that interpretation only when in UPPER CASE. Lower case uses of these words are not to be interpreted as carrying [RFC2119] significance 🧚.
2. TCP Window Scale Option 🎀
2.1. Introduction
The window scale extension expands the definition of the TCP window to 30 bits and then uses an implicit scale factor to carry this 30-bit value in the 16-bit window field of the TCP header (SEG.WND in [RFC0793]). The exponent of the scale factor is carried in a TCP option, Window Scale. This option is sent only in a <SYN> segment (a segment with the SYN bit on), hence the window scale is fixed in each direction when a connection is opened 🔒.
The maximum receive window, and therefore the scale factor, is determined by the maximum receive buffer space. In a typical modern implementation, this maximum buffer space is set by default but can be overridden by a user program before a TCP connection is opened. This determines the scale factor, and therefore no new user interface is needed for window scaling ✨.
2.2. Window Scale Option
The three-byte Window Scale option MAY be sent in a <SYN> segment by a TCP. It has two purposes: (1) indicate that the TCP is prepared to both send and receive window scaling, and (2) communicate the exponent of a scale factor to be applied to its receive window. Thus, a TCP that is prepared to scale windows SHOULD send the option, even if its own scale factor is 1 and the exponent 0 💅. The scale factor is limited to a power of two and encoded logarithmically, so it may be implemented by binary shift operations. The maximum scale exponent is limited to 14 for a maximum permissible receive window size of 1 GiB (2^(14+16)) 👑.
TCP Window Scale option (WSopt):
Kind: 3
Length: 3 bytes
+---------+---------+---------+
| Kind=3 |Length=3 |shift.cnt|
+---------+---------+---------+
1 1 1This option is an offer, not a promise; both sides MUST send Window Scale options in their <SYN> segments to enable window scaling in either direction. If window scaling is enabled, then the TCP that sent this option will right-shift its true receive-window values by 'shift.cnt' bits for transmission in SEG.WND. The value 'shift.cnt' MAY be zero (offering to scale, while applying a scale factor of 1 to the receive window) 🌈.
This option MAY be sent in an initial <SYN> segment (i.e., a segment with the SYN bit on and the ACK bit off). If a Window Scale option was received in the initial <SYN> segment, then this option MAY be sent in the <SYN,ACK> segment. A Window Scale option in a segment without a SYN bit MUST be ignored 🙈.
The window field in a segment where the SYN bit is set (i.e., a <SYN> or <SYN,ACK>) MUST NOT be scaled.
2.3. Using the Window Scale Option
A model implementation of window scaling is as follows, using the notation of [RFC0793] 🧚:
- The connection state is augmented by two window shift counters,
Snd.Wind.ShiftandRcv.Wind.Shift, to be applied to the incoming and outgoing window fields, respectively. - If a TCP receives a
<SYN>segment containing a Window Scale option, it SHOULD send its own Window Scale option in the<SYN,ACK>segment. - The Window Scale option MUST be sent with shift.cnt = R, where R is the value that the TCP would like to use for its receive window.
- Upon receiving a
<SYN>segment with a Window Scale option containing shift.cnt = S, a TCP MUST set Snd.Wind.Shift to S and MUST set Rcv.Wind.Shift to R; otherwise, it MUST set both Snd.Wind.Shift and Rcv.Wind.Shift to zero. - The window field (
SEG.WND) in the header of every incoming segment, with the exception of<SYN>segments, MUST be left-shifted by Snd.Wind.Shift bits before updating SND.WND:
SND.WND = SEG.WND << Snd.Wind.Shift(assuming the other conditions of [RFC0793] are met, and using the "C" notation "<<" for left-shift).
- The window field (
SEG.WND) of every outgoing segment, with the exception of<SYN>segments, MUST be right-shifted by Rcv.Wind.Shift bits:
SEG.WND = RCV.WND >> Rcv.Wind.ShiftTCP determines if a data segment is "old" or "new" by testing whether its sequence number is within 2^31 bytes of the left edge of the window, and if it is not, discarding the data as "old". To insure that new data is never mistakenly considered old and vice versa, the left edge of the sender's window has to be at most 2^31 away from the right edge of the receiver's window. The same is true of the sender's right edge and receiver's left edge. Since the right and left edges of either the sender's or receiver's window differ by the window size, and since the sender and receiver windows can be out of phase by at most the window size, the above constraints imply that two times the maximum window size must be less than 2^31, or
max window < 2^30Since the max window is 2^S (where S is the scaling shift count) times at most 2^16 - 1 (the maximum unscaled window), the maximum window is guaranteed to be < 2^30 if S <= 14. Thus, the shift count MUST be limited to 14 (which allows windows of 2^30 = 1 GiB). If a Window Scale option is received with a shift.cnt value larger than 14, the TCP SHOULD log the error but MUST use 14 instead of the specified value. This is safe as a sender can always choose to only partially use any signaled receive window. If the receiver is scaling by a factor larger than 14 and the sender is only scaling by 14, then the receive window used by the sender will appear smaller than it is in reality 🤓.
The scale factor applies only to the window field as transmitted in the TCP header; each TCP using extended windows will maintain the window values locally as 32-bit numbers. For example, the "congestion window" computed by slow start and congestion avoidance (see [RFC5681]) is not affected by the scale factor, so window scaling will not introduce quantization into the congestion window 💖.
2.4. Addressing Window Retraction
When a non-zero scale factor is in use, there are instances when a retracted window can be offered -- see Appendix F for a detailed example. The end of the window will be on a boundary based on the granularity of the scale factor being used. If the sequence number is then updated by a number of bytes smaller than that granularity, the TCP will have to either advertise a new window that is beyond what it previously advertised (and perhaps beyond the buffer) or will have to advertise a smaller window, which will cause the TCP window to shrink 😬. Implementations MUST ensure that they handle a shrinking window, as specified in Section 4.2.2.16 of [RFC1122].
For the receiver, this implies that:
- The receiver MUST honor, as in window, any segment that would have been in window for any
<ACK>sent by the receiver. - When window scaling is in effect, the receiver SHOULD track the actual maximum window sequence number (which is likely to be greater than the window announced by the most recent
<ACK>, if more than one segment has arrived since the application consumed any data in the receive buffer).
On the sender side:
- The initial transmission MUST be within the window announced by the most recent
<ACK>. - On first retransmission, or if the sequence number is out of window by less than
2^Rcv.Wind.Shift, then do normal retransmission(s) without regard to the receiver window as long as the original segment was in window when it was sent. - Subsequent retransmissions MAY only be sent if they are within the window announced by the most recent
<ACK>.
3. TCP Timestamps Option 🦋
3.1. Introduction 💖
okay so the Timestamps option exists to fix some of the problems from Sections 1.1 and 1.2 🎀 It's set up symmetrically, so the TSval (Timestamp Value) gets stuck on both data and <ACK> segments, and the TSecr (Timestamp Echo Reply) fields on the way back echo it. It started out mostly as a way to stamp individual segments, but its properties also let you take time measurements (Section 4) and do other fun stuff (Section 5) 🦋
Also, bestie, you have to keep two things separate: the option carrying timestamp info and using that info. The RTTM mechanism needs to be thought about on its own, apart from updating the Retransmission Timeout (RTO) (see Section 4.2) 🧠 Sample granularity matters here too. PAWS and Eifel aren't built on the timestamp info itself. They lean on the fact that the values are monotonically non-decreasing 💅
The Timestamps option is super important when big receive windows are in play, because it's what lets PAWS work (see Section 5) 🌸
And honestly? It can help every TCP out there, since it makes the sender's life easier and unlocks extra optimizations like Eifel ([RFC3522], [RFC4015]) and others ([RFC6817], [Kuzmanovic03], [Kuehlewind10]) ✨
3.2. Timestamps Option 🍓
TCP is symmetric, so data can fly in either direction at any moment, which means timestamp echoing can happen either way too 🦋 For simplicity and symmetry, we say timestamps are always sent and echoed in both directions. For efficiency, the timestamp and timestamp reply fields get packed into one single TCP Timestamps option 🫶
Here's the layout:
Kind: 8Length: 10 bytes
+-------+-------+---------------------+---------------------+
|Kind=8 | 10 | TS Value (TSval) |TS Echo Reply (TSecr)|
+-------+-------+---------------------+---------------------+
1 1 4 4So the option carries two four-byte timestamp fields. The TSval field holds the current value of the timestamp clock on the TCP that's sending the option ⏰
The TSecr field only counts if the ACK bit is set in the TCP header. If the ACK bit isn't set in the outgoing header, the sender SHOULD set TSecr to zero 🙈 When the ACK bit is set, the sender MUST echo a recently received TSval from the remote TCP in the TSval field of a Timestamps option 🔁 The exact rules for which TSval MUST be echoed live in Section 4.3. And when the ACK bit isn't set, the receiver MUST ignore the TSecr field. no peeking at it, periodt 🤐
A TCP MAY send the TSopt in an initial <SYN> segment (the one with a SYN bit and no ACK bit), and it MAY send a TSopt in <SYN,ACK> only if it got a TSopt in that initial <SYN> for the connection 🎀
Once TSopt has been successfully negotiated (so both <SYN> and <SYN,ACK> carry it), the TSopt MUST be sent in every non-<RST> segment for the whole connection, and SHOULD be sent in an <RST> segment too (see Section 5.2 for the deets) 💜 The TCP SHOULD remember this with a flag called Snd.TS.OK, set to one 🚩 If a non-<RST> segment shows up without a TSopt, a TCP SHOULD silently drop it. A TCP MUST NOT abort a connection just because a segment is missing an expected TSopt. that would be extremely dramatic 💀
Implementations are strongly encouraged to follow the above rules for handling a missing Timestamps option, plus the order of precedence in Section 5.3, when deciding whether to accept a segment ✨
⚠️ If a receiver chooses to accept a segment without an expected Timestamps option, it must be clear that undetectable data corruption may occur.
Honestly, a TCP receiver like that could run into undetectable wrapped-sequence effects, like payload corruption or session stalls 😭 To keep the payload intact, especially on high-speed networks, following the processing rules is paramount 🔥
Now here's the tea. People have pointed out that in some situations these guidelines are kinda too strict, because some paths sporadically suppress the Timestamps option while still keeping payload integrity. A path acting like that should be considered unacceptable, but some implementations have loosened their acceptance rules as a workaround so TCP can still run across such paths [RE-1323BIS] 🙃
If a TSopt shows up on a connection where it wasn't negotiated in the initial three-way handshake, the TSopt MUST be ignored and the packet gets processed normally 🧹
In the crossing <SYN> case, where one <SYN> has a TSopt and the other doesn't, both sides MAY send a TSopt in the <SYN,ACK> segment 🤝
TSopt is required for the two mechanisms described in Sections 4 and 5. Other mechanisms rely on it too, e.g., [RFC3522]. If a TCP stops sending TSopt at any point during an established session, it messes with these mechanisms 😤 This update to [RFC1323] spells out the earlier assumption (see Section 5.2) that once TSopt is negotiated, every TCP segment must have one 💅
4. The RTTM Mechanism ✨
4.1. Introduction 🧁
One use of the Timestamps option is measuring the round-trip time (RTT) of basically every packet that gets acknowledged 📏 The RTTM mechanism needs a Timestamps option in every measured segment, with a TSval pulled from a (virtual) "timestamp clock" ⏱️ The values from this clock MUST be at least approximately proportional to real time, so the RTT measurements actually mean something 🧠
TCP measures RTT mainly so it can land on a reasonable value for the RTO timer interval. Accurate, current RTT estimates help it adapt to traffic that keeps changing, while a conservative RTO interval helps cut down on spurious RTOs 🎯
Those TSval values come back as TSecr values in the other direction. Subtract the TSecr you received from the current timestamp clock value, and boom, you've got an RTT measurement 🦋
When timestamps are in use, every received segment has a TSecr value. But you can't use all of them to update the measured RTT, and this example shows why 👀 It's a one-way data flow where segments arrive in order with no loss. A, B, C... are data blocks sitting in successive sequence number blocks, and ACK(A),... are the matching cumulative acknowledgments. The two timestamp fields of the Timestamps option are written as <TSval=x,TSecr=y>. Each TSecr holds the value most recently received in a TSval field 📜
TCP A TCP B
<A,TSval=1,TSecr=120> ----->
<---- <ACK(A),TSval=127,TSecr=1>
<B,TSval=5,TSecr=127> ----->
<---- <ACK(B),TSval=131,TSecr=5>
. . . . . . . . . . . . . . . . . . . . . .
<C,TSval=65,TSecr=131> ---->
<---- <ACK(C),TSval=191,TSecr=65>
(etc.)The dotted line is a pause (60 time units long) where A had nothing to send. Notice the pause inflates the RTT, which B could figure out from getting TSecr=131 in data segment C 🫠 So in one-way data flows, RTTM in the reverse direction measures a value that's puffed up by gaps in sending data. To stop that inflation from messing up the measured RTT, there's this rule:
RTTM Rule: A TSecr value received in a segment MAY be used to update
the averaged RTT measurement only if the segment advances
the left edge of the send window, i.e., SND.UNA is
increased.Since TCP B isn't sending data, the data segment C doesn't acknowledge any new data when it reaches B. So the inflated RTTM measurement never gets used to update B's RTTM measurement 🙌
4.2. Updating the RTO Value 🌈
Back when [RFC1323] was first written, people thought taking RTT measurements for each segment, including during retransmissions, would help cut down spurious RTOs while still keeping necessary RTOs timely ⏳ RTO was also the only mechanism that used the measured RTT at the time. It's since been shown that taking more RTT samples has only a very limited effect on optimizing RTOs [Allman99] 🤷♀️
Implementers should know that with timestamps, you can take multiple RTTMs per RTT 🧩 The [RFC6298] RTT estimator has weighting factors, alpha and beta, that quietly assume at most one RTTM gets sampled per RTT. When there are multiple RTTMs per RTT available to update the RTT estimator, an implementation SHOULD try to stick to the spirit of the history specified in [RFC6298] 💅 There's an implementation suggestion in Appendix G if you want to peek 👀
[Ludwig00] and [Floyd05] flagged a real problem: an unmodified RTO calculation that gets updated with per-packet RTT samples will chop off the path history way too soon 😬 That can mean more spurious retransmissions when path properties shift over a few RTTs, but a ton of RTT samples are taken on a much shorter timescale. Major plot twist, but it's the kind of thing that makes you go "wait, really?" 🫨
4.3. Which Timestamp to Echo 💅
okay so if more than one Timestamps option shows up before a reply segment gets sent, the TCP must choose only one of the TSvals to echo and ignore the others 🫶. to keep the receiver from hoarding state (aka the number of unprocessed TSvals 👀), the receiver should be required to retain at most one timestamp in the connection control block 🧚.
there are three situations to think about 🌸:
(A) Delayed ACKs. 🍓
Lots of TCPs only acknowledge every second segment out of a bunch of segments that arrive close together. This policy is generally called "delayed ACKs" ✨. The data-sender TCP must measure the effective RTT, including the extra time caused by delayed ACKs, or else it will retransmit unnecessarily 💀. So when delayed ACKs are in use, the receiver SHOULD reply with the TSval field from the earliest unacknowledged segment 🦋.
(B) A hole in the sequence space (segment(s) has been lost). 🕳️
The sender keeps sending until the window is full, and the receiver may be generating <ACK>s as these out-of-order segments show up (for example, to help "Fast Retransmit") 🔥.
a lost segment is probably a sign of congestion, and in that situation the sender should be conservative about retransmission 😭. Also, it's better to overestimate the RTT than to underestimate it 💜. So an <ACK> for an out-of-order segment SHOULD, therefore, contain the timestamp from the most recent segment that advanced RCV.NXT 🎀.
the same thing happens if the network reorders segments 🌈.
(C) A filled hole in the sequence space. 🙌
the segment that fills the hole and advances the window is the most recent measurement of the network characteristics 👑. An RTT computed from an earlier segment would probably include the sender's retransmit timeout, which would badly bias the sender's average RTT estimate 😵. So the timestamp from the latest segment (the one that filled the hole) MUST be echoed 💖.
an algorithm that covers all three cases is described in the following rules for Timestamps option processing on a synchronized connection 🧠:
(1) The connection state gets two new 32-bit slots 🎀: TS.Recent holds a timestamp to be echoed in TSecr whenever a segment is sent, and Last.ACK.sent holds the ACK field from the last segment sent. Last.ACK.sent will equal RCV.NXT except when <ACK>s have been delayed 🦋.
(2) If:
SEG.TSval >= TS.Recent and SEG.SEQ <= Last.ACK.sentthen SEG.TSval is copied to TS.Recent; otherwise, it is ignored 💅.
(3) When a TSopt is sent, its TSecr field is set to the current TS.Recent value 🫶.
these examples show how the rules play out 🔍. Here A, B, C... are data segments that occupy successive blocks of sequence numbers, and ACK(A),... are the matching acknowledgment segments. Notice that ACK(A) has the same sequence number as B. We only show one direction of timestamp echoing, for clarity ✨.
- Segments arrive in sequence, and some of the <ACK>s are delayed.
By case (A), the timestamp from the oldest unacknowledged segment gets echoed 🍓.
TS.Recent
<A, TSval=1> ------------------->
1
<B, TSval=2> ------------------->
1
<C, TSval=3> ------------------->
1
<---- <ACK(C), TSecr=1>
(etc.)- Segments arrive out of order, and every segment is acknowledged.
By case (B), the timestamp from the last segment that advanced the left window edge is echoed until the missing segment arrives. After that, it's echoed according to case (C) 🌸. The same sequence would happen if segments B and D were lost and retransmitted 🔁.
TS.Recent
<A, TSval=1> ------------------->
1
<---- <ACK(A), TSecr=1>
1
<C, TSval=3> ------------------->
1
<---- <ACK(A), TSecr=1>
1
<B, TSval=2> ------------------->
2
<---- <ACK(C), TSecr=2>
2
<E, TSval=5> ------------------->
2
<---- <ACK(C), TSecr=2>
2
<D, TSval=4> ------------------->
4
<---- <ACK(E), TSecr=4>
(etc.)5. PAWS - Protection Against Wrapped Sequences ✨
5.1. Introduction 🎀
the Timestamps option has another job, and it's called PAWS 🧚. Section 5.2 describes a simple mechanism for rejecting old duplicate segments that could corrupt an open TCP connection 🙅♀️. PAWS works inside a single TCP connection and uses state saved in the connection control block 💜. Section 5.8 and Appendix H talk about what PAWS means for avoiding old duplicates from earlier incarnations of the same connection 👀.
5.2. The PAWS Mechanism 💖
PAWS uses the TCP Timestamps option we went over earlier, and it assumes that every received TCP segment (including data and <ACK> segments) carries a timestamp SEG.TSval whose values are monotonically non-decreasing in time ⏰. The big idea is that a segment can be thrown away as an old duplicate if it arrives with a timestamp SEG.TSval that's less than some timestamps recently received on this connection 🗑️.
in the PAWS mechanism, the "timestamps" are 32-bit unsigned integers in a modular 32-bit space 🔢. So "less than" is defined the same way it is for TCP sequence numbers, and the same implementation techniques apply 🤓. If s and t are timestamp values:
s < t if 0 < (t - s) < 2^31,this is computed in unsigned 32-bit arithmetic ✨.
the incoming timestamps saved for this comparison MUST guarantee a value that is monotonically non-decreasing 🫶. For example, an implementation might save the timestamp from the segment that last advanced the left edge of the receive window, which is the most recent in-sequence segment 🌸. For simplicity, though, the value TS.Recent (introduced in Section 4.3) is used instead, since using one common value for both PAWS and RTTM keeps the implementation simpler 🧠. As Section 4.3 explained, TS.Recent only differs from the timestamp of the last in-sequence segment when <ACK>s are delayed, and then only by less than one window 🪟. Either choice will, therefore, protect against sequence number wrap-around 🛡️.
PAWS runs every incoming segment through the same test, so it protects against duplicate <ACK> segments as well as data segments 🔁. (A non-symmetric alternative could protect against old duplicate <ACK>s: the sender of data would reject incoming <ACK> segments whose TSecr values were less than the TSecr saved from the last segment whose ACK field advanced the left edge of the send window. That idea was judged to lack economy of mechanism and symmetry 🙄.)
TSval timestamps sent on <SYN> and <SYN,ACK> segments are used to initialize PAWS 🚀. PAWS protects against old duplicate non-<SYN> segments and duplicate <SYN> segments received while there is a synchronized connection 🔐. Duplicate <SYN> and <SYN,ACK> segments received when there is no connection get thrown out by the normal 3-way handshake and sequence number checks of TCP 👋.
[RFC1323] recommended that <RST> segments NOT carry timestamps, and that they be acceptable regardless of their timestamp 🙃. Back then, the thinking was that old duplicate <RST> segments should be super unlikely, and their cleanup job should win out over timestamps 🧹. More recently, discussions about various blind attacks on TCP connections have raised the idea that if the Timestamps option is present, SEG.TSecr could be used to give <RST> segments stricter acceptance tests 🕵️♀️.
this is still under discussion, but to let researchers explore this area, it is now RECOMMENDED that when generating an <RST>, if the segment causing the <RST> to be generated contains a Timestamps option, the <RST> also contains a Timestamps option 🧪. In the <RST> segment, SEG.TSecr SHOULD be set to SEG.TSval from the incoming segment, and SEG.TSval SHOULD be set to zero 🎯. If an <RST> is being generated because of a user abort, and Snd.TS.OK is set, then a Timestamps option SHOULD be included in the <RST> 🙌. When an <RST> segment is received, it MUST NOT be subjected to the PAWS check by verifying an acceptable value in SEG.TSval, and information from the Timestamps option MUST NOT be used to update connection state information 🚫. SEG.TSecr MAY be used to provide stricter <RST> acceptance checks 💅.
5.3. Basic PAWS Algorithm ✨
okay bestie, this is the core PAWS routine and it is literally the main character of this whole section 💅 If PAWS is used, the following processing MUST be performed on all incoming segments for a synchronized connection. PAWS processing MUST take precedence over the regular TCP acceptability check (Section 3.3 in [RFC0793]), which runs after the received Timestamps option has been verified 🫶
R1) If there is a Timestamps option in the arriving segment,
SEG.TSval < TS.Recent, TS.Recent is valid (see later
discussion), and if the RST bit is not set, then treat the
arriving segment as not acceptable:
Send an acknowledgment in reply as specified in Section 3.9
of [RFC0793], page 69, and drop the segment.
Note: it is necessary to send an <ACK> segment in order to
retain TCP's mechanisms for detecting and recovering from
half-open connections. For an example, see Figure 10 of
[RFC0793].
R2) If the segment is outside the window, reject it (normal TCP
processing).
R3) If an arriving segment satisfies SEG.TSval >= TS.Recent and
SEG.SEQ <= Last.ACK.sent (see Section 4.3), then record its
timestamp in TS.Recent.
R4) If an arriving segment is in sequence (i.e., at the left window
edge), then accept it normally.
R5) Otherwise, treat the segment as a normal in-window,
out-of-sequence TCP segment (e.g., queue it for later delivery
to the user).Steps R2, R4, and R5 are just the normal TCP processing steps from [RFC0793], so the only new stuff here is R1 and R3 🌸
It's super important that the timestamp MUST be checked only when a segment first arrives at the receiver 🚨 It doesn't matter if the segment is in sequence or has to be queued for later, the check happens once, right at arrival.
Here's the example they give, and it's so good 🦋
Suppose the segment sequence: A.1, B.1, C.1, ..., Z.1 has been
sent, where the letter indicates the sequence number and the digit
represents the timestamp. Suppose also that segment B.1 has been
lost. The timestamp in TS.Recent is 1 (from A.1), so C.1, ...,
Z.1 are considered acceptable and are queued. When B is
retransmitted as segment B.2 (using the latest timestamp), it
fills the hole and causes all the segments through Z to be
acknowledged and passed to the user. The timestamps of the queued
segments are *not* inspected again at this time, since they have
already been accepted. When B.2 is accepted, TS.Recent is set to
2.So B.1 gets lost, C.1 through Z.1 pile up in the queue, and when B.2 finally shows up, everything gets acked and handed over to the user. The queued ones don't get re-checked, and TS.Recent just becomes 2 once B.2 is accepted. 🧚
This rule gives reasonable performance under loss, and it's lowkey genius. A full window of data is in transit at all times. After a loss, a full window less one segment will show up out of sequence and get queued at the receiver (e.g., up to ~2^30 bytes of data). The Timestamps option must not result in discarding that data 💜
Now the spicy part: rules R1 through R5 could, in very unlikely circumstances, throw away some segments they didn't need to. Check this one out 👀
Suppose again that segments: A.1, B.1, C.1, ..., Z.1 have been
sent in sequence and that segment B.1 has been lost. Furthermore,
suppose delivery of some of C.1, ... Z.1 is delayed until *after*
the retransmission B.2 arrives at the receiver. These delayed
segments will be discarded unnecessarily when they do arrive,
since their timestamps are now out of date.This case is very unlikely to happen 😭 If the retransmission was triggered by a timeout, some of C.1 through Z.1 must have been delayed longer than the RTO time. That's presumably rare, or there would be tons of spurious timeouts and retransmissions. If B's retransmission was triggered by the "Fast Retransmit" algorithm (aka duplicate <ACK>s), then the queued segments that caused those <ACK>s must have already been received, so no drama there.
Even if a segment got delayed past the RTO, the Fast Retransmit mechanism [Jacobson90c] will cause the delayed segments to be retransmitted at the same time as B.2. That avoids an extra RTT, so the penalty is very small 🙈
And the bottom line is: they know of no case with a significant chance of happening where timestamps cause performance problems by needlessly discarding segments. Not me being impressed, but also kind of me being impressed 🔥
5.4. Timestamp Clock 🕰️
It's really important to get this one: PAWS does NOT need the sender and receiver to have synchronized clocks 🙅♀️ The sender's timestamp clock is just a source of monotonic non-decreasing values to stamp segments. The receiver treats the timestamp value as a monotonically non-decreasing serial number, with no connection to time at all. From the receiver's point of view, the timestamp is basically a logical extension of the high-order bits of the sequence number. Iconic 💅
The receiver algorithm does put some requirements on how fast the timestamp clock ticks.
- (a) The timestamp clock must not be "too slow". It MUST tick at least once for each 2^31 bytes sent. In fact, so it's useful to the sender for round-trip timing, the clock SHOULD tick at least once per window's worth of data. And even with the window extension defined in Section 2.2, 2^31 bytes must be at least two windows.
- To make this more concrete: any clock faster than 1 tick/sec will reject old duplicate segments for link speeds of ~8 Gbps. A 1 ms timestamp clock will work at link speeds up to 8 Tbps (8*10^12) bps! 🚀
- (b) The timestamp clock must not be "too fast". The recycling time of the timestamp clock MUST be greater than MSL seconds. Since the clock (timestamp) is 32 bits and the worst-case MSL is 255 seconds, the maximum acceptable clock frequency is one tick every 59 ns.
- It's still desirable to have a much longer recycle period. That lets you handle outdated timestamps on idle connections (see Section 5.5) and relax the MSL requirement for preventing sequence number wrap-around. With a 1 ms timestamp clock, the 32-bit timestamp will wrap its sign bit in 24.8 days. So it will reject old duplicates on the same connection if MSL is 24.8 days or less. That's a very safe figure, and an MSL of 24.8 days or longer can probably be assumed on the Internet without precise MSL enforcement 🌈
Based on all this, they choose a timestamp clock frequency in the range 1 ms to 1 sec per tick ⏱️ That range also matches what RTTM needs, since RTTM doesn't need much more resolution than the retransmit timer's granularity, e.g., tens or hundreds of milliseconds.
PAWS also puts a strong monotonicity requirement on the sender's timestamp clock 👑 How you get there depends on the system's hardware and software. Some options they list:
- Some hosts have a hardware clock that is guaranteed to be monotonic between hardware resets.
- A clock interrupt may be used to simply increment a binary integer by 1 periodically.
- The timestamp clock may be derived from a system clock that is subject to being abruptly changed by adding a variable offset value. This offset is initialized to zero. When a new timestamp clock value is needed, the offset can be adjusted as necessary to make the new value equal to or larger than the previous value (which was saved for this purpose).
- A random offset may be added to the timestamp clock on a per-connection basis. See [RFC6528], Section 3, on randomizing the initial sequence number (ISN). The same function with a different secret key can be used to generate the per-connection timestamp offset.
5.5. Outdated Timestamps 🥀
Okay, so if a connection sits idle long enough for the other TCP's timestamp clock to wrap its sign bit, the value saved in TS.Recent gets too old 😬 Then PAWS will reject every subsequent segment, and the connection freezes until that clock wraps its sign bit again. Major dead-chat energy.
With the chosen range of timestamp clock frequencies (1 sec to 1 ms), the time to wrap the sign bit is between 24.8 days and 24800 days. A TCP connection that's idle for more than 24 days and then comes back to life is super unusual. Still, it's undesirable in principle to put any limit on TCP connection lifetimes, so they've decided:
We therefore require that an implementation of PAWS include a mechanism to "invalidate" the TS.Recent value when a connection is idle for more than 24 days. (An alternative would've been sending keep-alive segments at a very low rate, but still more often than the wrap-around time for timestamps, e.g., once a day. That would impose negligible overhead. However, the TCP specification has never included keep-alives, so the invalidation solution was chosen.) 🫠
Note that a TCP doesn't know the frequency, and therefore the wrap-around time, of the other TCP, so it has to assume the worst 🤷♀️ The validity of TS.Recent only needs to be checked if the basic PAWS timestamp check fails, i.e., only if SEG.TSval < TS.Recent. If TS.Recent turns out to be invalid, the segment is accepted no matter that the timestamp check failed, and rule R3 updates TS.Recent with the TSval from the new segment ✨
To detect how long the connection has been idle, the TCP MAY update a clock or timestamp value associated with the connection whenever TS.Recent is updated, for example. The details are implementation dependent 💖
5.6. Header Prediction 🔮
okay so "header prediction" [Jacobson90a] 🔥 is a high-performance transport protocol trick, and it's literally most important for high-speed links 🚀. The whole idea is to optimize for the most common case: receiving a segment correctly and in order 💅. So the receiver asks, "Is this segment the next in sequence?" 🤔 and that can be answered in fewer machine instructions than "Is this segment within the window?" ⚡ efficiency is the main character here fr
Adding header prediction to our timestamp procedure gives us this recommended sequence for processing an arriving TCP segment 📋:
- H1) Check timestamp (same as step R1 above). ⏰
- H2) Do header prediction: if the segment is next in sequence and if there are no special conditions requiring additional processing, accept the segment, record its timestamp, and skip H3. 🎯
- H3) Process the segment normally, as specified in RFC 793. This includes dropping segments that are outside the window and possibly sending acknowledgments, and queuing in-window, out-of-sequence segments. 🧺
Another possibility would be to interchange steps H1 and H2, i.e., to perform the header prediction step H2 first, and perform H1 and H3 only when header prediction fails. 🤷♀️ This could be a performance improvement, since the timestamp check in step H1 is very unlikely to fail, and it requires unsigned modulo arithmetic. But doing this check on every single segment is contrary to the philosophy of header prediction 🙃. We believe that this change might produce a measurable reduction in CPU time for TCP protocol processing on high-speed networks ✨.
However, putting H2 first would create a hazard 💀: a segment from 2^32 bytes in the past might arrive at exactly the wrong time and be accepted mistakenly by the header-prediction step. The following reasoning was introduced in [RFC1185] to show that the probability of this failure is negligible 🧮.
If all segments are equally likely to show up as old duplicates, then the probability of an old duplicate exactly matching the left window edge is the maximum segment size (MSS) divided by the size of the sequence space. This ratio must be less than 2^-16, since MSS must be < 2^16; for example, it will be (2^12)/(2^32) = 2^-20 for [a 100 Mbit/s] link. However, the older a segment is, the less likely it is to be retained in the Internet, and under any reasonable model of segment lifetime the probability of an old duplicate exactly at the left window edge must be much smaller than 2^-16. The 16 bit TCP checksum also allows a basic unreliability of one part in 2^16. A protocol mechanism whose reliability exceeds the reliability of the TCP checksum should be considered "good enough", i.e., it won't contribute significantly to the overall error rate. We therefore believe we can ignore the problem of an old duplicate being accepted by doing header prediction before checking the timestamp. [Note: the notation for exponentiation has been changed from how it appeared in RFC 1185.]
However, this probabilistic argument is not universally accepted 🫠, and the consensus at present is that the performance gain does not justify the hazard in the general case. It is therefore recommended that H2 follow H1.
5.7. IP Fragmentation 🧩
At high data rates, the protection against old segments provided by PAWS can be circumvented by errors in IP fragment reassembly (see [RFC4963]). 😬 The only way to protect against incorrect IP fragment reassembly is to not allow the segments to be fragmented. This is done by setting the Don't Fragment (DF) bit in the IP header.
Setting the DF bit implies the use of Path MTU Discovery as described in [RFC1191], [RFC1981], and [RFC4821]; thus, any TCP implementation that implements PAWS MUST also implement Path MTU Discovery. 📢
5.8. Duplicates from Earlier Incarnations of Connection 👯
The PAWS mechanism protects against errors due to sequence number wrap-around on high-speed connections. Segments from an earlier incarnation of the same connection are also a potential cause of old duplicate errors. In both cases, the TCP mechanisms to prevent such errors depend upon the enforcement of an MSL by the Internet (IP) layer (see the Appendix of RFC 1185 for a detailed discussion). 📚 Unlike the case of sequence space wrap-around, the MSL required to prevent old duplicate errors from earlier incarnations does not depend upon the transfer rate. If the IP layer enforces the recommended 2-minute MSL of TCP, and if the TCP rules are followed, TCP connections will be safe from earlier incarnations, no matter how high the network speed. 🛡️ Thus, the PAWS mechanism is not required for this case.
We may still ask whether the PAWS mechanism can provide additional security against old duplicates from earlier connections, allowing us to relax the enforcement of MSL by the IP layer. 🤔 Appendix B explores this question, showing that further assumptions and/or mechanisms are required, beyond those of PAWS. This is not part of the current extension.
6. Conclusions and Acknowledgments 🎉
This memo presented a set of extensions to TCP to provide efficient operation over large bandwidth * delay product paths and reliable operation over very high-speed paths. 🚄 These extensions are designed to provide compatible interworking with TCP stacks that do not implement the extensions.
These mechanisms are implemented using TCP options for scaled windows and timestamps. The timestamps are used for two distinct mechanisms: RTTM and PAWS. 💖
The Window Scale option was originally suggested by Mike St. Johns of USAF/DCA. The present form of the option was suggested by Mike Karels of UC Berkeley in response to a more cumbersome scheme defined by Van Jacobson. Lixia Zhang helped formulate the PAWS mechanism description in [RFC1185]. 🙌
Finally, much of this work originated as the result of discussions within the End-to-End Task Force on the theoretical limitations of transport protocols in general and TCP in particular. 🧠 Task force members and others on the end2end-interest list have made valuable contributions by pointing out flaws in the algorithms and the documentation. Continued discussion and development since the publication of [RFC1323] originally occurred in the IETF TCP Large Windows Working Group, later on in the End-to-End Task Force, and most recently in the IETF TCP Maintenance Working Group. The authors are grateful for all these contributions. 🫶
7. Security Considerations 🚨
The TCP sequence space is a fixed size, and as the window becomes larger, it becomes easier for an attacker to generate forged packets that can fall within the TCP window and be accepted as valid segments. 😱 While use of timestamps and PAWS can help to mitigate this, when using PAWS, if an attacker is able to forge a packet that is acceptable to the TCP connection, a timestamp that is in the future would cause valid segments to be dropped due to PAWS checks. Hence, implementers should take care to not open the TCP window drastically beyond the requirements of the connection.
See [RFC5961] for mitigation strategies to blind in-window attacks. 🔒
A naive implementation that derives the timestamp clock value directly from a system uptime clock may unintentionally leak this information to an attacker. This does not directly compromise any of the mechanisms described in this document. However, this may be valuable information to a potential attacker. It is therefore RECOMMENDED to generate a random, per-connection offset to be used with the clock source when generating the Timestamps option value (see Section 5.4). By carefully choosing this random offset, further improvements as described in [RFC6191] are possible. 🎲
Expanding the TCP window beyond 64 KiB for IPv6 allows Jumbograms [RFC2675] to be used when the local network supports packets larger than 64 KiB. When larger TCP segments are used, the TCP checksum becomes weaker. 🥀
Mechanisms to protect the TCP header from modification should also protect the TCP options. 🛡️
Middleboxes and TCP options:
Some middleboxes have been known to remove the TCP options described in this document from TCP segments [Honda11]. Middleboxes that remove TCP options described in this document from the <SYN> segment interfere with the selection of parameters appropriate for the session. Removing any of these options in a <SYN,ACK> segment will leave the end hosts in a state that destroys the proper operation of the protocol. 💥
- If a Window Scale option is removed from a
<SYN,ACK>segment, the end hosts will not negotiate the window scaling factor correctly. Middleboxes must not remove or modify the Window Scale option from<SYN,ACK>segments. - If a stateful firewall uses the window field to detect whether a received segment is inside the current window, and does not support the Window Scale option, it will not be able to correctly determine whether or not a packet is in the window. These middle boxes must also support the Window Scale option and apply the scale factor when processing segments. If the window scale factor cannot be determined, it must not do window-based processing. 🧱
- If the Timestamps option is removed from the
<SYN>or<SYN,ACK>segments, high speed connections that need PAWS would not have that protection. Successful negotiation of the Timestamps option enforces a stricter verification of incoming segments at the receiver. If the Timestamps option was removed from a subsequent data segment after a successful negotiation (e.g., as part of resegmentation), the segment is discarded by the receiver without further processing. Middleboxes should not remove the Timestamps option. ⏱️ - It must be noted that [RFC1323] doesn't address the case of the Timestamps option being dropped or selectively omitted after being negotiated, and that the update in this document may cause some broken middlebox behavior to be detected (potentially unresponsive TCP sessions). 👀
Implementations that depend on PAWS could provide a mechanism for the application to determine whether or not PAWS is in use on the connection and choose to terminate the connection if that protection doesn't exist. This is not just to protect the connection against middleboxes that might remove the Timestamps option, but also against remote hosts that do not have Timestamp support. 🧯
7.1. Privacy Considerations 🙈
The TCP options described in this document do not expose individual user's data. 🫣 However, a naive implementation simply using the system clock as a source for the Timestamps option will reveal characteristics of the TCP, potentially allowing more targeted attacks. It is therefore RECOMMENDED to generate a random, per-connection offset to be used with the clock source when generating the Timestamps option value (see Section 5.4).
Furthermore, the combination, relative ordering, and padding of the TCP options described in Sections 2.2 and 3.2 will reveal additional clues to allow the fingerprinting of the system. 🕵️♀️
8. IANA Considerations ✨
Okay so the TCP options in this doc are already super well known, because they came from the old [RFC1323] (now superseded, RIP 🕊️). IANA updated the "TCP Option Kind Numbers" table under "TCP Parameters" so it points at this document, RFC 7323, for both "Window Scale" and "Timestamps". Iconic paperwork moment 📋💅
9. References
9.1. Normative References
[RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981. 🌸
[RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, November 1990.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 🔑
9.2. Informative References
[Allman99] Allman, M. and V. Paxson, "On Estimating End-to-End Network Path Properties", Proceedings of the ACM SIGCOMM Technical Symposium, Cambridge, MA, September 1999, <http://aciri.org/mallman/papers/estimation-la.pdf>. 📄
[Floyd05] Floyd, S., "Subject: Re: [tcpm] RFC 1323: Timestamps option", message to the TCPM mailing list, 26 January 2007, <http://www.ietf.org/mail-archive/web/tcpm/current/msg02508.html>. 💌
[Garlick77] Garlick, L., Rom, R., and J. Postel, "Issues in Reliable Host-to-Host Protocols", Proceedings of the Second Berkeley Workshop on Distributed Data Management and Computer Networks, March 1977, <http://www.rfc-editor.org/ien/ien12.txt>. 🦋
[Honda11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., Handley, M., and H. Tokuda, "Is it Still Possible to Extend TCP?", Proceedings of the ACM Internet Measurement Conference (IMC) '11, November 2011. 🧐
[Jacobson88a] Jacobson, V., "Congestion Avoidance and Control", SIGCOMM '88, Stanford, CA, August 1988, <http://ee.lbl.gov/papers/congavoid.pdf>. ✨
[Jacobson90a] Jacobson, V., "4BSD Header Prediction", ACM Computer Communication Review, April 1990. 🎀
[Jacobson90c] Jacobson, V., "Subject: modified TCP congestion avoidance algorithm", message to the End2End-Interest mailing list, 30 April 1990, <ftp://ftp.isi.edu/end2end/end2end-interest-1990.mail>. 📬
[Karn87] Karn, P. and C. Partridge, "Estimating Round-Trip Times in Reliable Transport Protocols", Proceedings of SIGCOMM '87, August 1987. ⏱️
[Kuehlewind10] Kuehlewind, M. and B. Briscoe, "Chirping for Congestion Control - Implementation Feasibility", November 2010, <http://bobbriscoe.net/projects/netsvc_i-f/chirp_pfldnet10.pdf>. 🐦
[Kuzmanovic03] Kuzmanovic, A. and E. Knightly, "TCP-LP: Low-Priority Service via End-Point Congestion Control", 2003, <www.cs.northwestern.edu/~akuzma/doc/TCP-LP-ToN.pdf>. 🐢
[Ludwig00] Ludwig, R. and K. Sklower, "The Eifel Retransmission Timer", ACM SIGCOMM Computer Communication Review Volume 30 Issue 3, July 2000, <http://ccr.sigcomm.org/archive/2000/july00/LudwigFinal.pdf>. 🔍
[Martin03] Martin, D., "Subject: [Tsvwg] RFC 1323.bis", message to the TSVWG mailing list, 30 September 2003, <http://www.ietf.org/mail-archive/web/tsvwg/current/msg04435.html>. 💬
[Medina04] Medina, A., Allman, M., and S. Floyd, "Measuring Interactions Between Transport Protocols and Middleboxes", Proceedings of the ACM SIGCOMM/USENIX Internet Measurement Conference, October 2004, <http://www.icir.net/tbit/tbit-Aug2004.pdf>. 🌐
[Medina05] Medina, A., Allman, M., and S. Floyd, "Measuring the Evolution of Transport Protocols in the Internet", ACM Computer Communication Review Volume 35, No. 2, April 2005, <http://icir.net/floyd/papers/TCPevolution-Mar2005.pdf>. 📈
[RE-1323BIS] Oppermann, A., "Subject: Re: [tcpm] I-D Action: draft-ietf.tcpm-1323bis-13.txt", message to the TCPM mailing list, 01 June 2013, <http://www.ietf.org/mail-archive/web/tcpm/current/msg08001.html>. 📨
[RFC1072] Jacobson, V. and R. Braden, "TCP extensions for long-delay paths", RFC 1072, October 1988. 🛰️
[RFC1122] Braden, R., "Requirements for Internet Hosts - Communication Layers", STD 3, RFC 1122, October 1989. 🏗️
[RFC1185] Jacobson, V., Braden, B., and L. Zhang, "TCP Extension for High-Speed Paths", RFC 1185, October 1990. 🚄
[RFC1323] Jacobson, V., Braden, B., and D. Borman, "TCP Extensions for High Performance", RFC 1323, May 1992. 👑
[RFC1981] McCann, J., Deering, S., and J. Mogul, "Path MTU Discovery for IP version 6", RFC 1981, August 1996. 🧭
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP Selective Acknowledgment Options", RFC 2018, October 1996. 🧩
[RFC2675] Borman, D., Deering, S., and R. Hinden, "IPv6 Jumbograms", RFC 2675, August 1999. 🐘
[RFC2883] Floyd, S., Mahdavi, J., Mathis, M., and M. Podolsky, "An Extension to the Selective Acknowledgement (SACK) Option for TCP", RFC 2883, July 2000. 🧩
[RFC3522] Ludwig, R. and M. Meyer, "The Eifel Detection Algorithm for TCP", RFC 3522, April 2003. 🕵️
[RFC4015] Ludwig, R. and A. Gurtov, "The Eifel Response Algorithm for TCP", RFC 4015, February 2005. 🛠️
[RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU Discovery", RFC 4821, March 2007. 📦
[RFC4963] Heffner, J., Mathis, M., and B. Chandler, "IPv4 Reassembly Errors at High Data Rates", RFC 4963, July 2007. 💥
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681, September 2009. 🚦
[RFC5961] Ramaiah, A., Stewart, R., and M. Dalal, "Improving TCP's Robustness to Blind In-Window Attacks", RFC 5961, August 2010. 🛡️
[RFC6191] Gont, F., "Reducing the TIME-WAIT State Using TCP Timestamps", BCP 159, RFC 6191, April 2011. ⏳
[RFC6298] Paxson, V., Allman, M., Chu, J., and M. Sargent, "Computing TCP's Retransmission Timer", RFC 6298, June 2011. ⏲️
[RFC6528] Gont, F. and S. Bellovin, "Defending against Sequence Number Attacks", RFC 6528, February 2012. 🔐
[RFC6675] Blanton, E., Allman, M., Wang, L., Jarvinen, I., Kojo, M., and Y. Nishida, "A Conservative Loss Recovery Algorithm Based on Selective Acknowledgment (SACK) for TCP", RFC 6675, August 2012. 🌷
[RFC6691] Borman, D., "TCP Options and Maximum Segment Size (MSS)", RFC 6691, July 2012. 📏
[RFC6817] Shalunov, S., Hazel, G., Iyengar, J., and M. Kuehlewind, "Low Extra Delay Background Transport (LEDBAT)", RFC 6817, December 2012. 🌙
Appendix A. Implementation Suggestions 🛠️
TCP Option Layout
The following layout is recommended for sending options on non-<SYN> segments to achieve maximum feasible alignment of 32-bit and 64-bit machines. 🧱
+--------+--------+--------+--------+
| NOP | NOP | TSopt | 10 |
+--------+--------+--------+--------+
| TSval timestamp |
+--------+--------+--------+--------+
| TSecr timestamp |
+--------+--------+--------+--------+Interaction with the TCP Urgent Pointer
The TCP Urgent Pointer, like the TCP window, is a 16-bit value. Some of the original discussion for the TCP Window Scale option included proposals to increase the Urgent Pointer to 32 bits. As it turns out, this is unnecessary. There are two observations that should be made:
(1) With IP version 4, the largest amount of TCP data that can be sent in a single packet is 65495 bytes (64 KiB - 1 - size of fixed IP and TCP headers).
(2) Updates to the Urgent Pointer while the user is in "urgent mode" are invisible to the user.
This means that if the Urgent Pointer points beyond the end of the TCP data in the current segment, then the user will remain in urgent mode until the next TCP segment arrives. That segment will update the Urgent Pointer to a new offset, and the user will never have left urgent mode. 🚨
Thus, to properly implement the Urgent Pointer, the sending TCP only has to check for overflow of the 16-bit Urgent Pointer field before filling it in. If it does overflow, then a value of 65535 should be inserted into the Urgent Pointer. 🙌
The same technique applies to IP version 6, except in the case of IPv6 Jumbograms. When IPv6 Jumbograms are supported, [RFC2675] requires additional steps for dealing with the Urgent Pointer; these steps are described in Section 5.2 of [RFC2675]. 🐘
Appendix B. Duplicates from Earlier Connection Incarnations 🦋
There are two cases to be considered: (1) a system crashing (and losing connection state) and restarting, and (2) the same connection being closed and reopened without a loss of host state. These will be described in the following two sections. 🔄
B.1. System Crash with Loss of State 💥
TCP's quiet time of one MSL upon system startup handles the loss of connection state in a system crash/restart. For an explanation, see, for example, "Knowing When to Keep Quiet" in the TCP protocol specification [RFC0793]. The MSL that is required here does not depend upon the transfer speed. The current TCP MSL of 2 minutes seemed acceptable as an operational compromise, when many host systems used to take this long to boot after a crash. Current host systems can boot considerably faster. 🚀
The Timestamps option may be used to ease the MSL requirements (or to provide additional security against data corruption). If timestamps are being used and if the timestamp clock can be guaranteed to be monotonic over a system crash/restart, i.e., if the first value of the sender's timestamp clock after a crash/restart can be guaranteed to be greater than the last value before the restart, then a quiet time is unnecessary. ⏰
To dispense totally with the quiet time would require that the host clock be synchronized to a time source that is stable over the crash/restart period, with an accuracy of one timestamp clock tick or better. We can back off from this strict requirement to take advantage of approximate clock synchronization. Suppose that the clock is always resynchronized to within N timestamp clock ticks and that booting (extended with a quiet time, if necessary) takes more than N ticks. This will guarantee monotonicity of the timestamps, which can then be used to reject old duplicates even without an enforced MSL. 🎀
B.2. Closing and Reopening a Connection 🎀
okay so when a TCP connection closes, the 2*MSL delay in TIME-WAIT state keeps that socket pair tied up for 4 minutes 😭 (see Section 3.5 of [RFC0793]). apps that close one connection and open a fresh one, like an FTP data transfer connection in Stream mode, must choose a new socket pair every single time 💅. the TIME-WAIT delay has two jobs, bestie 👑
(a) Implement the full-duplex reliable close handshake of TCP. ✨
the timing for that final close step isn't really about the MSL. it depends on the RTO for the FIN segments, and that means it depends on the RTT of the path 🦋 (it could be argued that the side sending a FIN knows how much reliability it needs, so it should be able to pick the TIME-WAIT length for the FIN's recipient. this could be done with an appropriate TCP option in FIN segments 🧚)
there's no formal upper bound on RTT, but common network engineering practice makes an RTT over 1 minute super unlikely 🙈. so the 4-minute delay in TIME-WAIT state works just fine for a reliable full-duplex TCP close. again, this has nothing to do with MSL enforcement or network speed 💖
the TIME-WAIT state could cause an indirect performance problem if an app needed to close one connection and open another over and over at a really high frequency, because a host has fewer than 2^16 TCP ports available 💀. but high network speeds aren't the main cause of this. the RTT is the thing limiting how fast connections can be opened and closed 🍓 so this problem won't get any worse at high transfer speeds ✨
(b) Allow old duplicate segments to expire. 🌸
to replace this job of the TIME-WAIT state, a mechanism would have to work across connections. PAWS is defined strictly within a single connection 🙅♀️. the last timestamp (TS.Recent) lives in the connection control block and gets thrown away when the connection closes 🥀
an extra mechanism could be added to TCP: a per-host cache of the last timestamp received from any connection 🫶. this value could then be used in the PAWS mechanism to reject old duplicate segments from earlier incarnations of the connection, but only if the timestamp clock is guaranteed to have ticked at least once since the old connection was opened ⏰. that means the TIME-WAIT delay plus the RTT together must be at least one tick of the sender's timestamp clock 🔥. this kind of extension is not part of the proposal in this RFC, so don't go thinking it is 🙈
and yes, this is a variant on the mechanism proposed by Garlick, Rom, and Postel [Garlick77], which made each host keep connection records with the highest sequence numbers on every connection 📚. using timestamps instead means you only need to keep one quantity per remote host, no matter how many simultaneous connections go to that host 👑🌈
Appendix C. Summary of Notation ✨
the notation used throughout this document is all listed here, so you can keep it handy like a cheat sheet 🧚
Options 🎀
WSopt: TCP Window Scale option
TSopt: TCP Timestamps optionOption Fields 💖
shift.cnt: Window scale byte in WSopt
TSval: 32-bit Timestamp Value field in TSopt
TSecr: 32-bit Timestamp Reply field in TSoptOption Fields in Current Segment 🦋
SEG.TSval: TSval field from TSopt in current segment
SEG.TSecr: TSecr field from TSopt in current segment
SEG.WSopt: 8-bit value in WSoptClock Values ⏰
my.TSclock: System-wide source of 32-bit timestamp values
my.TSclock.rate: Period of my.TSclock (1 ms to 1 sec)
Snd.TSoffset: An offset for randomizing Snd.TSclock
Snd.TSclock: my.TSclock + Snd.TSoffsetPer-Connection State Variables 🌸
TS.Recent: Latest received Timestamp
Last.ACK.sent: Last ACK field sent
Snd.TS.OK: 1-bit flag
Snd.WS.OK: 1-bit flag
Rcv.Wind.Shift: Receive window scale exponent
Snd.Wind.Shift: Send window scale exponent
Start.Time: Snd.TSclock value when the segment being timed
was sent (used by code from before RFC 1323).Procedure 🔥
Update_SRTT(m) Procedure to update the smoothed RTT and RTT
variance estimates, using the rules of
[Jacobson88a], given m, a new RTT measurementSend Sequence Variables 💅
SND.UNA: Send unacknowledged
SND.NXT: Send next
SND.WND: Send window
ISS: Initial send sequence numberReceive Sequence Variables 🫶
RCV.NXT: Receive next
RCV.WND: Receive window
IRS: Initial receive sequence numberAppendix D. Event Processing Summary 🌸
okay so this appendix is where we get super specific about the algorithms 💅 It does that by showing the Event Processing rules from Section 3.9 of RFC 793 with the changes marked. Any line with a change bar (|) is different from RFC 793, and everything else is the same as before. no cap, it's like a highlighter pass over an old essay 🖍️
OPEN Call 🚪
When you OPEN a connection, you still pick an initial send sequence number (ISS), but now your SYN segment carries two new options. The timestamp and window scale stuff lives in the change-bar line:
| Send a <SYN>
| segment of the form:
|
| <SEQ=ISS><CTL=SYN><TSval=Snd.TSclock><WSopt=Rcv.Wind.Shift>The rest of the OPEN call is the usual RFC 793 business (the ... bits are just the unchanged parts) ✨
SEND Call 📨
Depending on what state you're in, SEND does different things. The changes all show up in the LISTEN and ESTABLISHED states, so let's look at those.
CLOSED STATE (i.e., TCB does not exist) and SYN-SENT STATE / SYN-RECEIVED STATE are unchanged from RFC 793 💖
LISTEN STATE
If you're active and the foreign socket is specified, you switch the connection from passive to active and select an ISS. Then you send a SYN with two new options. The change-bar part looks like this:
| Send a SYN
| segment containing the options: <TSval=Snd.TSclock> and
| <WSopt=Rcv.Wind.Shift>. Set SND.UNA to ISS, SND.NXT to ISS+1.After that you enter SYN-SENT state. The ... means the rest is the same as before 🫶
ESTABLISHED STATE and CLOSE-WAIT STATE
You segmentize the buffer and send it with a piggybacked acknowledgment (acknowledgment value = RCV.NXT). If the urgent flag is set, the usual RFC 793 steps apply. The new stuff is the timestamps and the window scaling:
| If the Snd.TS.OK flag is set, then include the TCP Timestamps
| option <TSval=Snd.TSclock,TSecr=TS.Recent> in each data
| segment.
|
| Scale the receive window for transmission in the segment
| header:
|
| SEG.WND = (RCV.WND >> Rcv.Wind.Shift).Translation: if the timestamps flag is on, every data segment gets a timestamps option, and the window is shifted right by the scale factor before it goes in the header. bestie that's the whole glow-up 🌈
SEGMENT ARRIVES 📬
When a segment shows up, the first thing you check depends on your state. Here's the LISTEN branch, which is where most of the new options get handled ✨
LISTEN branch
First you check for an RST, then an ACK. For a SYN, you check the security first. If the SYN bit is set, you check the security, and if the SEG.PRC is less than the TCB.PRC, you keep going. The big new part is the option parsing:
| Check for a Window Scale option (WSopt); if one is found,
| save SEG.WSopt in Snd.Wind.Shift and set Snd.WS.OK flag on.
| Otherwise, set both Snd.Wind.Shift and Rcv.Wind.Shift to
| zero and clear Snd.WS.OK flag.
|
| Check for a TSopt option; if one is found, save SEG.TSval in
| the variable TS.Recent and turn on the Snd.TS.OK bit.Then you set RCV.NXT to SEG.SEQ+1, IRS is set to SEG.SEQ, and any other control or text gets queued for later. You pick an ISS and send a SYN, ACK segment shaped like this:
<SEQ=ISS><ACK=RCV.NXT><CTL=SYN,ACK>With the new options turned on, you add them to that segment:
| If the Snd.WS.OK bit is on, include a WSopt
| <WSopt=Rcv.Wind.Shift> in this segment. If the Snd.TS.OK
| bit is on, include a TSopt <TSval=Snd.TSclock,
| TSecr=TS.Recent> in this segment. Last.ACK.sent is set to
| RCV.NXT.SND.NXT becomes ISS+1 and SND.UNA becomes ISS, and the connection moves to SYN-RECEIVED. Any other control or data that came in with the SYN gets processed in SYN-RECEIVED, but SYN and ACK processing should not be repeated. If the listen wasn't fully specified (the foreign socket wasn't fully specified), fill in the blanks now. The ... part for "fourth other text or control" is unchanged 💖
SYN-SENT branch
If the SYN bit is on and the security/compartment and precedence are acceptable, you set RCV.NXT to SEG.SEQ+1 and IRS to SEG.SEQ. SND.UNA gets advanced to SEG.ACK (if there is an ACK), and any segments on the retransmission queue that got acknowledged are removed. The option handling here is:
| Check for a Window Scale option (WSopt); if it is found,
| save SEG.WSopt in Snd.Wind.Shift; otherwise, set both
| Snd.Wind.Shift and Rcv.Wind.Shift to zero.
|
| Check for a TSopt option; if one is found, save SEG.TSval in
| variable TS.Recent and turn on the Snd.TS.OK bit in the
| connection control block. If the ACK bit is set, use
| Snd.TSclock - SEG.TSecr as the initial RTT estimate.If SND.UNA > ISS (meaning your SYN got ACKed), you move to ESTABLISHED and send an ACK:
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>Turn the options on, and it looks like this:
| and send it. If the Snd.TS.OK bit is on, include a TSopt
| option <TSval=Snd.TSclock,TSecr=TS.Recent> in this <ACK>
| segment. Last.ACK.sent is set to RCV.NXT.Queued data or controls can go along for the ride. If there's other control or text in the segment, you keep going at the sixth step (where the URG bit is checked). Otherwise you just return. ✨
If our SYN hasn't been ACKed yet, you enter SYN-RECEIVED and send a SYN, ACK:
<SEQ=ISS><ACK=RCV.NXT><CTL=SYN,ACK>| and send it. If the Snd.TS.OK bit is on, include a TSopt
| option <TSval=Snd.TSclock,TSecr=TS.Recent> in this segment.
| If the Snd.WS.OK bit is on, include a WSopt option
| <WSopt=Rcv.Wind.Shift> in this segment. Last.ACK.sent is
| set to RCV.NXT.If there's other control or text, queue it for processing once ESTABLISHED has been reached, then return.
Last step for this branch: if neither the SYN nor RST bit is set, drop the segment and return 🗑️
Otherwise branch
First you check the sequence number, and this covers SYN-RECEIVED, ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, and TIME-WAIT. Segments get processed in sequence. Initial checks on arrival throw out old duplicates, but everything else is handled in SEG.SEQ order. If a segment straddles the old/new boundary, only the new parts get processed.
The new bit here is rescaling the received window and checking timestamps:
| Rescale the received window field:
|
| TrueWindow = SEG.WND << Snd.Wind.Shift,
|
| and use "TrueWindow" in place of SEG.WND in the following
| steps.
|
| Check whether the segment contains a Timestamps option and
| if bit Snd.TS.OK is on. If so:
|
| If SEG.TSval < TS.Recent and the RST bit is off:
|
| If the connection has been idle more than 24 days,
| save SEG.TSval in variable TS.Recent, else the segment
| is not acceptable; follow the steps below for an
| unacceptable segment.
|
| If SEG.TSval >= TS.Recent and SEG.SEQ <= Last.ACK.sent,
| then save SEG.TSval in variable TS.Recent.There are four cases for the acceptability test (the ... covers the usual RFC 793 cases). If a segment isn't acceptable, you send an ACK back, unless the RST bit is set (in which case you drop it and return):
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>With the option attached:
| Last.ACK.sent is set to SEG.ACK of the acknowledgment. If
| the Snd.TS.OK bit is on, include the Timestamps option
| <TSval=Snd.TSclock,TSecr=TS.Recent> in this <ACK> segment.
Set Last.ACK.sent to SEG.ACK and send the <ACK> segment.
After sending the acknowledgment, drop the unacceptable
segment and return.Fifth step: check the ACK field 🫧
If the ACK bit is off, drop the segment and return. If the ACK bit is on, then in ESTABLISHED, when SND.UNA < SEG.ACK <= SND.NXT, set SND.UNA <- SEG.ACK. You also compute a new round-trip time estimate. If Snd.TS.OK is on, you use Snd.TSclock - SEG.TSecr. Otherwise, you use the elapsed time since the first segment in the retransmission queue was sent. Any segments on the retransmission queue that are entirely acknowledged by this get removed:
| If SND.UNA < SEG.ACK <= SND.NXT then, set SND.UNA <-
| SEG.ACK. Also compute a new estimate of round-trip time.
| If Snd.TS.OK bit is on, use Snd.TSclock - SEG.TSecr;
| otherwise, use the elapsed time since the first segment
| in the retransmission queue was sent. Any segments on
the retransmission queue that are thereby entirely
acknowledged...Seventh step: process the segment text 📝
This is for ESTABLISHED, FIN-WAIT-1, and FIN-WAIT-2. When you send an acknowledgment, the usual form is:
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>If Snd.TS.OK is on, the Timestamps option gets added to it:
| If the Snd.TS.OK bit is on, include the Timestamps option
| <TSval=Snd.TSclock,TSecr=TS.Recent> in this <ACK> segment.
| Set Last.ACK.sent to SEG.ACK of the acknowledgment, and send
| it. This acknowledgment should be piggybacked on a segment
being transmitted if possible without incurring undue delay.Appendix E. Timestamps Edge Cases 🕰️
okay so the RTTM rules are right most of the time, but there are some edge cases where the RTTM gets calculated wrong 🙈 Every single one of them involves segment loss. These scenarios are considered rare. And if one does happen, it just makes a single RTTM measurement inflated, which limits how much it messes up the RTO calculations.
[Martin03] describes two similar cases where the returning <ACK> is lost. Before the retransmission timer fires, another returning <ACK> arrives that acknowledges the data. In that situation the RTTM gets inflated:
clock
tc=1 <A, TSval=1> ------------------->
tc=2 (lost) <---- <ACK(A), TSecr=1, win=n>
(RTTM would have been 1)
(receive window opens, window update is sent)
tc=5 <---- <ACK(A), TSecr=1, win=m>
(RTTM is calculated at 4)One thing to know: it's kind of bounded by RTO + RTT, which caps how far off the RTTM calculation can be. More complicated scenarios can make bigger inflations (say, when retransmissions also get lost), but those involve multiple segment losses. And at that point the connection has way bigger problems than an inflated RTTM in the RTO calculation 💀
Appendix F. Window Retraction Example 🪟
Picture an established TCP connection with a scale factor of 128, where Snd.Wind.Shift=7 and Rcv.Wind.Shift=7. It's running with a really small window, because the receiver is bottlenecked and both ends are doing small reads and writes.
Here are the ACKs coming back. Look at the BEYOND BUFFER and RETRACTED WINDOW rows at the bottom, they're the drama 🎭
SEG.ACK SEG.WIN computed SND.WIN receiver's actual window
1000 2 1256 1300The sender writes 40 bytes, and the receiver ACKs:
1040 2 1296 1300Then the sender writes 5 more bytes and the receiver has a problem. There are two choices:
1045 2 1301 1300 - BEYOND BUFFER
1045 1 1173 1300 - RETRACTED WINDOWThis is a general problem, and it can happen any time the sender does a write that's smaller than the window scale factor 😭
In most stacks, it's at least partially hidden when the window is bigger than a few segments. That's because stacks like to announce windows that are a whole number of segments, rounded up to the next scale factor. Combine that with silly window suppression and you get fewer, bigger window updates. If the window was rounded down to a segment size instead, there's more room to advance the window (the BEYOND BUFFER case above) rather than having to retract it 🦋
Appendix G. RTO Calculation Modification ✨
okay so taking multiple RTT samples per window would shorten the history that the RTO mechanism in [RFC6298] calculates 🙈 The algorithm below is built to keep a similar history to what [RFC6298] originally intended 🫶
It's roughly known how many samples one congestion window's worth of data will yield, not counting ACK compression and ACK losses, ngl. Those events mean more of the path's history ends up in the final RTO value, and they're uncritical 💅 This modification makes sure a similar amount of time is taken into account for the RTO estimation, no matter how many samples you take per window:
ExpectedSamples = ceiling(FlightSize / (SMSS * 2))
alpha' = alpha / ExpectedSamples
beta' = beta / ExpectedSamplesNote that the factor 2 in ExpectedSamples is due to "Delayed ACKs" 🌸
Instead of using alpha and beta in the algorithm of [RFC6298], use alpha' and beta' instead 🎀
RTTVAR <- (1 - beta') * RTTVAR + beta' * |SRTT - R'|
SRTT <- (1 - alpha') * SRTT + alpha' * R'
(for each sample R')Appendix H. Changes from RFC 1323 🦋
Several important updates and clarifications to the specification in RFC 1323 are made in this document. Here's the tea on the technical changes ☕
- (a) A wrong reference to
SND.WNDwas corrected toSEG.WNDin Section 2.3. - (b) Section 2.4 was added describing the unavoidable window retraction issue and explicitly describing the mitigation steps necessary.
- (c) In Section 3.2, the wording on how the Timestamps option negotiation is to be performed was updated with RFC2119 wording. Also, a number of paragraphs were added to clarify the expected behavior with a compliant implementation using
TSopt, because RFC 1323 left room for interpretation, e.g., potential late enablement ofTSopt🧚 - (d) The description of which
TSecrvalues can be used to update the measured RTT has been clarified. Specifically, with timestamps, the Karn algorithm [Karn87] is disabled. The Karn algorithm disables all RTT measurements during retransmission, since it is ambiguous whether the<ACK>is for the original segment or the retransmitted segment. With timestamps, that ambiguity is removed, since theTSecrin the<ACK>will contain theTSvalfrom whichever data segment made it to the destination 💜 - (e) RTTM update processing explicitly excludes segments not updating
SND.UNA. The original text could be interpreted to allow taking RTT samples when SACK acknowledges some new, non-continuous data 👀 - (f) In RFC 1323, Section 3.4, step (2) of the algorithm to control which timestamp is echoed was incorrect in two regards:
- (1) It failed to update
TS.Recentfor a retransmitted segment that resulted from a lost<ACK>. - (2) It failed if
SEG.LEN = 0.
- (1) It failed to update
In the new algorithm, the case of SEG.TSval >= TS.Recent is included for consistency with the PAWS test 🍓
- (g) It is now recommended that the Timestamps option is included in
<RST>segments if the incoming segment contained a Timestamps option. - (h)
<RST>segments are explicitly excluded from PAWS processing 🚫 - (i) Text was added to clarify the precedence between regular TCP [RFC0793] and this document's Timestamps option / PAWS processing. Discussion about combined acceptability checks is ongoing 🧐
- (j)
Snd.TSoffsetandSnd.TSclockvariables have been added.Snd.TSclockis the sum ofmy.TSclockandSnd.TSoffset. This lets the starting points for timestamp values be randomized on a per-connection basis. SettingSnd.TSoffsetto zero yields the same results as [RFC1323]. Text was added to guide implementers to the proper selection of these offsets, because entirely random offsets for each new connection will conflict with PAWS 🔥 - (k) Appendix A has been expanded with information about the TCP Urgent Pointer. An earlier revision contained text around the TCP MSS option, which was split off into [RFC6691] ✨
- (l) One correction was made to the Event Processing Summary in Appendix D. In SEND CALL/ESTABLISHED STATE,
RCV.WNDis used to fill in theSEG.WNDvalue, notSND.WND💅 - (m) Appendix G was added to exemplify how an RTO calculation might be updated to properly take the much higher RTT sampling frequency enabled by the Timestamps option into account 🌈
Editorial changes to the document, the kind that don't impact the implementation or function of the mechanisms described in it, include:
- (a) Removed much of the discussion in Section 1 to streamline the document. However, detailed examples and discussions in Sections 2, 3, and 5 are kept as guidelines for implementers 📚
- (b) Added short text that the use of WS increases the chances of sequence number wrap, so the PAWS mechanism is required in certain environments 🌙
- (c) Removed references to "new" options, since the options were introduced in [RFC1323] already. Changed the text in Section 1.3 to specifically address TS and WS options.
- (d) Section 1.4 was added for [RFC2119] wording. Normative text was updated with the appropriate phrases 🎀
- (e) Added < > brackets to mark specific types of segments, and replaced most occurrences of "packet" with "segment" where TCP segments are referred to 🦋
- (f) Updated the text in Section 3 to take into account what has been learned since [RFC1323] 🧠
- (g) Removed some unused references 🧹
- (h) Removed the list of changes between [RFC1323] and prior versions. These changes are mentioned in Appendix C of [RFC1323] 📜
- (i) Moved "Changes from RFC 1323" to the end of the appendices for easier lookup. The entries were also split into a technical and an editorial part, and sorted to roughly correspond with the sections in the text where they apply 🗂️
Authors' Addresses 💖
David Borman
Quantum Corporation
Mendota Heights, MN 55120
USA
EMail: david.borman@quantum.comBob Braden
University of Southern California
4676 Admiralty Way
Marina del Rey, CA 90292
USA
EMail: braden@isi.eduVan Jacobson
Google, Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043
USA
EMail: vanj@google.comRichard Scheffenegger (editor)
NetApp, Inc.
Am Euro Platz 2
Vienna, 1120
Austria
EMail: rs@netapp.com📦 where this page comes from
- ietf: rfc7323 2014-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms