RFC 3449
TCP Performance Implications of Network Path Asymmetry
TCP Performance Implications of Network Path Asymmetry 🦋
okay bestie, this is RFC 3449 🎀, a Best Current Practice (BCP 69) from the Network Working Group, dated December 2002. It was written by H. Balakrishnan (MIT LCS), V. N. Padmanabhan (Microsoft Research), G. Fairhurst, and M. Sooriyabandara (University of Aberdeen, U.K.). The title block doesn't list anything it obsoletes or updates. ✨
Status of this Memo 📜
This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Distribution of this memo is unlimited. 💅
Copyright Notice ©️
Copyright (C) The Internet Society (2002). All Rights Reserved. 🌸
Abstract 🔥
ok so this doc is about the TCP performance problems that show up when paths are asymmetric 🫶. These problems pop up in a bunch of access networks, including bandwidth-asymmetric networks and packet radio subnetworks, and they happen for different underlying reasons. but the end result on TCP performance is the same in both cases: things often get way slower because the ACK feedback going from the receiver back to the sender is imperfect and kind of unpredictable 😭.
the document goes through several mitigations for this, which have either been proposed or evaluated in the literature, or are already deployed in real networks. they use a mix of local link-layer techniques, subnetwork techniques, and end-to-end mechanisms. there are three big groups: (i) managing the channel used by the upstream bottleneck link that carries the ACKs, usually with header compression or by sending fewer TCP ACKs, (ii) handling that reduced ACK frequency so the TCP sender still gets its acknowledgment-triggered self-clocking 🧚, and (iii) scheduling the data and ACK packets in the reverse direction so two-way traffic performs better 🍓. each technique gets described along with its known issues and recommendations for use, and there's a summary of the recommendations at the end of the document. ✨
Table of Contents 🗂️
1. Conventions used in this Document ...............................3
2. Motivation ....................................................4
2.1 Asymmetry due to Differences in Transmit
and Receive Capacity .........................................4
2.2 Asymmetry due to Shared Media in the Reverse Direction .......5
2.3 The General Problem ..........................................5
3. How does Asymmetry Degrade TCP Performance? .....................5
3.1 Asymmetric Capacity ..........................................5
3.2 MAC Protocol Interactions ....................................7
3.3 Bidirectional Traffic ........................................8
3.4 Loss in Asymmetric Network Paths ............................10
4. Improving TCP Performance using Host Mitigations ...............10
4.1 Modified Delayed ACKs .......................................11
4.2 Use of Large MSS ............................................12
4.3 ACK Congestion Control ......................................13
4.4 Window Prediction Mechanism .................................14
4.5 Acknowledgement based on Cwnd Estimation. ...................14
4.6 TCP Sender Pacing ...........................................14
4.7 TCP Byte Counting ...........................................15
4.8 Backpressure ................................................16
5. Improving TCP performance using Transparent Modifications ......17
5.1 TYPE 0: Header Compression ..................................18
5.1.1 TCP Header Compression ..................................18
5.1.2 Alternate Robust Header Compression Algorithms ..........19
5.2 TYPE 1: Reverse Link Bandwidth Management ...................19
5.2.1 ACK Filtering ...........................................20
5.2.2 ACK Decimation ..........................................21
5.3 TYPE 2: Handling Infrequent ACKs ............................22
5.3.1 ACK Reconstruction ......................................23
5.3.2 ACK Compaction and Companding ...........................25
5.3.3 Mitigating TCP packet bursts generated by
Infrequent ACKs .........................................26
5.4 TYPE 3: Upstream Link Scheduling ............................27
5.4.1 Per-Flow queuing at the Upstream Bottleneck Link ........27
5.4.2 ACKs-first Scheduling ...................................28
6. Security Considerations ........................................29
7. Summary ........................................................30
8. Acknowledgments ................................................32
9. References .....................................................32
10. IANA Considerations ...........................................37
Appendix: Examples of Subnetworks Exhibiting Network Path
Asymmetry ...............................................38
Authors' Addresses ................................................40
Full Copyright Statement ..........................................411. Conventions used in this Document 📚
FORWARD DIRECTION: the dominant direction of data transfer over an asymmetric network path. it's the direction with the better characteristics in terms of capacity, latency, error rate, etc. data transfer in the forward direction is called "forward transfer" 💖. packets traveling forward follow the forward path through the IP network.
REVERSE DIRECTION: the direction in which the acknowledgments of a forward TCP transfer flow. data transfer could also happen this way (and it's termed "reverse transfer"), but it's usually way less voluminous than the forward direction 🙈. the reverse direction typically has worse characteristics than the forward direction. packets traveling in reverse follow the reverse path through the IP network.
UPSTREAM LINK: the specific bottleneck link that normally has much less capability than the corresponding downstream link. congestion isn't confined to this link alone, and it may also happen at any point along the forward and reverse directions (e.g., because of sharing with other traffic flows) 🚧.
DOWNSTREAM LINK: a link on the forward path, corresponding to the upstream link. 🍑
ACK: a cumulative TCP acknowledgment [RFC791]. in this document, the term refers to a TCP segment that carries a cumulative acknowledgement (ACK), but no data 🫧.
DELAYED ACK FACTOR, d: the number of TCP data segments acknowledged by a TCP ACK. the minimum value of d is 1, since at most one ACK should be sent for each data packet [RFC1122, RFC2581] 💅.
STRETCH ACK: stretch ACKs are acknowledgements that cover more than 2 segments of previously unacknowledged data (d>2) [RFC2581]. stretch ACKs can occur by design (although this is not standard), due to implementation bugs [All97b, RFC2525], or due to ACK loss [RFC2760] 😬.
NORMALIZED BANDWIDTH RATIO, k: the ratio of the raw bandwidth (capacity) of the forward direction to the return direction, divided by the ratio of the packet sizes used in the two directions [LMS97] 🧮.
SOFTSTATE: per-flow state established in a network device that is used by the protocol [Cla88]. the state expires after a period of time (so it isn't required to be explicitly deleted when a session expires), and it's continuously refreshed while a flow continues. that means lost state can be rebuilt without needing to exchange additional control messages 🔄✨.
2. Motivation 🌈
ok so asymmetric characteristics show up in several network technologies, including cable data networks (like DOCSIS cable TV networks [DS00, DS01]), direct broadcast satellite (e.g., an IP service using Digital Video Broadcast, DVB, [EN97] with an interactive return channel), Very Small Aperture satellite Terminals (VSAT), Asymmetric Digital Subscriber Line (ADSL) [ITU02, ANS01], and several packet radio networks 📡. these networks are increasingly being deployed as high-speed Internet access networks, so it's highly desirable to get good TCP performance on them. the asymmetry of the network paths often makes that really challenging 😤. examples of networks that exhibit asymmetry are in the Appendix 🎀.
asymmetry can show up as a difference in transmit and receive capacity, an imbalance in the packet loss rate, or differences between the transmit and receive paths [RFC3077]. for example, when capacity is asymmetric, so that there's reduced capacity on the reverse path used by TCP ACKs, slow or infrequent ACK feedback degrades TCP performance in the forward direction 🐢. similarly, asymmetry in the underlying Medium Access Control (MAC) and Physical (PHY) protocols could make it expensive to transmit TCP ACKs (out of proportion to their size), even when capacity is symmetric 💀.
2.1. Asymmetry due to Differences in Transmit and Receive Capacity 🚀
network paths may be asymmetric because the upstream and downstream links operate at different rates and/or are implemented using different technologies 🧃.
the asymmetry in capacity may get a lot bigger when best effort IP flows carrying TCP ACKs share the available upstream capacity with other traffic flows, e.g., telephony, especially flows that have reserved upstream capacity 📞. this includes service guarantees at the IP layer (e.g., the Guaranteed Service [RFC2212]) or at the subnet layer (e.g., support of Voice over IP [ITU01] using the Unsolicited Grant service in DOCSIS [DS01], or CBR virtual connections in ATM over ADSL [ITU02, ANS01]) 🔐.
when multiple upstream links exist, the asymmetry may be reduced by dividing upstream traffic between a number of available upstream links 🛣️✨.
2.2. Asymmetry due to Shared Media in the Reverse Direction 🎙️
in networks using centralized multiple access control, asymmetry may be a fundamental consequence of the hub-and-spokes architecture of the network (i.e., a single base node communicating with multiple downstream nodes) 🕸️. the central node often incurs less transmission overhead and doesn't incur latency in scheduling its own downstream transmissions. in contrast, upstream transmission is subject to additional overhead and latency (e.g., due to guard times between transmission bursts, and contention intervals). this can produce significant network path asymmetry 😩.
upstream capacity may be limited even further by the requirement that each node must first request per-packet bandwidth using a contention MAC protocol (e.g., the DOCSIS 1.0 MAC restricts each node to sending at most a single packet in each upstream time-division interval [DS00]). a satellite network employing dynamic Bandwidth on Demand (BoD) also consumes MAC resources for each packet sent (e.g., [EN00]). in these schemes, the available uplink capacity is a function of the MAC algorithm 🧠. the MAC and PHY schemes also add overhead per upstream transmission, and that overhead could be so significant that sending short packets (including TCP ACKs) becomes as costly as sending MTU-sized data packets 🫠.
2.3 The General Problem ✨
okay so even though capacity-dependent and MAC-dependent asymmetries are totally different tech 💅, both kinds of network path end up hurting TCP performance for the same core reason: ACK feedback is imperfect and kinda unpredictable 😭. This document digs into that problem in detail and walks through several techniques that may reduce or eliminate the constraints 🦋.
3. How does Asymmetry Degrade TCP Performance? 🎀
this section spills the tea on what network path asymmetry actually does to TCP performance 🫶. if you want more details and experimental results, go check out [BPK99, Bal98, Pad98, FSS01, Sam99] 📚.
3.1 Asymmetric Capacity
bestie, the problems that mess with one-way transfers when the forward and return paths have super different capacities depend on what the upstream link is like 👑. Two situations pop up for one-way traffic on these paths: when the upstream bottleneck link has enough queuing to stop packet (ACK) losses, and when it has a tiny buffer. we'll take them one at a time 💖.
If the upstream bottleneck has deep queues, so that it doesn't drop ACKs in the reverse direction, then performance is a strong function of the normalized bandwidth ratio, k. For example, with a 10 Mbps downstream link and a 50 Kbps upstream link, the raw capacity ratio is 200. With 1000-byte data packets and 40-byte ACKs, the ratio of the packet sizes is 25. So k is 200/25 = 8. That means if the receiver acknowledges more often than one ACK every 8 (k) data packets, the upstream link gets saturated before the downstream one, which limits forward throughput 🔥. Note that, the achieved TCP throughput is set by the minimum of the receiver advertised window or TCP congestion window, cwnd [RFC2581].
If ACKs aren't dropped (at the upstream bottleneck link) and k > 1 or k > 0.5 when delayed ACKs are used [RFC1122], TCP ACK-clocking breaks down 💀. Picture two data packets the sender fires off in quick succession. On the way to the receiver, they get spaced out based on the smallest bottleneck link in the forward direction. ACK clocking's whole vibe is that the ACKs sent back for those packets carry that spacing all the way to the sender, so it can send new data packets with the same spacing [Jac88]. With delayed ACKs, the spacing that gets reflected is between the data packets that actually trigger ACKs 🧚. But the limited upstream capacity and the queuing at the upstream bottleneck router mess with the inter-ACK spacing on the reverse path, and so with what the sender sees 😤. When ACKs arrive at the upstream bottleneck faster than the link can support, they get queued up behind one another. Coming out the other side, their spacing is dilated compared to the original, and it's a function of the upstream bottleneck capacity. So the TCP sender clocks out new data packets at a slower rate than if the ACKs hadn't been queued. The connection no longer depends on the downstream bottleneck alone; it's throttled by the rate of arriving ACKs 🐢. As a side effect, the sender's rate of cwnd growth also slows down.
A second side effect shows up when the upstream bottleneck on the reverse path is saturated. The saturated link causes persistent queuing of packets, which makes the path Round Trip Time (RTT) [RFC2998] keep climbing for every end host using that bottleneck 📈. That can mess with the protocol control loops, and it may also trigger false timeouts (where the sending host underestimates the path RTT) 🙈.
A different situation happens when the upstream bottleneck has a relatively small amount of buffer space for ACKs. As the transmission window grows, that queue fills up and ACKs get dropped. If the receiver acknowledged every packet, only one of every k ACKs would make it to the sender, and the remaining (k-1) would be dropped due to buffer overflow at the upstream link buffer (k is the normalized bandwidth ratio, same as before) 😩. In this case, the reverse bottleneck link capacity and slow ACK arrival rate aren't directly responsible for any degraded performance. Still, the infrequency of ACKs leads to three reasons for degraded performance:
- The sender transmits data in large bursts of packets, limited only by the available cwnd. If the sender receives only one ACK in k, it transmits data in bursts of k (or more) packets, because each ACK shifts the sliding window by at least k (acknowledged) data packets (TCP data segments). This increases the likelihood of data packet loss along the forward path, especially when k is large, because routers don't handle large bursts of packets well 💥.
- Current TCP sender implementations increase their cwnd by counting the number of ACKs they receive, not by how much data is actually acknowledged by each ACK. The latter approach, also known as byte counting (section 4.7), is a standard implementation option for cwnd increase during the congestion avoidance period [RFC2581]. So fewer ACKs imply a slower rate of growth of the cwnd, which degrades performance over long-delay connections 🐌.
- The sender TCP's Fast Retransmission and Fast Recovery algorithms [RFC2581] are less effective when ACKs are lost. The sender may possibly not receive the threshold number of duplicate ACKs even if the receiver transmits more than the DupACK threshold (> 3 DupACKs) [RFC2581]. Furthermore, the sender may possibly not receive enough duplicate ACKs to adequately inflate its cwnd during Fast Recovery 🥲.
3.2 MAC Protocol Interactions
The interaction of TCP with MAC protocols may degrade end-to-end performance. Variable round-trip delays and ACK queuing are the main symptoms of this problem 🌈.
One example is the impact on terrestrial wireless networks [Bal98]. A high per-packet overhead may arise from the need for communicating link nodes to first synchronise (e.g., via a Ready To Send / Clear to Send (RTS/CTS) protocol) before communication, along with the significant turn-around time for the wireless channel 📡. This overhead is variable, since the RTS/CTS exchange may need to back off exponentially when the remote node is busy (e.g., engaged in a conversation with a different node). This leads to large and variable communication latencies in packet-radio networks.
An asymmetric workload (more downstream than upstream traffic) may cause ACKs to be queued in some wireless nodes (especially in the end host modems), which makes the variable latency worse 😵. Queuing may also occur in other shared media, e.g., cable modem uplinks and BoD access systems often employed on shared satellite channels 🛰️.
Variable latency and ACK queuing reduce the smoothness of the TCP data flow 🌊. In particular, ACK traffic can interfere with the flow of data packets, increasing the traffic load of the system 🚦.
TCP measures the path RTT, and from this calculates a smoothed RTT estimate (srtt) and a linear deviation, rttvar. These are used to estimate a path retransmission timeout (RTO) [RFC2988], set to srtt + 4*rttvar. For most wired TCP connections, the srtt remains constant or has a low linear deviation. The RTO therefore tracks the path RTT, and the TCP sender will respond promptly when multiple losses occur in a window ⏱️. In contrast, some wireless networks exhibit a high variability in RTT, causing the RTO to significantly increase (e.g., on the order of 10 seconds) 😱. Paths traversing multiple wireless hops are especially vulnerable to this effect, because this increases the probability that the intermediate nodes may already be engaged in conversation with other nodes. The overhead in most MAC schemes is a function of both the number and size of packets. However, the MAC contention problem is a significant function of the number of packets (e.g., ACKs) transmitted rather than their size. In other words, there is a significant cost to transmitting a packet regardless of packet size 💅.
Experiments conducted on the Ricochet packet radio network in 1996 and 1997 demonstrated the impact of radio turnarounds and the corresponding increased RTT variability, which resulted in degraded TCP performance 📉. It was not uncommon for TCP connections to experience timeouts of 9 - 12 seconds, with the result that many connections were idle for a significant fraction of their lifetime (e.g., sometimes 35% of the total transfer time) 😴. This leads to under-utilization of the available capacity. These effects may also occur in other wireless subnetworks 🫠.
3.3 Bidirectional Traffic 💌
okay so bidirectional traffic is when TCP transfers are happening in both directions at once over an asymmetric path 🦋 like when you send an e-mail in the reverse direction while a web page loads in the forward direction. iykyk. to keep things simple we only look at one TCP connection in each direction 💅 but ngl, in lots of real cases several connections have to share the available capacity, so the congestion gets even worse 😭
bidirectional traffic makes the effects from section 3.1 hit way harder, because part of the upstream link bandwidth gets eaten by the reverse transfer 💀 so the bandwidth asymmetry basically feels bigger than it actually is 🙈 and then there's more drama. the data packets of the reverse transfer and the ACKs of the forward transfer start interacting in messy ways 🫠
picture this 🎬 the reverse TCP connection has already saturated the bottleneck upstream link with data packets when the forward TCP connection starts. there's then a high probability that many of the new forward connection's ACKs run into a full upstream link buffer and get dropped 😩 and even after those rough first moments, the ACKs of the forward connection can still get queued behind big data packets from the reverse connection 🍓 those large packets can have really long transmission times (it takes about 280 ms to transmit a 1 Kbyte data packet over a 28.8 kbps line, can you believe 😭). so the forward transfer stalls for long periods of time ⏳ the only moment it gets to make rapid progress and build up its cwnd is when the reverse connection loses packets (due to a buffer overflow at an intermediate router) and slows down 🏃♀️💨
when ACKs get queued behind other traffic for appreciable periods of time, the burst nature of TCP traffic and self-synchronizing effects can cause something called ACK Compression [ZSC91] 😱 and it reduces the throughput of TCP. it happens when a series of ACKs in one direction gets queued behind a burst of other packets (e.g., data packets traveling in the same direction) and gets squished together in time. that creates an intense burst of data packets in the other direction, in response to the burst of compressed ACKs arriving at the server 🌪️ this phenomenon has been investigated in detail for bidirectional traffic, and recent analytical work [LMS97] predicted that ACK Compression may also result from bidirectional transmission with asymmetry. it was even observed in practical asymmetric satellite subnetworks [FSS01] 🛰️ and in the case of extreme asymmetry (k>>1), the inter-ACK spacing can increase due to queuing (section 3.1), resulting in ACK dilation 🫨
the TL;DR 🫶 sharing the upstream bottleneck link with multiple flows (e.g., IP flows to the same end host, or flows to a number of end hosts sharing a common upstream link) increases the level of ACK Congestion. and bidirectional traffic makes the bandwidth asymmetry problem worse, because the big data packets of a reverse direction connection clash with the ACKs of a forward direction connection. not a vibe 💔
3.4 Loss in Asymmetric Network Paths 🕳️
loss can happen in either the forward or the reverse direction 🙃 for data transfer in the forward direction, that means losing data packets or losing ACK packets respectively. losing ACKs is way less of a big deal than losing data packets, because it generally just results in stretch ACKs [CR98, FSS01] 🌸
now, with long delay paths there's another complication 😵💫 a slow upstream link [RFC3150] can cause trouble when the end host uses TCP large windows [RFC1323] to maximize throughput in the forward direction. if data packets get lost on the forward path (due to congestion, or link loss, which is common for some wireless links), you get a big pile of back-to-back duplicate ACKs (or TCP SACK packets [RFC2018]) for each correctly received data packet that follows the loss 📚 the TCP sender uses Fast Retransmission and Recovery [RFC2581] to recover from the loss, and even if that works, the ACK for the retransmitted data segment can be seriously delayed by other duplicate ACKs still sitting in the upstream link buffer 🥲 this can end up causing a timeout [RFC2988] and a premature end to TCP Slow Start [RFC2581], which means poor forward path throughput. the mitigations for this are covered in Section 5.3 🧚
4. Improving TCP Performance using Host Mitigations 🛠️
okay, two key issues need fixing to improve TCP performance over asymmetric networks 🎯 the first is managing the capacity of the upstream bottleneck link, which is used by ACKs and possibly other traffic too. a bunch of techniques work by reducing the number of ACKs flowing in the reverse direction. but that has the side effect of potentially destroying the nice self-clocking property of the TCP sender, where sending new data packets is triggered by incoming ACKs 😤 so the second issue is avoiding any adverse impact of infrequent ACKs 💅
each of these issues can be handled by local link-layer solutions and/or by end-to-end techniques. this section talks about end-to-end modifications. some techniques need TCP receiver changes (sections 4.1, 4.4, 4.5), some need TCP sender changes (sections 4.6, 4.7), and a pair needs changes to both the TCP sender and receiver (sections 4.2, 4.3). one technique needs a sender modification at the receiving host (section 4.8) 🤯 the techniques can be used independently, however some sets of techniques are complementary, e.g., pacing (section 4.6) and byte counting (section 4.7), which have been bundled into a single TCP Sender Adaptation scheme [BPK99] 🧩
it's normally envisaged that these changes happen in the end hosts using the asymmetric path. however, they could, and have, been used in a middle-box or Protocol Enhancing Proxy (PEP) [RFC3135] employing split TCP 🪞 this document does not discuss the issues concerning PEPs. section 4 describes several techniques which do not require end-to-end changes ✨
4.1 Modified Delayed ACKs 🕰️
there are two standard methods TCP receivers can use to generate acknowledgments 🌈 the method outlined in [RFC793] generates an ACK for each incoming data segment (i.e., d=1). then [RFC1122] says hosts SHOULD use "delayed acknowledgments". with this algorithm, an ACK is generated for at least every second full-sized segment (d=2), or if a second full-sized segment does not arrive within a given timeout (which MUST NOT exceed 500 ms [RFC1122], and is typically less than 200 ms). relaxing the latter constraint (i.e., allowing d>2) may generate Stretch ACKs [RFC2760]. this is a possible mitigation, because it reduces the rate at which ACKs get returned by the receiver 🐢 an implementer SHOULD only deviate from this requirement after careful consideration of the implications [RFC2581] 🧐
reducing the number of ACKs per received data segment has a few undesirable effects, including:
- (i) Increased path RTT 😬
- (ii) Increased time for TCP to open the cwnd 🥱
- (iii) Increased TCP sender burst size, since cwnd opens in larger steps 💥
on top of that, a TCP receiver is often unable to figure out an optimum setting for a large d, since it normally won't know the details of the properties of the links that form the path in the reverse direction 🙈
RECOMMENDATION 📣 a TCP receiver MUST use the standard TCP algorithm for sending ACKs as specified in [RFC2581]. that means it may delay sending an ACK after it receives a data segment [RFC1122]. when ACKs are delayed, the receiver MUST generate an ACK within 500 ms, and the ACK SHOULD be generated for at least every second full sized segment (MSS) of received data [RFC2581]. this results in an ACK delay factor (d) that does not exceed a value of 2 ✅ changing the algorithm would require a host modification to the TCP receiver and awareness by the receiving host that it is using a connection with an asymmetric path. such a change has many drawbacks in the general case, so it is NOT RECOMMENDED for use within the Internet 🚫
4.2 Use of Large MSS 📏
a TCP sender that uses a large Maximum Segment Size (MSS) reduces the number of ACKs generated per transmitted byte of data 🫧
although individual subnetworks may support a large MTU, the majority of current Internet links employ an MTU of approx 1500 bytes (that of Ethernet) 🔌 by setting the Don't Fragment (DF) bit in the IP header, Path MTU (PMTU) discovery [RFC1191] may be used to determine the maximum packet size (and hence MSS) a sender can use on a given network path without being subjected to IP fragmentation. it also gives a way to automatically select a suitable MSS for a specific path 🗺️ this guarantees that routers will not perform IP fragmentation of normal data packets 🙌
by electing not to use PMTU Discovery, an end host may choose to use IP fragmentation by routers along the path in the forward direction [RFC793]. this allows an MSS larger than the smallest MTU along the path. however, this increases the unit of error recovery (TCP segment) above the unit of transmission (IP packet) 😵. this is NOT RECOMMENDED, since it can increase the number of retransmitted packets following the loss of a single IP packet, leading to reduced efficiency, and potentially aggravating network congestion [Ken87] 🚨 choosing an MSS larger than the forward path minimum MTU also lets the sender transmit more initial packets (a burst of IP fragments for each TCP segment) when a session starts or following RTO expiry. that increases the aggressiveness of the sender compared to standard TCP [RFC2581], and it can adversely impact other standard TCP sessions that share a network path 🥀
RECOMMENDATION:
a larger forward path MTU is desirable for paths with bandwidth asymmetry 💖 network providers MAY use a large MTU on links in the forward direction. TCP end hosts using Path MTU discovery may be able to take advantage of a large MTU by automatically selecting an appropriate larger MSS, without requiring modification. the use of Path MTU discovery [RFC1191] is therefore RECOMMENDED 🎀
increasing the unit of error recovery and congestion control (MSS) above the unit of transmission and congestion loss (the IP packet) by using a larger end host MSS and IP fragmentation in routers is NOT RECOMMENDED. ⚠️ bestie, please don't do this one 😭
4.3 ACK Congestion Control ✨
okay so ACK Congestion Control (ACC) is an experimental technique that works end to end 🦋 The whole point is that it extends congestion control to ACKs, because ACKs can actually eat up resources (think packet buffers and MAC transmission overhead) at an upstream bottleneck link. It has two parts: (a) a network mechanism that tells the receiver the ACK path is congested, and (b) what the receiver does about it 💅
so a router feeding an upstream bottleneck link can notice incipient congestion, for example with an algorithm based on RED (Random Early Detection) [FJ93]. It keeps track of the average queue size over a recent time window. If that average goes over a threshold, the router may pick a packet at random. If the packet's IP header has the Explicit Congestion Notification Capable Transport (ECT) bit set, the router may mark it, which means setting an Explicit Congestion Notification (ECN) [RFC3168] bit (or bits) in the IP header. If the ECT bit isn't set, the packet just gets dropped as usual 🙈 The ECN notification the end host gets is reflected back to the sending TCP end host, which triggers congestion avoidance [RFC3168] 🌸 ngl this part is so clever. Also, routers doing RED with ECN do not get rid of packet loss, and they may still drop a packet even when the ECT bit is set 💀 It's also possible to use some algorithm other than RED to decide when to set the ECN bit ✨
so ACC extends ECN so that both TCP data packets and ACKs set the ECT bit, which makes them candidates for an ECN mark too 🎀 That means, when a TCP receiver gets an ACK with the ECN bit set [RFC3168], it reduces how fast it sends ACKs. It keeps a dynamically varying delayed-ACK factor, d, and sends one ACK for every d data packets it receives. When it gets a packet with the ECN bit set, it multiplies d up, which multiplicatively decreases how often ACKs go out 😭 Then, for each later RTT (for example, one measured with the TCP RTTM option [RFC1323]) where no ECN shows up, it linearly lowers the factor d, so ACKs get sent more often again. The receiver basically copies the standard congestion control behavior of TCP senders in the way it sends ACKs. Main character energy, honestly 👑
The maximum value of d is set by the TCP sender window size, which could be passed to the receiver in a new (experimental) TCP option 🧚 The receiver should send at least one ACK (preferably more) for each window of data from the sender, i.e., d < (cwnd/mss), so the sender doesn't stall waiting for the receiver's delayed ACK timer to fire an ACK 🍓
RECOMMENDATION: ACK Congestion Control (ACC) is an experimental technique that requires TCP sender and receiver modifications. There is currently little experience of using such techniques in the Internet. Future versions of TCP may evolve to include this or similar techniques. These are the subject of ongoing research. ACC is not recommended for use within the Internet in its current form. 🫶
4.4 Window Prediction Mechanism 🌈
the Window Prediction Mechanism (WPM) is a TCP receiver side mechanism [CLP98] that uses a dynamic ACK delay factor (varying d), kind of like the ACC scheme in section 4.3 🦋 The TCP receiver rebuilds the congestion control behavior of the TCP sender by predicting a cwnd value. That value is used together with the allowed window to adjust the receiver's d. WPM also makes allowance for unnecessary retransmissions caused by losses due to link errors ✨
RECOMMENDATION: Window Prediction Mechanism (WPM) is an experimental TCP receiver side modification. There is currently little experience of using such techniques in the Internet. Future versions of TCP may evolve to include this or similar techniques. These are the subjects of ongoing research. WPM is not recommended for use within the Internet in its current form. 💅
4.5 Acknowledgement based on Cwnd Estimation. 🌸
Acknowledgement based on Cwnd Estimation (ACE) [MJW00] tries to measure the cwnd at the TCP receiver and keep a varying ACK delay factor (d) 🧚 The cwnd gets estimated by counting how many packets arrive during one path RTT. The technique may improve how accurately a suitable cwnd gets predicted 💖
RECOMMENDATION: Acknowledgement based on Cwnd Estimation (ACE) is an experimental TCP receiver side modification. There is currently little experience of using such techniques in the Internet. Future versions of TCP may evolve to include this or similar techniques. These are the subject of ongoing research. ACE is not recommended for use within the Internet in its current form. 🎀
4.6 TCP Sender Pacing 🍓
okay so reducing how often ACKs go out might ease congestion at the upstream bottleneck link, but it can make TCP sender bursts bigger (see section 4.1) 😬 That can slow the growth of cwnd, and it's not great on shared network paths, since it may really bump up the maximum number of packets sitting in the bottleneck link buffer, which could mean more network congestion. It may also lead to ACK Compression [ZSC91] 💀
TCP Pacing [AST00], usually called TCP Sender pacing, uses an adapted TCP sender to tame transmission burstiness 🌊 There's a limit on how many packets the TCP sender can send back-to-back (at local line rate), even when the window(s) would allow more data. If needed, more bursts of data packets get scheduled for later points in time, worked out from the transmission rate of the TCP connection. That rate may be estimated from the ratio cwnd/srtt. So big bursts of data packets get broken into smaller ones spread out over time ✨
A subnetwork may also provide pacing (for example, Generic Traffic Shaping (GTS)), but that means a big jump in per-packet processing overhead and buffer requirements at the router where the shaping happens (see section 5.3.3) 🙈
RECOMMENDATIONS: TCP Sender Pacing requires a change to the implementation of the TCP sender. It may be beneficial in the Internet and will significantly reduce the burst size of packets transmitted by a host. This successfully mitigates the impact of receiving Stretch ACKs. TCP Sender Pacing implies increased processing cost per packet, and requires a prediction algorithm to suggest a suitable transmission rate. There are hence performance trade-offs between end host cost and network performance. Specification of efficient algorithms remains an area of ongoing research. Use of TCP Sender Pacing is not expected to introduce new problems. It is an experimental mitigation for TCP hosts that may control the burstiness of transmission (for example, resulting from Type 1 techniques, see section 5.1.2), however it is not currently widely deployed. It is not recommended for use within the Internet in its current form. 🫶
4.7 TCP Byte Counting 🔥
the TCP sender can stop slowing the growth of cwnd by looking at how much data each ACK acknowledges, instead of opening the cwnd based on how many ACKs arrived 👑 So if an ACK acknowledges d data packets (or TCP data segments), cwnd grows as if d separate ACKs had come in. This is called TCP Byte Counting [RFC2581, RFC2760]. (You could also treat that single ACK as being worth d/2 instead of d ACKs, to copy the effect of the TCP delayed ACK algorithm 🙈) This works because cwnd growth is only tied to the available capacity in the forward direction, so the number of ACKs doesn't matter 💖
This may ease the impact of asymmetry when it's combined with other techniques (for example, TCP Pacing (section4.6) together with ACC (section 4.3), plus a duplicate ACK threshold at the receiver) ✨ The main issue is that TCP byte counting may generate undesirable long bursts of TCP packets at the sender host line rate 😭 An implementation also has to consider that data packets in the forward direction and ACKs in the reverse direction may both travel over network paths that do some amount of packet reordering. Reordering of IP packets is currently common, and it can come from various causes [BPS00] 🌀
RECOMMENDATION: TCP Byte Counting requires a small TCP sender modification. In its simplest form, it can generate large bursts of TCP data packets, particularly when Stretch ACKs are received. Unlimited byte counting is therefore not allowed [RFC2581] for use within the Internet.
It is therefore strongly recommended [RFC2581, RFC2760] that any byte counting scheme includes a method to mitigate the potentially large bursts of TCP data packets the algorithm can cause (for example, TCP Sender Pacing (section 4.6), ABC [abc-ID]). If the burst size or sending rate of the TCP sender can be controlled, then the scheme may be beneficial when Stretch ACKs are received. Working out safe algorithms is still an area of ongoing research. Further experimentation will then be required to assess how well these safeguards work, before they can be recommended for use in the Internet 🧐
4.8 Backpressure 🧚
Backpressure is a technique for boosting the performance of bidirectional traffic for end hosts directly connected to the upstream bottleneck link [KVR98] 🌈 A limit is set on how many data packets of upstream transfers can be queued at the upstream bottleneck link. In other words, the bottleneck link queue puts 'backpressure' on the TCP (sender) layer. This needs a modified implementation, compared to what's currently deployed in many TCP stacks. Backpressure makes sure that ACKs of downstream connections don't get starved at the upstream bottleneck, which improves the performance of those downstream connections 🦋 Similar generic schemes that may be implemented in hosts or routers are discussed in section 5.4 ✨
Backpressure can be unfair to a reverse direction connection, and it can make that connection's throughput highly sensitive to the dynamics of the forward connection(s) 😭
RECOMMENDATION: Backpressure requires an experimental modification to the sender protocol stack of a host directly connected to an upstream bottleneck link. Use of backpressure is an implementation issue, rather than a network protocol issue. Where backpressure is implemented, the optimizations described in this section could be desirable and can benefit bidirectional traffic for hosts. Specification of safe algorithms for providing backpressure is still a subject of ongoing research. The technique is not recommended for use within the Internet in its current form. 💅
5. Improving TCP performance using Transparent Modifications ✨
okay so this section is all about the sneaky fixes 🫣 Lots of link and network layer techniques have been suggested to soften the blow of an upstream bottleneck link. Some can work without touching the TCP sender or receiver at all, and some can team up with the schemes from section 4 🤝 They're called "transparent" [RFC3135] because at the transport layer, the TCP sender and receiver might not even know they exist 👀 That doesn't mean they're not messing with the pattern and timing of packets at the network layer, though. The techniques are sorted into three types based on where they show up in the path 🗂️
Most of these techniques need each individual TCP connection going over the bottleneck link to be identified separately, and they usually keep some per-flow state for active TCP connections 🧠 A link scheduler can also be brought in (section 5.4). The techniques (with one exception, ACK Decimation (section 5.2.2)) require:
- (i) Visibility of an unencrypted IP and TCP packet header (e.g., no use of IPSec with payload encryption [RFC2406]).
- (ii) Knowledge of IP/TCP options and ability to inspect packets with tunnel encapsulations (e.g., [RFC2784]) or to suspend processing of packets with unknown formats.
- (iii) Ability to demultiplex flows (by using address/protocol/port number, or an explicit flow-id).
[RFC3135] describes a class of network device that does way more than just forwarding packets 💅 It's called a Protocol Enhancing Proxy (PEP) and there's a huge range of them 🌈 On the simple end you've got things like ACK filtering, and on the fancy end you've got stateful devices that split one TCP connection into two separate parts 🤯 The techniques in section 5 of this document are the simpler kind 🎀 They don't inspect or modify any TCP or UDP payload data, and they don't modify port numbers or link addresses either. Lots of the scary risks that come with the complex PEPs just aren't there for these schemes 🙌 If you want the full tea on how PEPs operate and what risks come with them, check out [RFC3135] 📚
5.1 TYPE 0: Header Compression
A client may reduce the volume of bits used to send a single ACK by using compression [RFC3150, RFC3135]. Most modern dial-up modems support ITU-T V.42 bulk compression 📞 But header compression is a whole different vibe. It's known to be super effective at cutting the number of bits sent on the upstream link [RFC1144] 🔥 It works because most TCP packet headers only change in a few bit positions between packets in the same flow, and those changes can often be predicted 🔮
5.1.1 TCP Header Compression
TCP header compression [RFC1144] (sometimes known as V-J compression) is a Proposed Standard for use over low capacity links running SLIP or PPP [RFC3150] 🐢 It seriously shrinks ACKs on the reverse link when losses are rare (a situation that keeps the compressor and decompressor in sync) 🤝 Still, this alone doesn't fix all the asymmetry issues:
- (i) In some (e.g., wireless) subnetworks there is a significant per-packet MAC overhead that is independent of packet size (section 3.2). 📶
- (ii) A reduction in the size of ACKs does not prevent adverse interaction with large upstream data packets in the presence of bidirectional traffic (section 3.3). 😬
- (iii) TCP header compression cannot be used with packets that have IP or TCP options (including IPSec [RFC2402, RFC2406], TCP RTTM [RFC1323], TCP SACK [RFC2018], etc.). 🚫
- (iv) The performance of header compression described by RFC1144 is significantly degraded when compressed packets are lost. An improvement, which can still incur significant penalty on long network paths is described in [RFC2507]. This suggests it should only be used on links (or paths) that experience a low level of packet loss [RFC3150]. 💀
- (v) The normal implementation of Header Compression inhibits compression when IP is used to support tunneling (e.g., L2TP, GRE [RFC2794], IP-in-IP). The tunnel encapsulation complicates locating the appropriate packet headers. Although GRE allows Header Compression on the inner (tunneled) IP header [RFC2784], this is not recommended, since loss of a packet (e.g., due to router congestion along the tunnel path) will result in discard of all packets for one RTT [RFC1144]. 🙈
RECOMMENDATION: TCP Header Compression is a transparent modification performed at both ends of the upstream bottleneck link. It offers no benefit for flows employing IPSec [RFC2402, RFC2406], or when additional protocol headers are present (e.g., IP or TCP options, and/or tunnel encapsulation headers). The scheme is widely implemented and deployed and used over Internet links. It is recommended to improve TCP performance for paths that have a low-to-medium bandwidth asymmetry (e.g., k<10). 💖
In the form described in [RFC1144], TCP performance gets degraded over links (or paths) that may see noticeable rates of packet loss [RFC3150] 😩 It also may not give much improvement for upstream links with bidirectional traffic. So it's not the move for paths with a high bandwidth asymmetry (e.g., k>10) 🙅♀️
5.1.2 Alternate Robust Header Compression Algorithms
TCP header compression [RFC1144] and IP header compression [RFC2507] don't perform well when packet loss shows up 💔 They also don't compress packets with TCP option fields (e.g., SACK [RFC2018] and Timestamp (RTTM) [RFC1323]). BUT recent work on more robust schemes suggests a new generation of compression algorithms may be developed that are way more robust ✨ The IETF ROHC working group has specified compression techniques for UDP-based traffic [RFC3095] and is looking at a bunch of schemes that may provide improved TCP header compression 👀 These could be seriously helpful for asymmetric network paths 🫶
RECOMMENDATION: Robust header compression is a transparent modification that may be performed at both ends of an upstream bottleneck link. This class of techniques may also be suited to Internet paths that suffer low levels of re-ordering. The techniques benefit paths with a low-to-medium bandwidth asymmetry (e.g., k>10) and may be robust to packet loss. 🧚
Picking the right compression algorithms is still an area of ongoing research 🧪 It's possible that schemes could be derived that support IPSec authentication but not IPSec payload encryption. Those schemes alone don't provide significant improvement in asymmetric networks with a high asymmetry and/or bidirectional traffic 🙃
5.2 TYPE 1: Reverse Link Bandwidth Management
Once you get past Type 0 header compression, you need more techniques to deal with the performance problems caused by appreciable asymmetry (k>>1) 💅 One set of techniques runs at just one point on the reverse direction path, inside the router or host connected to the upstream bottleneck link 🎯 They use flow classes or per-flow queues at the upstream link interface to manage the queue of packets waiting to be transmitted on the bottleneck upstream link 🚦
This type of technique caps the size of the upstream link buffer queue, and uses an algorithm to remove (discard) excess ACKs from each queue ✂️ It works because of the cumulative nature of ACKs (section 4.1) 🧁 Two approaches are described that use this kind of mitigation.
5.2.1 ACK Filtering
ACK Filtering (AF) [DMT96, BPK99] (also known as ACK Suppression [SF98, Sam99, FSS01]) is a TCP-aware link-layer technique that reduces the number of ACKs sent on the upstream link 🧹 It's been deployed in specific production networks, like asymmetric satellite networks [ASB96] 🛰️ The challenge is making sure the sender doesn't stall waiting for ACKs, which can happen if ACKs get removed indiscriminately 😱
When an ACK from the receiver is about to be enqueued at an upstream bottleneck link interface, the router or the end host link layer (if the host is directly connected to the upstream bottleneck link) checks the transmit queue(s) for older ACKs belonging to the same TCP connection 🔍 If it finds some, it removes some (or all of them) from the queue, which reduces the number of ACKs 🗑️
Some ACKs also have other jobs in TCP [RFC1144], so they shouldn't be deleted if normal operation is going to keep working 🛑 AF should therefore not delete an ACK that has any data or TCP flags set (SYN, RST, URG, and FIN). It also should avoid deleting a series of 3 duplicate ACKs that signal the need for Fast Retransmission [RFC2581], or ACKs with the Selective ACK option (SACK) [RFC2018], from the queue, so TCP's data-driven loss recovery doesn't get messed up 🥺 Proper treatment is also needed to keep ECN feedback (carried in the TCP header) working correctly [RFC3168] 💜
A range of policies can be used to filter ACKs 🎲 They might be deterministic or random (kind of like a random-drop gateway, but they should take the semantics of the items in the queue into account). Algorithms have also been suggested to guarantee a minimum ACK rate so the TCP sender window keeps getting updated [Sam99, FSS01], and to limit how many data packets (TCP segments) a Stretch ACK acknowledges [Sam99, FSS01]. Per-flow state only needs to be kept for connections with at least one packet in the queue (similar to FRED [LM97]) 🧠 That state is soft [Cla88], and if needed it can easily be rebuilt from the contents of the queue 🔄
The undesirable effect of delayed DupACKs (section 3.4) can be reduced by deleting duplicate ACKs above a threshold value [MJW00, CLP98]. That allows Fast Retransmission while avoiding early TCP timeouts, which might otherwise happen because of excessive queuing of DupACKs ⏱️
Future schemes may include more advanced rules that allow removal of selected SACKs [RFC2018] 🔮 A scheme like that could stop the upstream link queue from getting filled by back-to-back ACKs with SACK blocks. A SACK packet is much bigger than an ACK, so otherwise it would add a lot to the path delay in the reverse direction 🐌 Picking suitable algorithms is still an ongoing area of research 🔬
RECOMMENDATION: ACK Filtering requires a modification to the upstream link interface. The scheme has been deployed in some networks where the extra processing overhead (per ACK) may be compensated for by avoiding the need to modify TCP. ACK Filtering can generate Stretch ACKs resulting in large bursts of TCP data packets. Therefore on its own, it is not recommended for use in the general Internet. 🚨
ACK Filtering used together with a scheme that mitigates the effect of Stretch ACKs (i.e., controls TCP sender burst size) is recommended for paths with appreciable asymmetry (k>1) and/or with bidirectional traffic 🫶 Suitable algorithms to support IPSec authentication, SACK, and ECN are still areas of ongoing research 🌸
5.2.2 ACK Decimation 🌸
okay so ACK Decimation is built on standard router mechanisms 💅 bestie. If the upstream bottleneck link has (small) per-flow queues and a chosen dropping policy (like Weighted Fair Queuing, WFQ), you can get a similar effect to AF (section 5.2.1), but ngl you have less control over which packets actually get dropped 🥺.
Here's the vibe: the router or host at the bottleneck's upstream link keeps per-flow queues and services them fairly (or with priorities) by queuing and scheduling ACKs and data packets in the reverse direction. A small queue threshold is kept so that excess ACKs get dropped from the tail of each queue, which helps cut down ACK Congestion ✨. The catch is that it can't identify special ACK packets (c.f., AF), so it might accidentally drop DupACKs, FIN/ACK, RST packets, or packets carrying ECN information [RFC3168]. Losing those doesn't really change network congestion, but it does hurt the TCP session that sees the loss 😭.
A WFQ scheduler can give interactive traffic a higher priority (if it has a way to identify that traffic) and share the rest of the capacity fairly with bulk traffic 🎀. With bidirectional traffic and a suitable scheduling policy, this may give fairer sharing for ACK and data packets. Over asymmetric links, the forward transmission rate goes up when the ACK Decimation rate goes up, which leads to Stretch ACKs being generated. Just like with AF, the TCP sender's burst size grows when Stretch ACKs are received, unless other techniques are used in combination with this one 🙈.
This technique has been deployed in specific networks, like a network with high bandwidth asymmetry supporting high-speed data services to in-transit mobile hosts [Seg00]. It's not optimal, but it offered a potential mitigation when the TCP header is hard to identify or not visible to the link layer (e.g., due to IPSec encryption) 🔐.
RECOMMENDATION: ACK Decimation uses standard router mechanisms at the upstream link interface to constrain the rate at which ACKs are fed to the upstream link. The technique is beneficial with paths having appreciable asymmetry (k>1). It is however suboptimal, in that it may lead to inefficient TCP error recovery (and hence in some cases degraded TCP performance), and provides only crude control of link behavior. It is therefore recommended that where possible, ACK Filtering should be used in preference to ACK Decimation.
When ACK Decimation is used on paths with an appreciable asymmetry (k>1) (or with bidirectional traffic), it increases the burst size of the TCP sender, so a scheme to mitigate the effect of Stretch ACKs or control burstiness is strongly recommended 💜.
5.3 TYPE 2: Handling Infrequent ACKs 🧚
TYPE 2 mitigations do TYPE 1 upstream link bandwidth management, but they also bring in a second active element that softens the hit from the reduced ACK rate and the burstiness of ACK transmission. This is desirable when end hosts use standard TCP sender implementations (e.g., those not implementing the techniques in sections 4.6, 4.7) 🫶.
Picture a path where a TYPE 1 scheme forwards a Stretch ACK covering d TCP packets (i.e., where the acknowledgement number is d*MSS larger than the last ACK received by the TCP sender). When the TCP sender gets this ACK, it can send a burst of d (or d+1) TCP data packets. The sender is also constrained by the current cwnd. Received ACKs also serve to increase cwnd (by at most one MSS) 👑.
A TYPE 2 scheme mitigates the impact of the reduced ACK frequency that comes from using a TYPE 1 scheme. It does this by interspersing additional ACKs before each received Stretch ACK. Those additional ACKs, together with the original ACK, give the TCP sender enough ACKs for the TCP cwnd to open the same way it would if each of the original ACKs sent by the TCP receiver had been forwarded over the reverse path 🦋. On top of that, by trying to restore the spacing between ACKs, such a scheme can also bring back the TCP self-clocking behavior and reduce the TCP sender burst size. These schemes need to ensure conservative behavior (i.e., they should not introduce more ACKs than were originally sent) and reduce the probability of ACK Compression [ZSC91].
The action happens at two points on the return path: the upstream link interface (where excess ACKs are removed), and a point further along the reverse path (after the bottleneck upstream link(s)), where replacement ACKs are inserted. This tries to rebuild the ACK stream the TCP receiver originally sent when it's used together with AF (section 5.2.1) or ACK Decimation (section 5.2.2) 🔥.
TYPE 2 mitigations may be performed locally at the receive interface directly after the upstream bottleneck link, or they may be applied at any point further along the reverse path. That reverse path doesn't have to be on the forward path, since asymmetric routing may use different forward and reverse internet paths 🌈. Because these techniques may generate multiple ACKs when each individual Stretch ACK comes in, it is strongly recommended that the expander implements a scheme to prevent exploitation as a "packet amplifier" in a Denial-of-Service (DoS) attack (e.g., to verify the originator of the ACK) 💀. Identifying the sender could be done with appropriately configured packet filters and/or tunnel authentication procedures (e.g., [RFC2402, RFC2406]). A limit on the number of reconstructed ACKs that may be generated from a single packet may also be desirable 🛡️.
5.3.1 ACK Reconstruction 🦄
ACK Reconstruction (AR) [BPK99] is used in conjunction with AF (section 5.2.1). AR deploys a soft-state [Cla88] agent called an ACK Reconstructor on the reverse path following the upstream bottleneck link. The soft state can be regenerated if it's lost, based on received ACKs. When a Stretch ACK comes in, AR adds extra ACKs by filling gaps in the ACK sequence. Some potential Denial-of-Service vulnerabilities may arise (section 6), and they need to be dealt with using appropriate security techniques 🔐.
The Reconstructor figures out how many additional ACKs to make by estimating the number of filtered ACKs. It does this using implicit information in the received ACK stream by watching the ACK sequence number of each received ACK ✨. An example implementation could set an ACK threshold, ackthresh, to twice the MSS (this assumes the chosen MSS is known by the link). The factor of two matches standard TCP delayed-ACK policy (d=2). So if successive ACKs arrive separated by delta, the Reconstructor regenerates a maximum of ((delta/ackthresh) - 2) ACKs.
To reduce the TCP sender burst size and let the cwnd increase at a rate governed by the downstream link, the reconstructed ACKs must be sent at a consistent rate (i.e., with temporal spacing between reconstructed ACKs). One method is for the Reconstructor to measure the arrival rate of ACKs with an exponentially weighted moving average estimator. That rate depends on the output rate from the upstream link and on whether other traffic is sharing the link. The output of the estimator shows the average temporal spacing for the ACKs (and the average rate at which ACKs would reach the TCP sender if there were no further losses or delays). The Reconstructor can use this to set the temporal spacing of reconstructed ACKs. The scheme may also be combined with TCP sender adaptation (e.g., a combination of the techniques in sections 4.6 and 4.7) 🎀.
The trade-off in AR is between less TCP sender burstiness and a better rate of cwnd increase, with a reduction in RTT variation, versus a modest increase in the path RTT. The technique cannot do reconstruction on connections using IPSec (AH [RFC2402] or ESP [RFC2406]), since it can't generate appropriate security information. It also can't regenerate other packet header information (e.g., the exact pattern of bits carried in the IP packet ECN field [RFC3168] or the TCP RTTM option [RFC1323]) 🙈.
An ACK Reconstructor operates correctly (i.e., it generates no spurious ACKs and preserves the end-to-end semantics of TCP) when all of the following are provided:
- (i) the TCP receiver uses ACK Delay (d=2) [RFC2581]
- (ii) the Reconstructor receives only in-order ACKs
- (iii) all ACKs are routed via the Reconstructor
- (iv) the Reconstructor correctly determines the TCP MSS used by the session
- (v) the packets do not carry additional header information (e.g., TCP RTTM option [RFC1323], or IPSec using AH [RFC2402] or ESP [RFC2406])
RECOMMENDATION: ACK Reconstruction is an experimental transparent modification performed on the reverse path following the upstream bottleneck link. It is designed to be used in conjunction with a TYPE 1 mitigation. It reduces the burst size of TCP transmission in the forward direction, which may otherwise increase when TYPE 1 schemes are used alone. It requires modification of equipment after the upstream link (including maintaining per-flow soft state). The scheme introduces implicit assumptions about the network path and has potential Denial-of-Service vulnerabilities (i.e., acting as a packet amplifier); these need to be better understood and addressed by appropriate security techniques.
Selecting appropriate algorithms to pace the ACK traffic remains an open research issue 🧪. There is also currently little experience of the implications of using such techniques in the Internet, so it is recommended that this technique should not be used within the Internet in its current form 🚫.
5.3.2. ACK Compaction and Companding ✨
Okay bestie, so ACK Compaction and ACK Companding [SAM99, FSS01] 🦋 are techniques that run at a point on the reverse path, right after the constrained ACK bottleneck. Like AR (section 5.3.1), they're used together with an AF technique (section 5.2.1) and they regenerate the filtered ACKs to restore the ACK stream. The difference is that they use a modified AF (called a compactor or compressor) that adds explicit information to every Stretch ACK it generates 🎀. That info is there to explicitly synchronize the reconstruction step, which is called expansion here 🧚.
The modified AF is two mods in one 👑. First, when the compressor deletes an ACK from the upstream bottleneck link queue, it attaches explicit information (a prefix) to the ACK that's left over. That leftover ACK gets marked so it doesn't get deleted later 🔒. The prefix holds details about the conditions under which ACKs were filtered before. A bunch of stuff can go in there, like the number of ACKs the AF deleted and the average number of bytes acknowledged 📊. An expander at the far end of the tunnel can use it later. Timing info can also be added to control the pacing of the regenerated ACKs [FSS01] ⏱️, and the temporal spacing of the filtered ACKs can be encoded too.
Encoding the prefix means the expander has to recognize a modified ACK header 🤓. Normally that would keep the expander to link-local operation (at the receive interface of the upstream bottleneck link). If expansion is needed further along the reverse path, a tunnel can pass the modified ACKs to a remote expander. Yes, the tunnel adds extra overhead, but networks with asymmetric capacity and symmetric routing already use tunnels a lot anyway (for example, in a UDLR network [RFC3077], the expander may be co-located with the feed router) 🌈.
ACK expansion is stateless, which means each received packet is processed on its own, without looking at packets that came before it 🫶. It takes the prefix info plus the acknowledgment field in the received ACK and produces the same number of ACKs that the compactor deleted. Those ACKs get forwarded to the original destination (the TCP sender), so normal TCP ACK clocking is preserved 💖. This is the best part: ACK Compaction, unlike AR, doesn't depend on specific ACK policies, and it doesn't need to see every ACK on the reverse path. It might even work with schemes like DAASS [RFC2760] 😭.
Some potential Denial-of-Service vulnerabilities may pop up (section 6) and need to be handled with appropriate security techniques 🚨. Also, the technique can't do reconstruction on connections using IPSec, because it can't regenerate the right security information 💀. It's possible to explicitly encode IPSec security information from the suppressed packets, which would allow operation with IPSec AH. However, that's still an open research issue, and it adds overhead per ACK 🥲.
RECOMMENDATION: ACK Compaction and Companding are experimental transparent modifications performed on the reverse path, after the upstream bottleneck link. They're designed to be used together with a modified TYPE 1 mitigation, and they reduce the burst size of TCP transmission in the forward direction. Without them, that burst size may otherwise increase when TYPE 1 schemes are used alone 🍓.
The technique is desirable, but it requires modifying equipment after the upstream bottleneck link (including processing a modified ACK header). Picking the right algorithms to pace the ACK traffic is still an open research issue 🔬. Some potential Denial-of-Service vulnerabilities may come up with any device that acts as a packet amplifier, and those need appropriate security techniques too. There's little experience with using this scheme over Internet paths. It's a subject of ongoing research, and it is not recommended for use within the Internet in its current form 🫠.
5.3.3. Mitigating TCP packet bursts generated by Infrequent ACKs 🌸
The bursts of data packets that show up when a Type 1 scheme runs on the reverse direction path can be mitigated by putting a router that supports Generic Traffic Shaping (GTS) on the forward path [Seg00]. GTS is a standard router mechanism, and lots of deployed routers already implement it 🛠️. This technique doesn't eliminate the bursts the TCP sender generates. Instead, it tries to smooth them out using scheduling and queuing, producing traffic that looks like what you'd get with TCP Pacing (section 4.6) ✨. The catch is that these techniques need per-flow soft-state in the router and add per-packet processing overhead. Some extra buffer capacity is needed to queue the packets being shaped 📦.
To perform GTS, the router has to pick appropriate traffic shaping parameters. Those depend on knowing the network policy, the connection behavior, and/or the downstream bottleneck characteristics 🧠. GTS can also enforce other network policies and promote fairness between competing TCP connections (and UDP and multicast flows too) 🤝. It also lowers the probability of ACK Compression [ZSC91] 📉.
Smoothing out packet bursts reduces the impact of TCP transmission bursts on routers and hosts located after the point where GTS is done. So it's desirable to do GTS near the sending host, or at least at a point before the first forward path bottleneck router 🏃♀️.
RECOMMENDATIONS: Generic Traffic Shaping (GTS) is a transparent technique employed at a router on the forward path. The algorithms to implement GTS are available in widely deployed routers and may be used on an Internet link. They do imply significant additional per-packet processing cost, though 💸.
Configuring GTS is a policy decision made by a network service provider 🧾. When it's configured appropriately, the technique reduces the size of TCP data packet bursts, which mitigates the effects of Type 1 techniques 💅. GTS is recommended for use in the Internet together with type 1 techniques such as ACK Filtering (section 5.2.1) and ACK Decimation (section 5.2.2) 💜.
5.4 TYPE 3: Upstream Link Scheduling 🎀
Lots of the above schemes imply using per-flow queues (or per-connection queues in the case of TCP) at the upstream bottleneck link 🧵. Per-flow queuing (for example, FQ or CBQ) helps on any slow link where the time to transmit a packet is a big chunk of the path RTT [RFC3150] 🐢. Type 3 schemes give extra benefit when they're used with one of the techniques above 🌟.
5.4.1. Per-Flow queuing at the Upstream Bottleneck Link 🦄
When bidirectional traffic shows up in a bandwidth asymmetric network, the competing ACK and packet data flows along the return path can hurt the performance of both the upstream and downstream flows [KVR98] 😵. So it's highly desirable to use a queuing strategy combined with a scheduling mechanism at the upstream link. This has also been called priority-based multiplexing [RFC3135] 👑.
On a slow upstream link, sending large data packets ahead of ACKs can introduce a lot of jitter [RFC3150] 🫨. A simple scheme is per-flow queuing with a fair scheduler (for example, round robin service to all flows, or priority scheduling) 🎠. A modified scheduler [KVR98] could limit the number of ACKs a host is allowed to transmit upstream before it sends a data packet (assuming at least one data packet is waiting in the upstream link queue). That guarantees at least a certain minimum share of the capacity to the reverse direction flows, while still letting the forward direction flows improve TCP throughput 💖.
Bulk (payload) compression, a small MTU, link level transparent fragmentation [RFC1991, RFC2686], or link level suspend/resume (where higher priority frames can pre-empt the transmission of lower priority frames) can all be used to reduce the impact (jitter) of bidirectional traffic on low speed links [RFC3150] 🧩. More advanced schemes (for example, WFQ) can also improve the performance of transfers with multiple ACK streams, such as http [Seg00] 🌈.
RECOMMENDATION: Per-flow queuing is a transparent modification performed at the upstream bottleneck link. Per-flow (or per-class) scheduling doesn't impact the congestion behavior of the Internet, so it can be used on any Internet link 🕊️. The scheme has particular benefits for slow links, and it's widely implemented and widely deployed on links running at less than 2 Mbps. It's recommended as a mitigation on its own or in combination with one of the other described techniques ⭐.
5.4.2. ACKs-first Scheduling 🚀
ACKs-first Scheduling is an experimental technique for improving the performance of bidirectional transfers 🧪. In this case, data packets and ACKs compete for resources at the upstream bottleneck link [RFC3150]. A single First-In First-Out (FIFO) queue for both data packets and ACKs could hurt the performance of forward transfers 😩. For example, if the upstream bottleneck link is a 28.8 kbps dialup line, transmitting a 1 Kbyte data packet takes about 280 ms. So even if just two such data packets get queued ahead of ACKs (not an uncommon occurrence, since data packets are sent out in pairs during slow start), they would shut out the ACKs for well over half a second 🫠. If more than two data packets are queued up ahead of an ACK, the ACKs get delayed even more [RFC3150] 📈.
One possible way to fix this is to schedule data and ACKs differently from FIFO. One algorithm in particular is ACKs-first scheduling, which gives ACKs a higher priority than data packets 🥇. The motivation is that it minimizes the idle time for the forward connection, by minimizing the time ACKs spend queued behind data packets at the upstream link 💫. At the same time, with Type 0 techniques such as header compression [RFC1144], the transmission time of ACKs becomes small enough that the impact on subsequent data packets is minimal 🪄. (Subnetworks where the per-packet overhead of the upstream link is large, such as packet radio subnetworks, are an exception; see section 3.2 🙈.) This scheduling scheme doesn't require the upstream bottleneck router/host to explicitly identify or keep state for individual TCP connections 🫶.
ACKs-first scheduling doesn't help avoid a delay caused by a data packet that's already in transmission 🚧. Link fragmentation or suspend/resume may be beneficial in that case 🧚.
RECOMMENDATION: ACKs-first scheduling is an experimental transparent modification performed at the upstream bottleneck link. If it's used without a mechanism to regulate the volume of ACKs (such as ACK Congestion Control (ACC), section 4.3), it could lead to starvation of data packets. That's a performance penalty experienced by the end hosts using the link, and it doesn't modify Internet congestion behavior 🍬. Experiments indicate that ACKs-first scheduling in combination with ACC is promising. However, there's little experience using the technique in the wider Internet. Further development of the technique remains an open research issue, so the scheme is not currently recommended for use within the Internet 🌸.
6. Security Considerations 🛡️
Okay bestie, this section is serious, but it's also lowkey fascinating 🔥. The recommendations in this document do not impact the integrity of TCP, introduce new security implications to the TCP protocol, or affect applications using TCP. ✨
Some of the security stuff here comes from using IPSec on end hosts or on routers along the return path. IPSec can prevent or complicate some of the mitigations, so here's how it plays out 👀
- (i) When IPSec ESP [RFC2406] is used to encrypt the IP payload, the TCP header can neither be read nor modified by intermediate entities. This rules out header compression, ACK Filtering, ACK Reconstruction, and the ACK Compaction. 💀
- (ii) The TCP header information may be visible when some forms of network layer security are used. For example, with IPSec AH [RFC2402], intermediaries can read the TCP header but not modify it. This may allow extensions that support ACK Filtering in future, but it rules out intermediaries generating new packets (e.g., ACK Reconstruction). The enhanced header compression scheme discussed in [RFC2507] would also work with IPSec AH. 🦋
🚨 Denial-of-Service (DoS) alert: there are potential DoS implications when using Type 2 schemes. Unless additional security mechanisms are used, a Reconstructor/expander could be exploited as a packet amplifier. A third party may inject unauthorized Stretch ACKs into the reverse path, which triggers the generation of additional ACKs. Those ACKs would eat up capacity on the return path and processing resources at the systems along the path, including the destination host. That gives attackers a potential platform for a DoS attack. The usual precautions must be taken to verify the correct tunnel end point, and to make sure applications cannot falsely inject packets that expand to generate unwanted traffic. Imposing a rate limit and bound on the delayed ACK factor(d) would also lessen the impact of any undetected exploitation. 😭
7. Summary 🌸
So here's the big picture. This document looks at several TCP performance constraints that come from asymmetry in the properties of the forward and reverse paths across an IP network. These constraints show up, e.g., because of bandwidth (capacity) asymmetry, asymmetric shared media in the reverse direction, and interactions with Media Access Control (MAC) protocols. Asymmetric capacity may cause TCP Acknowledgments (ACKs) to be lost or become inordinately delayed (e.g., when a bottleneck link is shared between many flows, or when there is bidirectional traffic). This effect may be made worse by media-access delays (e.g., in certain multi-hop radio subnetworks, or satellite Bandwidth on Demand access). Asymmetry, and especially high asymmetry, raises a whole set of TCP performance issues. 🫶
A set of techniques that improve performance is surveyed. These include techniques to ease ACK Congestion, and techniques that let a TCP sender cope with infrequent ACKs without wrecking TCP self-clocking. The techniques cover end-to-end, local link-layer, and subnetwork schemes. Many of them have been evaluated in detail through analysis, simulation, and/or implementation on asymmetric subnetworks that form part of the Internet. However, there's as yet not enough operational experience for some techniques, so those remain items of ongoing research and experimentation. 🧪
The following table summarizes the current recommendations. Mechanisms are classified as recommended (REC), not recommended (NOT REC) or experimental (EXP). Experimental techniques may not be well specified. These techniques will need further operational experience before they can be recommended for use in the public Internet. 💅
The recommendations for end-to-end host modifications are summarized in table 1. It lists each technique, the section where each one is discussed, and where it gets applied (S means the host sending TCP data packets in the forward direction, R means the host that receives those data packets). 🎀
+------------------------+-------------+------------+--------+
| Technique | Use | Section | Where |
+------------------------+-------------+------------+--------+
| Modified Delayed ACKs | NOT REC | 4.1 | TCP R |
| Large MSS & NO FRAG | REC | 4.2 | TCP SR |
| Large MSS & IP FRAG | NOT REC | 4.2 | TCP SR |
| ACK Congestion Control | EXP | 4.3 | TCP SR |
| Window Pred. Mech (WPM)| NOT REC | 4.4 | TCP R |
| Window Cwnd. Est. (ACE)| NOT REC | 4.5 | TCP R |
| TCP Sender Pacing | EXP *1 | 4.6 | TCP S |
| Byte Counting | NOT REC *2 | 4.7 | TCP S |
| Backpressure | EXP *1 | 4.8 | TCP R |
+------------------------+-------------+------------+--------+
Table 1: Recommendations concerning host modifications.
*1 Implementation of the technique may require changes to the
internal design of the protocol stack in end hosts.
*2 Dependent on a scheme for preventing excessive TCP transmission
burst.The recommendations for techniques that don't require the TCP sender and receiver to know they exist (i.e., transparent techniques) are summarized in table 2. Each technique is listed along with the section where it's discussed, and where the technique is applied (S means the sending interface prior to the upstream bottleneck link, R means the receiving interface following the upstream bottleneck link). 🦋
+------------------------+-------------+------------+--------+
| Mechanism | Use | Section | Type |
+------------------------+-------------+------------+--------+
| Header Compr. (V-J) | REC *1 | 5.1.1 | 0 SR |
| Header Compr. (ROHC) | REC *1 *2 | 5.1.2 | 0 SR |
+------------------------+-------------+------------+--------+
| ACK Filtering (AF) | EXP *3 | 5.2.1 | 1 S |
| ACK Decimation | EXP *3 | 5.2.2 | 1 S |
+------------------------+-------------+------------+--------+
| ACK Reconstruction (AR)| NOT REC | 5.3.1 | 2 *4 |
| ACK Compaction/Compand.| EXP | 5.3.2 | 2 S *4 |
| Gen. Traff. Shap. (GTS)| REC | 5.3.3 | 2 *5 |
+------------------------+-------------+------------+--------+
| Fair Queueing (FQ) | REC | 5.4.1 | 3 S |
| ACKs-First Scheduling | NOT REC | 5.4.2 | 3 S |
+------------------------+-------------+------------+--------+
Table 2: Recommendations concerning transparent modifications.
*1 At high asymmetry these schemes may degrade TCP performance, but
are not considered harmful to the Internet.
*2 Standardisation of new TCP compression protocols is the subject of
ongoing work within the ROHC WG, refer to other IETF RFCs on the
use of these techniques.
*3 Use in the Internet is dependent on a scheme for preventing
excessive TCP transmission burst.
*4 Performed at a point along the reverse path after the upstream
bottleneck link.
*5 Performed at a point along the forward path.8. Acknowledgments 🙏
This document has benefited from comments from members of the Performance Implications of Links (PILC) Working Group. Special thanks to the authors' favourites, John Border, Spencer Dawkins, Aaron Falk, Dan Grossman, Randy Katz, Jeff Mandin, Rod Ragland, Ramon Segura, Joe Touch, and Lloyd Wood for their useful comments. 💖 They also acknowledge the data provided by Metricom Inc., concerning operation of their packet data network. 📡
9. References 📚
References of the form RFCnnnn are Internet Request for Comments (RFC) documents available online at http://www.rfc-editor.org/.
9.1 Normative References
[RFC793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, October 1989.
[RFC1144] Jacobson, V., "Compressing TCP/IP Headers for Low-Speed
Serial Links", RFC 1144, February 1990.
[RFC1191] Mogul, J. and S. Deering, "Path MTU Discovery", RFC 1191,
November 1990.
[RFC2581] Allman, M., Paxson, V. and W. Stevens, "TCP Congestion
Control", RFC 2581, April 1999.
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D. and P. Traina,
"Generic Routing Encapsulation (GRE)", RFC 2784, March
2000.
[RFC3135] Border, J., Kojo, M., Griner, J., Montenegro, G. and Z.
Shelby, "Performance Enhancing Proxies Intended to Mitigate
Link-Related Degradations", RFC 3135, June 2001.9.2. Informative References ✨
okay bestie, these are the receipts 🧾 the extra reading list for everything this doc is built on. nothing here is required to get your head around it, but if you want to go down the rabbit hole, it's all right here 🦋
[abc-ID] Allman, M., "TCP Congestion Control with Appropriate Byte
Counting", Work in Progress.
[All97b] Allman, M., "Fixing Two BSD TCP Bugs", Technical Report
CR-204151, NASA Lewis Research Center, October 1997.
[ANS01] ANSI Standard T1.413, "Network to Customer Installation
Interfaces - Asymmetric Digital Subscriber Lines (ADSL)
Metallic Interface", November 1998.
[ASB96] Arora, V., Suphasindhu, N., Baras, J.S. and D. Dillon,
"Asymmetric Internet Access over Satellite-Terrestrial
Networks", Proc. AIAA: 16th International Communications
Satellite Systems Conference and Exhibit, Part 1,
Washington, D.C., February 25-29, 1996, pp.476-482.
[AST00] Aggarwal, A., Savage, S., and T. Anderson, "Understanding
the Performance of TCP Pacing", Proc. IEEE INFOCOM, Tel-
Aviv, Israel, V.3, March 2000, pp. 1157-1165.
[Bal98] Balakrishnan, H., "Challenges to Reliable Data Transport
over Heterogeneous Wireless Networks", Ph.D. Thesis,
University of California at Berkeley, USA, August 1998.
http://nms.lcs.mit.edu/papers/hari-phd/
[BPK99] Balakrishnan, H., Padmanabhan, V. N., and R. H. Katz, "The
Effects of Asymmetry on TCP Performance", ACM Mobile
Networks and Applications (MONET), Vol.4, No.3, 1999, pp.
219-241. An expanded version of a paper published at Proc.
ACM/IEEE Mobile Communications Conference (MOBICOM), 1997.
[BPS00] Bennett, J. C., Partridge, C., and N. Schectman, "Packet
Reordering is Not Pathological Network Behaviour", IEEE/ACM
Transactions on Networking, Vol. 7, Issue. 6, 2000,
pp.789-798.
[Cla88] Clark, D.D, "The Design Philosophy of the DARPA Internet
Protocols", ACM Computer Communications Review (CCR), Vol.
18, Issue 4, 1988, pp.106-114.
[CLC99] Clausen, H., Linder, H., and B. Collini-Nocker, "Internet
over Broadcast Satellites", IEEE Communications Magazine,
Vol. 37, Issue. 6, 1999, pp.146-151.
[CLP98] Calveras, A., Linares, J., and J. Paradells, "Window
Prediction Mechanism for Improving TCP in Wireless
Asymmetric Links". Proc. IEEE Global Communications
Conference (GLOBECOM), Sydney Australia, November 1998,
pp.533-538.
[CR98] Cohen, R., and Ramanathan, S., "Tuning TCP for High
Performance in Hybrid Fiber Coaxial Broad-Band Access
Networks", IEEE/ACM Transactions on Networking, Vol.6,
No.1, 1998, pp.15-29.
[DS00] Cable Television Laboratories, Inc., Data-Over-Cable
Service Interface Specifications---Radio Frequency
Interface Specification SP-RFIv1.1-I04-00407, 2000
[DS01] Data-Over-Cable Service Interface Specifications, Radio
Frequency Interface Specification 1.0, SP-RFI-I05-991105,
Cable Television Laboratories, Inc., November 1999.
[DMT96] Durst, R., Miller, G., and E. Travis, "TCP Extensions for
Space Communications", ACM/IEEE Mobile Communications
Conference (MOBICOM), New York, USA, November 1996, pp.15-
26.
[EN97] "Digital Video Broadcasting (DVB); DVB Specification for
Data Broadcasting", European Standard (Telecommunications
series) EN 301 192, 1997.
[EN00] "Digital Video Broadcasting (DVB); Interaction Channel for
Satellite Distribution Systems", Draft European Standard
(Telecommunications series) ETSI, Draft EN 301 790, v.1.2.1
[FJ93] Floyd, S., and V. Jacobson, "Random Early Detection
gateways for Congestion Avoidance", IEEE/ACM Transactions
on Networking, Vol.1, No.4, 1993, pp.397-413.
[FSS01] Fairhurst, G., Samaraweera, N.K.G, Sooriyabandara, M.,
Harun, H., Hodson, K., and R. Donardio, "Performance Issues
in Asymmetric Service Provision using Broadband Satellite",
IEE Proceedings on Communication, Vol.148, No.2, 2001,
pp.95-99.
[ITU01] ITU-T Recommendation E.681, "Traffic Engineering Methods
For IP Access Networks Based on Hybrid Fiber/Coax System",
September 2001.
[ITU02] ITU-T Recommendation G.992.1, "Asymmetrical Digital
Subscriber Line (ADSL) Transceivers", July 1999.
[Jac88] Jacobson, V., "Congestion Avoidance and Control", Proc. ACM
SIGCOMM, Stanford, CA, ACM Computer Communications Review
(CCR), Vol.18, No.4, 1988, pp.314-329.
[Ken87] Kent C.A., and J. C. Mogul, "Fragmentation Considered
Harmful", Proc. ACM SIGCOMM, USA, ACM Computer
Communications Review (CCR), Vol.17, No.5, 1988, pp.390-
401.
[KSG98] Krout, T., Solsman, M., and J. Goldstein, "The Effects of
Asymmetric Satellite Networks on Protocols", Proc. IEEE
Military Communications Conference (MILCOM), Bradford, MA,
USA, Vol.3, 1998, pp.1072-1076.
[KVR98] Kalampoukas, L., Varma, A., and Ramakrishnan, K.K.,
"Improving TCP Throughput over Two-Way Asymmetric Links:
Analysis and Solutions", Proc. ACM SIGMETRICS, Medison,
USA, 1998, pp.78-89.
[LM97] Lin, D., and R. Morris, "Dynamics of Random Early
Detection", Proc. ACM SIGCOMM, Cannes, France, ACM Computer
Communications Review (CCR), Vol.27, No.4, 1997, pp.78-89.
[LMS97] Lakshman, T.V., Madhow, U., and B. Suter, "Window-based
Error Recovery and Flow Control with a Slow Acknowledgement
Channel: A Study of TCP/IP Performance", Proc. IEEE
INFOCOM, Vol.3, Kobe, Japan, 1997, pp.1199-1209.
[MJW00] Ming-Chit, I.T., Jinsong, D., and W. Wang,"Improving TCP
Performance Over Asymmetric Networks", ACM SIGCOMM, ACM
Computer Communications Review (CCR), Vol.30, No.3, 2000.
[Pad98] Padmanabhan, V.N., "Addressing the Challenges of Web Data
Transport", Ph.D. Thesis, University of California at
Berkeley, USA, September 1998 (also Tech Report UCB/CSD-
98-1016). http://www.cs.berkeley.edu/~padmanab/phd-
thesis.html
[RFC1323] Jacobson, V., Braden, R. and D. Borman, "TCP Extensions for
High Performance", RFC 1323, May 1992.
[RFC2018] Mathis, B., Mahdavi, J., Floyd, S. and A. Romanow, "TCP
Selective Acknowledgment Options", RFC 2018, October 1996.
[RFC2402] Kent, S. and R. Atkinson, "IP Authentication Header", RFC
2402, November 1998.
[RFC2406] Kent, S. and R. Atkinson, "IP Encapsulating Security
Payload (ESP)", RFC 2406, November 1998.
[RFC2507] Degermark, M., Nordgren, B. and S. Pink, "IP Header
Compression", RFC 2507, February 1999.
[RFC2525] Paxson, V., Allman, M., Dawson, S., Heavens, I. and B.
Volz, "Known TCP Implementation Problems", RFC 2525, March
1999.
[RFC2686] Bormann, C., "The Multi-Class Extension to Multi-Link PPP",
RFC 2686, September 1999.
[RFC2760] Allman, M., Dawkins, S., Glover, D., Griner, J., Henderson,
T., Heidemann, J., Kruse, H., Ostermann, S., Scott, K.,
Semke, J., Touch, J. and D. Tran, "Ongoing TCP Research
Related to Satellites", RFC 2760, February 2000.
[RFC2988] Paxson, V. and M. Allman, "Computing TCP's Retransmission
Timer", RFC 2988, November 2000.
[RFC3077] Duros, E., Dabbous, W., Izumiyama, H., Fujii, N. and Y.
Zhang, "A link Layer tunneling mechanism for unidirectional
links", RFC 3077, March 2001.
[RFC3095] Bormann, C., Burmeister, C., Degermark, M., Fukushima, H.,
Hannu, H., Jonsson, E., Hakenberg, R., Koren, T., Le, K.,
Liu, Z., Martensson, A., Miyazaki, A., Svanbro, K., Wiebke,
T., Yoshimura, T. and H. Zheng, "RObust Header Compression
(ROHC): Framework and four profiles: RTP, UDP ESP and
uncompressed", RFC 3095, July 2001.
[RFC3150] Dawkins, S., Montenegro, G., Kojo, M. and V. Magret, "End-
to-end Performance Implications of Slow Links", BCP 48, RFC
3150, July 2001.
[RFC3168] Ramakrishnan K., Floyd, S. and D. Black, "A Proposal to add
Explicit Congestion Notification (ECN) to IP", RFC 3168,
September 2001.
[Sam99] Samaraweera, N.K.G, "Return Link Optimization for Internet
Service Provision Using DVB-S Networks", ACM Computer
Communications Review (CCR), Vol.29, No.3, 1999, pp.4-19.
[Seg00] Segura R., "Asymmetric Networking Techniques For Hybrid
Satellite Communications", NC3A, The Hague, Netherlands,
NATO Technical Note 810, August 2000, pp.32-37.
[SF98] Samaraweera, N.K.G., and G. Fairhurst. "High Speed Internet
Access using Satellite-based DVB Networks", Proc. IEEE
International Networks Conference (INC98), Plymouth, UK,
1998, pp.23-28.
[ZSC91] Zhang, L., Shenker, S., and D. D. Clark, "Observations and
Dynamics of a Congestion Control Algorithm: The Effects of
Two-Way Traffic", Proc. ACM SIGCOMM, ACM Computer
Communications Review (CCR), Vol 21, No 4, 1991, pp.133-
147.10. IANA Considerations 💅
the IANA section is giving very short and sweet energy 🌈 bestie, there are no IANA considerations tied to this doc. nothing to register, nothing to assign, nobody has to go update a registry because of this one 🎀 ✨
Appendix - Examples of Subnetworks Exhibiting Network Path Asymmetry 🦋
okay bestie, this appendix is basically a "where does asymmetry actually show up in real life?" list 💅 Here's the intro: this appendix provides a list of some subnetworks that are known to experience network path asymmetry ✨ The capacity asymmetry on these paths can require mitigations to get acceptable overall performance 🫶 Examples include the following:
- IP service over some wide area and local area wireless networks 📡 Here the main asymmetry comes from the hub-and-spokes setup (think one base station talking to a bunch of mobile stations) 🧚 That means you need a Ready To Send / Clear To Send (RTS/CTS) protocol and a Medium Access Control (MAC) protocol that can handle the big turn-around time for the radios. A high per-packet transmission overhead can lead to significant network path asymmetry too 😭
- IP service over a forward satellite link using Digital Video Broadcast (DVB) transmission [EN97] 🛰️ (around 38-45 Mbps), paired with a slower upstream link on terrestrial tech like a dial-up modem, line of sight microwave, or cellular radio [CLC99] 🐢 The asymmetry here comes from the gap between upstream and downstream link capacities.
- Certain military networks [KSG98] that give Internet access to in-transit or isolated hosts [Seg00] 🎖️ They use a high-capacity downstream satellite link (around 2-3 Mbps) with a narrowband upstream link (around 2.4-9.6 kbps), either with Demand Assigned Multiple Access (DAMA) or fixed-rate satellite links 🌈 The main factor is the difference between upstream and downstream link capacities. Some differences between forward and reverse paths can also come from how upstream link capacity gets allocated 👀
- Most data over cable TV networks (e.g., DOCSIS [ITU01, DS00]) 📺 Here the analogue channels assigned for upstream traffic (the reverse direction) are narrower and can be noisier than the ones used downstream, so the two links end up with different transmission rates 💖 For example, in DOCSIS 1.0 [DS00], the downstream transmission rate is either 27 or 52 Mbps. Upstream rates can be dynamically picked from a series of rates ranging from 166 kbps to 9 Mbps ✨ Operators can also assign multiple upstream channels per downstream channel 🍓 Physical layer (PHY) overhead, which tags along with upstream transmissions but isn't there on the downstream link, can make the asymmetry worse too 💅 The Best Effort service, which is usually what carries TCP, uses a contention/reservation MAC protocol 🙈 A cable modem (CM) sending an isolated packet (like a TCP ACK) upstream has to contend with other CMs to request capacity from the central cable modem termination system (CMTS). The CMTS then grants timeslots to a CM for its upstream transmission 🎀 A CM can "piggyback" follow-up requests onto upstream packets, skipping contention cycles, so the spacing of TCP ACKs can get dramatically altered by small load changes on the cable network and by the inter-arrival times of TCP DATA packets 🌸 Plenty of other complexities can add to or mitigate the asymmetry in rate and access latency that upstream packets see compared to downstream packets in DOCSIS 🔥 The asymmetry end hosts experience can also change dynamically (for instance, with network load), and it gets extra messy when best effort services share capacity with services that have symmetric reserved capacity, like IP telephony over the Unsolicited Grant service [ITU01] 😭
- Asymmetric Digital Subscriber Line (ADSL) 📞 By definition, ADSL offers a downstream link transmission rate higher than its upstream rate 🌈 The available rates depend on channel quality and system configuration. For example, one widely deployed ADSL technology [ITU02, ANS01] runs at multiples of 32 kbps (up to 6.144 Mbps) downstream, and up to 640 kbps upstream 🧚 The asymmetry end hosts see can get worse when best effort services, like Internet access over ADSL, share the available upstream capacity with reserved services (such as constant bit rate voice telephony) 💜
Authors' Addresses 🗺️
Hari Balakrishnan
Laboratory for Computer Science
200 Technology Square
Massachusetts Institute of Technology
Cambridge, MA 02139
USA
Phone: +1-617-253-8713
EMail: hari@lcs.mit.edu
Web: http://nms.lcs.mit.edu/~hari/
Venkata N. Padmanabhan
Microsoft Research
One Microsoft Way
Redmond, WA 98052
USA
Phone: +1-425-705-2790
EMail: padmanab@microsoft.com
Web: http://www.research.microsoft.com/~padmanab/
Godred Fairhurst
Department of Engineering
Fraser Noble Building
University of Aberdeen
Aberdeen AB24 3UE
UK
EMail: gorry@erg.abdn.ac.uk
Web: http://www.erg.abdn.ac.uk/users/gorry
Mahesh Sooriyabandara
Department of Engineering
Fraser Noble Building
University of Aberdeen
Aberdeen AB24 3UE
UK
EMail: mahesh@erg.abdn.ac.uk
Web: http://www.erg.abdn.ac.uk/users/maheshFull Copyright Statement ⚖️
okay so this part is the legal stuff, and I'm keeping every word of it because it matters 🫶
Copyright (C) The Internet Society (2002). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.Acknowledgement 🌷
Funding for the RFC Editor function is currently provided by the Internet Society. 💖
📦 where this page comes from
- ietf: rfc3449 2002-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms