RFC 6077

RFC 6077: Open Research Issues in Internet Congestion Control 🦋

Part 1 of 12 ✨

• Number and title: RFC 6077, "Open Research Issues in Internet Congestion Control" 💖
• Stream: Internet Research Task Force (IRTF) 🌸
• Category: Informational 📚
• ISSN: 2070-1721
• Date: February 2011 📅
• Authors: D. Papadimitriou, Ed. (Alcatel-Lucent), M. Welzl (University of Oslo), M. Scharf (University of Stuttgart), and B. Briscoe (BT & UCL) 👯‍♀️
• Obsoletes / Updates: the header doesn't list any, so nothing is obsoleted or updated 🤷‍♀️

---

Abstract ✨

okay so this doc is all about the open problems in Internet congestion control that people still don't have answers for 🙈 some of them are brand-new challenges that show up as the network keeps growing, and some have been annoying folks for years. these are considered open research topics, which means they might need more study or some innovative techniques before anyone can confidently engineer and deploy Internet-scale solutions 🔥

Status of This Memo 📝

this doc is NOT an Internet Standards Track specification. it's published for informational purposes 💅

it's a product of the Internet Research Task Force (IRTF), which publishes the results of Internet-related research and development. those results might not be suitable for deployment 🧪 this RFC represents the consensus of the Internet Congestion Control Research Group (ICCRG) of the IRTF 🫶 documents approved for publication by the IRSG are not a candidate for any level of Internet Standard, see Section 2 of RFC 5741 🌈

for info on the current status of this doc, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc6077 🔗

Copyright Notice ©️

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. please read these carefully, because they spell out your rights and restrictions for this doc 📖

Table of Contents 🗂️

   1. Introduction ....................................................3
   2. Global Challenges ...............................................5
      2.1. Heterogeneity ..............................................5
      2.2. Stability ..................................................7
      2.3. Fairness ...................................................8
   3. Detailed Challenges ............................................10
      3.1. Challenge 1: Network Support ..............................10
           3.1.1. Performance and Robustness .........................14
           3.1.2. Granularity of Network Component Functions .........15
           3.1.3. Information Acquisition ............................16
           3.1.4. Feedback Signaling .................................17
      3.2. Challenge 2: Corruption Loss ..............................17
      3.3. Challenge 3: Packet Size ..................................19
      3.4. Challenge 4: Flow Startup .................................24
      3.5. Challenge 5: Multi-Domain Congestion Control ..............26
           3.5.1. Multi-Domain Transport of Explicit
                  Congestion Notification ............................26
           3.5.2. Multi-Domain Exchange of Topology or
                  Explicit Rate Information ..........................27
           3.5.3. Multi-Domain Pseudowires ...........................28
      3.6. Challenge 6: Precedence for Elastic Traffic ...............30
      3.7. Challenge 7: Misbehaving Senders and Receivers ............31
      3.8. Other Challenges ..........................................33
           3.8.1. RTT Estimation .....................................33
           3.8.2. Malfunctioning Devices .............................35
           3.8.3. Dependence on RTT ..................................36
           3.8.4. Congestion Control in Multi-Layered Networks .......36
           3.8.5. Multipath End-to-End Congestion Control and
                  Traffic Engineering ................................37
           3.8.6. ALGs and Middleboxes ...............................37
   4. Security Considerations ........................................38
   5. References .....................................................39
      5.1. Informative References ....................................39
   6. Acknowledgments ................................................50
   7. Contributors ...................................................50

---

1. Introduction 🌷

this doc is the result of the Internet Congestion Control Research Group (ICCRG), and it describes some of the open research topics in Internet congestion control that are known today 💖 we'll start by looking at some proposed definitions of congestion and congestion control, based on what people currently understand 🧠

so congestion is a state or condition that happens when network resources get overloaded, and that causes impairments for network users, which you can measure objectively through the probability of loss and/or delay ⚠️ the overload reduces utility in networks that support both spatial and temporal multiplexing but have no reservation [Keshav07]. congestion control is a (typically distributed) algorithm that shares network resources among competing traffic sources 🤝

there are two components of distributed congestion control that were defined in the context of primal-dual modeling [Kelly98] 👑 primal congestion control is the algorithm the traffic sources run to control their sending rates or window sizes. it's normally closed-loop, meaning the operation depends on feedback, and TCP algorithms fall into this category 🔁 dual congestion control is implemented by the routers, which gather information about the traffic going through them. a dual algorithm updates, implicitly or explicitly, a congestion measure or congestion rate, and then sends it back, implicitly or explicitly, to the traffic sources that use that link 📡 queue management algorithms like Random Early Detection (RED) [Floyd93] or Random Exponential Marking (REM) [Ath01] are in the "dual" category 🎀

congestion control gives you a core set of mechanisms for keeping the Internet stable and efficient 🌈 it's been associated with TCP since Van Jacobson's work in 1988, but there's congestion control outside of TCP too, like for real-time multimedia applications, multicast, and router-based mechanisms [RFC5783] 🦋 the Van Jacobson end-to-end congestion control algorithms [Jacobson88] [RFC2581] [RFC5681] are used by the Internet transport protocol TCP [RFC4614]. they've been super successful for many years, but they're starting to hit their limits 😭 the heterogeneity of the data link and physical layers on one side, and of applications on the other, is pulling TCP congestion control past its natural operating regime. it performs poorly as bandwidth or delay increases 📉 a side effect is that a growing share of hosts use non-standardized congestion control enhancements. for example, many Linux distributions have shipped with "CUBIC" [Ha08] as the default TCP congestion control mechanism 🐧

the original Van Jacobson algorithm needs no congestion-related state in routers, but newer modifications have departed from a strict application of the end-to-end principle [Saltzer84] in order to avoid congestion collapse 🙈 Active Queue Management (AQM) in routers, like RED and variants such as Adaptive RED (ARED), improves performance by keeping queues small (implicit feedback via dropped packets) 🧹 Explicit Congestion Notification (ECN) [Floyd94] [RFC3168] passes one bit of congestion information back to senders when an AQM would otherwise drop a packet 💅 also worth noting: other RED-based variants are for quality enforcement, like Weighted RED (WRED) and RED with In/Out (RIO) [Clark98], while Stabilized RED (SRED) and CHOKe [Pan00] and its extension XCHOKe [Chhabra02] are flow policers 🚦 and [Bonald00] analytically evaluated RED performance 🧮

these measures do improve performance, but there's only so much you can get without more information from routers 😤 the requirement of extreme scalability together with robustness has been a tough hurdle for speeding up this information flow 🧗‍♀️ the stability and equilibrium properties of primal-dual TCP/AQM distributed algorithms have been studied extensively (cf. [Low02], [Low03.1], [Low03.2], [Kelly98], and [Kelly05]) 🤓

congestion control has lots of new challenges that are becoming important as the network grows, on top of the issues known for years 🌸 these are generally considered open research topics that may need more study or innovative techniques before Internet-scale solutions can be confidently engineered and deployed ✨ so next up, we'll give an overview of some of these challenges 🫶

2. Global Challenges 🌍

this section covers the global challenges that need to be addressed in the domain of Internet congestion control 💜

2.1. Heterogeneity ✨

okay so the Internet is literally a huge mix of IP networks 🌈 built with a zillion different technologies, which means the link and path characteristics are all over the place. capacity can be super scarce on very-slow-speed radio links (several kbps), or totally abundant on high-speed optical links (several gigabit per second). latency is just as wild, ranging from local interconnects (much less than a millisecond) to certain wireless and satellite links with very large latencies up to or over a second). even higher latencies can happen in space communication 🚀 so both the available bandwidth and the end-to-end delay in the Internet can vary over many orders of magnitude, and it's likely that the range of parameters will keep increasing in the future. the vibes are chaotic and I'm obsessed 💀

also, neither the available bandwidth nor the end-to-end delay stays constant 😭 at the IP layer, competing cross-traffic, traffic management in routers, and dynamic routing can cause sudden changes in the characteristics of an end-to-end path. link layer mechanisms add more drama too: shared-media access (e.g., in wireless networks), changes to new links due to mobility (horizontal/vertical handovers), topology modifications (e.g., in ad hoc or meshed networks), link layer error correction, and dynamic bandwidth provisioning schemes. so yeah, path characteristics can undergo substantial changes within short time frames. main character energy of the network, fr fr 🎀

congestion control algorithms have to handle all this variety in an efficient and stable way 🫶 the congestion control principles introduced by Van Jacobson kinda assume a rather static scenario, and they implicitly target configurations where the bandwidth-delay product is of the order of some dozens of packets at most. they've proved to work on the Internet for almost two decades, but much larger bandwidth-delay products and increased dynamics are challenging them more and more 😩 there are lots of situations where today's congestion control algorithms react in a suboptimal way, which results, among other things, in low resource utilization. not ideal, bestie 💅

so there's a whole multitude of new proposals for congestion control algorithms now 🦋 for instance, the Additive Increase Multiplicative Decrease (AIMD) behavior of TCP is too conservative in practical environments when the congestion window is large, so several high-speed congestion control extensions have been developed. BUT these new algorithms may be less robust or starve legacy flows in certain situations they weren't designed for 💀 and at the time of writing, there is no common agreement in the IETF on which algorithm(s) and protocol(s) to choose. the group chat is still debating 🙈

it's always possible to tune congestion control parameters based on some knowledge of the environment and the application scenario 💜 however, the interaction between multiple congestion control techniques isn't well understood yet. the big fundamental challenge is whether it's possible to define one congestion control mechanism that operates reasonably well across a whole range of scenarios that exist in the Internet. so the important research questions are how new Internet congestion control mechanisms would have to be designed, which maximum degree of dynamics they can efficiently handle, and whether they can keep the generality of the existing end-to-end solutions. the big brain questions ngl 🧠✨

some improvements to congestion control could come from simple changes to single functions in end-systems or optimizations of network components 🍓 but new mechanism(s) might also require a fundamental redesign of the overall network architecture, and they may even affect the design of Internet applications. that can imply significant interoperability and backward compatibility challenges and/or create network accessibility obstacles 😬 especially, networks and/or applications that don't use or support a new congestion control mechanism could get penalized with significantly worse performance compared to what they'd get if everybody used the existing mechanisms (cf. the discussion on fairness in Section 2.3). [RFC5033] defines several criteria to evaluate whether a new congestion control mechanism is appropriate 📋 but a key issue is how much performance deterioration is acceptable for "legacy" applications. this tradeoff between performance and cost has to be very carefully examined for all new congestion control schemes. it's giving careful, slow, serious energy 🫶

2.2. Stability 🌸

control theory is a mathematical tool for describing dynamic systems 🧚 it lends itself to modeling congestion control, since TCP is a perfect example of a typical "closed loop" system that can be described in control theoretic terms. however, control theory has had to be extended to model the interactions between multiple control loops in a network [Vinnic02]. in control theory there's a mathematically defined notion of system stability: in a stable system, for any bounded input over any amount of time, the output will also be bounded. for congestion control, what's actually meant by global stability is typically asymptotic stability, meaning a mechanism should converge to a certain state irrespective of the initial state of the network. local stability means that if the system gets perturbed from its stable state, it will quickly return toward the locally stable state. the math is giving precision and I respect it 🙌

some fundamental facts from control theory are useful as guidelines when designing a congestion control mechanism ✨ for instance, a controller should only be fed a system state that reflects its output. a (low-pass) filter function should be used so that only states expected to last long enough for its action to be meaningful get passed to the controller [Jain88]. action should be carried out whenever such feedback arrives, because it's a fundamental principle of control that the control frequency should ideally be equal to the feedback frequency. reacting faster leads to oscillations and instability, while reacting more slowly makes the system tardy [Jain90]. too fast is chaotic, too slow is sleepy, just right is iconic 👑

control theoretic modeling of a realistic network can be quite difficult, especially when you take distinct packet sizes and heterogeneous round-trip times (RTTs) into account 😵‍💫

so it has become common practice to model simpler cases and leave the more complicated (realistic) situations for simulations 🎀 clearly, if a mechanism isn't stable in a simple scenario, it's generally useless, so this method helps eliminate faulty congestion control candidates at an early stage. BUT a mechanism that's found to be stable in simulations still can't be safely deployed in real networks, since simulation scenarios make simplifying assumptions. the sim said yes, reality said "lol no" 💀

TCP stability can be attributed to two key aspects that were introduced in [Jacobson88]. first, there's the AIMD control law during congestion avoidance, which is based on a simple, vector-based analysis of two controllers sharing one resource with synchronous RTTs [Chiu89]. second, there's the "conservation of packets principle", which, once the control has reached "steady state", tries to maintain an equal amount of packets in flight at any time by only sending a packet into the network when a packet has left the network (as indicated by an ACK arriving at the sender). the latter aspect has guided many decisions about changes made to TCP over the years, so it's honestly iconic 🦋

the reasoning in [Jacobson88] assumes all senders are acting at the same time 👯‍♀️ the stability of TCP under more realistic network conditions has been investigated in a large number of follow-up works, but they've led to no clear conclusion that TCP would also be asymptotically stable under arbitrary network conditions 😬 on the other hand, research has concluded that stability can be assured with constraints on dynamics that are less stringent than the "conservation of packets principle". from control theory, only the rate increase (not the target rate) needs to be inversely proportional to RTT, whereas window-based control converges on a target rate that's inversely proportional to RTT. so a congestion control mechanism can converge on a rate that's independent of RTT, as long as its dynamics depend on RTT (e.g., FAST TCP [Jin04]). the plot twist of the century 🌈✨

in the stability analysis of TCP and of these more modern controls, the impact of slow-start on stability is not entirely clear 🙈 (which can be significant, since short-lived HTTP flows often never leave this phase). the mystery continues 💜

2.3. Fairness 🎀

Okay so the way the Internet community thinks about fairness has been getting called out super hard lately [Bri07] 😭. A big chunk of people basically mean "flows that go through the same congestion should get roughly equal rates," which is the vibe in TCP [RFC2581] [RFC5681] and TCP-Friendly Rate Control (TFRC) [RFC5348]. [RFC3714] literally names this situation "The Amorphous Problem of Fairness" 💀.

There's a parallel tradition built on [Kelly98] 🧠. As long as each user is accountable for the cost their rate causes to others [MacK95], the set of rates everyone ends up choosing counts as fair (cost fairness). Why? Because any other set of choices would make people lose more value than they gain overall.

Honestly, the debate between max-min, proportional, and TCP fairness is about mere details 🙄. All three share the assumption that equal flow rates are desirable, and they only differ on the second-order question of how to share out excess capacity in a network of many bottlenecks. Cost fairness, on the other hand, should lead to extremely unequal flow rates by design 🤯. Equivalently, equal flow rates would typically be considered extremely unfair. Main character energy for the cost side, ngl ✨.

The two traditional approaches aren't protocol options you can mix and match in different parts of an internetwork, bestie 💅. They lead to research agendas that have different objectives, which means a different set of open issues.

If we assume TCP-friendliness as a goal with flow rate as the metric, the open issues would be:

• Should flow fairness depend on the packet rate or the bit rate? 🤔
• Should the target flow rate depend on RTT (as in TCP) or should only flow dynamics depend on RTT (e.g., as in FAST TCP [Jin04])?
• How should we estimate whether a particular flow start strategy is fair, or whether a particular fast recovery strategy after a reduction in rate due to congestion is fair?
• Should we judge what is reasonably fair if an application needs, for example, even smoother flows than TFRC, or it needs to burst occasionally, or with any other application behavior?
• During brief congestion bursts (e.g., due to new flow arrivals), how should we judge at what point it becomes unfair for some flows to continue at a smooth rate while others reduce their rate?
• Which mechanism(s) could be used to enforce approximate flow rate fairness?
• Should we introduce some degree of fairness that takes into account different users' flow activity over time?
• How should we judge the fairness of applications using a large number of flows over separate paths (e.g., via an overlay)?

If we assume cost fairness as a goal with congestion-volume as the metric, the open issues would be:

• Can one application's sensitivity to instantaneous congestion really be protected by longer-term accountability of competing applications? 🥺
• Which protocol mechanism(s) are needed to give accountability for causing congestion?
• How might we design one or two weighted transport protocols (such as TCP, UDP, etc.) with the addition of application policy control over the weight?
• Which policy enforcement might be used by networks, and what are the interactions between application policy and network policy enforcement?
• How should we design a new policy enforcement framework that will appropriately compete with existing flows aiming for rate equality (e.g., TCP)?

---

Figuring out how to reason about fairness is a prerequisite for agreeing on the research agenda 📚. If the relevant metric is flow rate, the constraints show up at protocol design time. If the metric is congestion-volume, the constraints move to run-time, and design-time constraints can be relaxed [Bri08]. But here's the tea: that question doesn't need more research in itself 🍵. It's a debate that needs to be settled by studying existing research, by figuring out how bad fairness problems could get if we don't address them rigorously, and by checking whether we can rely on trust to keep approximate fairness without requiring policing complexity [RFC5290]. The latter points might lead to more research, but it's also accepted that more research won't necessarily convince either side to change their minds. More debate would be needed, periodt 💅.

It also seems that if the architecture is built to support cost fairness, then equal instantaneous cost rates for flows sharing a bottleneck result in flow-rate fairness. In other words, flow-rate fairness can be seen as a special case of cost fairness 💖. One can be used to build the other, but not vice-versa 🦋.

3. Detailed Challenges 🌈

3.1. Challenge 1: Network Support 🌸

okay so this one is giving most critical energy, bestie 😭💅 If the work gets split differently between the endpoints and the network equipment, the per-datagram data plane interface between the transport and network layers might need to change too. Network equipment vendors need to know any new interface is stable enough (we're talking decade timescales!!) to bake into firmware and hardware 🔥. Operating-system vendors won't touch a new interface unless it's likely to get widely deployed 🙈

Network components can help with congestion control in two ways ✨ First, they can implicitly optimize their functions, like queue management and scheduling strategies, so end-to-end congestion control can do its thing. Second, network components can join in through explicit signaling mechanisms. Explicit signaling, whether in-band or out-of-band, needs communication between network components and end-systems. Signals realized within or over the IP layer only make sense to network components that process IP packets. That always includes routers and potentially middleboxes too, but not pure link layer devices 🧚 The next section draws a clear line between "network component" and "router", and "router" is only used whenever processing IP packets is explicitly required. One big challenge with network-supported congestion control is that typically not every network component along a path is a router (cf. Section 3.1.3) 🫶

The first (optimizing) category of implicit mechanisms can live in any network component that processes and stores packets. Various approaches have been proposed and some are deployed, like different AQM techniques. Even though these implicit techniques are known to improve network performance during congestion phases, they're still only partly deployed in the Internet. This may be because finding optimal and robust parameterizations for these mechanisms is a non-trivial problem 💀. Honestly, the issue with various AQM schemes is figuring out the correct parameter values that affect how the queuing scheme performs (since the number of sources, the capacity, and the feedback delay all vary) [Firoiu00] [Hollot01] [Zhang03]. Many AQM schemes (RED, REM, BLUE, and PI-Controller, but also Adaptive Virtual Queue (AVQ)) don't define a systematic rule for setting their parameters 😭

The second class of approaches uses explicit signaling. With explicit feedback from the network, connection endpoints can get more accurate info about current network characteristics on the path 👀 That lets endpoints make more precise decisions that control congestion better 💖

Explicit feedback techniques fall into three broad categories:

• Explicit congestion feedback: one-bit Explicit Congestion Notification (ECN) [RFC3168] or proposals for more than one bit [Xia05];
• Explicit per-datagram rate feedback: the eXplicit Control Protocol (XCP) [Katabi02] [Falk07], or the Rate Control Protocol (RCP) [Dukki05];
• Explicit rate feedback: by means of in-band signaling, such as by Quick-Start [RFC4782], or by means of out-of-band signaling, e.g., Congestion Avoidance with Distributed Proportional Control/Performance Transparency Protocol (CADPC/PTP) [Welzl03].

Explicit router feedback can fix some of the inherent shortcomings of TCP. For instance, XCP was developed to overcome the inefficiency and instability TCP suffers from when the per-flow bandwidth-delay product increases. By decoupling resource utilization/congestion control from fairness control, XCP achieves equal bandwidth allocation, high utilization, a small standing queue size, and near-zero packet drops, with both steady and highly varying traffic 🦋. Importantly, XCP doesn't maintain any per-flow state in routers and needs few CPU cycles per packet, which makes it potentially applicable in high-speed routers 🚀. However, XCP is still subject to research: as [Andrew05] has pointed out, XCP is locally stable but globally unstable when the maximum RTT of a flow is much larger than the mean RTT. This instability can be removed by changing the update strategy for the estimation interval, but that makes the system vulnerable to erroneous RTT advertisements 🙃. The authors of [Pap02] have shown that when flows with different RTTs are applied, XCP sometimes discriminates among heterogeneous traffic flows, even though XCP generally equalizes rates among different flows. [Low05] provides a complete characterization of the XCP equilibrium properties 🤓.

Several other explicit router feedback schemes have been developed with different design objectives. For instance, RCP uses per-packet feedback similar to XCP. But unlike XCP, RCP focuses on reducing flow completion times [Dukki06], taking an optimistic approach to flows likely to arrive in the next RTT and tolerating larger instantaneous queue sizes [Dukki05]. XCP, on the other hand, gives very poor flow completion times for short flows 😬.

Both implicit and explicit router support should be considered in the context of the end-to-end argument [Saltzer84], which is one of the key design principles of the Internet 👑. It suggests that functions that can be realized both in the end-systems and in the network should be implemented in the end-systems. This principle makes sure the network provides a general service and stays as simple as possible (any extra complexity goes above the IP layer, i.e., at the edges) so it stays evolvable, reliable, and robust. Furthermore, the fate-sharing principle ([Clark88], "Design Philosophy of the DARPA Internet Protocols") says an end-to-end Internet protocol design should not rely on maintaining any per-flow state (i.e., information about the state of the end-to-end communication) inside the network, and that the network state (e.g., routing state) maintained by the Internet should minimize its interaction with the states maintained at the endpoints/hosts [RFC1958] 🧠.

However, as discussed in [Moors02] for instance, congestion control can't be a pure end-to-end function only. Congestion is an inherent network phenomenon and can only be resolved efficiently by some cooperation between end-systems and the network 🤝. Congestion control in today's Internet protocols follows the end-to-end design principle insofar as only minimal feedback from the network is used, e.g., packet loss and delay. The end-systems just decide how to react and how to avoid congestion. The crux is that, on one hand, there would be substantial benefit from more help from the network, but, on the other hand, that network support could lead to duplication of functions, which might even harmfully interact with end-to-end protocol mechanisms 😵‍💫. The different requirements of applications (cf. the fairness discussion in Section 2.3) call for a variety of different congestion control approaches, but putting that per-flow behavior inside the network should be avoided, since such a design would clearly be at odds with the end-to-end and fate-sharing design principles 🚫.

The end-to-end and fate-sharing principles are generally regarded as the key ingredients for a scalable and survivable network design. To make sure new congestion control mechanisms are scalable, violating these principles should therefore be avoided 🌈.

For instance, protocols like XCP and RCP seem not to require flow state in the network, but that's only the case if the network trusts i) the receiver not to lie when feeding back the network's delta to the requested rate; ii) the source not to lie when declaring its rate; and iii) the source not to cheat when setting its rate in response to the feedback [Katabi04]. Yikes, trust issues 😳.

Solving these problems for non-cooperative environments like the public Internet requires flow state, at least on a sampled basis. However, because flows can create new identifiers whenever they want, sampling doesn't deter anything. A flow can simply cheat until it gets discovered, then switch to a whitewashed identifier [Feldman04], and keep cheating until it's discovered again ([Bri09], S7.3) 🕵️‍♀️.

However, holding flow state in the network only seems to solve these policing problems in single autonomous system settings. A multi-domain system would seem to need a completely different protocol structure, since the information required for policing is only seen as packets leave the internetwork, but the networks where packets enter will also want to police compliance 🌍.

Even if a new protocol structure were found, it seems unlikely that network flow state could be avoided, given the network's per-packet flow rate instructions would need to be compared against variations in the actual flow rate, which is inherently not a per-packet metric 📉. These issues have been outstanding ever since integrated services (IntServ) was identified as unscalable in 1997 [RFC2208]. All subsequent attempts to involve network elements in limiting flow rates (XCP, RCP, etc.) will run into the same open issue if anyone tries to standardize them for use on the public Internet 🥲.

In general, network support of congestion control raises many issues that haven't been completely solved yet 💜.

---

3.1.1. Performance and Robustness 🎀

Congestion control comes with some tradeoffs: on one hand, it must allow high link utilizations and fair resource sharing, but on the other hand, the algorithms must also be robust ⚖️.

Router support can help improve performance, but it can also add complexity and more control loops. That requires careful algorithm design to ensure stability and avoid, e.g., oscillations. Another challenge is that feedback information may be imprecise. For instance, severe congestion can delay feedback signals. Also, in-network measurement of parameters such as RTTs or data rates may contain estimation errors. Even though there has been significant progress in providing fundamental theoretical models for these effects, research hasn't completely explored the whole problem space yet 🔍.

Open questions are:

• How much can network elements theoretically improve performance in the complete range of communication scenarios that exist in the Internet without damaging or impacting end-to-end mechanisms already in place? 🤔
• Is it possible to design robust congestion control mechanisms that offer significant benefits with minimum additional risks, even if Internet traffic patterns change in the future? 🔮
• What is the minimum support needed from the network in order to achieve significantly better performance than with end-to-end mechanisms, given the current IP header limitations that provide at most unary ECN signals? 🧐

3.1.2. Granularity of Network Component Functions ✨

okay so there are a bunch of ways network entities can get involved, and they run all the way from a few extra steps in network management 🧚 on one end to extra per-packet processing on the other end of the solution space 💅. routers can also hold different amounts of state: no per-flow state, partial per-flow state, soft state, or hard state 🎀. ngl that extra router work is a real challenge for Internet scalability, and it could also push up end-to-end latencies 😭.

there are tons of research proposals that don't need per-flow state, so they don't cause big processing overhead 🌸. BUT there are no known full solutions (like, including anti-cheating 🙈) that don't need per-flow processing. scalability can also get messy because of synchronization mechanisms for state info between parallel processing entities. that shows up in high-speed router hardware designs, e.g., 🔥

open questions are:

• what granularity of router processing can be realized without affecting Internet scalability? 🦋
• how can additional processing efforts be kept to a minimum? 💜

3.1.3. Information Acquisition 🍓

so, to support congestion control, network components have to get at least a subset of the following info. and getting that info can turn into complex tasks 💀

1. Capacity of (outgoing) links 🌈

link characteristics depend on how the lower protocol layers are realized. routers working at the IP layer don't necessarily know the link layer network topology and link capacities, and those aren't always constant (e.g., on shared wireless links or bandwidth-on-demand links) 🥺. depending on the network technology, there can be queues or bottlenecks that aren't directly visible at the IP networking layer 👀.

it gets harder with IP-in-IP tunnels [RFC2003], IPsec tunnels [RFC4301], IP encapsulated in the Layer Two Tunneling Protocol (L2TP) [RFC2661], Generic Routing Encapsulation (GRE) [RFC1701] [RFC2784], the Point-to-Point Tunneling Protocol (PPTP) [RFC2637], or Multiprotocol Label Switching (MPLS) [RFC3031] [RFC3032] 🫠. in these cases, link info could be determined by cross-layer information exchange, but that needs interfaces that can process link layer technology specific information 🔧. an alternative could be online measurements, but this can cause significant additional network overhead 😩. it's an open research question as to how much, if any, online traffic measurement would be acceptable (at run-time) 🤔. encapsulation and decapsulation of explicit congestion information have been specified for IP-in-IP tunnelling [RFC6040] and for MPLS-in-MPLS or MPLS-in-IP [RFC5129] ✨.

2. Traffic carried over (outgoing) links 📊

accurate online measurement of data rates is challenging when traffic is bursty 💥. for instance, measuring a "current link load" requires defining the right measurement interval / sampling interval. this is a challenge for proposals that need knowledge about, e.g., the current link utilization 🫨.

3. Internal buffer statistics 🪣

some proposals use buffer statistics, like a virtual queue length, to trigger feedback. however, network components can include multiple distributed buffer stages, and that makes it hard to obtain such metrics 🙃.

open questions are:

• can and should this information be made available, e.g., by additional interfaces or protocols? 💌
• which information is so important to higher-layer controllers that machine architecture research should focus on designing to provide it? 🧠

3.1.4. Feedback Signaling 📣

explicit notification mechanisms can be realized either by in-band signaling (notifications piggybacked along with the data traffic) or by out-of-band signaling [Sarola07]. the latter case needs additional protocols and a secure binding between the signals and the packets they refer to 🔐. out-of-band signaling can be further subdivided into path-coupled and path-decoupled approaches 🌿.

open questions concerning feedback signaling include:

• at which protocol layer should the feedback signaling occur (IP/network layer assisted, transport layer assisted, hybrid solutions, shim layer, intermediate sub-layer, etc.)? should the feedback signaling be path-coupled or path-decoupled? 🎀
• what is the optimal frequency of feedback (only in case of congestion events, per RTT, per packet, etc.)? ⏱️
• what direction should feedback take (from network resource via receiver to sender, or directly back to sender)? 🔄

3.2. Challenge 2: Corruption Loss 🌧️

it's super common for congestion control mechanisms to read packet loss as a sign of congestion. that's totally fair when packets get dropped in routers because a queue overflowed, but there are other reasons packets get dropped too 👀. especially in wireless networks, packets can be dropped because of corruption loss, which makes the typical congestion control reaction inappropriate 🙅‍♀️. so non-congestive loss may be more prevalent in these networks due to corruption loss (when the wireless link can't be conditioned to properly control its error rate, or due to transient wireless link interruption in areas of poor coverage) 📶.

TCP over wireless and satellite has been investigated for a long time [Krishnan04] 🛰️. some proposals have the congestion control mechanism react as if a packet had not been dropped in the presence of corruption (cf. TCP HACK [Balan01]). but discussions in the IETF have shown that there's no agreement on whether that reaction is appropriate 🤷‍♀️. see, for instance, the discussion from April 2003 on the Datagram Congestion Control Protocol (DCCP) working group list http://www.ietf.org/mail-archive/web/dccp/current/mail6.html 📬. it's been said that congestion can show up as corruption on shared wireless links, and it's questionable whether a source sending packets that keep getting messed up by link noise should keep sending at a high rate, because it has lost the integrity of the feedback loop 🫣.

generally, two questions must be addressed when designing a congestion control mechanism that takes corruption loss into account:

1. How is corruption detected? 🔍
2. What should be the reaction? 🎬

in addition to question 1 above, it may be useful to consider detecting the reason for corruption, but this hasn't been done yet, to the best of our knowledge 🧐.

corruption detection can be done using an in-band or out-of-band signaling mechanism, much like what's described for Challenge 1 🪄. there's also implicit detection to consider: link layers sometimes retransmit erroneous frames, which can make the end-to-end delay go up. but from the perspective of a sender at the transport layer, there are lots of other possible reasons for that effect 🌀.

header checksums are another implicit detection possibility. if a checksum only covers all the necessary header fields and this checksum shows no error, it's possible for errors to be found in the payload using a second checksum 🧩. that kind of error detection is possible with UDP-Lite and DCCP. it worked well over a General Packet Radio Service (GPRS) network in one study [Chester04], and poorly over a WiFi network in another study [Rossi06] [Welzl08] 📡. note that while UDP-Lite and DCCP enable the detection of corruption, the specifications of these protocols don't foresee any specific reaction to it for the time being 🫶.

having a transport endpoint detect corruption and react (or not) to it raises a bunch of interesting questions about cross-layer interactions 🕸️. IP is designed to operate over arbitrary link layers, so it's difficult to design a congestion control mechanism on top of it that appropriately reacts to corruption. that's especially true because the specific data link layers in use along an end-to-end path are typically unknown to entities at the transport layer 🫠.

while the IETF hasn't yet specified how a congestion control mechanism should react to corruption, proposals exist in the literature, e.g., [Tickoo05] 📚. for instance, TCP Westwood [Mascolo01] sets the congestion window equal to the measured bandwidth at the time of congestion, in response to three DupACKs or a timeout. this measurement is obtained by counting and filtering the ACK rate 📈. this setting gives a significant goodput improvement in noisy channels, because the "blind" by half window reduction of standard TCP is avoided, i.e., the window isn't reduced by too much 🙌.

open questions concerning corruption loss include:

• how should corruption loss be detected? 🕵️‍♀️
• how should a source react when it is known that corruption has occurred? 🤔
• can an ECN-capable flow infer that loss must be due to corruption just from lack of explicit congestion notifications around a loss episode [Tickoo05]? or could this inference be dangerous, given the transport doesn't know whether all queues on the path are ECN-capable or not? ⚠️

3.3. Challenge 3: Packet Size 📏

TCP doesn't take packet size into account when responding to losses or ECN 🙄. over the past years, the performance of TCP congestion avoidance algorithms has been studied a lot 📖. the well-known "square root formula" gives an estimate of the performance of the TCP congestion avoidance algorithm for TCP Reno [RFC2581]. [Padhye98] enhances the model to account for timeouts, receiver window, and delayed ACKs 💅.

for the sake of this discussion, we'll assume that the TCP throughput is expressed using the simplified formula. with this formula, the TCP throughput B is proportional to the segment size and inversely proportional to the RTT and the square root of the drop probability:

               S     1
        B ~ C --- -------
              RTT sqrt(p)

where

    C     is a constant
    S     is the TCP segment size (in bytes)
    RTT   is the end-to-end round-trip time of the TCP
          connection (in seconds)
    p     is the packet drop probability

neglecting the fact that the TCP rate depends linearly on it, choosing the ideal packet size is a tradeoff between high throughput (the larger a packet, the smaller the relative header overhead) and low packet latency (the smaller a packet, the shorter the time needed until it's filled with data) 🧁. observing that TCP isn't optimal for applications with streaming media (since reliable in-order delivery and congestion control can cause arbitrarily long delays), this tradeoff hasn't usually been considered for TCP applications 🎬. so the influence of packet size on the sending rate hasn't typically been seen as a significant issue, given there are still few paths through the Internet that support packets larger than the 1500 bytes common with Ethernet 🌐.

the situation is already different for the Datagram Congestion Control Protocol (DCCP) [RFC4340] 🚨, which was designed to enable unreliable but congestion-controlled datagram transmission, avoiding the arbitrary delays associated with TCP. DCCP is intended for applications such as streaming media that can benefit from control over the tradeoffs between delay and reliable in-order delivery 💖.

DCCP has a whole menu of modular congestion control mechanisms 🍓, and you pick yours with Congestion Control Identifiers (CCIDs) ✨. Three profiles are currently specified, and each one has its own energy:

• DCCP Congestion Control ID 2 (CCID 2) [RFC4341]: TCP-like Congestion Control 💅. CCID 2 sends data with a close approximation of TCP's congestion control, and it also brings in a variant of Selective Acknowledgment (SACK) [RFC2018] [RFC3517]. It's a great fit for senders that can handle the sudden swings in congestion window that come with TCP's AIMD congestion control, and it's especially useful for senders who want to grab the available bandwidth when conditions are changing super fast 🔥.
• DCCP Congestion Control ID 3 (CCID 3) [RFC4342]: TCP-Friendly Rate Control (TFRC) [RFC5348] is a congestion control mechanism designed for unicast flows running in a best-effort Internet environment 🌸. When it's competing for bandwidth, its window looks a lot like TCP flows, but its throughput varies way less over time than TCP's does. That makes it perfect for things like streaming media, where a relatively smooth sending rate is really important 🎀. CCID 3 is the move for flows that want to minimize abrupt changes in the sending rate, including streaming media apps with small or moderate receiver buffering before playback ✨.
• DCCP Congestion Control ID 4 (CCID 4) [RFC5622]: TFRC Small Packets (TFRC-SP) [RFC4828] is a variant of the TFRC mechanism, and it was built for apps that exchange small packets 🧚. Its goal is to get the same bandwidth in bits per second as a TCP flow using packets of up to 1500 bytes. TFRC-SP enforces a minimum interval of 10 ms between data packets, so one flow can't send small packets arbitrarily often 🙈. CCID 4 was designed to be used either by apps with a small fixed segment size, or by apps that change their sending rate by varying the segment size 💜. Because CCID 4 is meant for apps that use a fixed small segment size, or that vary their segment size in response to congestion, the transmit rate derived from the TCP throughput equation gets reduced by a factor that accounts for the packet header size, as specified in [RFC4828] 🫶.

Okay, so the resulting open questions are:

• How does TFRC-SP operate under various network conditions? 🤔
• How can congestion control be designed so that it scales with packet size (the dependency of the congestion algorithm on packet size)? 👑

Today, lots of network resources are built so that packet processing can't get overloaded, even at incoming loads running at the maximum bit rate of the line 💖. If packet processing can handle a sustained load of r [packet per second], and the minimum packet size is h [bit] (meaning frame, packet, and transport headers with no payload), then a line rate of x [bit per second] will never be able to overload packet processing as long as x =< r*h.

Realistic equipment, though, is often only designed to cope with a near-worst-case workload with a few larger packets in the mix, rather than the true worst case of all minimum-size packets 😭. In that case, x = r*(h + e) for some small value of e. So packet congestion isn't impossible for runs of small packets (think TCP ACKs, or denial-of-service (DoS) attacks with TCP SYNs or small UDP datagrams) 💀. But if we set those weird anomalous workloads aside, equipment vendors at the 2008 ICCRG meeting believed that equipment could still be designed so that any congestion would be due to bit overload and not packet overload ✨.

This observation brings up some extra open issues:

• Can bit congestion remain prevalent?

Being able to assume that congestion is generally due to excess bits and not excess packets is a super useful simplifying assumption when designing congestion control protocols 🌈. Can we keep relying on it in the future? A different take is that in-network processing will become commonplace, so per-packet processing will be just as likely to be the bottleneck as per-bit transmission [Shin08] 🤯.

Over the last three decades, performance gains have mostly come from increased packet rates, not bigger packets 📈. But if bigger maximum segment sizes do become more prevalent, tiny segments (e.g., ACKs) won't stop being widely used, and that leads to a widening range of packet sizes 🍰.

So the open question is whether packet processing rates (r) will keep up with growth in transmission rates (x). A superficial look at Moore's Law-type trends would suggest that processing (r) will keep outstripping growth in transmission (x). But predictions based on actual knowledge of technology futures would be way more useful 🔮. Another open question is whether there are likely to be more small packets in the average packet mix. If the answers to either of these questions predict that packet congestion could become prevalent, congestion control protocols will have to be more complicated 😬.

• Confusable causes of loss

There's a considerable body of research on how to tell whether packet drops are due to transmission corruption or to congestion 🕵️‍♀️. But the full list of confusable causes of loss is longer, and it includes transmission corruption loss, congestion loss (bit congestion and packet congestion), and policing loss 🙃.

If congestion is due to excess bits, the bit rate should be reduced. If congestion is due to excess packets, the packet rate can be reduced without reducing the bit rate, by using larger packets 🎀. However, if the transport can't tell which of these causes led to a specific packet drop, its only safe response is to reduce the bit rate. That's why the Internet would be more complicated if packet congestion were prevalent: reducing the bit rate normally also reduces the packet rate, while reducing the packet rate doesn't necessarily reduce the bit rate 💅.

Given that distinguishing corruption loss from congestion is already an open issue (Section 3.2), if that problem ever gets solved, a further open issue would be whether to standardize a solution that distinguishes all the above causes of loss, and not just two of them 🧩.

Still, even if we find a way for network equipment to explicitly distinguish which sort of loss has occurred, we'll never be able to assume that such a smart AQM solution is deployed at every congestible resource throughout the Internet. That includes every higher-layer device like firewalls, proxies, and servers, and every lower-layer device like low-end hubs, DSLAMs, Wireless LAN (WLAN) cards, cellular base-stations, and so on 🥺. So transport protocols will always have to cope with packet drops from unpredictable causes, and we should always treat AQM as an optimization, since it will never be ubiquitous across the public Internet 🌍.

• What does a congestion notification on a packet of a certain size mean?

The open issue here is whether a loss or explicit congestion mark should be interpreted as a single congestion event no matter the size of the packet lost or marked, or whether the strength of the congestion notification should be weighted by the packet's size 🧐. This issue gets discussed at length in [Bri10], along with other aspects of packet size and congestion control 📚.

[Bri10] makes the strong recommendation that network equipment should drop or mark packets with a probability independent of each specific packet's size, while congestion controls should respond to dropped or marked packets in proportion to the packet's size ⭐.

• Packet size and congestion control protocol design

If that recommendation is correct (that the packet size of a congestion notification should be taken into account when the transport reads the notification, and not when the network writes it), it opens up a significant problem of protocol engineering and re-engineering 🛠️. Honestly, TCP doesn't take packet size into account when responding to losses or ECN. Right now that isn't a pressing problem, because 1500 byte data segments are super prevalent for TCP, and alternative maximum segment sizes aren't common. But we should design the Internet's protocols so they will scale with packet size 🌈. So an open issue is whether we should evolve TCP to be sensitive to packet size, or expect new protocols to take over 🦋.

As we keep standardizing new congestion control protocols, we'll have to face the question of how they should account for packet size 💭. It's still an open research issue to establish whether TCP was correct in not taking packet size into account. If it's determined that TCP was wrong about this, we should discourage future protocol designs from following TCP's example ⚖️. For example, as explained above, the small-packet variant of TCP-friendly rate control (TFRC-SP [RFC4828]) is an experimental protocol that aims to take packet size into account. Whatever packet size it uses, it makes sure its rate approximately equals that of a TCP using 1500 byte segments 🎯. That raises a further question: will TCP with 1500 byte segments be a suitable long-term gold standard, or do we need a more thorough review of what it means for a congestion control mechanism to scale with packet size? 👑✨

3.4. Challenge 4: Flow Startup 🚀

okay so the very beginning of a data transfer is its own special kind of chaos 😭 when a connection to a brand new destination gets set up, the end-systems basically know nothing about the path between them or how much bandwidth is actually available. there's no obvious move for how to start sending, and it's giving "walking into a party where you don't know anyone" 💀 a similar vibe happens after relatively long idle times too, because the congestion control state stops matching what the network is actually doing now. that's called the flow restart problem 🔄

Van Jacobson [Jacobson88] suggested using the slow-start mechanism for both flow startup and flow restart, and this is today's standard solution [RFC2581] [RFC5681] ✨ per [RFC5681], slow-start runs when the congestion window (cwnd) < slow-start threshold (ssthresh). the ssthresh starts out arbitrarily high (e.g., set to the size of the largest possible advertised window) and gets reduced when congestion shows up 📉 during slow-start, TCP bumps the cwnd by at most Sender Maximum Segment Size (MSS) bytes for each ACK that cumulatively acknowledges new data. slow-start ends when cwnd exceeds ssthresh or when congestion is observed.

BUT, bestie, slow-start is not optimal in a lot of situations 😤 first, it can take forever for a sender to fully use the available bandwidth on a path. second, that exponential growth can be too aggressive and cause multiple packet losses once really big congestion windows are reached (slow-start overshooting 💥). and finally, slow-start doesn't guarantee that new flows quickly converge to a reasonable share of resources, especially when they're competing with long-lived flows and leave slow-start early (slow-start vs overshoot tradeoff 🥲). this convergence problem could even get worse if more aggressive congestion control variants become widely used 😬

the slow-start and how it interacts with the congestion avoidance phase were mostly designed by intuition [Jacobson88] 🧠 so far there's little theory built up to understand the flow startup problem and how it affects congestion control stability and fairness. there's also no established methodology for evaluating whether new flow startup mechanisms are appropriate or not 🙈

so yeah, fixing the shortcomings of slow-start is a non-trivial task 🫠 several experimental enhancements have been proposed, like congestion window validation [RFC2861] and limited slow-start [RFC3742]. there's also ongoing research on things like bandwidth estimation techniques, delay-based congestion control, and rate-pacing mechanisms ✨ but any alternative end-to-end flow startup approach has to deal with the inherent problem that there's no information (or only a little) about the path at the start of a transfer. this uncertainty could be reduced by more expressive feedback signaling (cf. Section 3.1) 🎀 for instance, a source could learn path characteristics faster with the Quick-Start mechanism [RFC4782] ⚡ but even if the source knew exactly what rate it should aim for, it still wouldn't necessarily be safe to jump straight to that rate 😳 the end-system still doesn't know how changing its own rate will affect the path, and the path might also become congested in less than one RTT. more research would be useful to understand the effect of reducing uncertainty through explicit feedback, kept separate from control theoretic stability questions 🔬 flow startup also raises fairness questions too. for instance, it's unclear whether it could be reasonable to use a faster startup when an end-system detects that a path isn't currently congested 🤔

in summary, there are several topics for further research on flow startup 💖

• Better theoretical understanding of the design and evaluation of flow startup mechanisms, concerning their impact on congestion risk, stability, and fairness. 🧁
• Evaluating whether it may be appropriate to allow alternative starting schemes, e.g., to allow higher initial rates under certain constraints [Chu10]; this also requires refining the definition of fairness for startup situations. 🌸
• Better theoretical models for the effects of decreasing uncertainty by additional network feedback, particularly if the path characteristics are very dynamic. 🦋

---

3.5. Challenge 5: Multi-Domain Congestion Control 🌍

transport protocols like TCP operate over the Internet, which is split up into autonomous systems. these systems are really heterogeneous, because IP networks are built out of a huge mix of technologies 🧩

3.5.1. Multi-Domain Transport of Explicit Congestion Notification

different conditions and their variations lead to correlation effects between policers that regulate traffic against certain conformance criteria 🎀

with the rise of techniques that detect congestion early, packet loss isn't the only metric of congestion anymore ✨ ECN (Explicit Congestion Notification) marks packets (set by active queue management techniques) to convey congestion information, trying to prevent packet losses (packet loss and the number of marked packets give an indication of the level of congestion) 📊 using TCP ACKs to feed that information back lets hosts realign their transmission rate, which encourages them to use the network efficiently 🫶 in IP, ECN uses the two least significant bits of the (former) IPv4 Type of Service (TOS) octet or the (former) IPv6 Traffic Class octet [RFC2474] [RFC3260]. and ECN in TCP uses two bits in the TCP header that were previously defined as reserved [RFC793] 🌈

ECN [RFC3168] is an example of a congestion feedback mechanism going from the network toward hosts. the congestion-based feedback scheme, however, has limitations when it's applied across domains 😬 honestly, Sections 8 and 19 of [RFC3168] go into how two possible attacks would play out:

1. non-compliance: a network erasing a Congestion Experienced (CE) codepoint that was introduced earlier on the path, and
2. subversion: a network changing Not ECN-Capable Transport (Not-ECT) to ECT.

both of these problems could let an attacking network cause excess congestion in an upstream network, even if the transports were behaving correctly 💀 there are, to date, two possible solutions to the non-compliance problem (number 1 above): the ECN-nonce [RFC3540] and the [CONEX] work item, which was inspired by the re-ECN incentive system [Bri09] 🔐 still, accidental (rather than malicious) erasure of ECN is an issue for IPv6, because the absence of an IPv6 header checksum means corruption of ECN could be more impactful than in the IPv4 case 🙈

fragmentation is another issue! the ECN-nonce can't protect against misbehaving receivers that hide marked fragments, so some protection is lost in situations where path MTU discovery is disabled 😭 also, the ECN-nonce wouldn't protect against the subversion issue (number 2 above), because by definition a Not-ECT packet comes from a source without ECN enabled, and therefore without the ECN-nonce enabled either. so there's still room for improvement on the ECN mechanism when it's operating in multi-domain networks 🛠️

operational and deployment experience is still needed to figure out how big these problems actually are 📚 the second problem is mainly tied to deployment and usage practices, and it doesn't seem to create any specific research challenge 🌷

another controversial solution in a multi-domain environment might be the TCP rate controller (TRC), a traffic conditioner that regulates the TCP flow at the ingress node in each domain by controlling packet drops and delays of the packets in a flow 🎢 the outgoing traffic from a TRC-controlled domain is shaped so that no packets get dropped at the policer ✨ however, the TRC interferes with the end-to-end TCP model, so it would interfere with the past and future diversity of TCP implementations (violating the end-to-end principle) 🚫 in particular, the TRC bakes in the flow rate equality view of fairness, and would prevent evolution toward forms of fairness based on congestion-volume (Section 2.3) 💅

3.5.2. Multi-Domain Exchange of Topology or Explicit Rate Information

security is a big challenge for multi-domain exchange of explicit rate signals, whether they're in-band or out-of-band 🔒 at domain boundaries, authentication and authorization issues can pop up whenever congestion control information gets exchanged. from this perspective, the Internet doesn't have any security architecture for this problem yet 😱

the future evolution of Internet inter-domain operation will show whether more multi-domain information exchange can be realized effectively 🔮 this really matters for congestion control schemes that use explicit per-datagram rate feedback (e.g., RCP or XCP), explicit rate feedback using in-band congestion signaling (e.g., Quick-Start), or out-of-band signaling (e.g., CADPC/PTP) 🛰️ explicit signaling exchanges at the inter-domain level that trigger local domain actions are currently absent from the Internet 🫥 from this perspective, security issues caused by limited trust between different administrative units lead to policy enforcement that makes it even harder to exchange explicit feedback congestion control information between domains 🧱 note that even though authentication mechanisms could be extended for this purpose (by recognizing that explicit rate schemes such as RCP or XCP have the same inter-domain security requirements and structure as IntServ), they suffer from the same scalability problems identified in [RFC2208] 📉 in-band rate signaling, or out-of-band per-flow traffic specification signaling (like in the Resource Reservation Protocol (RSVP)), leads to similar scalability issues (see Section 3.1) 🌀

also, many autonomous systems only exchange a limited amount of information about their internal state (topology hiding principle) 🤐 even though having more precise information could be super beneficial for congestion control 🙌 revealing the internal network structure is highly sensitive in multi-domain network operations, so it's also a concern when it comes to whether congestion control schemes can actually be deployed 🫣 for instance, a network-assisted congestion control scheme with explicit signaling could reveal more about the internal network dimensioning than TCP does today 👀✨

3.5.3. Multi-Domain Pseudowires ✨

okay so pseudowires (PWs) [RFC3985] are the things that let you carry non-TCP data flows, like Time-Division Multiplexing (TDM) traffic or Constant Bit Rate (CBR) ATM traffic, across a multi-domain IP network 🧚. and this is where it gets messy bestie 😭. Structure-Agnostic TDM over Packet (SAToP) [RFC4553], Circuit Emulation Service over Packet Switched Network (CESoPSN) [RFC5086], and TDM over IP (TDMoIP) [RFC5087] are not responsive to congestion control as discussed in [RFC2914] (see also [RFC5033]). the same vibe applies to ATM circuit emulating services (CESs) connecting CBR equipment, like Private Branch Exchanges (PBX), across a Packet Switched Network (PSN) 📞💀

it's not giving "just slow down" 🙅‍♀️. you can't simply reduce the flow rate of a TDM PW or an ATM PW when packet loss shows up. providers can rate-control the incoming traffic, but they may not be able to tell that the PWs are carrying TDM or CBR ATM traffic at all (mechanisms for characterizing the traffic's temporal properties may not necessarily be supported) 🫠

here's a picture so you can see what I mean 🎀

            ...........       ............
           .           .     .
    S1 --- E1 ---      .     .
           .     |     .     .
           .      === E5 === E7 ---
           .     |     .     .     |
    S2 --- E2 ---      .     .     |
           .           .     .     |      |
            ...........      .     |      v
   .                                    ----- R --->
            ...........      .     |      ^
           .           .     .     |      |
    S3 --- E3 ---      .     .     |
           .     |     .     .     |
           .      === E6 === E8 ---
           .     |     .     .
    S4 --- E4 ---      .     .
           .           .     .
            ...........       ............

           \---- P1 ---/     \---------- P2 -----

so sources S1, S2, S3, and S4 are originating TDM over IP traffic 📡. P1 provider edges E1, E2, E3, and E4 are rate-limiting that traffic 🚦. the Service Level Agreement (SLA) of provider P1 with transit provider P2 says the latter assumes a BE traffic pattern, and that the distribution looks like typical BE traffic (elastic, non-real time, non-interactive) 💅

the problem is that transit provider P2 can't tell that its IP packets are carrying constant-bit-rate service traffic, for which the only useful congestion control would rely on implicit or explicit admission control, meaning self-blocking or enforced blocking, respectively 🙈

now, if P1 is rate-limiting BE traffic, a transit P2 router R can get seriously congested, because all the TDM PWs cross the same router 🔥. TCP-friendly traffic (like each flow inside another PW) follows TCP's AIMD algorithm, which means it halves its sending rate for every packet drop 📉. but the TDM PWs could grab all the available capacity while the more TCP-friendly, or generally congestion-responsive, traffic shrinks to nothing 😤. and this can just happen because S4 suddenly switches on extra TDM channels 🧃

it's neither possible nor desirable to assume edge routers will soon be able to detect whether the carried traffic is responsive, buuut it's still important for transit providers to be able to police a fair, robust, responsive, and efficient congestion control technique so they don't mess with congestion-responsive Internet traffic 🫶. however, we must not require only certain specific responses to congestion to be embedded within the network, because that would harm evolvability 🦋. so designing the corresponding mechanisms in the data and control planes still needs more investigation 🔍

3.6. Challenge 6: Precedence for Elastic Traffic 👑

traffic started by so-called elastic applications adapts to the available bandwidth using feedback about the state of the network 🌸

for elastic apps, the transport keeps adjusting the data sending rate to different network conditions 🎢. examples include short-lived elastic traffic like HTTP and instant-messaging, plus long file transfers with FTP and the applications targeted by [LEDBAT] 📂. honestly, elastic data apps can have super different requirements and traffic characteristics 💖

the idea of sorting best-effort traffic into several classes is pretty old, because it would be useful to handle the different delay sensitivities of different elastic apps 🕰️. the notion of traffic precedence was already introduced in [RFC791], and it was broadly defined as "An independent measure of the importance of this datagram" 💅. for example, lower-precedence traffic should get lower average throughput than higher-precedence traffic 📉. but a few questions pop up here: what does "relative" even mean? and what's the role of the transport layer? 🤔

giving higher-precedence traffic preferential treatment, combined with the right congestion control mechanisms, is still an open issue 🚧. depending on the solution, it may affect both the host and the network's precedence awareness, and therefore congestion control 🧠. [RFC2990] points out that the interactions between congestion control and DiffServ [RFC2475] went unaddressed until recently 🙃

recently, a study and a potential solution were proposed that introduce Guaranteed TFRC (gTFRC) [Lochin06] ✨. gTFRC is an adaptation of TCP-Friendly Rate Control that provides throughput guarantees for unicast flows over the DiffServ/Assured Forwarding (AF) class 🎀. the point of gTFRC is to separate the guaranteed part from the best-effort part of the traffic that results from AF conditioning 🧩. this proposed congestion control has been specified and tested inside DCCP/CCID 3 for DiffServ/AF networks [Lochin07] [Jourjon08] 🧪

but there's still work to do for lower-precedence traffic 🥺. these are data transfers that are useful, yet not important enough to justify significantly impairing other traffic 🍓. examples of apps that could use this include web caches and web browsers (e.g., for pre-fetching) as well as peer-to-peer apps 🌈. there are proposals for achieving low precedence on a pure end-to-end basis (e.g., TCP Low Priority (TCP-LP) [Kuzmanovic03]), and there's a specification for achieving it via router mechanisms [RFC3662] 🛠️. it seems, though, that network-based lower-precedence mechanisms aren't a common service on the Internet yet 🌐. since early 2010, end-to-end mechanisms for lower precedence, e.g., [Shal10], have become common -- at least when competing with other traffic within their own queues (e.g., in a home router) 🏠. but it's less clear whether users will be willing to make their background traffic yield to other people's foreground traffic, unless the right incentives are created 🤷‍♀️

there's an issue about reconciling two divergent views of how traffic class precedence and congestion control relate 😵‍💫. one view says congestion signals (losses or explicit notifications) in one traffic class are independent of those in another 🦄. the other ties the marking of the classes together within the active queue management (AQM) mechanism [Gibbens02] 🔗. in the independent case, using a higher-precedence class gives higher scheduling precedence and generally a lower congestion level 📶. in the linked case, using a higher-precedence class still gives higher scheduling precedence, but results in a higher level of congestion 🌋. that higher congestion level reflects the extra congestion the higher-precedence traffic causes to both classes combined 💥. the linked case separates scheduling precedence from rate control 🎯. the end-to-end congestion control algorithm can choose to take a higher rate by responding less to the higher congestion level 🏎️. this second approach could become prevalent if weighted congestion controls became common 🏋️. however, it's still an open issue how the two approaches might co-exist, or how one might evolve into the other 🌀

3.7. Challenge 7: Misbehaving Senders and Receivers 😈

in the current Internet architecture, congestion control depends on parties acting against their own interests 🙃. a receiver isn't incentivized to honestly return feedback about congestion on the path, which would effectively mean asking for a slower transfer 🐢. a sender isn't incentivized to reduce its rate in response to congestion if it can count on others to do so 🫣. and on top of that, networks may have strategic reasons to make other networks look congested 🚨

there are lots of strategies to improve congestion control that have already been identified 💡. the IETF has especially focused on misbehaving TCP receivers that could trick a compliant sender into assigning excessive network and/or server resources to that receiver (e.g., [Savage99], [RFC3540]) 👀. but although these strategies are scarily powerful, they don't seem common yet (however, evidence of attack prevalence is itself a research requirement) 🕵️‍♀️

a growing share of Internet traffic comes from apps designed not to use congestion control at all, or worse, apps that add more forward error correction as they experience more losses 😱. some people believe the Internet was designed to allow that kind of freedom, so it can hardly be called misbehavior 🤷‍♀️. but others consider it misbehavior to abuse this freedom [RFC3714], since one person's freedom can constrain everyone else's (congestion is literally this conflict of interests) ⚖️. honestly, leaving freedom unchecked might result in congestion collapse in parts of the Internet 💀. proportionally, large volumes of unresponsive voice traffic could be that kind of threat, especially for countries with less generous provisioning [RFC3714] 📞. also, Internet video on demand services that transfer much greater data rates without congestion control are becoming super popular 🍿. in general, it's recommended that such UDP applications use some form of congestion control [RFC5405] 🙏

the problem isn't just misbehavior driven by a self-interested desire for more bandwidth 🙅‍♀️. congestion control can also be attacked by someone who gains nothing for themselves, other than the satisfaction of harming others (see Security Considerations in Section 4) 😈💅

open research questions from all this 📝:

• by design, new congestion control protocols need to let one end check the other for protocol compliance 🔎. how would such mechanisms be designed? 🤔
• which congestion control primitives could safely satisfy more demanding applications (smoother than TFRC, faster than high-speed TCPs), so that app developers and users don't turn congestion control off to get the rate they expect and need? 🏃‍♀️💨

also, self-restraint could disappear from the Internet 😬. so it may no longer be enough to rely on developers and users voluntarily submitting themselves to congestion control 🙅‍♀️. as a result, mechanisms to enforce fairness (see Sections 2.3, 3.4, and 3.5) need more emphasis in the research agenda 📣🔥

3.8. Other Challenges 🧁

this section covers additional challenges and open research issues that are not (at this point in time) deemed so significant, or that are of a different nature compared to the main challenges covered so far 🌷

3.8.1. RTT Estimation ✨

okay so, a bunch of congestion control schemes need to know the round-trip time (RTT) of a path, like, precisely 😤. RTT is basically how much delay is on the network right now 🌸. It's defined as the time between sending a packet and getting back a corresponding response, if the receiver echoes it back immediately on arrival. That's the one-way delay of the packet plus the (possibly different) one-way delay of the response. Any RTT measurement also picks up some extra delay from packet processing at both end-systems 💅.

There are a couple of ways to measure it. Active measurements shove special probe packets into the network and time the response (using, e.g., ICMP) 🧚. Passive measurements figure out the RTT from ongoing communication, without sending any extra packets 🦋.

The connection endpoints of transport protocols like TCP, the Stream Control Transmission Protocol (SCTP), and DCCP, plus several application protocols, keep track of RTT so they can dynamically adjust things like the retransmission timeout (RTO) or the rate-control equation 💖. They can measure it implicitly on the sender side by watching the time gap between sending data and the acknowledgments coming back. For TCP, that's the default procedure, used together with Karn's algorithm, which prohibits RTT measurements from retransmitted segments [RFC2988] 🙈. Traditionally, TCP takes one RTT measurement at a time (about once per RTT). The TCP timestamp option [RFC1323] allows way more frequent explicit measurements, since a sender can safely get an RTT sample from every acknowledgment it receives ✨. Protocols other than TCP use similar measurement mechanisms too 🌈.

Sometimes it'd be super useful to know the RTT not just at the sender but also at the receiver, e.g., to find the one-way variation in delay caused by one-way congestion 👀. A passive receiver can deduce some RTT info by analyzing the sequence numbers of received segments, but that's error-prone and only works if the sender is permanently sending data 😬. Other network entities on the path can use similar heuristics to approximate a connection's RTT, but that's protocol-specific and needs per-connection state. Honestly, in the current Internet there's no simple and safe way for network entities other than the sender to figure out a connection's RTT 💀. The bigger question is whether network elements even need to measure or know the RTT at all 🤔.

As we talked about earlier in this document, RTT isn't a constant value. For a given path there's a theoretical minimum, set by the minimum transmission, processing, and propagation delay on that path. But extra variable delays can come from congestion, cross-traffic, shared-media access control schemes, recovery procedures, or other sub-IP layer mechanisms. And a change of path (like route flapping or a hand-over in mobile networks) can result in completely different delay characteristics 🌪️.

Because of all that variability, one single measured RTT value is barely enough to describe a path 🫠. That's why many protocols use RTT estimators that derive an averaged value and keep track of some history of previous samples. For instance, TCP endpoints derive a smoothed round-trip time (SRTT) from an exponentially weighted moving average [RFC2988]. This low-pass filter makes sure measurement noise and single outliers don't really mess with the estimate. Still, the drawback of low-pass filters is that the averaged value reacts more slowly to sudden changes in the measured RTT 😭. There are workarounds: the standard TCP retransmission timeout calculation looks at not only the SRTT but also a measure of how variable the RTT measurements are [RFC2988]. Since that algorithm isn't well suited for frequent timestamp-based measurements, some implementations tweak the weight factors (e.g., [Sarola02]). There are also proposals for fancier estimators, like Kalman filters or estimators that mainly use peak values 👑.

However, open questions about RTT estimation in the Internet are still hanging around 🥺:

• Optimal measurement frequency: there's no theory or common understanding yet of the right time scale for RTT measurement 🤷. The necessity for rather frequent measurements (e.g., per packet) isn't well understood. There's some empirical evidence that such frequent sampling may not have a significant benefit [Allman99].
• Filter design: a closely related question is how to design good filters for the measured samples. The existing algorithms are known to be robust, but they're far from perfect. The fundamental problem is that there's no single set of RTT values that could characterize the Internet as a whole, so it's hard to define a design target 🎯.
• Default values: RTT estimators can fail in certain scenarios, e.g., when any feedback is missing. In that case, default values have to be used. Today, most default values are set to conservative values that may not be optimal for most Internet communication. Still, the impact of more aggressive settings isn't well understood 🤨.
• Clock granularities: RTT estimation depends on the clock granularities of the protocol stacks. Even though there's a trend toward higher-precision timers, limited granularity (particularly on low-cost devices) may still prevent highly accurate RTT estimations ⏰.

3.8.2. Malfunctioning Devices 🛠️

there's a long history of malfunctioning devices harming the rollout of new and potentially beneficial functionality on the Internet 😩. Sometimes these devices drop packets or even crash completely when a certain mechanism is used. That makes users pick reliability over performance and turn the mechanism off, or makes operating-system vendors disable it by default 🙃. One iconic example is ECN, whose deployment was long hindered by malfunctioning firewalls and is still hindered by malfunctioning home-hubs. But there are tons of other examples too (e.g., the Window Scaling option of TCP) [Thaler07] 💀.

As new congestion control mechanisms get developed with the goal of eventually deploying them on the Internet, it'd be really helpful to collect info about failures caused by devices like this, analyze the reasons for them, and figure out whether there are ways for such devices to do what they're meant to do without causing unintended failures 🔍. Recommendations for vendors of these devices could come from that analysis. It would also be useful to see whether failures caused by such devices can become more visible to endpoints, or to the maintainers of those devices 👀.

A possible way to reduce these problems in the future would be guidelines for standards authors to make sure "forward compatibility" is considered in all IETF work 🧩. That means the default behavior of a device should be precisely defined for all possible values and combinations of protocol fields, not just the minimum needed for the protocol being defined. Then, when previously unused or reserved fields start getting used by newer devices to comply with a new standard, older devices that run into those unusual fields should at least behave predictably 🎀.

3.8.3. Dependence on RTT 📏

AIMD window algorithms that aim for packet conservation end up converging on a rate that's inversely proportional to RTT. However, control theoretic approaches to stability have shown that only the increase in rate (acceleration), and not the target rate, needs to be inversely proportional to RTT [Jin04] 🧠.

It's possible to have more aggressive behaviors for some demanding applications, as long as they're part of a mix with less aggressive transports [Key04] 🤝. This beneficial effect of transport type mixing is probably how the Internet currently manages to stay stable even with TCP slow-start, which is more aggressive than the theory allows for stability 😮. Research giving deeper insight into these aspects would be very useful 📚.

3.8.4. Congestion Control in Multi-Layered Networks 🥞

a network of IP nodes is just as vulnerable to congestion in the lower layers between IP-capable nodes as it is to congestion on the IP-capable nodes themselves 🚧. If network elements take a bigger part in congestion control (ECN, XCP, RCP, etc. -- see Section 3.1), these techniques will either need to be deployed at lower layers as well, or they'll need to interwork with lower-layer mechanisms 🔗.

[RFC5129] shows how to propagate ECN from lower layers upwards, but only for the specific case of MPLS. To the authors' knowledge, though, the layering problem hasn't been addressed for explicit rate protocol proposals such as XCP and RCP 🙈. Some issues are straightforward matters of interoperability (e.g., how exactly to copy fields up the layers), while others are way less obvious (e.g., re-framing issues: if RCP were deployed in a lower layer, how might multiple small RCP frames, all with different rates in their headers, be assembled into a larger IP layer datagram?) 🤯.

Multi-layer considerations also mess with many mechanisms that try to discover whether every node on the path supports a new congestion control protocol 😵‍💫. For instance, some proposals maintain a secondary Time to Live (TTL) field parallel to the one in the IP header. Any nodes that support the new behavior update both TTL fields, whereas legacy IP nodes only update the IP TTL field. This lets the endpoints check whether all IP nodes on the path support the new behavior, in which case both TTLs will be equal at the receiver. But mechanisms like these overlook nodes at lower layers that might not support the new behavior 👀.

A further related issue is congestion control across overlay networks of relays [Hilt08] [Noel07] [Shen08] 🌐.

Section 3.5.3 deals with inelastic multi-domain pseudowires (PWs), where the identity of the pseudowire itself implies the characteristics of the traffic crossing the multi-domain PSN (independently of the actual characteristics of the traffic carried in the PW) 📦. A more complex situation arises when inelastic traffic is carried as part of a pseudowire (e.g., inelastic traffic over Ethernet PW over PSN) whose edges don't have the means to characterize the properties of the traffic encapsulated in the Ethernet frames 😬. In this case, the problem explained in Section 3.5.3 isn't limited to multi-domain pseudowires, but more generally arises from a "pseudowire carrying inelastic traffic" (whether over a single- or multi-domain PSN) 🫶.

The problem gets even more intricate when the Ethernet PW carries both inelastic and elastic traffic 🌀. Addressing this issue further supports our observation that a general framework to efficiently deal with congestion control problems in multi-layer networks, without harming evolvability, is absolutely necessary 🔥.

3.8.5. Multipath End-to-End Congestion Control and Traffic Engineering 🦋

Okay so recent work has shown that multipath endpoint congestion control [Kelly05] is seriously iconic, with considerable benefits for resilience and how efficiently resources get used 🔥. The IETF has since started a work item on multipath TCP [MPTCP]. The best part is that by pooling resources across all the paths, even nodes that aren't using multiple paths themselves still benefit from the ones that are 💖.

There's still a ton of research to do here, especially around how this interacts with route provisioning and traffic engineering that network operators control 🤔. Like, is multipath congestion control actually able to do better traffic engineering than the network itself, given the right incentives [Arkko09]? Total mystery, bestie 👀.

3.8.6. ALGs and Middleboxes 🧚

More and more application layer gateways (ALGs), middleboxes, and proxies (see Section 3.6 of [RFC2775]) are showing up at domain boundaries 🚪. They check conformance, but they also filter traffic and control flows. One big motivation is stopping information beyond routing data from leaking between autonomous systems. The problem is that these systems split up end-to-end TCP connections and disrupt end-to-end congestion control 😭. And transport over encrypted tunnels may not let other network entities join in on congestion control at all.

Basically, these systems mess with the primal and dual congestion control components 💀. End-to-end congestion control can get swapped out for flow-control backpressure mechanisms on the split connections. A huge variety of ALGs and middleboxes use these to make apps faster (Performance Enhancing Proxies, Application Accelerators, etc.) 🚀. The catch? Those mechanisms are often proprietary and undocumented, so their implications haven't been studied systematically so far 🙈.

There are two levels of interference:

• The "transparent" case, meaning the endpoint address from the sender's perspective is still visible to the receiver (the destination IP address). Relay systems that intercept payloads but don't relay congestion control information are one example. Middleboxes like these can prevent end-to-end congestion control from working 😤.
• The "non-transparent" case, which causes fewer problems for congestion control 💅. Even though these devices interfere with end-to-end network transparency, they correctly terminate network, transport, and application layer protocols on both sides, and each of those can be congestion controlled individually ✨.

---

4. Security Considerations 🛡️

Misbehavior can come from pure malice, or malice can be driven by wider selfish interests, e.g., using distributed denial-of-service (DDoS) attacks to gain rewards by extortion [RFC4948] 😬. DDoS attacks are possible both because of vulnerabilities in operating systems and because the Internet delivers packets without requiring congestion control. Not a vibe 💀.

To date, compliance with congestion control rules and being fair require endpoints to cooperate. The possibility of uncooperative behavior can be regarded as a security issue, and its implications are discussed throughout these documents in a scattered fashion 📚.

Right now, the research agenda against denial of service is mostly about identifying attack packets that target machines and the networks hosting them, with a particular focus on mitigating source address spoofing 🕵️. But if mechanisms to enforce congestion control fairness were robust to both selfishness and malice [Bri06], they would also naturally mitigate denial of service against the network. From the perspective of a well-behaved Internet user, that can be considered a congestion control enforcement problem 🧠. Even some denial-of-service attacks on hosts (rather than the network) could be considered a congestion control enforcement issue at the higher layer. BUT clearly there are also denial-of-service attacks that enforcing congestion control would not solve 😭. Not everything can be fixed by one trick, periodt.

Sections 3.5 and 3.7 on multi-domain issues and misbehaving senders and receivers also discuss some information security issues suffered by various congestion control approaches 🔐.

---

5. References

5.1. Informative References 📖

[Allman99]  Allman, M. and V. Paxson, "On Estimating End-to-End
            Network Path Properties", Proceedings of ACM SIGCOMM'99,
            September 1999.

[Andrew05]  Andrew, L., Wydrowski, B., and S. Low, "An Example of
            Instability in XCP", Manuscript available at
            <http://netlab.caltech.edu/maxnet/XCP_instability.pdf>.

[Arkko09]   Arkko, J., Briscoe, B., Eggert, L., Feldmann, A., and M.
            Handley, "Dagstuhl Perspectives Workshop on End-to-End
            Protocols for the Future Internet," ACM SIGCOMM Computer
            Communication Review, Vol. 39, No. 2, pp. 42-47, April
            2009.

[Ath01]     Athuraliya, S., Low, S., Li, V., and Q. Yin, "REM: Active
            Queue Management", IEEE Network Magazine, Vol. 15, No. 3,
            pp. 48-53, May 2001.

[Balan01]   Balan, R.K., Lee, B.P., Kumar, K.R.R., Jacob, L., Seah,
            W.K.G., and A.L. Ananda, "TCP HACK: TCP Header Checksum
            Option to Improve Performance over Lossy Links",
            Proceedings of IEEE INFOCOM'01, Anchorage (Alaska), USA,
            April 2001.

[Bonald00]  Bonald, T., May, M., and J.-C. Bolot, "Analytic
            Evaluation of RED Performance", Proceedings of IEEE
            INFOCOM'00, Tel Aviv, Israel, March 2000.

[Bri06]     Briscoe, B., "Using Self-interest to Prevent Malice;
            Fixing the Denial of Service Flaw of the Internet",
            Workshop on the Economics of Securing the Information
            Infrastructure, October 2006,
            <http://wesii.econinfosec.org/draft.php?paper_id=19>.

[Bri07]     Briscoe, B., "Flow Rate Fairness: Dismantling a
            Religion", ACM SIGCOMM Computer Communication Review,
            Vol. 37, No. 2, pp. 63-74, April 2007.

[Bri08]     Briscoe, B., Moncaster, T. and L. Burness, "Problem
            Statement: Transport Protocols Don't Have To Do
            Fairness", Work in Progress, July 2008.

[Bri09]     Briscoe, B., "Re-feedback: Freedom with Accountability
            for Causing Congestion in a Connectionless Internetwork",
            UCL PhD Thesis (2009).

[Bri10]     Briscoe, B. and J. Manner, "Byte and Packet Congestion
            Notification," Work in Progress, October 2010.

[Chester04] Chesterfield, J., Chakravorty, R., Banerjee, S.,
            Rodriguez, P., Pratt, I., and J. Crowcroft, "Transport
            level optimisations for streaming media over wide-area
            wireless networks", WIOPT'04, March 2004.

[Chhabra02] Chhabra, P., Chuig, S., Goel, A., John, A., Kumar, A.,
            Saran, H., and R. Shorey, "XCHOKe: Malicious Source
            Control for Congestion Avoidance at Internet Gateways,"
            Proceedings of IEEE International Conference on Network
            Protocols (ICNP'02), Paris, France, November 2002.

[Chiu89]    Chiu, D.M. and R. Jain, "Analysis of the increase and
            decrease algorithms for congestion avoidance in computer
            networks", Computer Networks and ISDN Systems, Vol. 17,
            pp. 1-14, 1989.

[Clark88]   Clark, D., "The design philosophy of the DARPA internet
            protocols", ACM SIGCOMM Computer Communication Review,
            Vol. 18, No. 4, pp. 106-114, August 1988.

[Clark98]   Clark, D. and W. Fang, "Explicit Allocation of Best-
            Effort Packet Delivery Service", IEEE/ACM Transactions on
            Networking, Vol. 6, No. 4, pp. 362-373, August 1998.

[Chu10]     Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis,
            "Increasing TCP's Initial Window", Work in Progress,
            October 2010.

[CONEX]     IETF WG Action: Congestion Exposure (conex).

[Dukki05]   Dukkipati, N., Kobayashi, M., Zhang-Shen, R., and N.
            McKeown, "Processor Sharing Flows in the Internet",
            Proceedings of International Workshop on Quality of
            Service (IWQoS'05), Passau, Germany, June 2005.

[Dukki06]   Dukkipati, N. and N. McKeown, "Why Flow-Completion Time
            is the Right Metric for Congestion Control", ACM SIGCOMM
            Computer Communication Review, Vol. 36, No. 1, January
            2006.

[ECODE]     "ECODE Project", European Commission Seventh Framework
            Program, Grant No. 223936, <http://www.ecode-project.eu>.

[Falk07]    Falk, A., Pryadkin, Y., and D. Katabi, "Specification for
            the Explicit Control Protocol (XCP)", Work in Progress,
            January 2007.

[Feldman04]
            Feldman, M., Papadimitriou, C., Chuang, J., and I.
            Stoica, "Free-Riding and Whitewashing in Peer-to-Peer
            Systems" Proceedings of ACM SIGCOMM Workshop on Practice
            and Theory of Incentives in Networked Systems (PINS'04)
            2004.

[Firoiu00]  Firoiu, V. and M. Borden, "A Study of Active Queue
            Management for Congestion Control", Proceedings of IEEE
            INFOCOM'00, Tel Aviv, Israel, March 2000.

[Floyd93]   Floyd, S. and V. Jacobson, "Random early detection
            gateways for congestion avoidance", IEEE/ACM Transactions
            on Networking, Vol. 1, No. 4, pp. 397-413, August 1993.

[Floyd94]   Floyd, S., "TCP and Explicit Congestion Notification",
            ACM Computer Communication Review, Vol. 24, No. 5,
            pp. 10-23, October 1994.

[Gibbens02] Gibbens, R. and Kelly, F., "On Packet Marking at Priority
            Queues", IEEE Transactions on Automatic Control, Vol. 47,
            No. 6, pp. 1016-1020, 2002.

[Ha08]      Ha, S., Rhee, I., and L. Xu, "CUBIC: A new TCP-friendly
            high-speed TCP variant", ACM SIGOPS Operating System
            Review, Vol. 42, No. 5, pp. 64-74, 2008.

[Hilt08]    Hilt, V. and I. Widjaja, "Controlling Overload in
            Networks of SIP Servers", Proceedings of IEEE
            International Conference on Network Protocols (ICNP'08),
            Orlando (Florida), USA, October 2008.

[Hollot01]  Hollot, C., Misra, V., Towsley, D., and W.-B. Gong, "A
            Control Theoretic Analysis of RED", Proceedings of IEEE
            INFOCOM'01, Anchorage (Alaska), USA, April 2001.

[Jacobson88]
            Jacobson, V., "Congestion Avoidance and Control",
            Proceedings of ACM SIGCOMM'88 Symposium, August 1988.

[Jain88]    Jain, R. and K. Ramakrishnan, "Congestion Avoidance in
            Computer Networks with a Connectionless Network Layer:
            Concepts, Goals, and Methodology", Proceedings of IEEE
            Computer Networking Symposium, Washington DC, USA, April
            1988.

[Jain90]    Jain, R., "Congestion Control in Computer Networks:
            Trends and Issues", IEEE Network, pp. 24-30, May 1990.

[Jin04]     Jin, Ch., Wei, D.X., and S. Low, "FAST TCP: Motivation,
            Architecture, Algorithms, Performance", Proceedings of
            IEEE INFOCOM'04, Hong-Kong, China, March 2004.

[Jourjon08] Jourjon, G., Lochin, E., and P. Senac, "Design,
            Implementation and Evaluation of a QoS-aware Transport
            Protocol", Elsevier Computer Communications, Vol. 31,
            No. 9, pp. 1713-1722, June 2008.

Okay, the reference list keeps going, and every single entry stays exactly as written 📚💖

   [Katabi02]  Katabi, D., M. Handley, and C. Rohrs, "Internet
               Congestion Control for Future High Bandwidth-Delay
               Product Environments", Proceedings of ACM SIGCOMM'02
               Symposium, August 2002.

   [Katabi04]  Katabi, D., "XCP Performance in the Presence of Malicious
               Flows", Proceedings of PFLDnet'04 Workshop, Argonne
               (Illinois), USA, February 2004.

   [Kelly05]   Kelly, F. and Th. Voice, "Stability of end-to-end
               algorithms for joint routing and rate control", ACM
               SIGCOMM Computer Communication Review, Vol. 35, No. 2,
               pp. 5-12, April 2005.

   [Kelly98]   Kelly, F., Maulloo, A., and D. Tan, "Rate control in
               communication networks: shadow prices, proportional
               fairness, and stability", Journal of the Operational
               Research Society, Vol. 49, pp. 237-252, 1998.

   [Keshav07]  Keshav, S., "What is congestion and what is congestion
               control", Presentation at IRTF ICCRG Workshop, PFLDnet
               2007, Los Angeles (California), USA, February 2007.

   [Key04]     Key, P., Massoulie, L., Bain, A., and F. Kelly, "Fair
               Internet Traffic Integration: Network Flow Models and
               Analysis", Annales des Telecommunications, Vol. 59,
               No. 11-12, pp. 1338-1352, November-December 2004.

   [Krishnan04]
               Krishnan, R., Sterbenz, J., Eddy, W., Partridge, C., and
               M. Allman, "Explicit Transport Error Notification (ETEN)
               for Error-Prone Wireless and Satellite Networks",
               Computer Networks, Vol. 46, No. 3, October 2004.

   [Kuzmanovic03]
               Kuzmanovic, A. and E.W. Knightly, "TCP-LP: A Distributed
               Algorithm for Low Priority Data Transfer", Proceedings of
               IEEE INFOCOM'03, San Francisco (California), USA, April
               2003.

   [LEDBAT]    IETF WG Action: Low Extra Delay Background Transport
               (ledbat).

   [Lochin06]  Lochin, E., Dairaine, L., and G. Jourjon, "Guaranteed TCP
               Friendly Rate Control (gTFRC) for DiffServ/AF Network",
               Work in Progress, August 2006.

   [Lochin07]  Lochin, E., Jourjon, G., and L. Dairaine, "Study and
               enhancement of DCCP over DiffServ Assured Forwarding
               class", 4th Conference on Universal Multiservice Networks
               (ECUMN 2007), Toulouse, France, February 2007.

   [Low02]     Low, S., Paganini, F., Wang, J., Adlakha, S., and J.C.
               Doyle, "Dynamics of TCP/RED and a Scalable Control",
               Proceedings of IEEE INFOCOM'02, New York (New Jersey),
               2002.

   [Low03.1]   Low, S., "A duality model of TCP and queue management
               algorithms", IEEE/ACM Transactions on Networking,
               Vol. 11, No. 4, pp. 525-536, August 2003.

   [Low03.2]   Low, S., Paganini, F., Wang, J., and J. Doyle, "Linear
               stability of TCP/RED and a scalable control", Computer
               Networks Journal, Vol. 43, No. 5, pp. 633-647, December
               2003.

   [Low05]     Low, S., Andrew, L., and B. Wydrowski, "Understanding
               XCP: equilibrium and fairness", Proceedings of IEEE
               INFOCOM'05, Miami (Florida), USA, March 2005.

   [MacK95]    MacKie-Mason, J. and H. Varian, "Pricing Congestible
               Network Resources", IEEE Journal on Selected Areas in
               Communications, Advances in the Fundamentals of
               Networking, Vol. 13, No. 7, pp. 1141-1149, 1995.

   [Mascolo01] Mascolo, S., Casetti, Cl., Gerla M., Sanadidi, M.Y., and
               R. Wang, "TCP Westwood: Bandwidth estimation for enhanced
               transport over wireless links", Proceedings of MOBICOM
               2001, Rome, Italy, July 2001.

   [Moors02]   Moors, T., "A critical review of "End-to-end arguments in
               system design"", Proceedings of IEEE International
               Conference on Communications (ICC) 2002, New York City
               (New Jersey), USA, April/May 2002.

   [MPTCP]     IETF WG Action: Multipath TCP (mptcp).

   [Noel07]    Noel, E. and C. Johnson, "Initial Simulation Results That
               Analyze SIP Based VoIP Networks Under Overload",
               International Teletraffic Congress (ITC'07), Ottawa,
               Canada, June 2007.

   [Padhye98]  Padhye, J., Firoiu, V., Towsley, D., and J. Kurose,
               "Modeling TCP Throughput: A Simple Model and Its
               Empirical Validation", University of Massachusetts
               (UMass), CMPSCI Tech. Report TR98-008, February 1998.

   [Pan00]     Pan, R., Prabhakar, B., and K. Psounis, "CHOKe: a
               stateless AQM scheme for approximating fair bandwidth
               allocation", Proceedings of IEEE INFOCOM'00, Tel Aviv,
               Israel, March 2000.

   [Pap02]     Papadimitriou, I. and G. Mavromatis, "Stability of
               Congestion Control Algorithms using Control Theory with
               an application to XCP", Technical Report, 2002.
               <http://www.stanford.edu/class/ee384y/projects/
               reports/ionnis.pdf>.

   [RFC791]    Postel, J., "Internet Protocol", STD 5, RFC 791,
               September 1981.

   [RFC793]    Postel, J., "Transmission Control Protocol", STD 7,
               RFC 793, September 1981.

   [RFC1323]   Jacobson, V., Braden, R., and D. Borman, "TCP Extensions
               for High Performance", RFC 1323, May 1992.

   [RFC1701]   Hanks, S., Li, T., Farinacci, D., and P. Traina, "Generic
               Routing Encapsulation (GRE)", RFC 1701, October 1994.

   [RFC1958]   Carpenter, B., Ed., "Architectural Principles of the
               Internet", RFC 1958, June 1996.

   [RFC2003]   Perkins, C., "IP Encapsulation within IP", RFC 2003,
               October 1996.

   [RFC2018]   Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
               Selective Acknowledgment Options", RFC 2018, October
               1996.

   [RFC2208]   Mankin, A., Ed., Baker, F., Braden, B., Bradner, S.,
               O'Dell, M., Romanow, A., Weinrib, A., and L. Zhang,
               "Resource ReSerVation Protocol (RSVP) -- Version 1
               Applicability Statement Some Guidelines on Deployment",
               RFC 2208, September 1997.

   [RFC2474]   Nichols, K., Blake, S., Baker, F., and D. Black,
               "Definition of the Differentiated Services Field (DS
               Field) in the IPv4 and IPv6 Headers", RFC 2474, December
               1998.

   [RFC2475]   Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
               and W. Weiss, "An Architecture for Differentiated
               Service", RFC 2475, December 1998.

   [RFC2581]   Allman, M., Paxson, V., and W. Stevens, "TCP Congestion
               Control", RFC 2581, April 1999.

   [RFC2637]   Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little,
               W., and G. Zorn, "Point-to-Point Tunneling Protocol
               (PPTP)", RFC 2637, July 1999.

   [RFC2661]   Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn,
               G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"",
               RFC 2661, August 1999.

   [RFC2775]   Carpenter, B., "Internet Transparency", RFC 2775,
               February 2000.

   [RFC2784]   Farinacci, D., Li, T., Hanks, S., Meyer, D., and P.
               Traina, "Generic Routing Encapsulation (GRE)", RFC 2784,
               March 2000.

   [RFC2861]   Handley, M., Padhye, J., and S. Floyd, "TCP Congestion
               Window Validation", RFC 2861, June 2000.

   [RFC2914]   Floyd, S., "Congestion Control Principles", BCP 41,
               RFC 2914, September 2000.

   [RFC2988]   Paxson, V. and M. Allman, "Computing TCP's Retransmission
               Timer", RFC 2988, November 2000.

   [RFC2990]   Huston, G., "Next Steps for the IP QoS Architecture",
               RFC 2990, November 2000.

   [RFC3031]   Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
               Label Switching Architecture", RFC 3031, January 2001.

   [RFC3032]   Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
               Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
               Encoding", RFC 3032, January 2001.

   [RFC3168]   Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
               of Explicit Congestion Notification (ECN) to IP",
               RFC 3168, September 2001.

   [RFC3260]   Grossman, D., "New Terminology and Clarifications for
               Diffserv", RFC 3260, April 2002.

   [RFC3517]   Blanton, E., Allman, M., Fall, K., and L. Wang, "A
               Conservative Selective Acknowledgment (SACK)-based Loss
               Recovery Algorithm for TCP", RFC 3517, April 2003.

   [RFC3540]   Spring, N., Wetherall, D., and D. Ely, "Robust Explicit
               Congestion Notification (ECN) Signaling with Nonces",
               RFC 3540, June 2003.

   [RFC3662]   Bless, R., Nichols, K., and K. Wehrle, "A Lower Effort
               Per-Domain Behavior (PDB) for Differentiated Services",
               RFC 3662, December 2003.

   [RFC3714]   Floyd, S., Ed., and J. Kempf, Ed., "IAB Concerns
               Regarding Congestion Control for Voice Traffic in the
               Internet", RFC 3714, March 2004.

   [RFC3742]   Floyd, S., "Limited Slow-Start for TCP with Large
               Congestion Windows", RFC 3742, March 2004.

   [RFC3985]   Bryant, S., Ed., and P. Pate, Ed., "Pseudo Wire Emulation
               Edge-to-Edge (PWE3) Architecture", RFC 3985, March 2005.

   [RFC4301]   Kent, S. and K. Seo, "Security Architecture for the
               Internet Protocol", RFC 4301, December 2005.

   [RFC4340]   Kohler, E., Handley, M., and S. Floyd, "Datagram
               Congestion Control Protocol (DCCP)", RFC 4340, March
               2006.

   [RFC4341]   Floyd, S. and E. Kohler, "Profile for Datagram Congestion
               Control Protocol (DCCP) Congestion Control ID 2: TCP-like
               Congestion Control", RFC 4341, March 2006.

   [RFC4342]   Floyd, S., Kohler, E., and J. Padhye, "Profile for
               Datagram Congestion Control Protocol (DCCP) Congestion
               Control ID 3: TCP-Friendly Rate Control (TFRC)",
               RFC 4342, March 2006.

   [RFC4553]   Vainshtein, A., Ed., and YJ. Stein, Ed., "Structure-
               Agnostic Time Division Multiplexing (TDM) over Packet
               (SAToP)", RFC 4553, June 2006.

   [RFC4614]   Duke, M., Braden, R., Eddy, W., and E. Blanton, "A
               Roadmap for Transmission Control Protocol (TCP)
               Specification Documents", RFC 4614, September 2006.

   [RFC4782]   Floyd, S., Allman, M., Jain, A., and P. Sarolahti,
               "Quick-Start for TCP and IP", RFC 4782, January 2007.

   [RFC4828]   Floyd, S. and E. Kohler, "TCP Friendly Rate Control (TFRC):
               The Small-Packet (SP) Variant", RFC 4828, April
               2007.

   [RFC4948]   Andersson, L., Davies, E., and L. Zhang, "Report from the
               IAB workshop on Unwanted Traffic March 9-10, 2006",
               RFC 4948, August 2007.

   [RFC5033]   Floyd, S. and M. Allman, "Specifying New Congestion
               Control Algorithms", BCP 133, RFC 5033, August 2007.

   [RFC5086]   Vainshtein, A., Ed., Sasson, I., Metz, E., Frost, T., and
               P. Pate, "Structure-Aware Time Division Multiplexed (TDM)
               Circuit Emulation Service over Packet Switched Network
               (CESoPSN)", RFC 5086, December 2007.

   [RFC5087]   Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi,
               "Time Division Multiplexing over IP (TDMoIP)", RFC 5087,
               December 2007.

   [RFC5129]   Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
               Marking in MPLS", RFC 5129, January 2008.

That's the whole reference list, bestie, every citation exactly as it was 🫶🦋

[RFC5290] Floyd, S. and M. Allman, "Comments on the Usefulness of Simple Best-Effort Traffic", RFC 5290, July 2008.

[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP Friendly Rate Control (TFRC): Protocol Specification", RFC 5348, September 2008.

[RFC5405] Eggert, L. and G. Fairhurst, "Unicast UDP Usage Guidelines for Application Designers", BCP 145, RFC 5405, November 2008.

[RFC5622] Floyd, S. and E. Kohler, "Profile for Datagram Congestion Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate Control for Small Packets (TFRC-SP)", RFC 5622, August 2009.

[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681 (Obsoletes RFC 2581), September 2009.

[RFC5783] Welzl, M. and W. Eddy, "Congestion Control in the RFC Series", RFC 5783, February 2010.

[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion Notification", RFC 6040, November 2010.

[Rossi06] Rossi, M., "Evaluating TCP with Corruption Notification in an IEEE 802.11 Wireless LAN", Master Thesis, University of Innsbruck, November 2006. Available from http://heim.ifi.uio.no/michawe/research/projects/corruption/.

[Saltzer84] Saltzer, J., Reed, D., and D. Clark, "End-to-end arguments in system design", ACM Transactions on Computer Systems, Vol. 2, No. 4, November 1984.

[Sarola02] Sarolahti, P. and A. Kuznetsov, "Congestion Control in Linux TCP", Proceedings of the USENIX Annual Technical Conference, 2002.

[Sarola07] Sarolahti, P., Floyd, S., and M. Kojo, "Transport-layer Considerations for Explicit Cross-layer Indications", Work in Progress, March 2007.

[Savage99] Savage, S., Cardwell, N., Wetherall, D., and T. Anderson, "TCP Congestion Control with a Misbehaving Receiver", ACM SIGCOMM Computer Communication Review, 1999.

[Shal10] Shalunov, S., Hazel, G., and J. Iyengar, "Low Extra Delay Background Transport (LEDBAT)", Work in Progress, October 2010.

[Shen08] Shen, C., Schulzrinne, H., and E. Nahum, "Session Initiation Protocol (SIP) Server Overload Control: Design and Evaluation, Principles", Systems and Applications of IP Telecommunications (IPTComm'08), Heidelberg, Germany, July 2008.

[Shin08] Shin, M., Chong, S., and I. Rhee, "Dual-Resource TCP/AQM for Processing-Constrained Networks", IEEE/ACM Transactions on Networking, Vol. 16, No. 2, pp. 435-449, April 2008.

[Thaler07] Thaler, D., Sridharan, M., and D. Bansal, "Implementation Report on Experiences with Various TCP RFCs", Presentation to the IETF Transport Area, March 2007. <http://www.ietf.org/proceedings/07mar/slides/tsvarea-3/>.

[Tickoo05] Tickoo, O., Subramanian, V., Kalyanaraman, S., and K.K. Ramakrishnan, "LT-TCP: End-to-End Framework to Improve TCP Performance over Networks with Lossy Channels", Proceedings of International Workshop on QoS (IWQoS), Passau, Germany, June 2005.

[TRILOGY] "Trilogy Project", European Commission Seventh Framework Program (FP7), Grant No: 216372, <http://www.trilogy-project.org>.

[Vinnic02] Vinnicombe, G., "On the stability of networks operating TCP-like congestion control," Proceedings of IFAC World Congress, Barcelona, Spain, 2002.

[Welzl03] Welzl, M., "Scalable Performance Signalling and Congestion Avoidance", Springer (ISBN 1-4020-7570-7), September 2003.

[Welzl08] Welzl, M., Rossi, M., Fumagalli, A., and M. Tacca, "TCP/IP over IEEE 802.11b WLAN: the Challenge of Harnessing Known-Corrupt Data", Proceedings of IEEE International Conference on Communications (ICC) 2008, Beijing, China, May 2008.

[Xia05] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman, "One more bit is enough", ACM SIGCOMM Computer Communication Review, Vol. 35, No. 4, pp. 37-48, 2005.

[Zhang03] Zhang, H., Towsley, D., Hollot, C., and V. Misra, "A Self-Tuning Structure for Adaptation in TCP/AQM Networks", Proceedings of ACM SIGMETRICS'03 Conference, San Diego (California), USA, June 2003.

---

6. Acknowledgments 🎀

Huge thanks to the people who gave feedback and comments that helped shape this doc 🫶: Keith Moore, Jan Vandenabeele, and Larry Dunn! Larry's comments at the Manchester ICCRG, plus chats with him, helped with the packet-congestibility section 💜

Dimitri Papadimitriou's work was partly funded by [ECODE], a Seventh Framework Program (FP7) research project sponsored by the European Commission ✨

Bob Briscoe's work was partly funded by [TRILOGY], a research project supported by the European Commission 🌈

Michael Scharf is now with Alcatel-Lucent 🦋

---

7. Contributors 💖

These extra cuties also contributed to the document 🧚:

• Wesley Eddy <weddy@grc.nasa.gov>
• Bela Berde <bela.berde@gmx.de>
• Paulo Loureiro <loureiro.pjg@gmail.com>
• Chris Christou <christou_chris@bah.com>

---

Authors' Addresses 🦋

Dimitri Papadimitriou (editor)
Alcatel-Lucent
Copernicuslaan, 50
2018 Antwerpen, Belgium

Phone: +32 3 240 8491
EMail: dimitri.papadimitriou@alcatel-lucent.com


Michael Welzl
University of Oslo, Department of Informatics
PO Box 1080 Blindern
N-0316 Oslo, Norway

EMail: michawe@ifi.uio.no


Michael Scharf
University of Stuttgart
Pfaffenwaldring 47
70569 Stuttgart, Germany

EMail: michael.scharf@googlemail.com


Bob Briscoe
BT & UCL
B54/77, Adastral Park
Martlesham Heath
Ipswich IP5 3RE, UK

EMail: bob.briscoe@bt.com