RFC 3124

The Congestion Manager

📅 published June 2001, from ietf

The Congestion Manager (RFC 3124) 🎀 Part 1 of 6

Okay so this is RFC 3124, and it's called The Congestion Manager ✨. It was written by H. Balakrishnan of MIT LCS and S. Seshan of CMU, both in the Network Working Group. It's on the Standards Track 🦋, and the date is June 2001. It doesn't obsolete or update anything.

Status of this Memo

This document specifies an Internet standards track protocol for the whole Internet community 🌐, and it's literally asking for discussion and suggestions 💬. For the current standardization state and status of this protocol, check the "Internet Official Protocol Standards" (STD 1) 📚. Distribution of this memo is unlimited 💖.

Copyright Notice

Abstract

This doc describes the Congestion Manager (CM) 🔥, an end-system module that does two big jobs:

(i) It lets a bunch of concurrent streams from one sender, all going to the same receiver and sharing the same congestion properties, do proper congestion avoidance and control together 🧚
(ii) It lets applications easily adapt to network congestion 💅

1. Conventions used in this document: ✨

Okay bestie, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in RFC-2119 [Bradner97] 🫶. Those are the ones to watch for.

Here are the terms you need to know:

STREAM: A group of packets that all share the same source and destination IP address, IP type-of-service, transport protocol, and source and destination transport-layer port numbers. 🎀

MACROFLOW: A group of CM-enabled streams that all use the same congestion management and scheduling algorithms, and share congestion state information. Right now, streams going to different receivers belong to different macroflows. Streams going to the same receiver MAY belong to different macroflows. When the Congestion Manager is in use, streams that experience identical congestion behavior and use the same congestion control algorithm SHOULD belong to the same macroflow. It's giving group chat 💜

APPLICATION: Any software module that uses the CM. This includes user-level applications such as Web servers or audio/video servers, as well as in-kernel protocols such as TCP [Postel81] that use the CM for congestion control. 🦋

WELL-BEHAVED APPLICATION: An application that only transmits when allowed by the CM and accurately accounts for all data that it has sent to the receiver by informing the CM using the CM API. No cap, this is the good kind of app 🌈

PATH MAXIMUM TRANSMISSION UNIT (PMTU): The size of the largest packet that the sender can transmit without it being fragmented en route to the receiver. It includes the sizes of all headers and data except the IP header. 🍓

CONGESTION WINDOW (cwnd): A CM state variable that modulates the amount of outstanding data between sender and receiver. 💀

OUTSTANDING WINDOW (ownd): The number of bytes that has been transmitted by the source, but not known to have been either received by the destination or lost in the network. 🙈

INITIAL WINDOW (IW): The size of the sender's congestion window at the beginning of a macroflow. ⭐

DATA TYPE SYNTAX: We use "u64" for unsigned 64-bit, "u32" for unsigned 32-bit, "u16" for unsigned 16-bit, "u8" for unsigned 8-bit, "i32" for signed 32-bit, "i16" for signed 16-bit quantities, "float" for IEEE floating point values. The type "void" is used to indicate that no return value is expected from a call. Pointers are referred to using "*" syntax, following C language convention. 👑

We really emphasize that all the API functions described in this document are "abstract" calls, and that conformant CM implementations may differ in specific implementation details. The vibe is: the idea is locked in, but the exact build is up to the implementer 🧚

2. Introduction ✨

okay so this section is the big picture, bestie 🦋 The CM framework plugs congestion management into all the apps and transport protocols on a machine. It keeps track of the congestion stuff (how much bandwidth is free overall and per stream, the round-trip times to each receiver, and so on) and gives apps an API to do five things: learn about the network, pass info to the CM, share congestion info with each other, and schedule when they send data. This document is mostly about apps and transport protocols that keep their own per-byte or per-packet sequence numbers, and it doesn't need changes to the receiver's protocol stack 💅 The catch is that the receiving app has to tell the sending app what arrived and what got lost, and the sender is expected to use the CM API to update the CM's state. The doc also doesn't cover networks with reservations or service differentiation 🎀

The CM is an end-system module that lets a whole group (an "ensemble") of concurrent streams do stable congestion avoidance and control, and it makes it easy for apps to adjust their sending to whatever the network is doing right now ✨ Like the previous paragraph, it integrates congestion management across every app and transport protocol, keeps those congestion parameters (aggregate and per-stream bandwidth, per-receiver round-trip times, and so on), and exposes that same API for learning, passing info, sharing congestion info, and scheduling. The main rule when the CM is in use: every data transmission that's subject to the CM has to happen with the CM's explicit consent through this API, or the congestion behaviour won't be proper 😭

Systems MAY choose to use CM, and if so they MUST follow this specification. 🌸

This document is focused on applications and networks where these conditions hold:

Applications are well-behaved, with their own independent per-byte or per-packet sequence number information, and they use the CM API to update internal state in the CM.
Networks are best-effort, with no service discrimination or reservations. In particular, this doc doesn't handle situations where different streams between the same pair of hosts take paths with different characteristics 🙈

The Congestion Manager framework can be extended to support applications that don't provide their own feedback, and to differentially-served networks. Those extensions will get their own documents later, so stay tuned 🔥

The CM has two main goals, and they're both giving main character energy:

(i) Enable efficient multiplexing. Right now the internet trend is for unicast data senders (think Web servers) to send all kinds of different data to receivers, from unreliable real-time streaming to reliable Web pages and applets. So lots of logically different streams end up sharing the same path between sender and receiver 🧚 For the internet to stay stable, every one of those streams needs its own control protocol that safely probes for spare bandwidth and backs off when there's congestion. The problem is that these concurrent streams usually compete with each other for network resources instead of sharing them well, and they don't learn from each other about the state of the network 💀 Even when each one implements congestion control on its own (like a bunch of TCP connections each running the algorithms in [Jacobson88, Allman99]), the whole group tends to be more aggressive under congestion than a single TCP connection running standard TCP congestion control and avoidance [Balakrishnan98]. Rude.

(ii) Enable application adaptation to congestion. Real-time streaming apps are getting super popular, and they usually run over UDP with their own user-level transport protocols because that gives good app performance. But most of them today don't adapt or react properly to network congestion 😤 The CM fixes this by providing a stable control algorithm and an adaptation API, which makes it easy for apps to adapt to congestion. The apps then adjust the data they send to match the current network conditions 🦋

The CM framework builds on some earlier ideas: TCP control block sharing [Touch97], integrated TCP congestion control (TCP-Int) [Balakrishnan98], and TCP sessions [Padmanabhan98]. [Touch97] argues for sharing some of the state in the TCP control block to improve transient transport performance, and it describes sharing across a group of TCP connections. [Balakrishnan98], [Padmanabhan98], and [Eggert00] describe experiments that measure the benefits of sharing congestion state, including better stability under congestion and better loss recovery. Integrating loss recovery across concurrent connections helps a lot, because a loss on one connection can be spotted when later data sent on a different connection gets received and acknowledged 🌈 The CM extends these ideas in two big ways: (i) it brings congestion management to non-TCP streams, which are getting really common and often don't do proper congestion management at all, and (ii) it gives apps an API so they can adapt what they send to current network conditions. If you want the longer version of the motivation, architecture, API, and algorithms, check out [Balakrishnan99]. For an implementation and its performance results, see [Andersen00].

The end-host protocol architecture at the sender is shown in Figure 1. The CM keeps the network stable by running stable congestion avoidance and control algorithms that are "TCP-friendly" [Mahdavi98], based on algorithms described in [Allman99]. It doesn't try to enforce proper congestion behaviour for every app, though it doesn't stop a policer on the host from doing that job 👑 The network still needs protecting against compromises of the CM and the end-host policer, which takes router machinery [Floyd99a]. This document doesn't go further into that issue.

   |--------| |--------| |--------| |--------|       |--------------|
   |  HTTP  | |  FTP   | |  RTP 1 | |  RTP 2 |       |              |
   |--------| |--------| |--------| |--------|       |              |
       |          |         |  ^       |  ^          |              |
       |          |         |  |       |  |          |   Scheduler  |
       |          |         |  |-------|--+->|   |   |              |
       |          |         |          |     |   |<--|              |
       v          v         v          v     |   |   |--------------|
   |--------| |--------|  |-------------|    |   |           ^
   |  TCP 1 | |  TCP 2 |  |    UDP 1    |    | A |           |
   |--------| |--------|  |-------------|    |   |           |
      ^   |      ^   |              |        |   |   |--------------|
      |   |      |   |              |        | P |-->|              |
      |   |      |   |              |        |   |   |              |
      |---|------+---|--------------|------->|   |   |  Congestion  |
          |          |              |        | I |   |              |
          v          v              v        |   |   |  Controller  |
     |-----------------------------------|   |   |   |              |
     |               IP                  |-->|   |   |              |
     |-----------------------------------|   |   |   |--------------|
                                             |---|

                                      Figure 1

The key parts of the CM framework are (i) the API, (ii) the congestion controller, and (iii) the scheduler 🍓 The API is partly motivated by the requirements of application-level framing (ALF) [Clark90], and it's covered in Section 4. The CM internals (Section 5) include a congestion controller (Section 5.1) and a scheduler that orchestrates data transmissions between concurrent streams in a macroflow (Section 5.2). The congestion controller adjusts the aggregate transmission rate between sender and receiver based on its estimate of congestion in the network, and it gets feedback about past transmissions from the apps themselves through the API. The scheduler splits up the available bandwidth among the different streams within each macroflow and tells apps when they're allowed to send data 🌸 This doc focuses on well-behaved apps. A future document will describe the sender-receiver protocol and header formats for apps that don't provide their own feedback to the CM.

3. CM API 💖

By convention, the IETF doesn't treat Application Programming Interfaces as standards track 🙈 But it's considered important to have the CM API and the CM algorithm requirements in one coherent document, so here we are. This section on the CM API uses the terms MUST, SHOULD, etc., but they only apply within the context of an implementation of the CM API. They don't apply to congestion control implementations in general, just to the ones that offer the CM API 🎀

With the CM API, streams can figure out their share of the available bandwidth, request and get their data transmissions scheduled, tell the CM about successful transmissions, and get notified when the CM's estimate of path bandwidth changes. So the CM takes the load off apps, which don't have to track the congestion state or the available bandwidth on any path themselves 🧚

The function prototypes below follow standard C convention 💅 These API functions are abstract calls, and conformant CM implementations can differ in the details, as long as they provide equivalent functionality.

When an app creates a new stream, it passes some info to the CM with the cm_open(stream_info) API call. Right now, stream_info consists of: (i) the source IP address, (ii) the source port, (iii) the destination IP address, (iv) the destination port, and (v) the IP protocol number ✨

3.1 State maintenance

Open: All applications MUST call cm_open(stream_info) before using the CM API. It returns a handle, cm_streamid, which the app uses for all further CM API calls for that stream. If the returned cm_streamid is -1, then cm_open() failed and that stream can't use the CM 😭

All other calls to the CM for a stream use the cm_streamid returned from the cm_open() call 🫶

Close: When a stream terminates, the application SHOULD invoke cm_close(cm_streamid) to tell the CM the stream is done 🌈
Packet size: cm_mtu(cm_streamid) returns the estimated PMTU of the path between sender and receiver. Internally, this info SHOULD be obtained via path MTU discovery [Mogul90]. It MAY be statically configured if there's no such mechanism 💜

3.2 Data transmission ✨

okay bestie, so the CM supports two types of adaptive senders 💅. that means apps can change their content based on what the network is doing right now 🦋, and it also supports ALF-based apps 🌸

1. Callback-based transmission 🎀

this one is giving "the stream is in charge" energy 👑. the CM does not buffer any data at all. instead it gives streams the chance to adapt at the very last possible instant 😭. so when a stream learns about a rate change, it can "pull out" and repacketize its data, which is super hard to do once the data is already buffered 💀. the CM must implement a cm_request(i32 cm_streamid) call for streams that want to send this way. after some time (depending on the rate), the CM MUST invoke a callback using cmapp_send(), which is a grant to send up to PMTU bytes 🧚. this callback-style API is the recommended choice for ALF-based streams 🫶. also, cm_request() doesn't take a number of bytes or MTU-sized units as an argument. every call is an implicit request to send up to PMTU bytes 🍓. the CM MAY provide an alternate interface, cm_request(int k). the cmapp_send callback for that request gets the right to send up to k PMTU sized segments ✨. Section 4.3 talks about how long the transmission grant stays valid, and Section 5.2 explains how these requests get scheduled and how the callbacks happen 📅.

2. Synchronous-style 🌈

the callback API above covers the "asynchronous" ALF streams. those don't send on a periodic clock. they send when something triggers them, like a file read or a captured frame 📂. then there are the "synchronous" transmitters, which send periodically based on their own internal timers (an audio sender that sends at a constant sampling rate is the classic example 🎧). the CM callbacks could be set up to periodically interrupt these, but the transmit loop of those apps is way less messy if they just keep their original timer-based loop 🙈. plus, it would make the CM API more complicated if a stream had to express how periodic and how granular its callbacks should be 😤. so the CM MUST export an API that lets these streams hear about rate changes through the cmapp_update(u64 newrate, u32 srtt, u32 rttdev) callback. here, newrate is the new rate in bits per second for this stream, srtt is the current smoothed round-trip time estimate in microseconds, and rttdev is the smoothed linear deviation in the round-trip time estimate, calculated with the same algorithm TCP uses [Paxson00] 📈. the newrate value is an instantaneous rate. it's calculated, for example, by taking the ratio of cwnd and srtt and dividing by the fraction of that ratio given to the stream 🧮.

in response, the stream MUST adapt its packet size or change its timer interval so it conforms to (meaning doesn't exceed) the allowed rate 🫧. it's allowed to use less than all of that rate, though 💜. also, the CM isn't on the data path of the actual transmission, so it's not touching your packets 🙌.

to avoid pointless cmapp_update() callbacks that the app would just ignore 🙄, the CM MUST provide a cm_thresh(float rate_downthresh, float rate_upthresh, float rtt_downthresh, float rtt_upthresh) function that a stream can use at any stage of its execution 🎯. then the CM SHOULD invoke the callback only when the rate drops below (rate_downthresh * lastrate) or rises above (rate_upthresh * lastrate), where lastrate is the rate last notified to the stream. or it can invoke it when the round-trip time changes by the required thresholds 📉. this is just a hint to the CM though, so cmapp_update() can still get called even when none of these conditions are met 🤷‍♀️.

the CM MUST implement a cm_query(i32 cm_streamid, u64* rate, u32* srtt, u32* rttdev) so an application can check the current CM state 🔍. this sets rate to the current rate estimate in bits per second, srtt to the current smoothed round-trip time estimate in microseconds, and rttdev to the mean linear deviation. if the CM doesn't have valid estimates for the macroflow, it fills in negative values for rate, srtt, and rttdev 🚫.

a stream can use more than one of these transmission APIs at the same time 🎪. knowing the sustainable rate is useful for asynchronous streams as well as synchronous ones. for example, an asynchronous Web server sending images over TCP might use cmapp_send() to schedule its transmissions and cmapp_update() to decide whether to send a low-resolution or high-resolution image 🖼️. Section 6.1.1 describes a TCP implementation that uses the CM, and that's where the benefit of the cm_request() callback API for TCP really shines ✨.

you'll notice the basic CM API doesn't have an interface for buffered congestion-controlled transmissions. that's on purpose, because this mode can be built on top of the callback-based primitive 🧱. Section 6.1.2 shows how congestion-controlled UDP sockets can be built using the CM API 🧩.

3.3 Application notification 📣

when a stream gets feedback from receivers, it MUST use cm_update(i32 cm_streamid, u32 nrecd, u32 nlost, u8 lossmode, i32 rtt) to tell the CM about things like congestion losses, successful receptions, the type of loss (a timeout event, Explicit Congestion Notification [Ramakrishnan99], and so on), and round-trip time samples 💌. the nrecd parameter says how many bytes the receiver successfully got since the last cm_update call. the original then says the nrecd parameter identifies how many bytes were received were lost during the same time period 🫠.

aside from me: that second nrecd is written exactly like that in the original, and it's almost certainly meant to be a different parameter (the loss count). go check the source before you implement it 🙏

the rtt value is the round-trip time measured while those bytes were in flight ⏱️. the rtt value must be set to -1 if the app didn't get a valid round-trip sample 🙅‍♀️. the lossmode parameter tells the CM how a loss was detected 🕵️‍♀️. a value of CM_NO_FEEDBACK means the app got no feedback at all for its outstanding data and is reporting exactly that 😶. this is what a TCP would send after a timeout, for example. a value of CM_LOSS_FEEDBACK means the app saw some loss that it thinks is from congestion, but not everything outstanding was lost 📉. a TCP segment loss spotted through duplicate (selective) acknowledgments or other data-driven techniques falls in this bucket 🧠. CM_EXPLICIT_CONGESTION means the receiver echoed an explicit congestion notification message 🚨. and CM_NO_CONGESTION means no congestion-related loss happened at all 🌤️. the lossmode parameter MUST be reported as a bit-vector, where the bits correspond to CM_NO_FEEDBACK, CM_LOSS_FEEDBACK, CM_EXPLICIT_CONGESTION, and CM_NO_CONGESTION 🔢. also, over links (paths) that lose packets for reasons other than congestion, an application SHOULD tell the CM about the losses with the CM_NO_CONGESTION field set 🫶.

cm_notify(i32 cm_streamid, u32 nsent) MUST be called whenever data gets transmitted from the host (for example, in the IP output routine) 📤. it tells the CM that nsent bytes were just sent on a given stream. that lets the CM keep its estimate of the outstanding bytes for the macroflow and for the stream up to date 📊.

a cmapp_send() grant from the CM is only valid until an expiration time. that time is the larger of the round-trip time and an implementation-dependent threshold passed as an argument to the cmapp_send() callback function ⏳. the application MUST NOT send data based on that callback after the time has expired 🛑. and if the app decides not to send after getting the callback, it SHOULD call cm_notify(stream_info, 0) so the CM can let other streams in the macroflow transmit 🙋‍♀️. the CM congestion controller MUST be robust to applications that forget to call cm_notify(stream_info, 0) correctly, or to applications that crash or disappear after making a cm_request() call 💀🛡️. this part matters a lot, so don't skip it, bestie 🔥.

3.4 Querying 🔎

if apps want to know the per-stream available bandwidth and round-trip time, they can call the CM's cm_query(i32 cm_streamid, i64* rate, i32* srtt, i32* rttdev), which fills in the values they asked for 🎯. just like before, if the CM doesn't have valid estimates for the macroflow, it fills in negative values for rate, srtt, and rttdev 🚫.

one of the decisions the CM has to make is the granularity of a macroflow. that means deciding which streams belong to the same macroflow and share congestion information 🤝. the API gives apps two functions to decide which of their streams should be in the same macroflow 👯.

cm_getmacroflow(i32 cm_streamid) returns a unique i32 macroflow identifier 🪪. cm_setmacroflow(i32 cm_macroflowid, i32 cm_streamid) sets the macroflow of the stream cm_streamid to cm_macroflowid. if the cm_macroflowid passed to cm_setmacroflow() is -1, a new macroflow gets constructed and returned to the caller 🆕. every call to cm_setmacroflow() overrides the stream's previous macroflow association, if it had one 🔄.

the suggested default aggregation method is by destination IP address. so all streams going to the same destination address get bundled into a single macroflow by default 📦. cm_getmacroflow() and cm_setmacroflow() can then change that whenever needed ✏️. there are cases where this might not be ideal, even over best-effort networks 🌫️. for example, if a group of receivers sits behind a NAT device, the sender sees them all as one address. and if the hosts behind the NAT are actually connected over different bottleneck links, some of them could end up with worse performance than before 😬. it's possible to detect those hosts using delay and loss estimates, but the specific mechanisms for doing that are out of scope for this document 🙅‍♀️.

the goal of this interface is to set up sharing groups, not the sharing policy of relative weights of streams in a macroflow ⚖️. the policy part needs the scheduler to provide an interface for setting it. but since the CM wants to support many different schedulers (each of which might need different information to set policy), it doesn't specify a complete API to the scheduler (see Section 5.2 though 👀). a later guideline document is expected to describe a few simple schedulers, like weighted round-robin and hierarchical scheduling, along with the APIs they export for relative prioritization 📚.

4. CM internals 🛠️

this section describes the internal components of the CM 🔧. it includes a Congestion Controller and a Scheduler, and both have well-defined, abstract interfaces that they export 💖.

4.1 Congestion controller 💅

okay so every macroflow has a congestion control algorithm attached to it 🦋, and all of those algorithms together make up the congestion controller of the CM. the control algorithm is the one deciding when a macroflow can send and how much it can send 💖. it's giving brain. it listens to application notifications (Section 4.3) from all the streams riding on the same macroflow, and uses them to figure out what the congestion state of the network path looks like 👀.

the congestion controller MUST implement a "TCP-friendly" [Mahdavi98] congestion control algorithm. ✨ several macroflows MAY (and honestly, they often will) use the same congestion control algorithm, but each macroflow still keeps its own state about the network its streams are using 🌸.

the congestion control module MUST implement the following abstract interfaces. ✨ lowkey important: these aren't visible to applications at all. they live inside the context of a macroflow, and they're different from the CM API functions from Section 4. 🙈

void query(u64 *rate, u32 *srtt, u32 *rttdev): this returns the estimated rate (in bits per second) and the smoothed round trip time (in microseconds) for the macroflow. 🍓
void notify(u32 nsent): this MUST be used to tell the congestion control module whenever data is sent by an application. the nsent parameter is the number of bytes the application just sent. 📤
void update(u32 nsent, u32 nrecd, u32 rtt, u32 lossmode): this gets called whenever any of the CM streams on a macroflow spots that data either reached the receiver or got lost en route 📦. nrecd is the number of bytes that just arrived at the receiver. nsent is the sum of the bytes just received and the bytes identified as lost en route. rtt is the estimated round trip time in microseconds during the transfer. lossmode is an indicator of how a loss was detected (section 4.3).

although these interfaces aren't visible to applications, the congestion controller MUST implement these abstract interfaces so that it can play nicely with different separately-developed schedulers 🫶. that's the whole modular inter-operability vibe ✨.

the congestion control module MUST also call the associated scheduler's schedule function (section 5.2) when it believes that the current congestion state allows an MTU-sized packet to be sent. 🚦

4.2 Scheduler 🧚

okay this one's the other half of the team 👑. while the congestion control module is in charge of deciding when and how much data can be transmitted, the scheduler module of a macroflow is in charge of deciding which of the streams gets the chance to transmit data. it's a whole division of labour and I'm obsessed 💜.

The Scheduler MUST implement the following interfaces:

void schedule(u32 num_bytes): when the congestion control module determines that data can be sent, the schedule() routine MUST be called with no more than the number of bytes that can be sent. in turn, the scheduler MAY call the cmapp_send() function that CM applications must provide. 🎀
float query_share(i32 cm_streamid): this call returns the described stream's share of the total bandwidth available to the macroflow. combined with the query call of the congestion controller, this gives the information needed to satisfy an application's cm_query() request 💅.
void notify(i32 cm_streamid, u32 nsent): this interface is used to notify the scheduler module whenever data is sent by a CM application. the nsent parameter is the number of bytes the application just sent 📨.

The Scheduler MAY implement many additional interfaces. ✨ as experience with CM schedulers grows, future documents may make additions and/or changes to some parts of the scheduler API. the future is bright 🌈.

5. Examples 🎀

5.1 Example applications 🦋

this section walks through three possible uses of the CM API by applications 🌸. two of them are asynchronous: an implementation of a TCP sender, and an implementation of congestion-controlled UDP sockets. the third one is synchronous: a streaming audio server 🎧. if you want more details on these apps and the CM implementation optimizations that make them run efficiently, check out [Andersen00] 💖.

all applications that use the CM MUST incorporate feedback from the receiver. 💌 for example, an app must periodically (typically once or twice per round trip time) figure out how many of its packets arrived at the receiver. when the source gets this feedback, it MUST use cm_update() to tell the CM about this new information 📬. this results in the CM updating ownd, and it may result in the CM changing its estimates and calling cmapp_update() of the streams of the macroflow 🔄.

the protocols in this section are examples and suggestions for implementation 💡. they are not requirements for any conformant implementation, so no pressure, bestie 🫶.

5.1.1 TCP 🌊

a TCP implementation that uses CM should use the cmapp_send() callback API. TCP only figures out what data it should send when an acknowledgement arrives or a timer expires 🕰️. as a result, it needs tight control over when and if new data or retransmissions get sent 🔥.

when TCP either connects to or accepts a connection from another host, it performs a cm_open() call to associate the TCP connection with a cm_streamid 🔗.

once a connection is established, the CM controls the transmission of outgoing data ✨. the CM gets rid of the need for TCP to track and react to congestion itself, because the CM and its transmission API make sure the congestion behaviour is proper 👑. loss recovery is still handled by TCP, using fast retransmissions and recovery as well as timeouts 🛠️. TCP is also modified to keep its own outstanding window estimate, called tcp_ownd. whenever data segments are sent from its cmapp_send() callback, TCP updates its tcp_ownd value 📈. the ownd variable also gets updated after each cm_update() call. TCP keeps a count of outstanding segments too, called pkt_cnt 🔢. at any time, TCP can calculate the average packet size, avg_pkt_size, as tcp_ownd/pkt_cnt 🧮. TCP uses avg_pkt_size to help estimate the amount of outstanding data. note that none of this is needed if the SACK option is used on the connection, since that information is explicitly available 🙌.

the TCP output routines are modified as follows:

all congestion window (cwnd) checks are removed. bye bye, congestion checks 👋.
when application data is available, the TCP output routines perform all the non-congestion checks (Nagle algorithm, receiver-advertised window check, etc). if these checks pass, the output routine queues the data and calls cm_request() for the stream 📝.
if incoming data or timers result in a loss being detected, the retransmission is also placed in a queue, and cm_request() is called for the stream 🔁.
the cmapp_send() callback for TCP is set to an output routine. if any retransmission is enqueued, the routine outputs the retransmission. otherwise, the routine outputs as much new data as the TCP connection state allows. however, cmapp_send() never sends more than a single segment per call 🍰. this routine also arranges for the other output computations to be done, such as header and options computations ⚙️.

the IP output routine on the host calls cm_notify() when the packets are actually sent out 📤. because it doesn't know which cm_streamid is responsible for the packet, cm_notify() takes the stream_info as an argument (see Section 4 for what the stream_info should contain). because cm_notify() reports the IP payload size, TCP keeps track of the total header size and folds these updates in 🧾.

the TCP input routines are modified as follows:

RTT estimation is done as normal, using either timestamps or Karn's algorithm. any rtt estimate that gets generated is passed to the CM via the cm_update call ⏱️.
all cwnd and slow start threshold (ssthresh) updates are removed 🗑️.
upon the arrival of an ack for new data, TCP computes the value of in_flight (the amount of data in flight) as snd_max-ack-1 (i.e., MAX Sequence Sent - Current Ack - 1). TCP then calls cm_update(streamid, tcp_ownd - in_flight, 0, CM_NO_CONGESTION, rtt) 📊.
upon the arrival of a duplicate acknowledgement, TCP must check its dupack count (dup_acks) to decide what to do 🕵️. if dup_acks < 3, TCP does nothing. if dup_acks == 3, TCP assumes that a packet was lost and that at least 3 packets arrived to generate these duplicate acks. so it calls cm_update(streamid, 4 * avg_pkt_size, 3 * avg_pkt_size, CM_LOSS_FEEDBACK, rtt). the average packet size is used because the acknowledgments don't indicate exactly how much data reached the other end 🤷‍♀️. most TCP implementations interpret a duplicate ACK as an indication that a full MSS has reached its destination. once a new ACK is received, these TCP sender implementations may resynchronize with the TCP receiver. the CM API doesn't provide a mechanism for TCP to pass information from this resynchronization, so TCP can only infer the arrival of an avg_pkt_size amount of data from each duplicate ack 🧩. TCP also enqueues a retransmission of the lost segment and calls cm_request(). if dup_acks > 3, TCP assumes that a packet has reached the other end and caused this ack to be sent. as a result, it calls cm_update(streamid, avg_pkt_size, avg_pkt_size, CM_NO_CONGESTION, rtt) ✅.
upon the arrival of a partial acknowledgment (one that does not exceed the highest segment transmitted at the time the loss occurred, as defined in [Floyd99b]), TCP assumes that a packet was lost and that the retransmitted packet has reached the recipient 🎯. so it calls cm_update(streamid, 2 * avg_pkt_size, avg_pkt_size, CM_NO_CONGESTION, rtt). CM_NO_CONGESTION is used since the loss period has already been reported 🙃. TCP also enqueues a retransmission of the lost segment and calls cm_request().

when the TCP retransmission timer expires, the sender identifies that a segment has been lost 😱. it calls cm_update(streamid, avg_pkt_size, 0, CM_NO_FEEDBACK, 0) to signify that no feedback has been received from the receiver, and that one segment is sure to have "left the pipe" 🚀. TCP also enqueues a retransmission of the lost segment and calls cm_request().

5.1.2 Congestion-controlled UDP 🦋

okay so congestion-controlled UDP is a super useful CM app, and we're going to explain it using Berkeley sockets [Stevens94] 💅. These sockets give you the exact same functionality as normal Berkeley UDP sockets, but there's a twist. Instead of immediately shipping data from the kernel packet queue down to the lower layers, the buffered socket implementation calls the CM API inside the kernel and gets callbacks back from the CM. 🔥

Here's how the flow goes. When a CM UDP socket gets created, it's bound to one particular stream. Later, when data gets added to the packet queue, cm_request() is called on the stream linked to that socket. When the CM schedules that stream for transmission, it calls udp_ccappsend() in the UDP module. That function sends one MTU from the packet queue and schedules the transmission of whatever packets are left over. ✨

The in-kernel implementation of the CM UDP API should not require any additional data copies, and it should support all standard UDP options. 🎀 Modifying existing applications to use congestion-controlled UDP means implementing a new socket option on the socket.

Now, to work correctly, the sender has to get feedback about congestion, and there are at least two ways to make that happen 🌸:

(i) the UDP receiver application can give feedback to the sender application, which then informs the CM of network conditions using cm_update();
(ii) the UDP receiver implementation can give feedback straight to the sending UDP.

ngl the second option is the trickier one. It needs changes to the receiver's network stack, and the sender UDP cannot assume that every receiver supports this option without explicit negotiation. 🙈

5.1.3 Audio server 🎧

A typical audio application often has access to the sample in a multitude of data rates and qualities. 🍓 The goal is to deliver the highest possible quality of audio (typically the highest data rate) to its clients. The choice of which version of audio to transmit should be based on the current congestion state of the network. On top of that, the source will want audio delivered to its users at a consistent sampling rate. So it has to send data at a regular rate, minimizing delayed transmissions and reducing buffering before playback. To meet these requirements, this application can use the synchronous sender API (Section 4.2). 👑

When the source first starts up, it uses the cm_query() call to get an initial estimate of network bandwidth and delay. If some other streams on that macroflow have already been active, then it gets an initial estimate that is valid. Otherwise, it gets negative values, which it ignores. ✨ It then picks an encoding that does not exceed these estimates (or, in the case of an invalid estimate, uses application-specific initial values) and begins transmitting data.

The application also implements the cmapp_update() callback. When the CM determines that network characteristics have changed, it calls the application's cmapp_update() function and passes it a new rate and round-trip time estimate. The application must change its choice of audio encoding so that it does not exceed these new estimates. 🫶

5.2 Example congestion control module 🧚

To illustrate the responsibilities of a congestion control module, here are some of the actions of a simple TCP-like congestion control module that implements Additive Increase Multiplicative Decrease congestion control (AIMD_CC). It's giving TCP vibes, bestie 💜.

query(): AIMD_CC returns the current congestion window (cwnd) divided by the smoothed rtt (srtt) as its bandwidth estimate. It returns the smoothed rtt estimate as srtt.
notify(): AIMD_CC adds the number of bytes sent to its outstanding data window (ownd).
update(): AIMD_CC subtracts nsent from ownd. If the value of rtt is non-zero, AIMD_CC updates srtt using the TCP srtt calculation. If the update indicates that data has been lost, AIMD_CC sets cwnd to 1 MTU if the loss_mode is CM_NO_FEEDBACK, and to cwnd/2 (with a minimum of 1 MTU) if the loss_mode is CM_LOSS_FEEDBACK or CM_EXPLICIT_CONGESTION. AIMD_CC also sets its internal ssthresh variable to cwnd/2. If no loss had occurred, AIMD_CC mimics TCP slow start and linear growth modes. It increments cwnd by nsent when cwnd < ssthresh (bounded by a maximum of ssthresh-cwnd), and by nsent * MTU/cwnd when cwnd > ssthresh. 📈
When cwnd or ownd are updated and indicate that at least one MTU may be transmitted, AIMD_CC calls the CM to schedule a transmission. ⭐

5.3 Example Scheduler Module 🌈

To clarify the responsibilities of a scheduler module, here are some of the actions of a simple round robin scheduler module (RR_sched). It's very fair, very organized, very not-drama 💅.

schedule(): RR_sched schedules as many streams as possible in round robin fashion.
query_share(): RR_sched returns 1/(number of streams in macroflow).
notify(): RR_sched does nothing. Round robin scheduling is not affected by the amount of data sent. 🙃

6. Security Considerations 🚨

okay this section is SERIOUS, bestie 😭. The CM provides many of the same services that congestion control in TCP provides, so it is vulnerable to many of the same security problems. For example, incorrect reports of losses and transmissions will give the CM an inaccurate picture of the network's congestion state. By giving the CM a high estimate of congestion, an attacker can degrade the performance observed by applications. For example, a stream on a host can arbitrarily slow down any other stream on the same macroflow, which is a form of denial of service. 💀

The more dangerous form of attack is when an application gives the CM a low estimate of congestion. That would make the CM overly aggressive and allow data to be sent much more quickly than sound congestion control policies would allow. Not it. 🙅‍♀️

[Touch97] describes a number of the security problems that arise with congestion information sharing. There's also an additional vulnerability (not covered by [Touch97])) that comes from the fact that applications have access through the CM API to control shared state, and that shared state will affect other applications on the same computer. For instance, a poorly designed, possibly a compromised, or intentionally malicious UDP application could misuse cm_update() to cause starvation and/or too-aggressive behavior of others in the macroflow. Main character villain energy, and we really don't want that 💔.

7. References 📚

[Allman99] Allman, M. and Paxson, V., "TCP Congestion Control", RFC 2581, April 1999.
[Andersen00] Balakrishnan, H., System Support for Bandwidth Management and Content Adaptation in Internet Applications, Proc. 4th Symp. on Operating Systems Design and Implementation, San Diego, CA, October 2000. Available from http://nms.lcs.mit.edu/papers/cm-osdi2000.html
[Balakrishnan98] Balakrishnan, H., Padmanabhan, V., Seshan, S., Stemm, M., and Katz, R., "TCP Behavior of a Busy Web Server: Analysis and Improvements," Proc. IEEE INFOCOM, San Francisco, CA, March 1998.
[Balakrishnan99] Balakrishnan, H., Rahul, H., and Seshan, S., "An Integrated Congestion Management Architecture for Internet Hosts," Proc. ACM SIGCOMM, Cambridge, MA, September 1999.
[Bradner96] Bradner, S., "The Internet Standards Process --- Revision 3", BCP 9, RFC 2026, October 1996.
[Bradner97] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[Clark90] Clark, D. and Tennenhouse, D., "Architectural Consideration for a New Generation of Protocols", Proc. ACM SIGCOMM, Philadelphia, PA, September 1990.
[Eggert00] Eggert, L., Heidemann, J., and Touch, J., "Effects of Ensemble TCP," ACM Computer Comm. Review, January 2000.
[Floyd99a] Floyd, S. and Fall, K.," Promoting the Use of End-to-End Congestion Control in the Internet," IEEE/ACM Trans. on Networking, 7(4), August 1999, pp. 458-472.
[Floyd99b] Floyd, S. and T. Henderson,"The New Reno Modification to TCP's Fast Recovery Algorithm," RFC 2582, April 1999.
[Jacobson88] Jacobson, V., "Congestion Avoidance and Control," Proc. ACM SIGCOMM, Stanford, CA, August 1988.
[Mahdavi98] Mahdavi, J. and Floyd, S., "The TCP Friendly Website," http://www.psc.edu/networking/tcp_friendly.html
[Mogul90] Mogul, J. and S. Deering, "Path MTU Discovery," RFC 1191, November 1990.
[Padmanabhan98] Padmanabhan, V., "Addressing the Challenges of Web Data Transport," PhD thesis, Univ. of California, Berkeley, December 1998.
[Paxson00] Paxson, V. and M. Allman, "Computing TCP's Retransmission Timer", RFC 2988, November 2000.
[Postel81] Postel, J., Editor, "Transmission Control Protocol", STD 7, RFC 793, September 1981.
[Ramakrishnan99] Ramakrishnan, K. and Floyd, S., "A Proposal to Add Explicit Congestion Notification (ECN) to IP," RFC 2481, January 1999.
[Stevens94] Stevens, W., TCP/IP Illustrated, Volume 1. Addison-Wesley, Reading, MA, 1994.
[Touch97] Touch, J., "TCP Control Block Interdependence", RFC 2140, April 1997.

8. Acknowledgments 🫶

We thank David Andersen, Deepak Bansal, and Dorothy Curtis for their work on the CM design and implementation. We thank Vern Paxson for his detailed comments, feedback, and patience, and Sally Floyd, Mark Handley, and Steven McCanne for useful feedback on the CM architecture. Allison Mankin and Joe Touch provided several useful comments on previous drafts of this document. Such a supportive group chat 💖.

9. Authors' Addresses 💌

Hari Balakrishnan
Laboratory for Computer Science
200 Technology Square
Massachusetts Institute of Technology
Cambridge, MA 02139

EMail: hari@lcs.mit.edu
Web: http://nms.lcs.mit.edu/~hari/


Srinivasan Seshan
School of Computer Science
Carnegie Mellon University
5000 Forbes Ave.
Pittsburgh, PA 15213

EMail: srini@cmu.edu
Web: http://www.cs.cmu.edu/~srini/

Full Copyright Statement 💖

okay bestie, the legal bit at the end, and it's important so read it 🫶

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

ngl this is the part people skip, but the gist is: you can share and remix it (even turn it into your own explainer 💅), but you can't edit the original doc itself, and the permissions are forever 🌈

Acknowledgement 🫶

Funding for the RFC Editor function is currently provided by the Internet Society. 🌸 Literally the people keeping the whole RFC library running, so shoutout to them ✨

📦 where this page comes from

ietf: rfc3124 2001-06
💖 girlified by claude-haiku-5-5 on 2026-10-16: a language model's rewrite of the original, under its terms