MOD_CC(9)

💅 MOD_CC(9) — Modular Congestion Control (Kernel Developer's Manual, bestie edition)

okay so picture this: you're a kernel dev, you want to write a congestion control algorithm, and you do not want to rebuild the whole kernel for it. Enter mod_cc. This is the <pink>Kernel Developer's Manual</pink> page for it. Grab a snack, we're going in 🍓

---

✨ NAME ✨

mod_cc, DECLARE_CC_MODULE, CCV — Modular Congestion Control. That's the whole vibe.

---

🎀 SYNOPSIS 🎀

#include <netinet/tcp.h>
#include <netinet/cc/cc.h>
#include <netinet/cc/cc_module.h>

DECLARE_CC_MODULE(ccname, ccalgo);

CCV(ccv, what);

---

🦋 DESCRIPTION 🦋

So the mod_cc framework is literally the glow-up that lets congestion control algorithms be implemented as dynamically loadable kernel modules via the kld(4) facility. Transport protocols get to pick from the list of available algorithms connection-by-connection, or just vibe with the system default (peep mod_cc(4) for the deets).

mod_cc modules are identified by an ascii(7) name plus a set of hook functions, all wrapped up in a struct cc_algo, which has these members 👇

struct cc_algo {
        char    name[TCP_CA_NAME_MAX];
        int     (*mod_init) (void);
        int     (*mod_destroy) (void);
        size_t  (*cc_data_sz)(void);
        int     (*cb_init) (struct cc_var *ccv, void *ptr);
        void    (*cb_destroy) (struct cc_var *ccv);
        void    (*conn_init) (struct cc_var *ccv);
        void    (*ack_received) (struct cc_var *ccv, uint16_t type);
        void    (*cong_signal) (struct cc_var *ccv, uint32_t type);
        void    (*post_recovery) (struct cc_var *ccv);
        void    (*after_idle) (struct cc_var *ccv);
        int     (*ctl_output)(struct cc_var *, struct sockopt *, void *);
        void    (*rttsample)(struct cc_var *, uint32_t, uint32_t, uint32_t);
        void    (*newround)(struct cc_var *, uint32_t);
};

Let's meet the cast 🫶

• name — the unique name of the algorithm, and she should be no longer than TCP_CA_NAME_MAX-1 characters in length (the TCP_CA_NAME_MAX define lives in <netinet/tcp.h> for compatibility reasons — yes, that's where it's crashing, don't ask).
• mod_init — called when a new module is loaded into the system, but before the registration process is complete. Implement it if your module needs to set up some global state before new connections can use it. ⚠️ Returning a non-zero value from mod_init will cause the loading of the module to fail. No pressure.
• mod_destroy — called prior to unloading an existing module from the kernel. Implement it if your module needs to clean up any global state before getting yeeted out of the kernel. The return value is currently ignored (so, no consequences, but also no rewards 💀).
• cc_data_sz — called by the socket option code to get the size of data that the cb_init function needs. The socket option code then preallocates the module's memory so that cb_init will not fail (the socket option code uses M_WAITOK with no locks held to do this).
• cb_init — called when a TCP control block struct tcpcb is created. Implement it if your module needs to allocate memory for storing private per-connection state. ⚠️ Returning a non-zero value from cb_init will cause the connection set up to be aborted, terminating the connection as a result. Also, note that the ptr argument passed to the function should be checked to see if it is non-NULL — if so, it is preallocated memory that cb_init must use instead of calling malloc itself. Don't be that guy.
• cb_destroy — called when a TCP control block struct tcpcb is destroyed. Implement it if your module needs to free memory allocated in cb_init. Clean up after yourself, icon behavior.
• conn_init — called when a new connection has been established and variables are being initialised. Implement it to initialise congestion control algorithm variables for the newly established connection.
• ack_received — called when a TCP acknowledgement (ACK) packet is received. Modules use the type argument as an input to their congestion management algorithms. The ACK types currently reported by the stack are CC_ACK and CC_DUPACK. CC_ACK means the received ACK acknowledges previously unacknowledged data. CC_DUPACK means the received ACK acknowledges data we have already received an ACK for.
• cong_signal — called when a congestion event is detected by the TCP stack. Modules use the type argument as an input to their congestion management algorithms. The congestion event types currently reported by the stack are CC_ECN, CC_RTO, CC_RTO_ERR and CC_NDUPACK.
  • CC_ECN is reported when the TCP stack receives an explicit congestion notification (RFC3168).
  • CC_RTO is reported when the retransmission time out timer fires.
  • CC_RTO_ERR is reported if the retransmission time out timer fired in error (oop 🙈).
  • CC_NDUPACK is reported if N duplicate ACKs have been received back-to-back, where N is the fast retransmit duplicate ack threshold (N=3 currently as per RFC5681).
• post_recovery — called after the TCP connection has recovered from a congestion event. Implement it to adjust state as required. She's healed, now fix her makeup.
• after_idle — called when data transfer resumes after an idle period. Implement it to adjust state as required.
• ctl_output — called when getsockopt(2) or setsockopt(2) is called on a tcp(4) socket, with the struct sockopt pointer forwarded unmodified from the TCP control, and a void * pointer to algorithm specific argument.
• rttsample — called to pass round trip time information to the congestion controller. The additional arguments to the function include the microsecond RTT that is being noted, the number of times that the data being acknowledged was retransmitted, as well as the flightsize at send. For transports that do not track flightsize at send, this variable will be the current cwnd at the time of the call.
• newround — called each time a new round trip time begins. The monotonically increasing round number is also passed to the congestion controller as well. This can be used for various purposes by the congestion controller (e.g. Hystart++).

🚨 reality check bestie: currently not all TCP stacks call the rttsample and newround function, so dependency on these functions is also dependent upon which TCP stack is in use. Don't build your whole personality around them, k?

---

The DECLARE_CC_MODULE() macro is the convenience wrapper around the DECLARE_MODULE(9) macro, and it's what you use to register a mod_cc module with the mod_cc framework. The ccname argument specifies the module's name. The ccalgo argument points to the module's struct cc_algo.

mod_cc modules must instantiate a struct cc_algo, but are only required to set the name field, and optionally any of the function pointers. Note that if a module defines the cb_init function it also must define a cc_data_sz function — this is because when switching from one congestion control module to another, the socket option code will preallocate memory for the cb_init function. If no memory is allocated by the module's cb_init, then the cc_data_sz function should return 0.

The stack will skip calling any function pointer which is NULL, so there is no requirement to implement any of the function pointers (with the exception of the cb_init ↔ cc_data_sz dependency noted above — that one's non-negotiable). Using the C99 designated initialiser feature to set fields is encouraged ✨

---

Each function pointer which deals with congestion control state gets handed a pointer to a struct cc_var, which has these members:

struct cc_var {
        void            *cc_data;
        int             bytes_this_ack;
        tcp_seq         curack;
        uint32_t        flags;
        int             type;
        union ccv_container {
                struct tcpcb            *tcp;
                struct sctp_nets        *sctp;
        } ccvc;
        uint16_t        nsegs;
        uint8_t         labc;
};

struct cc_var groups congestion control related variables into a single, embeddable structure and adds a layer of indirection to accessing transport protocol control blocks. The eventual goal is to allow a single set of mod_cc modules to be shared between all congestion aware transport protocols — though currently only tcp(4) is supported. Baby steps, we'll get there 💜

To aid the eventual transition towards this goal, direct use of variables from the transport protocol's data structures is strongly discouraged. However, it is inevitable at the current time to require access to some of these variables, so the CCV() macro exists as a convenience accessor. The ccv argument points to the struct cc_var passed into the function by the mod_cc framework. The what argument specifies the name of the variable to access.

Apart from the type and ccv_container fields, the remaining fields in struct cc_var are for use by mod_cc modules 👇

• cc_data — available for algorithms requiring additional per-connection state to attach a dynamic memory pointer to. The memory should be allocated and attached in the module's cb_init hook function.
• bytes_this_ack — the number of new bytes acknowledged by the most recently received ACK packet. It is only valid in the ack_received hook function.
• curack — the sequence number of the most recently received ACK packet. It is only valid in the ack_received, cong_signal and post_recovery hook functions.
• flags — used to pass useful information from the stack to a mod_cc module.
  • The CCF_ABC_SENTAWND flag is relevant in ack_received and is set when appropriate byte counting (RFC3465) has counted a window's worth of bytes has been sent. It is the module's responsibility to clear the flag after it has processed the signal — yes, that's on you, no one else is gonna do it.
  • The CCF_CWND_LIMITED flag is relevant in ack_received and is set when the connection's ability to send data is currently constrained by the value of the congestion window. Algorithms should use the absence of this flag being set to avoid accumulating a large difference between the congestion window and send window. iykyk.
• nsegs — used to pass in how much compression was done by the local LRO system. So for example, if LRO pushed three in-order acknowledgements into one acknowledgement, the variable would be set to three.
• labc — used in conjunction with the CCF_USE_LOCAL_ABC flag to override what labc variable the congestion controller will use for this particular acknowledgement.

---

🌈 SEE ALSO 🌈

cc_cdg(4), cc_chd(4), cc_cubic(4), cc_dctcp(4), cc_hd(4), cc_htcp(4), cc_newreno(4), cc_vegas(4), mod_cc(4), tcp(4)

(the whole friend group, fr fr 👑)

---

🧚 ACKNOWLEDGEMENTS 🧚

Development and testing of this software were made possible in part by grants from the FreeBSD Foundation and Cisco University Research Program Fund at Community Foundation Silicon Valley. Legends, honestly 🫶

---

⭐ FUTURE WORK ⭐

Integrate with sctp(4). It's giving "on the roadmap", bestie. One day 💅

---

to be continued… part 2 is coming, don't go anywhere 🦋

🌸 HISTORY 🌸

okay so storytime, bestie 🎀 the modular Congestion Control (CC) framework? it first appeared in FreeBSD 9.0. like, that's its debut era.

but wait, the lore goes deeper 👀 the framework was first released in 2007 by James Healy and Lawrence Stewart, whilst (yes, whilst, we're being fancy) working on the NewTCP research project at Swinburne University of Technology's Centre for Advanced Internet Architectures, Melbourne, Australia 🇦🇺. and that whole thing was made possible in part by a grant from the Cisco University Research Program Fund at Community Foundation Silicon Valley. slay, funding 💅

more details are available at:

http://caia.swin.edu.au/urp/newtcp/

like, if you want the full director's cut, that's where it lives ✨

---

👑 AUTHORS 👑

the mod_cc framework was written by these absolute icons 🫶:

• Lawrence Stewart <lstewart@FreeBSD.org>
• James Healy <jimmy@deefa.com>
• David Hayes <david.hayes@ieee.org>

and this manual page (yes, the one you're reading rn, the one with the vibes) was written by:

• David Hayes <david.hayes@ieee.org>
• Lawrence Stewart <lstewart@FreeBSD.org>

periodt 💜

---

and that's the whole page, bestie!! this lil beauty is from the Debian crew, dated May 13, 2021 — go forth and congest responsibly 🦋💖