BPFTUNE-TCP-CONN(8)

💖 BPFTUNE-TCP-CONN 💖

okay bestie, listen up — this is BPFTUNE-TCP-CONN, a bpftune plugin living in section 8 of the System Manager's Manual. it does exactly one absolutely iconic thing: it auto-picks your congestion control algorithm for TCP, and it's so smart about it. no cap. 🦋

---

✨ DESCRIPTION ✨

so the TCP connection algorithm tuner sets the congestion control algorithm on TCP sockets. Linux ships with cubic as the default, and honestly? it works well in a wide range of settings. she's reliable, she's the group's responsible friend. 🫶

but — and this is the tea — when losses show up, cubic can underestimate network capacity, and as a result throughput can drop excessively. like, dramatically. 😭 in those cases, BBR is a good fit, because it continuously estimates bottleneck bandwidth and tries to fit the congestion algorithm to it.

when we have limited info about a destination — i.e. we have not had > REMOTE_HOST_MIN_INSTANCES connections involving it — the only auto-selection involved is to use BBR in cases where loss rates exceed 1/(2^DROP_SHIFT) (1.5%) of the packet sent rate. in those situations, BBR performs better than the other congestion control algorithms. periodt.

but for cases where we connect multiple times? we can try different algorithms to select the best one. 💅

---

🧠 the reinforcement learning of it all

when selecting the appropriate congestion control algorithm, a reinforcement learning-based method is used, where we choose the congestion control algorithm that best fits the optimal bandwidth delay product (BDP):

BDP = BottleneckBandwidth * MinRoundTripTime

the algorithm works as follows: BPF maintains a map of metrics keyed by remote gateway. for each remote gateway we track:

• the minimum RTT observed across all TCP connections, and
• the max bandwidth observed.

the former tells us — as closely as we can determine — what the true RTT of the link is. the latter estimates the bandwidth limit of the link. knowing both of these lets us determine the optimum operating state for a congestion control algorithm, where we feed the pipe enough to reach bandwidth limits but do not overwhelm it. 💖

tracking both of these allows us to determine that optimum BDP, so any loss function we use for optimization should drive us towards congestion control algorithms that realize that optimal BDP by being as close as possible to the minimum RTT and as close as possible to the maximum packet delivery rate. we cannot use raw BDP alone, because it's composed of the delivery rate and the RTT — so instead, the metric used is:

(current_min_rtt - overall_min_rtt)*S/overall_min_rtt +
(overall_max_delivery_rate - cong_alg_max_delivery_rate)*S/overall_max_delivery_rate

both denominators are scaled by a scaling factor S to make sure integer division yields nonzero values. see ../src/tcp_conn_tuner.h for the metric computation. (the original page has a lil typo there and honestly? we respect the typo. 🎀)

ngl, one thing to clock: while we use the current RTT for the connection, we use the maximum delivery rate observed for the congestion algorithm to compare with the overall maximum. the reasoning here is that because the delivery rate fluctuates so much for different connections (depending on service type etc), it's too unstable to use it on a per-connection basis. RTT is less variable across connections, so we can use the current RTT in metric calculation.

for a TCP connection with optimal BDP (minimum RTT + max delivery rate), the loss function yields 0. otherwise it yields a positive cost. this is used to update the cost for that congestion control algorithm via the usual reinforcement learning algorithm, i.e.:

cong_alg_cost = cong_alg_cost +
                learning_rate*(curr_cost - cong_alg_cost)

we use an epsilon-greedy approach, whereby the vast majority of the time the lowest-cost algorithm is used, but 5% of the time we randomly select an algorithm. this ensures that if network conditions change we can adapt accordingly — without this, we can get stuck and never discover that another algorithm is doing better. main character energy: always be exploring. 🔥

---

🧪 how does this work in practice?

to benchmark this we tested iperf3 performance between network namespaces on the same system, with a 10% loss rate imposed via netem. what we see is that bpftune converges to using BBR:

IPAddress      CongAlg     Metric    Count   Greedy   MinRtt MaxRtDlvr
192.168.168.1    cubic    2338876        9        9        3     1737
192.168.168.1      bbr     923173       61       59        3    10024
192.168.168.1     htcp    2318283        5        4        3      620
192.168.168.1    dctcp    3506360        3        1        9      160

not me noticing that we selected the BBR congestion control algorithm 61 out of 78 times and its associated cost was less than half of that of the other algorithms. that's because it exhibited the maximum delivery rate and the lowest RTT. she ate. 👑

iperf3 performance also improved as a result of selecting BBR — from a baseline of 58MBit/Sec (running the Linux default cubic algorithm) to 490MBit/Sec running bpftune and auto-selecting BBR. 📈✨

so this approach appears to find the right answer and converge quickly under loss conditions. but what about normal network conditions? 🤔

we might worry that grounding our model in assumptions closely tied to BBR's design might unduly favour BBR in all circumstances — do we see this in practice outside of conditions where BBR is optimal?

thankfully no; we see a convergence to dctcp as the optimal congestion control algorithm. again, it has the maximum delivery rate and minimum RTT:

IPAddress      CongAlg     Metric    Count   Greedy   MinRtt MaxRtDlvr
192.168.168.1    cubic    1710535        6        4        3     8951
192.168.168.1      bbr    2309881        1        1        7      206
192.168.168.1     htcp    3333333        3        3        3     8784
192.168.168.1    dctcp    1466296       71       70        3     9377

note however that it is a close-run thing 😬 — the metric for cubic is close, and it matches dctcp for minimum RTT (3us), and the maximum delivery rate is close (9377 for dctcp, 8951 for cubic).

in addition, when retransmits occur for a TCP connection, we enable TCP thin linear timeouts to improve responsiveness for thin TCP connections. slay. 💅

---

🌈 references 🌈

BBR: Congestion-Based Congestion Control

<https://queue.acm.org/detail.cfm?id=3022184>

---

👀 SEE ALSO 👀

bpf(2), bpftune(8),

---

okay that's the whole page, bestie — go forth and let the kernel pick its own congestion control algorithm, she's got this. 🦋🌸

this was BPFTUNE-TCP-CONN(8) from the System Manager's Manual. mwah. 💖 ✨ 🎀