RFC 9040

TCP Control Block Interdependence

📅 published July 2021, from ietf

RFC 9040: TCP Control Block Interdependence 🎀

okay bestie, this is the RFC about how TCP connections can borrow notes from each other so they get to a good speed faster 🦋

  • Who wrote it: J. Touch (Independent), M. Welzl, and S. Islam (University of Oslo) ✨
  • Category: Informational 💅
  • Date: July 2021 🌸
  • Series: IETF, ISSN 2070-1721 💜
  • Obsoletes: RFC 2140 🍓 (the abstract also says it updates and replaces RFC 2140's description of sharing TCP state)

Abstract ✨

so this memo gives TCP implementers some guidance to help connections settle into steady-state operation faster, and it does that without breaking interoperability 🫶 it updates and replaces RFC 2140's description of sharing TCP state, the stuff usually stored in TCP Control Blocks, between similar concurrent or consecutive connections 🔥

Status of This Memo 💅

this document is not an Internet Standards Track specification. it's published for informational purposes 🌈

it's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). not every document the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 for that 🧚

for the current status, errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc9040 🦋

Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. please read these carefully, since they explain your rights and restrictions for this document 🎀 Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License ⭐

Table of Contents 🦋

   1.  Introduction
   2.  Conventions Used in This Document
   3.  Terminology
   4.  The TCP Control Block (TCB)
   5.  TCB Interdependence
   6.  Temporal Sharing
     6.1.  Initialization of a New TCB
     6.2.  Updates to the TCB Cache
     6.3.  Discussion
   7.  Ensemble Sharing
     7.1.  Initialization of a New TCB
     7.2.  Updates to the TCB Cache
     7.3.  Discussion
   8.  Issues with TCB Information Sharing
     8.1.  Traversing the Same Network Path
     8.2.  State Dependence
     8.3.  Problems with Sharing Based on IP Address
   9.  Implications
     9.1.  Layering
     9.2.  Other Possibilities
   10. Implementation Observations
   11. Changes Compared to RFC 2140
   12. Security Considerations
   13. IANA Considerations
   14. References
     14.1.  Normative References
     14.2.  Informative References
   Appendix A.  TCB Sharing History
   Appendix B.  TCP Option Sharing and Caching
   Appendix C.  Automating the Initial Window in TCP over Long
           Timescales
     C.1.  Introduction
     C.2.  Design Considerations
     C.3.  Proposed IW Algorithm
     C.4.  Discussion
     C.5.  Observations
   Acknowledgments
   Authors' Addresses

1. Introduction ✨

okay so TCP is a connection-oriented reliable transport protocol that sits on top of IP [RFC0793] 💖 every TCP connection keeps its state in a data structure called the "TCP Control Block (TCB)". the TCB holds info about the connection state, the local process it belongs to, and feedback parameters about how the connection transmits 🌸 in the original spec and in most implementations, most TCB info is kept per connection 🧚

some implementations share certain TCB info across connections to the same host [RFC2140] 🫶 the goal is better overall transient performance, especially when there are loads of short-lived, simultaneous connections, like the ones the World Wide Web and other apps make [Be94] [Br02] 🔥 this sharing is meant to help TCP connections reach long-term behavior (assuming stable application load, the so-called "steady-state") faster, without hurting TCP interoperability 💅

this document updates RFC 2140's discussion of TCB state sharing and is a complete replacement for that document 🎀 the sharing only affects TCB initialization [RFC2140], so it has no effect on TCP's long-term behavior after a connection is established, or on interoperability 🦋 path info shared across SYN destination port numbers assumes that TCP segments with the same host-pair get the same path properties. in other words, traffic isn't routed differently based on port numbers or other connection parameters (more on that in Section 8.1) 🌈 the observations here apply to any protocol with congestion state, including the Stream Control Transmission Protocol (SCTP) [RFC4960] and the Datagram Congestion Control Protocol (DCCP) [RFC4340], and to individual subflows in Multipath TCP [RFC8684] ⭐

2. Conventions Used in This Document 💜

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🍓

the core of this document describes behavior TCP standards already permit. so it's informative guidance and doesn't use normative language, except when it quotes other documents 🌸 normative language does show up in Appendix C, where it's used as examples of requirements for future consideration 🧚

3. Terminology 🌈

these terms come up a lot in this doc. the ones with a "+" in front can be part of the state kept as TCP connection state in the TCB of associated connections, and they're the focus of the sharing described here 💖 terms are used as originally introduced where possible. sometimes the direction gets a suffix (_S for send, _R for receive) and sometimes it's spelled out (sendcwnd) 🫶

  • +cwnd: TCP congestion window size [RFC5681]
  • host: a source or sink of TCP segments associated with a single IP address
  • host-pair: a pair of hosts and their corresponding IP addresses
  • ISN: Initial Sequence Number
  • +MMS_R: maximum message size that can be received, the largest received transport payload of an IP datagram [RFC1122]
  • +MMS_S: maximum message size that can be sent, the largest transmitted transport payload of an IP datagram [RFC1122]
  • path: an Internet path between the IP addresses of two hosts
  • PCB: protocol control block, the data associated with a protocol as maintained by an endpoint; a TCP PCB is called a "TCB"
  • PLPMTUD: packetization-layer path MTU discovery, a mechanism that uses transport packets to discover the Path Maximum Transmission Unit (PMTU) [RFC4821]
  • +PMTU: largest IP datagram that can traverse a path [RFC1191] [RFC8201]
  • PMTUD: path-layer MTU discovery, a mechanism that relies on ICMP error messages to discover the PMTU [RFC1191] [RFC8201]
  • +RTT: round-trip time of a TCP packet exchange [RFC0793]
  • +RTTVAR: variation of round-trip times of a TCP packet exchange [RFC6298]
  • +rwnd: TCP receive window size [RFC5681]
  • +sendcwnd: TCP send-side congestion window (cwnd) size [RFC5681]
  • +sendMSS: TCP maximum segment size, a value transmitted in a TCP option that represents the largest TCP user data payload that can be received [RFC6691]
  • +ssthresh: TCP slow-start threshold [RFC5681]
  • TCB: TCP Control Block, the data associated with a TCP connection as maintained by an endpoint
  • TCP-AO: TCP Authentication Option [RFC5925]
  • TFO: TCP Fast Open option [RFC7413]
  • +TFO_cookie: TCP Fast Open cookie, state that is used as part of the TFO mechanism, when TFO is supported [RFC7413]
  • +TFO_failure: an indication of when TFO option negotiation failed, when TFO is supported
  • +TFOinfo: information cached when a TFO connection is established, which includes the TFO_cookie [RFC7413]

4. The TCP Control Block (TCB) 🦋

a TCB describes the data for each connection, meaning each association of a pair of applications across the network 💅 the TCB has at least the following info [RFC0793]:

      Local process state

         pointers to send and receive buffers
         pointers to retransmission queue and current segment
         pointers to Internet Protocol (IP) PCB

      Per-connection shared state

         macro-state
            connection state
            timers
            flags
            local and remote host numbers and ports
            TCP option state
         micro-state
            send and receive window state (size*, current number)
            congestion window size (sendcwnd)*
            congestion window size threshold (ssthresh)*
            max window size seen*
            sendMSS#
            MMS_S#
            MMS_R#
            PMTU#
            round-trip time and its variation#

the per-connection info is split into macro-state and micro-state, a term borrowed from [Co91] 🌈 macro-state covers the protocol for setting up the initial shared state of the connection. it includes the endpoint numbers and the parts (timers, flags) needed at the start that get used later to keep that state going 🫶 micro-state covers the protocol after a connection is up, keeping the data's reliability and congestion control on track ✨

there are two other classes of shared micro-state that are tied more to host-pairs than to application pairs 🔥 the first is clearly host-pair dependent (marked "#" above, like sendMSS, MMS_R, MMS_S, PMTU, RTT). that's because these parameters are defined by the endpoint or endpoint pair (for the examples given: sendMSS, MMS_R, MMS_S, RTT), or they're already cached and shared on that basis (for the example given: PMTU [RFC1191] [RFC4821]) 💖 the second is host-pair dependent in its aggregate (marked "*" above, like congestion window info, current window sizes, etc.) because they depend on the total capacity between the two endpoints 🌸

not all TCB state can be shared. some TCP options are only negotiated when the application layer asks for them, so their use may not match across connections 🍓 other options negotiate connection-specific parameters, which also can't be shared. Appendix B has more on these 🧚

last thing: rwnd isn't discussed further, since its value should depend on the send window size. that means send window sharing already covers it, and sharing doesn't affect it independently 💜

5. TCB Interdependence ⭐

there are two cases of TCB interdependence, and they're kinda the heart of this whole doc 👑 temporal sharing happens when the TCB of an earlier connection to a host (one that's now CLOSED) is used to initialize some parameters of a new connection to that same host. that's sharing in sequence 🎀 ensemble sharing happens when a currently active connection to a host is used to initialize another, concurrent connection to that host 🦋

6. Temporal Sharing ✨

okay so the TCB data cache has two main jobs bestie 🫶 it gets read to set up brand new TCBs, and it gets written whenever there's fresher per-host info to save 💅

6.1. Initialization of a New TCB 🌸

when a new connection starts, its TCB can borrow cached context from past connections, and the table below shows exactly what gets copied over 🎀

              +==============+=============================+
              | Cached TCB   | New TCB                     |
              +==============+=============================+
              | old_MMS_S    | old_MMS_S or not cached (2) |
              +--------------+-----------------------------+
              | old_MMS_R    | old_MMS_R or not cached (2) |
              +--------------+-----------------------------+
              | old_sendMSS  | old_sendMSS                 |
              +--------------+-----------------------------+
              | old_PMTU     | old_PMTU (1)                |
              +--------------+-----------------------------+
              | old_RTT      | old_RTT                     |
              +--------------+-----------------------------+
              | old_RTTVAR   | old_RTTVAR                  |
              +--------------+-----------------------------+
              | old_option   | (option specific)           |
              +--------------+-----------------------------+
              | old_ssthresh | old_ssthresh                |
              +--------------+-----------------------------+
              | old_sendcwnd | old_sendcwnd                |
              +--------------+-----------------------------+

              Table 1: Temporal Sharing - TCB Initialization

(1) ngl PMTU gets cached at the IP layer, see [RFC1191] [RFC4821] 🦋

(2) some values just don't get cached 🙈 MMS_R is computed locally, and MMS_S shows up in the connection itself (it's the MSS in the SYN) ✨

the next table is an overview of the option-specific stuff that can be shared 🧚 for more on certain TCP options and sharing, check Appendix B 💜

                   +=================+=================+
                   | Cached          | New             |
                   +=================+=================+
                   | old_TFO_cookie  | old_TFO_cookie  |
                   +-----------------+-----------------+
                   | old_TFO_failure | old_TFO_failure |
                   +-----------------+-----------------+

                        Table 2: Temporal Sharing -
                         Option Info Initialization

6.2. Updates to the TCB Cache 🔥

during a connection, the TCB cache can get updated based on what's happening with the current connections and their TCBs as time goes on, as shown in Table 3 👑

     +==============+===============+=============+=================+
     | Cached TCB   | Current TCB   | When?       | New Cached TCB  |
     +==============+===============+=============+=================+
     | old_MMS_S    | curr_MMS_S    | OPEN        | curr_MMS_S      |
     +--------------+---------------+-------------+-----------------+
     | old_MMS_R    | curr_MMS_R    | OPEN        | curr_MMS_R      |
     +--------------+---------------+-------------+-----------------+
     | old_sendMSS  | curr_sendMSS  | MSSopt      | curr_sendMSS    |
     +--------------+---------------+-------------+-----------------+
     | old_PMTU     | curr_PMTU     | PMTUD (1) / | curr_PMTU       |
     |              |               | PLPMTUD (1) |                 |
     +--------------+---------------+-------------+-----------------+
     | old_RTT      | curr_RTT      | CLOSE       | merge(curr,old) |
     +--------------+---------------+-------------+-----------------+
     | old_RTTVAR   | curr_RTTVAR   | CLOSE       | merge(curr,old) |
     +--------------+---------------+-------------+-----------------+
     | old_option   | curr_option   | ESTAB       | (depends on     |
     |              |               |             | option)         |
     +--------------+---------------+-------------+-----------------+
     | old_ssthresh | curr_ssthresh | CLOSE       | merge(curr,old) |
     +--------------+---------------+-------------+-----------------+
     | old_sendcwnd | curr_sendcwnd | CLOSE       | merge(curr,old) |
     +--------------+---------------+-------------+-----------------+

                Table 3: Temporal Sharing - Cache Updates

(1) PMTU is cached at the IP layer, see [RFC1191] [RFC4821] 🦋

so merge() is the function that mixes the current value with the previous (old) one 🎀 it can be different for each parameter in the cache, and this document doesn't pin down which one to use 🤷‍♀️ the examples given are windowed averages (the mean of the past N values, for some N) and exponential decay (new = (1-alpha)*old + alpha *new, where alpha is in the range [0..1]) ✨

Table 4 is the option-specific version of this, and it can be shared in a similar way 💖 the TFO cookie stays put until the client explicitly asks for it to be updated as a separate event 🍓

      +=================+=================+=======+=================+
      | Cached          | Current         | When? | New Cached      |
      +=================+=================+=======+=================+
      | old_TFO_cookie  | old_TFO_cookie  | ESTAB | old_TFO_cookie  |
      +-----------------+-----------------+-------+-----------------+
      | old_TFO_failure | old_TFO_failure | ESTAB | old_TFO_failure |
      +-----------------+-----------------+-------+-----------------+

              Table 4: Temporal Sharing - Option Info Updates

6.3. Discussion 💅

so like we said, there's no real perk to caching MMS_S and MMS_R, since the local IP stack reports them anyway 🙄 caching sendMSS and PMTU is pretty straightforward: reported values get cached (PMTU at the IP layer) and the newest values are what gets used 💜 the cache gets updated when the MSS option shows up in a SYN, or after PMTUD. PMTUD here means an ICMPv4 Fragmentation Needed [RFC1191] or an ICMPv6 Packet Too Big message [RFC8201] arrives, or the equivalent is inferred (like from PLPMTUD [RFC4821]). either way the cache always holds the most recent values from any connection 🔥

for sendMSS, the cache only gets checked when a connection is being established and isn't updated otherwise, so MSS options don't touch current connections 🙈 the default sendMSS is never saved. only reported MSS values update the cache, so you need an explicit override if you want to lower the sendMSS 🫶 and cached sendMSS only affects data sent in the SYN segment (so during client connection initiation or simultaneous open). every other segment's MSS is limited by whatever value got updated and included in the SYN 🦋

RTT values get updated by formulas that merge the old and new values, as Section 6.2 said 🧚 dynamic RTT estimation needs a bunch of RTT measurements in a row, so the cached RTT (and its variation) is an average of the old value and whatever the currently active TCB for that host has, and that only happens when a TCB closes 💖 RTT values only update when a connection closes. the merging method has to try to tone down the transient effects of new connections ✨

RTT, RTTVAR, and ssthresh updates all rely on existing info, meaning the old values. if no old values exist, the current values get cached instead 🎀

TCP options get copied or merged depending on the details of each option 🌈 for example, TFO state gets updated when a connection is established and gets read before a new connection is set up 🍓

Sections 8 and 9 talk about compatibility issues and the implications of sharing the specific info listed above, and Section 10 gives an overview of known implementations 👑

most cached TCB values get updated when a connection closes 💅 the exceptions are MMS_R and MMS_S (reported by IP [RFC1122]), PMTU (updated after Path MTU Discovery and also reported by IP [RFC1191] [RFC4821] [RFC8201]), and sendMSS (updated if the MSS option shows up in the TCP SYN header) 🫶

sharing sendMSS info only affects data in the SYN of the next connection, since sendMSS info usually gets included in most TCP SYN segments 🧚 caching PMTU can speed up PMTUD, but it can also cause black-holing until it's corrected if the value is wrong 😭 and caching MMS_R and MMS_S may not add much directly, since the local IP stack reports them anyway 🙈

the way state for other TCP options can be shared depends on that option's details 🔥 for example, TFO state includes the TCP Fast Open cookie [RFC7413] or, if TFO fails, a negative TCP Fast Open response. RFC 7413 says:

The client MUST cache negative responses from the server in order to avoid potential connection failures. Negative responses include the server not acknowledging the data in the SYN, ICMP error messages, and (most importantly) no response (SYN-ACK) from the server at all, i.e., connection timeout.

TFOinfo gets cached when a connection is established ✨

state for other TCP options might not be as easy to cache 🦋 for example, TCP-AO [RFC5925] success or failure between a host pair for a single SYN destination port might be useful to cache. TCP-AO success or failure for other SYN destination ports on that same host pair is never useful to cache, because TCP-AO security parameters can vary per service 💜

7. Ensemble Sharing 👑

sharing cached TCB data across concurrent connections means you have to pay attention to the aggregate nature of some of the shared state 🧚 for example, MSS and RTT values can be shared by copying, but it might not be right to just copy congestion window or ssthresh info. instead, the new values can be a function (f) of the cumulative values and the number of connections (N) 🔥

7.1. Initialization of a New TCB ✨

okay so when a brand-new TCB shows up, it can borrow cached context from the concurrent connections already hanging out with the same host 🫶 Here's the glow-up table:

              +===================+=========================+
              | Cached TCB        | New TCB                 |
              +===================+=========================+
              | old_MMS_S         | old_MMS_S               |
              +-------------------+-------------------------+
              | old_MMS_R         | old_MMS_R               |
              +-------------------+-------------------------+
              | old_sendMSS       | old_sendMSS             |
              +-------------------+-------------------------+
              | old_PMTU          | old_PMTU (1)            |
              +-------------------+-------------------------+
              | old_RTT           | old_RTT                 |
              +-------------------+-------------------------+
              | old_RTTVAR        | old_RTTVAR              |
              +-------------------+-------------------------+
              | sum(old_ssthresh) | f(sum(old_ssthresh), N) |
              +-------------------+-------------------------+
              | sum(old_sendcwnd) | f(sum(old_sendcwnd), N) |
              +-------------------+-------------------------+
              | old_option        | (option specific)       |
              +-------------------+-------------------------+

               Table 5: Ensemble Sharing - TCB Initialization

(1) Note that PMTU is cached at the IP layer [RFC1191] [RFC4821]. 🌸

In Table 5, the cached sum() is a total across all active connections, because these parameters act in aggregate 💜 And f() is the function that updates that sum using the new connection's values, which the table calls "N". It's giving teamwork.

Table 6 is the rundown on option-specific info that can get shared in the same way. The TFO_cookie only gets updated when the client explicitly asks for it, and that's its own separate event 🎀

                   +=================+=================+
                   | Cached          | New             |
                   +=================+=================+
                   | old_TFO_cookie  | old_TFO_cookie  |
                   +-----------------+-----------------+
                   | old_TFO_failure | old_TFO_failure |
                   +-----------------+-----------------+

                        Table 6: Ensemble Sharing -
                         Option Info Initialization

7.2. Updates to the TCB Cache 🦋

While a connection is running, the TCB cache can get updated based on what's changing in the concurrent connections and their TCBs. Main character energy for the cache, honestly 👑

      +==============+===============+===========+=================+
      | Cached TCB   | Current TCB   | When?     | New Cached TCB  |
      +==============+===============+===========+=================+
      | old_MMS_S    | curr_MMS_S    | OPEN      | curr_MMS_S      |
      +--------------+---------------+-----------+-----------------+
      | old_MMS_R    | curr_MMS_R    | OPEN      | curr_MMS_R      |
      +--------------+---------------+-----------+-----------------+
      | old_sendMSS  | curr_sendMSS  | MSSopt    | curr_sendMSS    |
      +--------------+---------------+-----------+-----------------+
      | old_PMTU     | curr_PMTU     | PMTUD+ /  | curr_PMTU       |
      |              |               | PLPMTUD+  |                 |
      +--------------+---------------+-----------+-----------------+
      | old_RTT      | curr_RTT      | update    | rtt_update(old, |
      |              |               |           | curr)           |
      +--------------+---------------+-----------+-----------------+
      | old_RTTVAR   | curr_RTTVAR   | update    | rtt_update(old, |
      |              |               |           | curr)           |
      +--------------+---------------+-----------+-----------------+
      | old_ssthresh | curr_ssthresh | update    | adjust sum as   |
      |              |               |           | appropriate     |
      +--------------+---------------+-----------+-----------------+
      | old_sendcwnd | curr_sendcwnd | update    | adjust sum as   |
      |              |               |           | appropriate     |
      +--------------+---------------+-----------+-----------------+
      | old_option   | curr_option   | (depends) | (option         |
      |              |               |           | specific)       |
      +--------------+---------------+-----------+-----------------+

                Table 7: Ensemble Sharing - Cache Updates
  • Note that the PMTU is cached at the IP layer [RFC1191] [RFC4821]. 🔥

In Table 7, rtt_update() is the function used to combine old and current values, for example as a windowed average or an exponentially decayed average 📈

Table 8 is the overview of option-specific info that can be shared the same way, so you can see the pattern 🌈

      +=================+=================+=======+=================+
      | Cached          | Current         | When? | New Cached      |
      +=================+=================+=======+=================+
      | old_TFO_cookie  | old_TFO_cookie  | ESTAB | old_TFO_cookie  |
      +-----------------+-----------------+-------+-----------------+
      | old_TFO_failure | old_TFO_failure | ESTAB | old_TFO_failure |
      +-----------------+-----------------+-------+-----------------+

              Table 8: Ensemble Sharing - Option Info Updates

7.3. Discussion 🧠

For ensemble sharing, TCB information should be cached as early as possible, sometimes even before a connection is closed. Otherwise, opening multiple concurrent connections may not lead to any TCB data sharing if no connection closes before the others open. The amount of work involved in updating the aggregate average should be minimized, but the resulting value should be equivalent to having all the values measured within a single connection. The function rtt_update in Table 7 indicates this operation, which happens whenever the RTT would have been updated in the individual TCP connection. So the cache ends up holding the shared RTT variables, which no longer need to live in the TCB 💅

Congestion window size and ssthresh aggregation are trickier in the concurrent case. When there's an ensemble of connections, we need to decide how that ensemble would have shared these variables, so we can derive initial values for new TCBs 🤔

Sections 8 and 9 discuss compatibility issues and implications of sharing the specific information listed above 📚

There are several ways to initialize the congestion window in a new TCB among an ensemble of current connections to a host. Current TCP implementations initialize it to 4 segments as standard [RFC3390] and to 10 segments experimentally [RFC6928]. These approaches assume that new connections should behave as conservatively as possible. The algorithm described in [Ba12] adjusts the initial cwnd depending on the cwnd values of ongoing connections. It's also possible to use sharing mechanisms over long timescales to adapt TCP's initial window automatically, as described further in Appendix C 🦋

8. Issues with TCB Information Sharing ⚠️

Here, we talk about the different kinds of problems that can pop up with TCB information sharing 🙈

For the congestion and current window information, the initial values computed by TCB interdependence may not be consistent with the long-term aggregate behavior of a set of concurrent connections between the same endpoints. Under conventional TCP congestion control, if the congestion window of a single existing connection has converged to 40 segments, two newly joining concurrent connections will assume initial windows of 10 segments [RFC6928], and the existing connection's window will not decrease to make room for this extra load. As a consequence, the three connections can mutually interfere 😭 One example is seen on low-bandwidth, high-delay links, where concurrent connections supporting Web traffic can collide because their initial windows were too large, even when set at 1 segment.

The authors of [Hu12] recommend caching ssthresh for temporal sharing only when flows are long. Some studies suggest that sharing ssthresh between short flows can deteriorate the performance of individual connections [Hu12] [Du16], although this may benefit aggregate network performance 💀

8.1. Traversing the Same Network Path 🛣️

TCP is sometimes used in situations where packets of the same host-pair don't always take the same path, such as when connection-specific parameters are used for routing (e.g., for load balancing). Multipath routing that relies on examining transport headers, such as ECMP and Link Aggregation Group (LAG) [RFC7424], may not result in repeatable path selection when TCP segments are encapsulated, encrypted, or altered -- for example, in some Virtual Private Network (VPN) tunnels that rely on proprietary encapsulation. Similarly, such approaches cannot operate deterministically when the TCP header is encrypted, e.g., when using IPsec Encapsulating Security Payload (ESP). (Although TCB interdependence among the entire set sharing the same endpoint IP addresses should work without problems when the TCP header is encrypted.) Measures to increase the probability that connections use the same path could be applied; for example, the connections could be given the same IPv6 flow label [RFC6437]. TCB interdependence can also be extended to sets of host IP address pairs that share the same network path conditions, such as when a group of addresses is on the same LAN (see Section 9) 🌐

Traversing the same path isn't important for host-specific information (e.g., rwnd), TCP option state (e.g., TFOinfo), or information that's already cached per-host (e.g., path MTU) ✨ When TCB information is shared across different SYN destination ports, path-related information can be incorrect. However, the impact of this error is potentially diminished if (as discussed here) TCB sharing affects only the transient event of a connection start, or if TCB information is shared only within connections to the same SYN destination port 🎀

In the case of temporal sharing, TCB information could also become invalid over time, i.e., indicating that although the path remains the same, path properties have changed. Because this is similar to the case when a connection becomes idle, mechanisms that address idle TCP connections (e.g., [RFC7661]) could also be applied to TCB cache management, especially when TCP Fast Open is used [RFC7413] 🧚

8.2. State Dependence 🌸

There may be additional considerations for the way TCB interdependence rebalances congestion feedback among the current connections. For example, it may be appropriate to consider the impact of a connection being in Fast Recovery [RFC5681], or some other similar unusual feedback state that could inhibit or affect the calculations described herein. Not every vibe is the same, bestie 🫶

8.3. Problems with Sharing Based on IP Address ✨

okay so sharing TCB info by IP address can go totally wrong bestie 🙈 like, if an IP gets handed to a brand new host (think IP address spinning, which ISPs do to stop people running servers), the old info is now about the wrong machine 💀. It can also be wrong if Network Address Translation (NAT) [RFC2663], Network Address and Port Translation (NAPT) [RFC2663], or any other IP sharing mechanism is in play 🦋. These are less likely to show up with IPv6, which is a vibe 🌈. Other ways to identify a host could also be considered to make correct TCB sharing more likely 💜. And here's the lowkey twist: some TCB info is about the dominant path rather than one specific host 🧚. IP addresses can be different and the relevant part of the path can still be the exact same, so main character energy goes to the path 👑.

9. Implications 🔥

There are a few implications when you bring TCB interdependence into TCP implementations, and they're honestly iconic 💅. First, it may reduce the need for application-layer multiplexing for performance enhancement [RFC7231]. Protocols like HTTP/2 [RFC7540] avoid connection re-establishment costs by serializing or multiplexing a set of per-host connections across a single TCP connection 🌸. This dodges TCP's per-connection OPEN handshake and also dodges recomputing the MSS, RTT, and congestion window values ✨. By avoiding the so-called "slow-start restart", performance can be optimized [Hu01] 🚀. TCB interdependence can give you that "slow-start restart avoidance" perk of multiplexing without needing a multiplexing mechanism at the application layer 🫶.

9.1. Layering 🎀

TCB interdependence pushes some of the TCP implementation from its usual home, solely inside the transport layer (in the ISO model), up to the network layer 🦋. This acknowledges that some components of state are per-host-pair, or can be per-path as indicated solely by that host-pair 💖. Transport protocols usually manage per-application-pair associations (per stream), and network protocols manage per-host-pair and path associations (routing) 🌈. Round-trip time, MSS, and congestion information could be more appropriately handled at the network layer, aggregated among concurrent connections, and shared across connection instances [RFC3124] ✨.

An earlier version of RTT sharing suggested implementing RTT state at the IP layer rather than at the TCP layer 🍓. Our observations describe sharing state among TCP connections, which avoids some of the difficulties in an IP-layer solution 💅. One problem with an IP-layer solution is figuring out the correspondence between packet exchanges using IP header information alone, and that correspondence is needed to compute RTT 💀. Because TCB sharing computes RTTs inside the TCP layer using TCP header information, it can be implemented more directly and simply than at the IP layer 😭. So this is a case where information should be computed at the transport layer but could be shared at the network layer 👑.

9.2. Other Possibilities 🌸

Per-host-pair associations are not the limit of these techniques, bestie 🧚. TCBs could be similarly shared between hosts on a subnet or within a cluster, because the predominant path can be subnet-subnet rather than host-host 🦋. Also, TCB interdependence can be applied to any protocol with congestion state, including SCTP [RFC4960] and DCCP [RFC4340], as well as to individual subflows in Multipath TCP [RFC8684] 💜.

There may be other information that can be shared between concurrent connections 🫶. For example, if another connection just tried to expand its window size and failed, a connection may not attempt to do the same for some period 🙈. The idea is that existing TCP implementations infer the behavior of all competing connections, including those within the same host or subnet 🔥. One possible optimization is to make that implicit feedback explicit, via extended information associated with the endpoint IP address and its TCP implementation, rather than per-connection state in the TCB ✨.

This document focuses on sharing TCB information at connection initialization 🎀. Since RFC 2140, there have been loads of approaches that try to coordinate ongoing state across concurrent connections, both within TCP and other congestion-reactive protocols, which are summarized in [Is18] 📚. These approaches are more complex to implement, and comparing them to steady-state TCP equivalence can be harder to establish, sometimes on purpose (i.e., they sometimes intend to provide a different kind of "fairness" than emerges from TCP operation) 😭.

10. Implementation Observations 💖

The observation that some TCB state is host-pair specific rather than application-pair dependent is not new, and it's a common engineering decision in layered protocol implementations 🌈. Although it's now deprecated, T/TCP [RFC1644] was the first to propose using caches to maintain TCB states (see Appendix A) 🧚.

Table 9 describes the current implementation status for TCB temporal sharing in Windows as of December 2020, Apple variants (macOS, iOS, iPadOS, tvOS, and watchOS) as of January 2021, Linux kernel version 5.10.3, and FreeBSD 12 📅. Ensemble sharing is not yet implemented 💀.

     +==============+=========================================+
     | TCB data     | Status                                  |
     +==============+=========================================+
     | old_MMS_S    | Not shared                              |
     +--------------+-----------------------------------------+
     | old_MMS_R    | Not shared                              |
     +--------------+-----------------------------------------+
     | old_sendMSS  | Cached and shared in Apple, Linux (MSS) |
     +--------------+-----------------------------------------+
     | old_PMTU     | Cached and shared in Apple, FreeBSD,    |
     |              | Windows (PMTU)                          |
     +--------------+-----------------------------------------+
     | old_RTT      | Cached and shared in Apple, FreeBSD,    |
     |              | Linux, Windows                          |
     +--------------+-----------------------------------------+
     | old_RTTVAR   | Cached and shared in Apple, FreeBSD,    |
     |              | Windows                                 |
     +--------------+-----------------------------------------+
     | old_TFOinfo  | Cached and shared in Apple, Linux,      |
     |              | Windows                                 |
     +--------------+-----------------------------------------+
     | old_sendcwnd | Not shared                              |
     +--------------+-----------------------------------------+
     | old_ssthresh | Cached and shared in Apple, FreeBSD*,   |
     |              | Linux*                                  |
     +--------------+-----------------------------------------+
     | TFO failure  | Cached and shared in Apple              |
     +--------------+-----------------------------------------+

                  Table 9: KNOWN IMPLEMENTATION STATUS

* Note: In FreeBSD, new ssthresh is the mean of curr_ssthresh and its previous value if a previous value exists; in Linux, the calculation depends on state and is max(curr_cwnd/2, old_ssthresh) in most cases. 🤓

In Table 9, "Apple" refers to all Apple OSes, i.e., macOS (desktop/laptop), iOS (phone), iPadOS (tablet), tvOS (video player), and watchOS (smart watch), which all share the same Internet protocol stack 🍎✨.

11. Changes Compared to RFC 2140 🎀

This document updates the description of TCB sharing in RFC 2140 and its associated impact on existing and new connection state, and it's a complete replacement for that document [RFC2140] 💅. It clarifies the previous description and terminology and extends the mechanism to its impact on new protocols and mechanisms, including multipath TCP, Fast Open, PLPMTUD, NAT, and the TCP Authentication Option 🌈.

The detailed impact on TCB state addresses TCB parameters with greater specificity 🔍. It separates the way MSS is used in both send and receive directions, it separates the way both of these MSS values differ from sendMSS, it adds both path MTU and ssthresh, and it addresses the impact on state associated with TCP options 💖.

New sections have been added to address compatibility issues and implementation observations 🧚. The relation of this work to T/TCP has been moved to Appendix A (which describes the history of TCB sharing), partly to reflect the deprecation of that protocol 🦋.

Appendix C has been added to discuss the potential to use temporal sharing over long timescales to adapt TCP's initial window automatically, avoiding the need to periodically revise a single global constant value ✨.

Finally, this document updates and significantly expands the referenced literature 📚👑.

12. Security Considerations 🔒✨

okay bestie, this is the spicy-but-responsible section 💅 Quick headline first: the stuff in this doc doesn't add any new risk for attacks aimed at one connection, like aborting it or injecting info into it 🙈 The usual denial-of-service problems still apply to individual connections, sharing or not, unless you secure them some other way 🛡️

Now the drama. Sharing TCB info can create extra denial-of-service attacks that hurt the performance of other connections by polluting the cached info 😭 This can happen across any set of connections that share a TCB. That could be connections inside a single host, or connections between hosts if TCB sharing is set up within a subnet (see "Implications" (Section 9)) 🌐

Some shared parameters are only used to create new TCBs, while others are shared among the TCBs of connections that are already running 🧠 New connections can join that ongoing set, for example to tune the send window size across a bunch of connections to the same host 🪟 And PMTU is already defined as shared at the IP layer, so it's already vulnerable to this kind of thing 🙃

Client SYN options are lowkey easier to forge than a full, two-way connection 👀 So their values might not be safe to fold into shared values until the three-way handshake is done 🤝 Patience is a whole virtue here, periodt ✨

Attacks on parameters that are only used for initialization only mess with the temporary performance of a TCP connection 🌸 For short connections, though, the impact can get close to a denial-of-service attack 💀 For example, if an application changes its TCB to have a fake, tiny window size, later connections will be slow until their window grows back to where it should be 🐢

Last but not least, TCB sharing reuses and mixes information from past and current connections 🎀 Reusing info could make fingerprinting hosts possible, but the mixing lowers that risk 🧚 There's been no evidence of fingerprinting based on this technique, so it's currently considered safe in that regard 💜 Also, info about how a TCP connection performed hasn't been considered private 🫶

13. IANA Considerations 🌈

This document has no IANA actions. Nothing to register, nothing to update, just vibes 🌸

14. References 📚

14.1. Normative References 📖

14.2. Informative References 📚

Okay, the informative refs bestie 🧚 Here's the whole list, with the tags exactly as they're written:

Appendix A. TCB Sharing History 🕰️

Okay, this is the tea on how TCB sharing got started 🍵. T/TCP wanted to stash TCB info in caches so it could carry over from one instance to the next (that's temporal sharing) 🦋. We're talking smoothed RTT, RTT variation, the congestion-avoidance threshold, and MSS [RFC1644]. Those values were on top of the connection counts T/TCP used to speed up data delivery before the full three-way handshake during an OPEN ✨. The big goal was to group TCB components together whenever they reflect one association, which is the host-pair, instead of splitting them up by connection 💖.

At least one T/TCP implementation saved the MSS and combined the RTT parameters across several connections, but it skipped caching the congestion window info [Br94], even though [RFC1379] originally specified it 🙈. Some T/TCP implementations updated MSS right away when the TCP MSS header option came in [Br94]. The concepts and functional specification [RFC1379] [RFC1644] didn't really address this, though 🤷‍♀️. Later T/TCP implementations only updated RTT values after a CLOSE, which doesn't help concurrent sessions 😭.

Temporal sharing of cached TCB data first showed up in the Sun OS 4.1.3 T/TCP extensions [Br94] and the FreeBSD port of them [FreeBSD] 🌸. Like I said before, only the MSS and RTT parameters were cached, as [RFC1379] originally specified 💅. Later chats about T/TCP suggested putting congestion control parameters in this cache too. For example, Section 3.1 of [RFC1644] hints at setting the congestion window back to the old window size 👑.

Appendix B. TCP Option Sharing and Caching 🎀

Besides the options that can be cached and shared, this memo also lists known TCP options [IANA] where keeping state is unsafe 🙅‍♀️. Heads up that this list isn't meant to be authoritative or exhaustive, so don't treat it as the final word 💜.

Obsolete (unsafe to keep state):

  • Echo
  • Echo Reply
  • Partial Order Connection Permitted
  • Partial Order Service Profile
  • CC
  • CC.NEW
  • CC.ECHO
  • TCP Alternate Checksum Request
  • TCP Alternate Checksum Data

No state to keep:

  • End of Option List (EOL)
  • No-Operation (NOP)
  • Window Scale (WS)
  • SACK
  • Timestamps (TS)
  • MD5 Signature Option
  • TCP Authentication Option (TCP-AO)
  • RFC3692-style Experiment 1
  • RFC3692-style Experiment 2

Unsafe to keep state:

  • Skeeter (DH exchange, known to be vulnerable)
  • Bubba (DH exchange, known to be vulnerable)
  • Trailer Checksum Option
  • SCPS capabilities
  • Selective Negative Acknowledgements (S-NACK)
  • Records Boundaries
  • Corruption experienced
  • SNAP
  • TCP Compression Filter
  • Quick-Start Response
  • User Timeout Option (UTO)
  • Multipath TCP (MPTCP) negotiation success (see below for negotiation failure)
  • TCP Fast Open (TFO) negotiation success (see below for negotiation failure)

Safe but optional to keep state:

  • Multipath TCP (MPTCP) negotiation failure (to avoid negotiation retries)
  • Maximum Segment Size (MSS)
  • TCP Fast Open (TFO) negotiation failure (to avoid negotiation retries)

Safe and necessary to keep state:

  • TCP Fast Open (TFO) Cookie (if TFO succeeded in the past)

Appendix C. Automating the Initial Window in TCP over Long Timescales ⏳

C.1. Introduction 🌸

okay so temporal sharing (the thing we covered earlier in this doc) is built on the idea that back-to-back connections between the same host pair are lowkey likely to see similar environment conditions, fr fr. The stored info can get less accurate over time though, so you have to account for that aging (more on this in Section 8.1). 💅 But there are also times it makes sense to track these values over much longer periods, watching how TCP connections behave so the trends in TCP parameters slowly evolve. This appendix is a giant example of that. 🦋

TCP's congestion control algorithm uses an initial window value (IW) as the starting point for new connections and also as the upper limit when restarting after an idle period [RFC5681] [RFC7661]. The IW has glowed up a lot over time 💖 It started at 1 maximum segment size (MSS), then went up to the lesser of 4 MSSs or 4,380 bytes [RFC3390] [RFC5681]. For a typical Internet connection with a maximum transmission unit (MTU) of 1500 bytes, that means 3 segments of 1,460 bytes each.

The IW was originally implied in the first TCP congestion control description and got documented as a standard in 1997 [RFC2001] [Ja88]. In 1998 it was updated experimentally, then moved to the Standards Track in 2002 [RFC2414] [RFC3390]. In 2013 it was experimentally bumped up to 10 [RFC6928]. Iconic glow-up behavior 👑

This appendix talks about how TCP can objectively measure when an IW is too large, and why that feedback should be used over long timescales to adjust the IW automatically. The result should be safer to deploy and might save us from having to keep revisiting the IW over time. 🎀

Heads up though: this mechanism tries to make the IW more adaptive over time. It can push the IW past what is currently recommended for wide-scale deployment, so its use should be carefully monitored. 🙈

C.2. Design Considerations 🧚

TCP's IW value has been static for over two decades, so any solution that adjusts the IW dynamically should have similarly stable, non-invasive effects on TCP's performance and complexity. To be fair, the IW should be similar for most machines on the public Internet. And a desirable goal is a self-correcting algorithm, so IW values that cause network problems can be avoided. So the design goals we're proposing are:

  • Impart little to no impact to TCP in the absence of loss, i.e., it should not increase the complexity of default packet processing in the normal case.
  • Adapt to network feedback over long timescales, avoiding values that persistently cause network problems.
  • Decrease the IW in the presence of sustained loss of IW segments, as determined over a number of different connections.
  • Increase the IW in the absence of sustained loss of IW segments, as determined over a number of different connections.
  • Operate conservatively, i.e., tend towards leaving the IW the same in the absence of sufficient information, and give greater consideration to IW segment loss than IW segment success.

We expect that, without other context, a good IW algorithm will converge to a single value, but this is not required. An endpoint with additional context or information, or one deployed in a constrained environment, can always use a different value. In particular, information from previous connections, or sets of connections with a similar path, can already be used as context for these decisions (as noted in the core of this document). ✨

That said, if a given IW value persistently causes packet loss during the initial burst of packets, it's clearly inappropriate and could be inducing unnecessary loss in other competing connections. This might happen for sites behind very slow boxes with small buffers, which may or may not be the first hop. 😭

C.3. Proposed IW Algorithm 💜

Below is a simple description of the proposed IW algorithm. It relies on the following parameters:

  • MinIW = 3 MSS or 4,380 bytes (as per [RFC3390])
  • MaxIW = 10 MSS (as per [RFC6928])
  • MulDecr = 0.5
  • AddIncr = 2 MSS
  • Threshold = 0.05

We assume that the minimum IW (MinIW) should be as currently specified as standard [RFC3390]. The maximum IW (MaxIW) can be set to a fixed value (we suggest using the experimental and now somewhat de facto standard in [RFC6928]) or set based on a schedule if trusted time references are available [Al10]. Here, we prefer a fixed value. We also propose to use an Additive Increase Multiplicative Decrease (AIMD) algorithm, with increases and decreases as noted. 🔥

Although these parameters are somewhat arbitrary, their initial values aren't important, except that the algorithm is AIMD and the MaxIW should not exceed what is recommended for other systems on the Internet. (Here, we selected the current de facto standard rather than the actual standard.) Current proposals, including default current operation, are degenerate cases of the algorithm below for given parameters, notably MulDec = 1.0 and AddIncr = 0 MSS, which disables the automatic part of the algorithm. 💅

The proposed algorithm is as follows:

  1. On boot:
IW = MaxIW; # assume this is in bytes and indicates an integer
            # multiple of 2 MSS (an even number to support
            # ACK compression)
  1. Upon starting a new connection:
CWND = IW;
conncount++;
IWnotchecked = 1; # true
  1. During a connection's SYN-ACK processing, if SYN-ACK includes ECN (as similarly addressed in Section 5 of ECN++ for TCP [Ba20]), treat it as if the IW is too large:
if (IWnotchecked && (synackecn == 1)) {
   losscount++;
   IWnotchecked = 0; # never check again
}
  1. During a connection, if retransmission occurs, check the seqno of the outgoing packet (in bytes) to see if the re-sent segment fixes an IW loss:
if (Retransmitting && IWnotchecked && ((seqno - ISN) < IW))) {
   losscount++;
   IWnotchecked = 0; # never do this entire "if" again
} else {
   IWnotchecked = 0; # you're beyond the IW so stop checking
}
  1. Once every 1000 connections, as a separate process (i.e., not as part of processing a given connection):
if (conncount > 1000) {
   if (losscount/conncount > threshold) {
      # the number of connections with errors is too high
      IW = IW * MulDecr;
   } else {
      IW = IW + AddIncr;
   }
}

As presented, this algorithm can yield a false positive when the sequence number wraps around. For example, the code might increment losscount in step 4 when no loss occurred, or fail to increment losscount when a loss did occur. This can be avoided using either Protection Against Wrapped Sequences (PAWS) [RFC7323] context or internal extended sequence number representations (as in TCP Authentication Option (TCP-AO) [RFC5925]). Alternately, false positives can be tolerated, because they are expected to be infrequent and thus won't significantly impact the algorithm. 🤓

There are a number of additional constraints that need to be imposed if this mechanism is implemented. They make sure it defaults to values that comply with current Internet standards, stays conservative in how it extends those values, and returns to those values in the absence of positive feedback (i.e., success). To that end, we recommend the following list of example constraints:

  • The automatic IW algorithm MUST initialize MaxIW a value no larger than the currently recommended Internet default in the absence of other context information.

Thus, if there are too few connections to make a decision, or if there is otherwise insufficient information to increase the IW, then the MaxIW defaults to the current recommended value.

  • An implementation MAY allow the MaxIW to grow beyond the currently recommended Internet default but not more than 2 segments per calendar year.

Thus, if an endpoint has a persistent history of successfully transmitting IW segments without loss, then it is allowed to probe the Internet to determine if larger IW values have similar success. This probing is limited and requires a trusted time source; otherwise, the MaxIW remains constant.

  • An implementation MUST adjust the IW based on loss statistics at least once every 1000 connections.

An endpoint needs to be sufficiently reactive to IW loss.

  • An implementation MUST decrease the IW by at least 1 MSS when indicated during an evaluation interval.

An endpoint that detects loss needs to decrease its IW by at least 1 MSS; otherwise, it is not participating in an automatic reactive algorithm.

  • An implementation MUST increase by no more than 2 MSSs per evaluation interval.

An endpoint that does not experience IW loss needs to probe the network incrementally.

  • An implementation SHOULD use an IW that is an integer multiple of 2 MSSs.

The IW should remain a multiple of 2 MSS segments to enable efficient ACK compression without incurring unnecessary timeouts.

  • An implementation MUST decrease the IW if more than 95% of connections have IW losses.

Again, this is to ensure an implementation is sufficiently reactive.

  • An implementation MAY group IW values and statistics within subsets of connections. Such grouping MAY use any information about connections to form groups except loss statistics.

There are some TCP connections that might not be counted at all, such as those to or from loopback addresses, or those within the same subnet as that of a local interface (for which congestion control is sometimes disabled anyway). This may also include connections that terminate before the IW is full, i.e., as a separate check at the time of the connection closing. 🍓

The period over which the IW is updated is intended to be a long timescale, e.g., a month or so, or 1,000 connections, whichever is longer. An implementation might check the IW once a month and simply not update the IW, or clear the connection counts, in months where the number of connections is too small. 🌈

C.4. Discussion 💖

okay so there are LOTS of parameters in this algorithm that still follow the requirements 🎀 that's the whole point, bestie. It lets people tweak the config and build it their own way while making sure every version stays reactive and safe ✨

This algorithm keeps counting in segments because that's the basis of most TCP implementations 🧱 It might be worth revisiting the specs to allow byte-based congestion once there's enough experience, ngl 🤔

The algorithm only checks for IW losses during the first IW after a connection starts. It does NOT check for them anywhere else the IW gets used, like during slow-start restarts 👀

  • An implementation MAY detect IW losses during slow-start restarts in addition to losses during the first IW of a connection. In this case, the implementation MUST count each restart as a "connection" for the purposes of connection counts and periodic rechecking of the IW value. 🌸

False positives can happen during some kinds of segment reordering. That can trigger spurious retransmissions even without a true segment loss 😭 But they aren't expected to be common enough to take over the algorithm and its conclusions, so we're good 🫶

This mechanism does need extra per-connection state. It's already common in some implementations and it's useful for other reasons too (e.g., the ISN is used in TCP-AO [RFC5925]) ✨ It also benefits from persistent state kept across reboots, which would be useful to other state sharing mechanisms as well (e.g., TCP Control Block Sharing per the main body of this document) 🦋

The receive window (rwnd) isn't part of this calculation at all 🙈 Its size is set by receiver resources and gives room for segment reordering. Also, rwnd has nothing to do with congestion control, and congestion control is the whole focus of how this appendix manages the IW 💅

C.5. Observations 🌈

The IW may not converge to a single global value. It also may not converge at all and could instead oscillate by a few MSSs as it keeps probing the Internet for larger IWs and failing 🎢 Both of these properties match TCP's behavior during each individual connection, so it's all very consistent, iykyk 💜

This mechanism assumes that losses during the IW are caused by IW size. Persistent errors that drop packets for other reasons, e.g., OS bugs, can cause false positives 🐛 Again, this lines up with TCP's basic assumption that loss is caused by congestion and needs backoff. This algorithm treats the IW of new connections as a long-timescale backoff system ⏳👑

Acknowledgments 🫶

The authors would like to thank Praveen Balasubramanian for information regarding TCB sharing in Windows, and Christoph Paasch for information regarding TCB sharing in Apple OSs 💻 Huge thanks also to Yuchung Cheng, Lars Eggert, Ilpo Jarvinen, and Michael Scharf for comments on earlier draft versions of this document, as well as the members of the TCPM WG 🌟 Earlier revisions of this work received funding from a collaborative research project between the University of Oslo and Huawei Technologies Co., Ltd., and were partly supported by USC/ISI's Postel Center 🎓

Authors' Addresses 🦋

Joe Touch
Manhattan Beach, CA 90266
United States of America

Phone: +1 (310) 560-0334
Email: touch@strayalpha.com
Michael Welzl
University of Oslo
PO Box 1080 Blindern
N-0316 Oslo
Norway

Phone: +47 22 85 24 20
Email: michawe@ifi.uio.no
Safiqul Islam
University of Oslo
PO Box 1080 Blindern
Oslo N-0316
Norway

Phone: +47 22 84 08 37
Email: safiquli@ifi.uio.no
📦 where this page comes from