RFC 6182

Architectural Guidelines for Multipath TCP Development

📅 published March 2011, from ietf

Architectural Guidelines for Multipath TCP Development 🎀

Okay bestie, the title block is where the receipts are 🧾. This is RFC 6182, filed under Informational (so it's guidance, not a standard 💅) and published in March 2011. The ISSN is 2070-1721.

The authors are:

  • A. Ford from Roke Manor Research
  • C. Raiciu from ~~nowhere~~ no other affiliation listed, so just the names 🌸
  • M. Handley from University College London
  • S. Barre from Universite catholique de Louvain
  • J. Iyengar from Franklin and Marshall College

It's from the Internet Engineering Task Force (IETF). The title block doesn't list anything it obsoletes or updates, so it's a standalone doc 🦋

Abstract ✨

Okay so hosts are often connected by multiple paths, but TCP only lets a connection use one path at a time 🙃 That's kinda sad, because the network's resources would be used way more efficiently if those paths could be used at the same time. Users would also get a better experience, with more resilience when a network fails and higher throughput 🚀

This doc lays out architectural guidelines for building a Multipath Transport Protocol, and shows how those pieces come together in Multipath TCP (MPTCP). It also lists some high-level design decisions that set the foundation for MPTCP, based on those architectural requirements 💜

Status of This Memo 📜

Bestie, this one is not an Internet Standards Track spec. It's published for informational purposes 🎀

It's a product of the IETF, and it represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so see Section 2 of RFC 5741 for that 🌈

For the current status, any errata, and how to give feedback, check http://www.rfc-editor.org/info/rfc6182 🔍

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved 💖

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please actually read these, because they spell out your rights and restrictions for this document 🙈 Any Code Components pulled out of this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License ⚖️

Table of Contents 🧚

   1. Introduction ....................................................4
      1.1. Requirements Language ......................................5
      1.2. Terminology ................................................5
      1.3. Reference Scenario .........................................6
   2. Goals ...........................................................6
      2.1. Functional Goals ...........................................6
      2.2. Compatibility Goals ........................................7
           2.2.1. Application Compatibility ...........................7
           2.2.2. Network Compatibility ...............................8
           2.2.3. Compatibility with Other Network Users .............10
      2.3. Security Goals ............................................10
      2.4. Related Protocols .........................................10
   3. An Architectural Basis for Multipath TCP .......................11
   4. A Functional Decomposition of MPTCP ............................12
   5. High-Level Design Decisions ....................................14
      5.1. Sequence Numbering ........................................14
      5.2. Reliability and Retransmissions ...........................15
      5.3. Buffers ...................................................17
      5.4. Signaling .................................................18
      5.5. Path Management ...........................................19
      5.6. Connection Identification .................................20
      5.7. Congestion Control ........................................21
      5.8. Security ..................................................21
   6. Software Interactions ..........................................23
      6.1. Interactions with Applications ............................23
      6.2. Interactions with Management Systems ......................23
   7. Interactions with Middleboxes ..................................23
   8. Contributors ...................................................25
   9. Acknowledgements ...............................................25
   10. Security Considerations .......................................26
   11. References ....................................................26
      11.1. Normative References .....................................26
      11.2. Informative References ...................................26

1. Introduction ✨

As the Internet evolves, demand on Internet resources keeps climbing 📈 but a lot of those resources (bandwidth especially) can't be fully used because of protocol limits, both on the end systems and inside the network 😩 If those resources could be used concurrently, the end user experience could be way better. These improvements would also cut down on the spending on network infrastructure that you'd otherwise need to get the same user experience boost 💅 Through resource pooling [3], these resources can be "pooled" so they look like a single logical resource to the user 🫶

Multipath transport tries to achieve some of the resource pooling goals by using multiple disjoint (or partially disjoint) paths across a network at once 🦋 There are two key benefits:

  • To increase the resilience of the connectivity by providing multiple paths, protecting end hosts from the failure of one.
  • To increase the efficiency of the resource usage, and thus increase the network capacity available to end hosts.

Multipath TCP is a modified version of TCP [1] that does multipath transport. It pools multiple paths within a single transport connection, transparently to the application 👑 Mostly it's about using multiple paths end-to-end, where one or both end hosts are multihomed. It might also work where multiple paths exist inside the network and an end host can manipulate them, like using different port numbers with Equal Cost MultiPath (ECMP) [4] 🌈

MPTCP, defined in [5], is the specific protocol that makes the Multipath TCP idea real 🎀 This doc looks at both the general architectural principles for a Multipath TCP that hits the goals in Section 2, and the key design decisions behind MPTCP, which get detailed in Section 5 💜

Multihoming and multipath aren't new for transport protocols (Stream Control Transmission Protocol (SCTP) [6] is a famous example), but MPTCP wants wide-scale deployment, so it cares a lot about application and network compatibility 🫶 These goals are covered in Section 2 and they're about how MPTCP looks to the network (so non-MPTCP-aware entities just see TCP) and to the application (it gives a service equivalent to TCP for non-MPTCP-aware applications) ✨

This doc has three key purposes 🌸:

  1. It describes goals for a multipath transport, which MPTCP is designed to meet.
  2. It lays out an architectural basis for MPTCP's design, and this discussion applies to other multipath transports too.
  3. It discusses and documents the high-level design decisions made in MPTCP's development, and considers their implications.

Companion docs to this architectural overview cover the protocol extensions [5], congestion control algorithms [7], and application-level considerations [8]. Put together, those pieces specify a complete Multipath TCP design 🧩 Specific components can be swapped out, as long as it follows the layer and functional decompositions discussed in this doc 🔁

1.1. Requirements Language 🔑

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [2] 💖

1.2. Terminology 📖

  • Regular/Single-Path TCP: The standard version of TCP [1] in use today, operating between a single pair of IP addresses and ports.
  • Multipath TCP: A modified version of the TCP protocol that supports the simultaneous use of multiple paths between hosts.
  • Path: A sequence of links between a sender and a receiver, defined in this context by a source and destination address pair.
  • Host: An end host either initiating or terminating a Multipath TCP connection.
  • MPTCP: The proposed protocol extensions specified in [5] to provide a Multipath TCP implementation.
  • Subflow: A flow of TCP segments operating over an individual path, which forms part of a larger Multipath TCP connection.
  • (Multipath TCP) Connection: A set of one or more subflows combined to provide a single Multipath TCP service to an application at a host.

1.3. Reference Scenario 🗺️

Figure 1 shows a typical Multipath TCP usage scenario 🦋 Two hosts, A and B, are talking to each other. Both are multihomed and multi-addressed, giving them two disjoint connections to the Internet. The addresses on each host are called A1, A2, B1, and B2, so there are up to four different paths between the two hosts: A1-B1, A1-B2, A2-B1, A2-B2 🌈

               +------+           __________           +------+
               |      |A1 ______ (          ) ______ B1|      |
               | Host |--/      (            )      \--| Host |
               |      |        (   Internet   )        |      |
               |  A   |--\______(            )______/--|   B  |
               |      |A2        (__________)        B2|      |
               +------+                                +------+

               Figure 1: Simple Multipath TCP Usage Scenario

The scenario can have any number of addresses (1 or more) on each host, as long as there are 2 or more paths between the two hosts (i.e., num_addr(A) * num_addr(B) > 1) 💅 The paths through the Internet don't have to be entirely disjoint. Potential fairness issues from shared bottlenecks need to be handled by the Multipath TCP congestion controller ⚖️ And honestly, the paths through the Internet often aren't pure end-to-end, since middleboxes like NATs and firewalls can get involved 🙃

2. Goals 🎯

This section outlines the primary goals that Multipath TCP aims to meet. They're broadly split into two groups: functional goals, which steer the services and features Multipath TCP must provide, and compatibility goals, which decide how Multipath TCP should appear to the entities that interact with it 🫶

2.1. Functional Goals ✨

okay so, Multipath TCP has two functional goals when it comes to using multiple paths 🛣️💖

  • Improve Throughput: Multipath TCP MUST support the concurrent use of multiple paths. To meet the minimum performance incentives for deployment, a Multipath TCP connection over multiple paths SHOULD achieve no worse throughput than a single TCP connection over the best constituent path. 🚀
  • Improve Resilience: Multipath TCP MUST support the use of multiple paths interchangeably for resilience purposes, by permitting segments to be sent and re-sent on any available path. It follows that, in the worst case, the protocol MUST be no less resilient than regular single-path TCP. 🛡️

Distribution of traffic among available paths and responses to congestion are done according to resource pooling principles [3]. So a secondary effect of hitting these goals is that widespread use of Multipath TCP over the Internet should improve overall network utility, by shifting load away from congested bottlenecks and by using spare capacity wherever possible. It's giving network-wide glow-up 🌈✨

Also, Multipath TCP SHOULD feature automatic negotiation of its use. A host that supports Multipath TCP and needs the other host to do the same must be able to reliably detect whether that other host really supports the required extensions. If it does, the connection uses them. If it doesn't, it automatically falls back to single-path TCP. No drama, just vibes 🫶

2.2. Compatibility Goals 🎀

Besides the functional goals above, a Multipath TCP must meet a bunch of compatibility goals so it can actually be deployed on today's Internet. These goals fall into the following categories.

2.2.1. Application Compatibility 💅

Application compatibility is about how Multipath TCP looks to the application. That covers both the API that can be used and the service model that gets provided.

Multipath TCP MUST follow the same service model as TCP [1]: in-order, reliable, and byte-oriented delivery. 📦 Furthermore, a Multipath TCP connection SHOULD provide the application with no worse throughput or resilience than it would expect from running a single TCP connection over any one of its available paths. A Multipath TCP may not, however, be able to provide the same level of consistency of throughput and latency as a single TCP connection. These and other application considerations are discussed in detail in [8]. 📚

A multipath-capable version of TCP MUST retain some level of backward compatibility with existing TCP APIs. That way, existing applications can use the newer transport just by upgrading the operating systems of the end hosts. 🙌 This does not preclude an advanced API that lets multipath-aware applications specify preferences. It also doesn't stop users from configuring their systems differently from the default, for example switching the automatic use of multipath extensions on or off. 🎛️

Regular TCP sessions can already survive brief breaks in connectivity today, by keeping state at the end hosts before a timeout happens. It would be desirable to support similar session continuity in MPTCP. However, the circumstances could be different. In regular TCP the IP addresses stay constant across the break in connectivity, but in MPTCP a different interface may show up. 😳 It is desirable (but not mandated) to support this kind of "break-before-make" session continuity. This places constraints on security mechanisms, though, as discussed in Section 5.8. Timeouts for this function would be locally configured. ⏱️

2.2.2. Network Compatibility 🌐

In the traditional Internet architecture, network devices operate at the network layer and below, and the layers above the network layer only exist at the end hosts. This architecture, shown in Figure 2, was mostly followed at first. But the layering no longer matches the "ground truth" of the Internet, because middleboxes have multiplied [9]. Middleboxes regularly interpose on the transport layer, and sometimes they completely terminate transport connections. That leaves the application layer as the first real end-to-end layer, as shown in Figure 3. 👀

   +-------------+                                       +-------------+
   | Application |<------------ end-to-end ------------->| Application |
   +-------------+                                       +-------------+
   |  Transport  |<------------ end-to-end ------------->|  Transport  |
   +-------------+   +-------------+   +-------------+   +-------------+
   |   Network   |<->|   Network   |<->|   Network   |<->|   Network   |
   +-------------+   +-------------+   +-------------+   +-------------+
      End Host           Router             Router          End Host

                Figure 2: Traditional Internet Architecture
   +-------------+                                       +-------------+
   | Application |<------------ end-to-end ------------->| Application |
   +-------------+                     +-------------+   +-------------+
   |  Transport  |<------------------->|  Transport  |<->|  Transport  |
   +-------------+   +-------------+   +-------------+   +-------------+
   |   Network   |<->|   Network   |<->|   Network   |<->|   Network   |
   +-------------+   +-------------+   +-------------+   +-------------+
                                          Firewall,
      End Host           Router         NAT, or Proxy      End Host

                        Figure 3: Internet Reality

Middleboxes that interpose on the transport layer cause a loss of "fate-sharing" [10]. In other words, they often hold "hard" state, and when that state is lost or corrupted, the end-to-end transport connection is lost or corrupted too. 💔

The network compatibility goal requires that the multipath extension to TCP stays compatible with the Internet as it exists today. That includes making reasonable efforts to get through the common middleboxes, such as firewalls, NATs, and performance-enhancing proxies [9]. This requirement exists because middleboxes are a big deployment bottleneck for any transport that isn't TCP or UDP. It also means Multipath TCP has to look like TCP on the wire and use established TCP extensions where needed. 🧱 To keep the transport "end-to-end," Multipath TCP MUST preserve fate-sharing without making any assumptions about middlebox behavior. 🔒

A detailed analysis of middlebox behavior and its impact on the Multipath TCP architecture is in Section 7. Network compatibility also has to hold up in one more way: Multipath TCP MUST fall back to regular TCP if there are insurmountable incompatibilities for the multipath extension on a path. 🙏

Middleboxes may also make some TCP features exist on one subflow but not another. These are typically subflow-level features, such as selective acknowledgment (SACK) [11], so they don't affect connection-level behavior. In the future, any proposed TCP connection-level extensions should consider how they can coexist with MPTCP. 🔮

The modifications that support Multipath TCP stay at the transport layer, although some knowledge of the underlying network layer is required. Multipath TCP SHOULD work with IPv4 and IPv6 interchangeably. That means one connection may operate over both IPv4 and IPv6 networks. 🌈

2.2.3. Compatibility with Other Network Users 🤝

As a corollary to both network and application compatibility, the architecture must let new Multipath TCP flows coexist nicely with existing single-path TCP flows. They shouldn't compete for bandwidth too aggressively or too timidly, unless the application specifically asks for low-precedence operation, like with LEDBAT. 🐢 The use of multiple paths MUST NOT unduly harm users of single-path TCP at shared bottlenecks, beyond the impact that another single-path TCP flow would cause. Multiple Multipath TCP flows on a shared bottleneck MUST share bandwidth between each other, with fairness similar to what happens at a shared bottleneck with single-path TCP. ⚖️

2.3. Security Goals 🔐

The extension of TCP with multipath capabilities brings a number of new threats, which are analyzed in detail in [12]. The security goal for Multipath TCP is to provide a service no less secure than regular single-path TCP. This will be achieved through a combination of existing TCP security mechanisms (potentially modified to line up with the Multipath TCP extensions) and protection against the new multipath threats identified. The design decisions that come from this goal are presented in Section 5.8. 🛡️💜

SCTP [6] and MPTCP have several similarities. Both can use multiple addresses at the end hosts to get some multipath capability. In SCTP, the main use case is redundancy and mobility for multihomed hosts (that is, a single path changes one of its end host addresses). Simultaneous use of multiple paths is not supported. Extensions to support simultaneous multipath transport have been proposed [13], but they haven't been standardized yet. 🕰️ By far the most widely used stream-based transport protocol is TCP [1], though, and SCTP does not meet the network and application compatibility goals in Section 2.2. On the network side, various middleboxes (especially NATs) don't know about SCTP and end up blocking it. On the application side, applications have to actively choose to use SCTP, and given those deployment issues, very few do. 😬 MPTCP's compatibility goals are partly based on these observations about SCTP's deployment problems. 💖

3. An Architectural Basis for Multipath TCP ✨

okay so this section is where the authors show ONE possible transport architecture that they think can actually support the MPTCP goals 💖. The model they borrow from is called Tng ("Transport next-generation") [14]. Tng isn't the only possible multipath setup, but it's packed with lessons from past transport research and real deployments, so it's a seriously strong starting point 🎀 for thinking about the current Internet and what any new transport or transport extension has to deal with 🧚.

          +------------------+
          |    Application   |
          +------------------+  ^ Application-oriented transport
          |                  |  | functions (Semantic Layer)
          + - - Transport - -+ ----------------------------------
          |                  |  | Network-oriented transport
          +------------------+  v functions (Flow+Endpoint Layer)
          |      Network     |
          +------------------+
            Existing Layers             Tng Decomposition

              Figure 4: Decomposition of Transport Functions

Tng loosely splits the transport layer into "application-oriented" and "network-oriented" layers (see Figure 4) 🦋. The application-oriented "Semantic" layer handles stuff driven mainly by supporting and protecting the app's end-to-end communication, while the network-oriented "Flow+Endpoint" layer does things like endpoint identification (using port numbers) and congestion control 💅. Those network-y functions traditionally lived in the "end-to-end" Transport layer, but in practice network operators and the middleboxes they deploy care a LOT about them 👀. Middleboxes use them to enforce network usage policies [15] [16] or to optimize communication performance [17] 🔥.

Figure 5 shows how middleboxes interact with each layer in this decomposed model 🌈: the application-oriented layer operates end-to-end, while the network-oriented layer operates "segment-by-segment" and can be interposed upon by middleboxes 👑.

+-------------+                                       +-------------+
| Application |<------------ end-to-end ------------->| Application |
+-------------+                                       +-------------+
|  Semantic   |<------------ end-to-end ------------->|  Semantic   |
+-------------+   +-------------+   +-------------+   +-------------+
|Flow+Endpoint|<->|Flow+Endpoint|<->|Flow+Endpoint|<->|Flow+Endpoint|
+-------------+   +-------------+   +-------------+   +-------------+
|   Network   |<->|   Network   |<->|   Network   |<->|   Network   |
+-------------+   +-------------+   +-------------+   +-------------+
                     Firewall         Performance
   End Host          or NAT        Enhancing Proxy      End Host

              Figure 5: Middleboxes in the New Internet Model

MPTCP's architectural design follows Tng's decomposition, as shown in Figure 6 🫶. MPTCP gives application compatibility by preserving TCP-like semantics, meaning global ordering of application data and reliability, so it's an instantiation of the "application-oriented" Semantic layer ✨. The subflow TCP component gives network compatibility by appearing and behaving as a TCP flow in the network, so it's an instantiation of the "network-oriented" Flow+Endpoint layer 🌸.

  +--------------------------+    +-------------------------------+
  |      Application         |    |          Application          |
  +--------------------------+    +-------------------------------+
  |        Semantic          |    |             MPTCP             |
  |------------+-------------|    + - - - - - - - + - - - - - - - +
  | Flow+Endpt | Flow+Endpt  |    | Subflow (TCP) | Subflow (TCP) |
  +------------+-------------+    +---------------+---------------+
  |   Network  |   Network   |    |       IP      |       IP      |
  +------------+-------------+    +---------------+---------------+

     Figure 6: Relationship between Tng (Left) and MPTCP (Right)

So as a protocol extension to TCP, MPTCP openly acknowledges middleboxes in its design 🙈. It specifies a protocol that works at two scales: the MPTCP component operates end-to-end, while it lets the TCP component operate segment-by-segment 💜.

4. A Functional Decomposition of MPTCP 🧩

the last two sections covered the goals and the decomposition basis, and this one breaks down the functional pieces inside the MPTCP design 🎀.

MPTCP uses what look like standard TCP sessions to the network, called "subflows", to provide the underlying transport per path, so network compatibility is kept 🦋. MPTCP-specific information gets carried in a TCP-compatible way. That mechanism is separate from the actual information being transferred, though, so it could evolve in future revisions 🌸. Figure 7 illustrates the layered architecture ✨.

                                 +-------------------------------+
                                 |           Application         |
    +---------------+            +-------------------------------+
    |  Application  |            |             MPTCP             |
    +---------------+            + - - - - - - - + - - - - - - - +
    |      TCP      |            | Subflow (TCP) | Subflow (TCP) |
    +---------------+            +-------------------------------+
    |      IP       |            |       IP      |      IP       |
    +---------------+            +-------------------------------+

    Figure 7: Comparison of Standard TCP and MPTCP Protocol Stacks

Sitting below the application, the MPTCP extension manages multiple TCP subflows underneath it 💅. To pull that off, it MUST implement the following functions:

  • Path Management 🗺️: This is the function that detects and uses multiple paths between two hosts. MPTCP treats the presence of multiple IP addresses at one or both hosts as the sign of that. The path management features are the mechanisms to signal alternative addresses to hosts, plus the mechanisms to set up new subflows joined to an existing MPTCP connection.
  • Packet Scheduling 📦: This breaks the byte stream received from the application into segments to transmit on one of the available subflows. The MPTCP design uses a data sequence mapping that associates segments sent on different subflows with a connection-level sequence numbering, so segments on different subflows can be correctly re-ordered at the receiver 🔁. The packet scheduler depends on information about path availability exposed by path management, and then uses the subflows to transmit queued segments. This function is also responsible for connection-level re-ordering when packets arrive from the TCP subflows, according to the attached data sequence mappings.
  • Subflow (single-path TCP) Interface 🔗: A subflow component takes segments from the packet-scheduling component and transmits them over the specified path, ensuring detectable delivery to the host.

MPTCP uses TCP underneath for network compatibility, and TCP ensures in-order, reliable delivery 💜. TCP adds its own sequence numbers to the segments, and those are used to detect and retransmit lost packets at the subflow layer. On receipt, the subflow passes its reassembled data up to the packet scheduling component for connection-level reassembly. The data sequence mapping from the sender's packet scheduling component lets the whole byte stream be re-ordered 🌈.

  • Congestion Control 🚦: This coordinates congestion control across the subflows. As specified, this congestion control algorithm MUST ensure that an MPTCP connection does not unfairly take more bandwidth than a single path TCP flow would take at a shared bottleneck. An algorithm to support this is specified in [7].

These functions fit together like this 🧃. Path management looks after discovering (and, if necessary, initializing) multiple paths between two hosts. The packet scheduler then gets a stream of data from the application headed for the network and does its thing on it, like segmenting the data into connection-level segments and adding a connection-level sequence number, before sending it on to a subflow. The subflow then adds its own sequence number and ACKs, and passes them to the network 🛫. The receiving subflow re-orders data (if necessary) and hands it to the packet scheduling component, which does connection-level re-ordering and sends the data stream up to the application 👑. Finally, the congestion control component exists as part of packet scheduling, so it can schedule which segments get sent at what rate on which subflow 🫶.

5. High-Level Design Decisions 🎯

there seems to be a wide range of choices when designing a multipath extension to TCP 🤔. BUT the goals we talked about earlier in this document constrain the possible solutions, so there's relatively little choice in a lot of areas 🙈. This section outlines high-level design choices that draw from the architectural basis discussed earlier in Section 3, which the design of MPTCP [5] takes into account 💖.

5.1. Sequence Numbering ✨

okay so MPTCP runs two levels of sequence spaces 🎀. There's a connection-level sequence number, and then each subflow gets its own sequence number too. This is what lets the connection do segmentation, reassembly, and retransmission of the same chunk of connection-level sequence space over different subflow-level sequence space. iconic, honestly 💅

the alternative would be one single connection-level sequence number that gets sent across multiple subflows, and bestie, it has two big problems 😭. First, each subflow looks to the network like a regular TCP session with gaps in its sequence space. That can confuse middleboxes like intrusion detection systems or certain transparent proxies, which means it would go against the network compatibility goal 🙈. Second, the sender couldn't tell which path a packet loss or reception belonged to when the same segment went out on multiple paths (aka retransmissions). not me getting lost in that 💀

the sender has to tell the receiver how to put the data back together for the application 🧩. so the receiver needs to figure out how the subflow-level data (the stuff carrying subflow sequence numbers) maps onto the connection level. this is called the "data sequence mapping" 🦋. the mapping can be written as a tuple of (data sequence number, subflow sequence number, length). so for a given number of bytes (that's the length), the subflow sequence space starting at the given sequence number maps onto the connection-level sequence space starting at the given data sequence number. where this info comes from could honestly vary a lot 🤷‍♀️

one option is to reuse the fields TCP already has (subflow sequence number, length) and just add the data sequence number to each segment, like as a TCP option. the catch is that middleboxes that re-segment or assemble data would mess with it, because there's no specified behavior for coalescing TCP options 😭. and if you signaled (data sequence number, length) instead, it would still break with middleboxes that coalesce segments without understanding MPTCP signaling, since they won't rewrite the options correctly 💔

because of all that drama, MPTCP's design decision is this: whenever a mapping for subflow data needs to be shared with the other host, all three pieces (data seq, subflow seq, length) must be sent 🫶. to cut down on overhead, it's allowed for the mapping to be sent periodically and cover more than one segment. more experimentation is still needed to figure out the tradeoffs for how often mappings should go out 🔍. and it could be skipped entirely for a connection that hasn't started using more than one subflow yet, since the data-level and subflow-level sequence spaces are the same at that point, literally no need 🌸

5.2. Reliability and Retransmissions 🦋

MPTCP has acknowledgements at the connection level and at the subflow level, which gives the application a super robust service 👑.

under normal behavior, MPTCP can use the data sequence mapping and the subflow ACKs to figure out when a connection-level segment has been received ✨. TCP ACKs for a subflow get handled entirely at the subflow level, because that keeps TCP semantics intact and lets subflow-level retransmissions trigger properly 🎀. but this has some end-to-end consequences, ngl. once a segment is ACKed at the subflow level, it can't be thrown out of the reorder buffer at the connection level anymore 😬. also, unlike regular TCP, a receiver can't just drop out-of-order segments when it needs to (say, because memory is under pressure). sometimes it might be worth dropping segments after they've been ACKed on the subflow but before they reach the app, and a connection-level ACK is what makes that possible 🧚

and honestly there are cases where connection-level ACKs make things more robust 💅. picture a subflow going through a transparent proxy: if the proxy ACKs a segment and then crashes, the sender won't retransmit that lost segment on another subflow, because it thinks it was already received 💀. the connection just grinds to a halt even though it has other working subflows, and the sender has no way to figure out what went wrong. one example where this could happen is moving between wireless access points that each run a transport-level proxy 📶. last but not least, as an optimization, a connection-level ACK might be sent over the path with the shortest Round-Trip Time (RTT), which could reduce send buffer requirements (see Section 5.3) 🍓

so, to build a fully robust multipath TCP solution under all those constraints, MPTCP for use on the public Internet MUST feature explicit connection-level acknowledgements, on top of the subflow-level ones 👑. a connection-level ACK is only really needed to signal when the receive window moves forward. the heuristics for when to use that signal are covered in more detail in the protocol specification [5] 📖

now retransmissions 🔁. it MUST be possible for a segment to be retransmitted on a different subflow from the one it was originally sent on 🫶. this is one of MPTCP's core goals, since it keeps data intact during temporary or permanent subflow failures, and the dual sequence number space is what makes it possible ✨

the scheduling of retransmissions is going to hugely affect the MPTCP user experience 🙈. the current MPTCP spec suggests that data still outstanding on subflows that have timed out should get rescheduled for transmission on other subflows. the goal is to keep disruption low when a path breaks, so it uses the first timeout as the trigger. more conservative versions would wait for the second or third timeout for the same segment 🐢

usually, a fast retransmit on one subflow won't trigger a retransmission on another subflow, although that might still be worth doing in certain cases, like to reduce receive buffer requirements 🧠. but in all cases where retransmissions go out on different subflows, the lost segments SHOULD still be sent on the path that lost them 🛤️. right now that's believed to be necessary for maintaining subflow integrity, which ties back to the network compatibility goal. the downside is that some efficiency gets lost, and at this point it's unclear what the optimal retransmit strategy actually is 🤔

so yeah, large-scale experiments are needed to figure out the most appropriate retransmission strategy 🧪. the recommendations will get refined once more information is available 💜

5.3. Buffers 🧺

MPTCP must use a connection-level receive buffer to make sure segments arrive in order 📦. segments sit in there until they're in order and the application can read them ✨

in regular, single-path TCP, it's usually recommended to set the receive buffer to 2*BDP, where BDP is the Bandwidth-Delay Product, meaning BDP = BW*RTT with BW being Bandwidth and RTT being Round-Trip Time 📐. one BDP covers segments being reordered by the network. the other BDP lets the connection keep going during a fast retransmit: when a segment gets fast retransmitted, the receiver has to be able to keep storing incoming data for one more RTT 🫧

for MPTCP, it gets a bit more complicated, not gonna lie 😭. the ultimate goal is that a packet loss or failure on one subflow shouldn't hurt the throughput of the other working subflows. so the receiver needs enough buffering to store all the data until the missing segment gets retransmitted and reaches the destination 🛡️

the worst-case scenario is when the subflow with the highest RTT/RTO (Round-Trip Time or Retransmission TimeOut) hits a timeout. in that case, the receiver has to buffer data from every subflow for the whole duration of the RTO 💀. so the smallest connection-level receive buffer that avoids stalling during subflow failures is sum(BW_i)*RTO_max, where BW_i is the Bandwidth of each subflow and RTO_max is the largest RTO across all the subflows 📊

this is an order of magnitude more than what a single connection needs, and it's probably too expensive to be practical 😩. so a more sensible requirement is avoiding stalls when there are no timeouts. therefore, the RECOMMENDED receive buffer is 2*sum(BW_i)*RTT_max, where RTT_max is the largest RTT across all subflows 🌈. this sizing makes sure subflows don't stall when a fast retransmit gets triggered on any subflow, which is so good 👑

the resulting buffer size should be small enough for practical use 🤏. but there can be extreme cases, like when fast, high-throughput paths (say 100 Mb/s with a 10 ms RTT) get used together with slow paths (say 1 Mb/s with a 1000 ms RTT) 🐌. in that case, the required receive buffer would be 12.5 MB, which is likely way too big 😵. in extreme cases like this one, it might be smart to only use the fastest available paths for the MPTCP connection, and keep the slow path(s) for backup only 🔄

Send Buffer: the RECOMMENDED send buffer is the same size as the recommended receive buffer, which is 2*sum(BW_i)*RTT_max 🧁. this is because the sender has to locally store the segments it sent that haven't been acknowledged by the connection-level ACK. the send buffer size really matters for hosts that keep a huge number of connections going at once 🖥️. if the required send buffer ends up too large, a host can choose to only send data on the fast subflows, and only use the slow subflows when something fails 💖

5.4. Signaling ✨

okay so MPTCP uses regular TCP for its subflows, which means an MPTCP connection starts out as just one normal TCP connection 🫶 but it still has to tell the other side "hey, I speak MPTCP and I want to use it on this connection" 💅 and the way TCP does that kind of thing is with a TCP option, so that's what we use. 🎀

more signaling is needed once the session is running too, like for putting data back together across multiple subflows, and for telling the other host about other IP addresses it could use 🦋

the MPTCP design uses TCP options for all of this extra signaling. this was picked as the option that fits best with the goals from Section 2 🌸 signaling in options gets sent separately from the data, so it can be created and processed on its own, and it keeps things architecturally compatible with network entities 💖

this decision came from the Working Group after some very detailed chats at IETF78, and the main reasons are:

  • TCP options are the traditional signaling method for TCP 🧚
  • A TCP option on a SYN is the most compatible way for an end host to signal that it is MPTCP capable 💜
  • If connection-level ACKs are signaled in the payload, they may suffer from packet loss and may be congestion-controlled 😭 which can affect forward data throughput and could lead to head-of-line blocking 💀
  • Middleboxes, such as NAT traversal helpers, can easily parse TCP options, e.g., to rewrite addresses 🌈

ngl there are downsides to TCP options compared with TLV encoding in the payload, though 🙈

  • there is limited space for signaling messages 🍓
  • a middlebox may, potentially, drop a packet with an unknown option 😬
  • the transport of control information in options is not necessarily reliable 💔

the detailed MPTCP design tackles these as much as it can by carefully thinking about how big MPTCP options are, and by falling back smoothly to regular TCP when control data gets lost ✨

both option and payload encoding might get in the way of offloading TCP processing to high-speed network interface cards, like segmentation, checksumming, and reassembly. for network cards that support MPTCP, signaling in TCP options should make offloading simpler, since MPTCP signaling and data get handled separately 🔥

5.5. Path Management 🦋

okay so right now the network doesn't show you path diversity between pairs of IP addresses 🙈 to get path diversity out of today's IP networks, MPTCP usually uses multiple addresses at one or both hosts to figure out that there are different paths across the network 🌸 the paths might not be totally non-overlapping, but they're expected to be disjoint enough for multipath to get better throughput and robustness 💅 using multiple IP addresses is a simple trick that doesn't need any extra features in the network, which is iconic 👑

so in most cases, the path selectors are different (source, destination) address pairs 🧚 but each path is identified by a standard five-tuple (source address, destination address, source port, destination port, protocol), and that opens the door to MPTCP using ports as well as addresses as path selectors ✨ hosts could then use port-based load balancing with MPTCP, for example if the network sends different ports over different paths (which can happen with things like Equal Cost MultiPath (ECMP) routing [4]) 🎀

heads up though: ISPs often do traffic engineering to make their resource use as efficient as possible, so ISPs and developers both need to be careful that MPTCP using pretty similar paths doesn't mess with that ⚠️

to get a better shot at setting up extra subflows (like when one end is stuck behind a firewall, NAT, or some other restrictive middlebox), either host SHOULD be able to add new subflows to an MPTCP connection. MPTCP MUST be able to handle paths that appear and disappear during the lifetime of a connection, for example when an extra network interface gets activated. 💖

path management is its own separate function from packet scheduling, the subflow interface, and congestion control, as covered in Section 4 💅 which means it's totally feasible to swap this IP-address-based design for a different path selection mechanism later on, with no major changes to the other functional components 🦋

5.6. Connection Identification 🔥

an MPTCP connection might not stay bound to a traditional 5-tuple (source address and port, destination address and port, protocol number) for its whole life 😭 so it's really useful to have a new way to identify connections 💜 that helps MPTCP-aware applications, the MPTCP implementation, and MPTCP-aware middleboxes all have one unique ID to link the multiple subflows together 🫶

so every MPTCP connection needs a connection identifier at each host, and it only has to be unique within that host 🌈 honestly this is kind of like an ephemeral port number in regular TCP 🧚 how that identifier shows up and what it's for is out of scope for this architecture document 🍓

non-MPTCP-aware applications won't have access to this identifier, so for them an MPTCP connection is identified by the 5-tuple of the first TCP subflow 🌸 what the MPTCP implementation should do if that first TCP subflow fails later is also out of scope for this document 🙈 if there are MPTCP-unaware applications that assume the initial address pair will keep existing, carrying on regardless could mess up their behavior 💀 it's expected to be a very small, maybe even negligible, group of applications though ✨

and MPTCP MUST NOT be used for applications that ask to bind to a specific address or interface 😤 those apps are deliberately picking a path, so MPTCP would be going against that choice 🔥

still, since application requirements aren't clear yet, it's not confirmed whether carrying on after losing the initial address pair would be a bad assumption to make 🤔 that behavior will be an implementation-specific solution, so implementors are expected to pick it once more research has looked at what its impact is 💅

5.7. Congestion Control 👑

as covered in the network-layer compatibility requirements in Section 2.2.3, an MPTCP implementation's congestion control algorithms have three goals 🫶 first, improve throughput, at least as well as a single-path TCP connection would perform 📈 second, do no harm to other network users, meaning don't take up more capacity on any one path than a single path flow using only that route would, which matters a lot for shared bottlenecks 🛡️ third, balance congestion by moving traffic away from the most congested paths 🌈

to hit these goals, the congestion control algorithms on each subflow have to be coupled somehow 🧚 a proposal for a suitable congestion control algorithm is given in [7] ✨

5.8. Security 💀

okay this one is super serious bestie 😭 a detailed threat analysis for Multipath TCP lives in a separate document [12] 🔥 that doc focuses on flooding attacks and hijacking attacks that could be launched against a Multipath TCP connection 🙈

the basic security goal of Multipath TCP, from Section 2.3, is: "provide a solution that is no worse than standard TCP" 💅

from the threat analysis, with that goal in mind, three key security requirements stand out 🎀 a multi-addressed Multipath TCP SHOULD be able to do the following:

  • Provide a mechanism to confirm that the parties in a subflow handshake are the same as in the original connection setup (e.g., require use of a key exchanged in the initial handshake in the subflow handshake, to limit the scope for hijacking attacks) 🔐
  • Provide verification that the peer can receive traffic at a new address before adding it (i.e., verify that the address belongs to the other host, to prevent flooding attacks) 🛑
  • Provide replay protection, i.e., ensure that a request to add/remove a subflow is 'fresh' 🍓

additional mechanisms have already been added to standard TCP stacks to resist Denial-of-Service (DoS) attacks 🧚 for example, there are various mechanisms to protect against TCP reset attacks [18], and Multipath TCP should keep supporting similar protection 💖 TCP SYN Cookies [19] were also made so a TCP server can put off creating session state in the SYN_RCVD state, staying stateless until the ESTABLISHED state is reached 🌸 Multipath TCP should, ideally, keep providing that too, and at minimum avoid a big computational burden before reaching the ESTABLISHED state (of the Multipath TCP connection as a whole) ✨

aspects of the Multipath TCP design space put some constraints on the security solution, though 😬

  • the use of TCP options really limits how much information can be carried in the handshake 📦
  • needing to work through middleboxes means packets' mutability has to be handled 🦋
  • the desire to support a 'break-before-make' (as well as a 'make-before-break') approach to adding subflows (within a limited time period) means a host can't rely on using a pre-existing subflow to support adding a new one 🌈

the MPTCP protocol will be designed with these security requirements in mind, and the protocol specification [5] will document how they get met 👑

6. Software Interactions 💅

6.1. Interactions with Applications 🫶

for applications that have used an existing API call to bind to a specific address or interface, the MPTCP extension MUST NOT be used 🙅 that's because those apps are clearly choosing a path, so they expect certain behavior that has to be kept, in order to stick to the application compatibility goals 💖

interactions with applications are covered in [8], including (but not limited to) performance changes you might expect, semantic changes, and new features that might be requested through an enhanced API 🎀

TCP can send "Urgent" data, and delivering that to the application may or may not be out-of-band 😬 using this feature is not recommended, because of security implications and implementation differences [20] 🚫 MPTCP needs contiguous data to support its data sequence mapping over multiple segments, so the Urgent pointer can't interrupt an existing mapping 🔥 an MPTCP implementation MAY choose to support sending Urgent data, and if it does, it SHOULD send the Urgent data on the soonest available unassigned subflow sequence space 🌈 incoming Urgent data SHOULD be mapped to connection-level sequence space and delivered to the application in the same way as Urgent data in regular TCP 🧚✨

6.2. Interactions with Management Systems 📡

okay so, to let TCP and network management systems actually talk to each other, the TCP [21] MIB and the TCP Extended Statistics (ESTATS) [22] MIB already exist 🌸. MPTCP should share these MIBs for the aspects that are designed to stay transparent to the application 💅.

It's anticipated that an MPTCP MIB will be defined in the future, once experience of experimental MPTCP deployments is gathered 🔮. This MIB would give access to MPTCP-specific properties, like whether MPTCP is enabled and the number and properties of the individual paths in use 🛣️✨.

7. Interactions with Middleboxes 🧱

As discussed in Section 2.2, it is a goal of MPTCP to be deployable today and thus compatible with the majority of middleboxes 🫶. This section summarizes the issues that may arise with NATs, firewalls, proxies, intrusion detection systems, and other middleboxes that, if not considered in the protocol design, may hinder its deployment.

This section is intended primarily as a description of options and considerations only. Protocol-specific solutions to these issues will be given in the companion documents 📝.

Multipath TCP will be deployed in a network that no longer provides just basic datagram delivery 🌐. A myriad of middleboxes are deployed to optimize various perceived problems with the Internet protocols: NATs primarily address IP address space shortage [15], Performance Enhancing Proxies (PEPs) optimize TCP for different link characteristics [17], firewalls [16] and intrusion detection systems try to block malicious content from reaching a host, and traffic normalizers [23] ensure a consistent view of the traffic stream to Intrusion Detection Systems (IDS) and hosts. It's giving a lot of middlebox main character energy 💜.

All these middleboxes optimize current applications at the expense of future applications 😤. In effect, future applications will often need to behave in a similar fashion to existing ones, in order to increase the chances of successful deployment. Further, the precise behavior of all these middleboxes is not clearly specified, and implementation errors make matters worse, raising the bar for the deployment of new technologies 💀.

The following list of middlebox classes documents behavior that could impact the use of MPTCP. This list is used in [5] to describe the features of the MPTCP protocol that are used to mitigate the impact of these middlebox behaviors 🦋.

  • NATs: Network Address Translators decouple the host's local IP address (and, in the case of NAPTs, port) with that which is seen in the wider Internet when the packets are transmitted through a NAT. This adds complexity, and reduces the chances of success, when signaling IP addresses.
  • PEPs: Performance Enhancing Proxies, which aim to improve the performance of protocols over low-performance (e.g., high-latency or high-error-rate) links. As such, they may "split" a TCP connection and behavior such as proactive ACKing may occur, and therefore it is no longer guaranteed that one host is communicating directly with another. PEPs, firewalls, or other middleboxes may also change the declared receive window size.
  • Traffic Normalizers: These aim to eliminate ambiguities and potential attacks at the network level, and amongst other things, are unlikely to permit holes in TCP-level sequence space (which has an impact on MPTCP's retransmission and subflow sequence numbering design choices) 🧹.
  • Firewalls: on top of preventing incoming connections, firewalls may also attempt additional protection such as sequence number randomization (so a sender cannot reliably know what TCP sequence number the receiver will see) 🔒.
  • IDSs: Intrusion Detection Systems may look for traffic patterns to protect a network and may have false positives with MPTCP and drop the connections during normal operation. Future MPTCP-aware middleboxes will require the ability to correlate the various paths in use 🚨.
  • Content-Aware Firewalls: Some middleboxes may actively change data in packets, such as rewriting URIs in HTTP traffic 🙈.

In addition, all classes of middleboxes may affect TCP traffic in the following ways 👀:

  • TCP Options: some middleboxes may drop packets with unknown TCP options or strip those options from the packets.
  • Segmentation and Coalescing: middleboxes (or even something as close to the end host as TCP Segmentation Offloading (TSO) on a Network Interface Card (NIC)) may change the packet boundaries from those that the sender intended. It may do this by splitting packets or coalescing them together. This leads to two major impacts: where a packet boundary will be cannot be guaranteed and what a middlebox will do with TCP options in these cases (they may be repeated, dropped, or sent only once) cannot be said for sure 🫠.

8. Contributors 🎀

The authors would like to acknowledge the contributions of Andrew McDonald and Bryan Ford to this document 🫶.

The authors would also like to thank the following people for detailed reviews: Olivier Bonaventure, Gorry Fairhurst, Iljitsch van Beijnum, Philip Eardley, Michael Scharf, Lars Eggert, Cullen Jennings, Joel Halpern, Juergen Quittek, Alexey Melnikov, David Harrington, Jari Arkko, and Stewart Bryant 👑.

9. Acknowledgements 🌈

Alan Ford, Costin Raiciu, Mark Handley, and Sebastien Barre are supported by Trilogy (http://www.trilogy-project.org), a research project (ICT-216372) partially funded by the European Community under its Seventh Framework Program. The views expressed here are those of the author(s) only. The European Commission is not liable for any use that may be made of the information in this document 🇪🇺.

10. Security Considerations 🔐

This part is seriously important bestie, so please read it carefully 🚨.

This informational document provides an architectural overview for Multipath TCP and so does not, in itself, raise any security issues. A separate threat analysis [12] lists threats that can exist with a Multipath TCP. However, a protocol based on the architecture in this document will have a number of security requirements. The high-level goals for such a protocol are identified in Section 2.3, whilst Section 5.8 provides more detailed discussion of security requirements and design decisions which are applied in the MPTCP protocol design [5]. So yes, the architecture itself is chill, but any protocol built on it has real security work to do 😭💅.

11. References 📚

11.1. Normative References

[1] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.

[2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.

11.2. Informative References

[3] Wischik, D., Handley, M., and M. Bagnulo Braun, "The Resource Pooling Principle", ACM SIGCOMM CCR vol. 38 num. 5, pp. 47-52, October 2008, <http://ccr.sigcomm.org/online/files/p47-handleyA4.pdf>.

[4] Hopps, C., "Analysis of an Equal-Cost Multi-Path Algorithm", RFC 2992, November 2000.

[5] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", Work in Progress, March 2011.

[6] Stewart, R., "Stream Control Transmission Protocol", RFC 4960, September 2007.

[7] Raiciu, C., Handley, M., and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", Work in Progress, March 2011.

[8] Scharf, M. and A. Ford, "MPTCP Application Interface Considerations", Work in Progress, March 2011.

[9] Carpenter, B. and S. Brim, "Middleboxes: Taxonomy and Issues", RFC 3234, February 2002.

[10] Carpenter, B., "Internet Transparency", RFC 2775, February 2000.

[11] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP Selective Acknowledgment Options", RFC 2018, October 1996.

[12] Bagnulo, M., "Threat Analysis for TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6181, March 2011.

[13] Becke, M., Dreibholz, T., Iyengar, J., Natarajan, P., and M. Tuexen, "Load Sharing for the Stream Control Transmission Protocol (SCTP)", Work in Progress, December 2010.

[14] Ford, B. and J. Iyengar, "Breaking Up the Transport Logjam", ACM HotNets, October 2008.

[15] Srisuresh, P. and K. Egevang, "Traditional IP Network Address Translator (Traditional NAT)", RFC 3022, January 2001.

[16] Freed, N., "Behavior of and Requirements for Internet Firewalls", RFC 2979, October 2000.

[17] Border, J., Kojo, M., Griner, J., Montenegro, G., and Z. Shelby, "Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations", RFC 3135, June 2001.

[18] Ramaiah, A., Stewart, R., and M. Dalal, "Improving TCP's Robustness to Blind In-Window Attacks", RFC 5961, August 2010.

[19] Eddy, W., "TCP SYN Flooding Attacks and Common Mitigations", RFC 4987, August 2007.

[20] Gont, F. and A. Yourtchenko, "On the Implementation of the TCP Urgent Mechanism", RFC 6093, January 2011.

[21] Raghunarayan, R., "Management Information Base for the Transmission Control Protocol (TCP)", RFC 4022, March 2005.

[22] Mathis, M., Heffner, J., and R. Raghunarayan, "TCP Extended Statistics MIB", RFC 4898, May 2007.

[23] Handley, M., Paxson, V., and C. Kreibich, "Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics", Usenix Security 2001, 2001, <http://www.usenix.org/events/sec01/full_papers/handley/handley.pdf>.

Authors' Addresses 💌

Alan Ford
Roke Manor Research
Old Salisbury Lane
Romsey, Hampshire  SO51 0ZN
UK
Phone: +44 1794 833 465
EMail: alan.ford@roke.co.uk

Costin Raiciu
University College London
Gower Street
London  WC1E 6BT
UK
EMail: c.raiciu@cs.ucl.ac.uk

Mark Handley
University College London
Gower Street
London  WC1E 6BT
UK
EMail: m.handley@cs.ucl.ac.uk

Sebastien Barre
Universite catholique de Louvain
Pl. Ste Barbe, 2
Louvain-la-Neuve  1348
Belgium
Phone: +32 10 47 91 03
EMail: sebastien.barre@uclouvain.be

Janardhan Iyengar
Franklin and Marshall College
Mathematics and Computer Science
PO Box 3003
Lancaster, PA  17604-3003
USA
Phone: 717-358-4774
EMail: jiyengar@fandm.edu
📦 where this page comes from