RFC 6824
TCP Extensions for Multipath Operation with Multiple Addresses
🎀 TCP Extensions for Multipath Operation with Multiple Addresses 🎀
RFC 6824 · Part 1 of 17 🦋 more parts coming, bestie
Okay so this doc is the Experimental spec for Multipath TCP (MPTCP) 💖 It's written by A. Ford (Cisco), C. Raiciu (U. Politechnica of Bucharest), M. Handley (U. College London), and O. Bonaventure (U. catholique de Louvain). It was published in January 2013 and its ISSN is 2070-1721. The header doesn't list any RFCs it obsoletes or updates, so it's standing on its own ✨
Abstract 📝
Okay, so TCP/IP connections are currently stuck on one path each, even though there are usually multiple paths between peers 🙈 Using several of those paths at once would make better use of network resources, which means higher throughput and better resilience when something in the network fails. Iconic, honestly.
So Multipath TCP lets peers use multiple paths simultaneously 🌈 This doc adds a set of extensions to regular TCP to make that work. It gives apps the same service TCP already does (a reliable bytestream 💅), and it has what's needed to set up and use multiple TCP flows across paths that might be totally disjoint.
Status of This Memo 📜
This document is not an Internet Standards Track spec. It's published so people can examine it, try it out in experimental implementations, and evaluate it 🧪
It defines an Experimental Protocol for the Internet community. It's a product of the IETF, it represents IETF consensus, it got public review, and the IESG approved it for publication. Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you want the full story ✨
For the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6824 🔗
Copyright Notice ©️
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please actually read these, bestie, because they spell out your rights and restrictions 🫶 Code Components pulled out of this document MUST include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License.
Table of Contents 🗺️
1. Introduction ....................................................4
1.1. Design Assumptions .........................................4
1.2. Multipath TCP in the Networking Stack ......................5
1.3. Terminology ................................................6
1.4. MPTCP Concept ..............................................7
1.5. Requirements Language ......................................8
2. Operation Overview ..............................................8
2.1. Initiating an MPTCP Connection .............................9
2.2. Associating a New Subflow with an Existing MPTCP
Connection .................................................9
2.3. Informing the Other Host about Another Potential Address ..10
2.4. Data Transfer Using MPTCP .................................11
2.5. Requesting a Change in a Path's Priority ..................11
2.6. Closing an MPTCP Connection ...............................12
2.7. Notable Features ..........................................12
3. MPTCP Protocol .................................................12
3.1. Connection Initiation .....................................14
3.2. Starting a New Subflow ....................................18
3.3. General MPTCP Operation ...................................23
3.3.1. Data Sequence Mapping ..............................25
3.3.2. Data Acknowledgments ...............................28
3.3.3. Closing a Connection ...............................29
3.3.4. Receiver Considerations ............................30
3.3.5. Sender Considerations ..............................31
3.3.6. Reliability and Retransmissions ....................32
3.3.7. Congestion Control Considerations ..................33
3.3.8. Subflow Policy .....................................34
3.4. Address Knowledge Exchange (Path Management) ..............35
3.4.1. Address Advertisement ..............................36
3.4.2. Remove Address .....................................39
3.5. Fast Close ................................................40
3.6. Fallback ..................................................41
3.7. Error Handling ............................................45
3.8. Heuristics ................................................45
3.8.1. Port Usage .........................................46
3.8.2. Delayed Subflow Start ..............................46
3.8.3. Failure Handling ...................................47
4. Semantic Issues ................................................48
5. Security Considerations ........................................49
6. Interactions with Middleboxes ..................................51
7. Acknowledgments ................................................55
8. IANA Considerations ............................................55
9. References .....................................................57
9.1. Normative References ......................................57
9.2. Informative References ....................................57
Appendix A. Notes on Use of TCP Options ...........................59
Appendix B. Control Blocks ........................................60
B.1. MPTCP Control Block .......................................60
B.1.1. Authentication and Metadata ........................60
B.1.2. Sending Side .......................................61
B.1.3. Receiving Side .....................................61
B.2. TCP Control Blocks ........................................62
B.2.1. Sending Side .......................................62
B.2.2. Receiving Side .....................................62
Appendix C. Finite State Machine ..................................631. Introduction 🌸
Okay so Multipath TCP (MPTCP) is a set of extensions to regular TCP [1] that gives you a Multipath TCP [2] service 🦋 It lets one transport connection run across multiple paths at the same time. This doc covers the protocol changes required to add multipath to TCP: signaling and setting up multiple paths (called "subflows"), managing those subflows, reassembling data, and tearing down sessions.
But this isn't the only info you need to build an MPTCP implementation, lowkey 🙈 It has three buddies that go with it:
- Architecture [2]: explains why MPTCP exists, talks through the high-level design decisions this design is built on, and explains the functional separation that lets an extensible MPTCP implementation get developed 🧩
- Congestion control [5]: presents a safe congestion control algorithm for coupling the behavior of multiple paths so it does "no harm" to other network users 🫶
- Application considerations [6]: discusses what MPTCP means for apps, what apps will want to do with it, and which API extensions an MPTCP implementation SHOULD present as a result 💅
1.1. Design Assumptions ✨
To keep the potentially huge design space from getting out of hand, the working group put two key constraints on the Multipath TCP design in this doc:
- It MUST be backwards-compatible with current, regular TCP, so it has a better shot at getting deployed 🚀
- You can assume that one or both hosts are multihomed and multiaddressed 🏠
To keep the design simple, we assume that having multiple addresses at a host is enough to mean there are multiple paths 🛣️ These paths don't have to be totally disjoint. They may share one or many routers between them. Even then, using multiple paths is still a win, since it improves resource use and resilience to some node failures 💪 The congestion control algorithms in [5] make sure this doesn't backfire. Also, there may be scenarios where different TCP ports on one host can give you disjoint paths (like with certain Equal-Cost Multipath (ECMP) implementations [7]), so the MPTCP design also supports using ports in path identifiers 🎀
Now the backwards-compatibility part has three aspects, which are discussed in more detail in [2] 🧐
- External Constraints: The protocol MUST function through the vast majority of existing middleboxes, like NATs, firewalls, and proxies 🛡️ So it MUST look as much like existing TCP as possible on the wire. And the protocol MUST NOT assume that the segments it sends arrive unmodified at the destination. They may be split or coalesced, and TCP options may be removed or duplicated. Chaotic, but that's the internet 💀
- Application Constraints: The protocol MUST be usable with no changes to existing apps that use the common TCP API. (Not every feature will be available to those legacy apps, and that's reasonable 🙃) The protocol MUST also provide the same service model to the app that regular TCP does 🌈
- Fallback: The protocol SHOULD be able to fall back to standard TCP with no interference from the user, so it can talk to legacy hosts 🔁
The companion application considerations doc [6] goes over the features an API needs for backwards-compatibility, plus API extensions that convey MPTCP's behavior at the same level of control and information you'd get from regular single-path TCP 👑
For more on the design constraints and the design decisions that come from them, check out the MPTCP Architecture document [2] and [8] 📚
1.2. Multipath TCP in the Networking Stack 🧚
okay so MPTCP lives at the transport layer and it's trying SO hard to be invisible to the layers above and below it 💅 Think of it as a bundle of extra features sitting on top of regular TCP. Figure 1 shows the layering. It's also built so legacy apps can use it with zero changes 🙈 The deep dive on how it chats with applications is in [6].
+-------------------------------+
| Application |
+---------------+ +-------------------------------+
| Application | | MPTCP |
+---------------+ + - - - - - - - + - - - - - - - +
| TCP | | Subflow (TCP) | Subflow (TCP) |
+---------------+ +-------------------------------+
| IP | | IP | IP |
+---------------+ +-------------------------------+
Figure 1: Comparison of Standard TCP and MPTCP Protocol Stacks1.3. Terminology 📖
this doc uses a bunch of terms that are MPTCP-specific or have a special meaning for MPTCP, so here they are 🎀
- Path: a sequence of links between a sender and a receiver. In this context it's defined by a 4-tuple of source and destination address/port pairs.
- Subflow: a flow of TCP segments running over one individual path, forming part of a bigger MPTCP connection. It starts and ends pretty much like a regular TCP connection.
- (MPTCP) Connection: a set of one or more subflows that an application uses to talk between two hosts. There's a one-to-one mapping between a connection and an application socket.
- Data-level: the payload data is nominally sent over a connection, which in turn rides on subflows. So "data-level" is the same thing as "connection level", as opposed to "subflow-level", which describes the properties of one individual subflow.
- Token: a locally unique identifier a host gives to a multipath connection. It may also be called a "Connection ID".
- Host: an end host running an MPTCP implementation that either starts or accepts an MPTCP connection.
Beyond these, note that MPTCP's take on regular single-path TCP semantics, and how it changes them, is covered in Section 4 🦋
1.4. MPTCP Concept 💡
this section is the high-level tour of how MPTCP normally works, shown with the scenario in Figure 2. The full, detailed version is in Section 3 ✨
- To a non-MPTCP-aware application, MPTCP behaves exactly like normal TCP 🫶 Extended APIs could give MPTCP-aware applications extra control [6]. An app starts by opening a TCP socket the normal way, and MPTCP signaling and operation are handled by the MPTCP implementation.
- An MPTCP connection starts out similarly to a regular TCP connection. Figure 2 shows an MPTCP connection being established between addresses A1 and B1 on Hosts A and B, respectively.
- If extra paths are available, additional TCP sessions (called MPTCP "subflows") get created on those paths. They're combined with the existing session, which still looks like one single connection to the applications at both ends. Figure 2 shows the extra session being created between Address A2 on Host A and Address B1 on Host B 🌈
- MPTCP spots multiple paths by noticing multiple addresses on hosts. Combinations of those addresses make up the additional paths. In the example, other potential paths that could be set up are A1<->B2 and A2<->B2. Even though the extra session is shown starting from A2, it could just as easily have started from B1.
- The discovery and setup of extra subflows is handled by a path management method. This document describes a mechanism where a host can start new subflows using its own additional addresses, or by signaling its available addresses to the other host 🧚
- MPTCP adds connection-level sequence numbers so it can reassemble segments arriving on multiple subflows, even when those subflows have different network delays.
- Subflows end like regular TCP connections, with a four-way FIN handshake. The MPTCP connection itself ends with a connection-level FIN 🏁
Host A Host B
------------------------ ------------------------
Address A1 Address A2 Address B1 Address B2
---------- ---------- ---------- ----------
| | | |
| (initial connection setup) | |
|----------------------------------->| |
|<-----------------------------------| |
| | | |
| (additional subflow setup) |
| |--------------------->| |
| |<---------------------| |
| | | |
| | | |
Figure 2: Example MPTCP Usage Scenario1.5. Requirements Language 🎀
okay, the key words here are the official ones: "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL". They're to be read as described in RFC 2119 [3] 💖
2. Operation Overview 🌸
this section is one single description of how MPTCP works in general, with references to the protocol operation. It's a high-level overview of the key functions, and the full spec follows in Section 3. Extensibility and negotiated features aren't covered here. A lot of the section talks about symbolic names for MPTCP options, which are subtypes of the IANA-assigned MPTCP option (see Section 8). Their formats are defined in the detailed protocol spec in Section 3 ✨
A Multipath TCP connection gives you a bidirectional bytestream between two hosts that talk like normal TCP, so it doesn't require any changes to the applications 🦋 The big difference is that Multipath TCP lets the hosts use different paths, with different IP addresses, to swap packets that belong to the MPTCP connection. To an application, it still looks like a normal TCP connection. To the network layer, though, each MPTCP subflow looks like a regular TCP flow whose segments carry a new TCP option type. Multipath TCP handles creating, removing, and using these subflows to send data. The number of subflows managed within a Multipath TCP connection isn't fixed, and it can fluctuate over the connection's lifetime 🌈
All MPTCP operations are signaled with a TCP option. That's a single numerical type for MPTCP, with "sub-types" for each MPTCP message. What follows is a summary of the purpose and rationale of those messages 💜
2.1. Initiating an MPTCP Connection 🦋
this is the same signaling as starting a normal TCP connection, except the SYN, SYN/ACK, and ACK packets also carry the MP_CAPABLE option. It's variable length and does a few jobs. First, it checks whether the remote host supports Multipath TCP. Second, it lets the hosts swap some info to authenticate setting up additional subflows later. More details are in Section 3.1 🫶
Host A Host B
------ ------
MP_CAPABLE ->
[A's key, flags]
<- MP_CAPABLE
[B's key, flags]
ACK + MP_CAPABLE ->
[A's key, B's key, flags]2.2. Associating a New Subflow with an Existing MPTCP Connection 🫶
the key exchange in the MP_CAPABLE handshake gives the endpoints material they can use to authenticate each other when new subflows get set up. New subflows start the same way as a normal TCP connection, except the SYN, SYN/ACK, and ACK packets also carry the MP_JOIN option 🔥
Host A starts a new subflow between one of its addresses and one of Host B's addresses. The token, which is generated from the key, says which MPTCP connection it's joining. The HMAC handles authentication. The Hash-based Message Authentication Code (HMAC) uses the keys exchanged in the MP_CAPABLE handshake, plus the random numbers (nonces) exchanged in these MP_JOIN options. MP_JOIN also carries flags and an Address ID, which can be used to refer to the source address without the sender needing to know whether a NAT has changed it. More details are in Section 3.2 💅
Host A Host B
------ ------
MP_JOIN ->
[B's token, A's nonce,
A's Address ID, flags]
<- MP_JOIN
[B's HMAC, B's nonce,
B's Address ID, flags]
ACK + MP_JOIN ->
[A's HMAC]
<- ACK2.3. Informing the Other Host about Another Potential Address 🍓
the set of IP addresses tied to a multihomed host can change during an MPTCP connection's lifetime. MPTCP supports adding and removing addresses on a host, both implicitly and explicitly ✨ If Host A has a subflow going from address IP#-A1 and wants to open a second subflow starting at address IP#-A2, it just starts setting up that subflow as explained above. The remote host is then implicitly told about the new address 🧚
Sometimes a host wants to tell the remote host about an address without setting up a new subflow, like when a NAT blocks setup in one direction. In the example below, Host A tells Host B about its alternative IP address (IP#-A2). Host B may later send an MP_JOIN to this new address. Because middleboxes may translate IP addresses, this option uses an address identifier to unambiguously identify an address on a host. More details are in Section 3.4.1 🌈
Host A Host B
------ ------
ADD_ADDR ->
[IP#-A2,
IP#-A2's Address ID]There's a matching signal for address removal, which uses the Address ID signaled in the add address handshake. More details in Section 3.4.2 💀
Host A Host B
------ ------
REMOVE_ADDR ->
[IP#-A2's Address ID]2.4. Data Transfer Using MPTCP ✨
okay so this is where it gets spicy 🌶️ subflows can literally pop up or vanish whenever they want, so how does MPTCP keep everything reliable and in order? 🫶 Every bit of data across the whole MPTCP connection gets numbered with a 64-bit data sequence number (DSN) 🔢. Each subflow keeps its own 32-bit sequence number space, and an MPTCP option maps that subflow space onto the data sequence space. The payoff is that if a path fails, the same data can be resent on a different subflow, and it still gets the same DSN 🦋 main character energy for retransmissions, ngl.
The "Data Sequence Signal" is the thing that carries the "Data Sequence Mapping" 💖. The mapping has the subflow sequence number, the data sequence number, and the length for which that mapping is valid. This option can also carry a connection-level acknowledgment (the "Data ACK") for the DSN that was received 🌸.
All the subflows share one receive buffer and advertise the same receive window 🎀. There are two levels of acknowledgment, and I'm obsessed with this part 😭. Regular TCP acknowledgments happen on each subflow, acknowledging the segments sent over that subflow no matter what their DSN is. On top of that, there are connection-level acknowledgments for the data sequence space 👑. These track how far the bytestream has advanced and slide the receiving window forward ✨.
More details are in Section 3.3.
Host A Host B
------ ------
DATA_SEQUENCE_SIGNAL ->
[Data Sequence Mapping]
[Data ACK]
[Checksum]2.5. Requesting a Change in a Path's Priority 🌈
hosts can tell each other at subflow setup whether a subflow should be a regular path or a backup path 🧚. A backup path only gets used when there are no regular paths available, so it's basically the bestie you call when everyone else is busy 💅. During the connection, Host A can ask to change a subflow's priority by sending the MP_PRIO signal to Host B 🔥.
More details are in Section 3.3.8.
Host A Host B
------ ------
MP_PRIO ->2.6. Closing an MPTCP Connection 🦋
when Host A is done sending data and wants Host B to know, it signals a "Data FIN" as part of the Data Sequence Signal (see above) 💀. This works just like a regular TCP FIN, except it happens at the connection level 💜. Once every bit of data on the MPTCP connection has been received successfully, Host B acknowledges it at the connection level with a DATA_ACK 🥹 and that's the iconic ending.
More details are in Section 3.3.3.
Host A Host B
------ ------
DATA_SEQUENCE_SIGNAL ->
[Data FIN]
<- (MPTCP DATA_ACK)2.7. Notable Features 🎀
it's worth hyping up how MPTCP's signaling was designed around a few key requirements 🙈
- To cope with NATs on the path, addresses are referred to by Address IDs, in case the IP packet's source address gets changed by a NAT. Setting up a new TCP flow is not possible if the passive opener is behind a NAT, so to let subflows be created when either end is behind a NAT, MPTCP uses the
ADD_ADDRmessage 🌍. - MPTCP falls back to ordinary TCP if MPTCP operation is not possible. For example, that happens if one host isn't MPTCP capable, or if a middlebox alters the payload 🧸.
- To meet the threats identified in [9], the following steps are taken: keys are sent in the clear in the
MP_CAPABLEmessages;MP_JOINmessages are secured with HMAC-SHA1 ([10], [4]) using those keys; and standard TCP validity checks are made on the other messages (ensuring sequence numbers are in-window) 🔐 security is giving main character, and I respect it.
3. MPTCP Protocol 🧚♀️
this section is the nitty-gritty of how MPTCP actually works, and it's split up into sections for each key part of the operation 📚.
All MPTCP operations are signaled using optional TCP header fields 💌. IANA assigned a single TCP option number ("Kind") for MPTCP (see Section 8). Individual messages are then identified by a "subtype", and those values are also kept in an IANA registry (and listed in Section 8 too) 🗂️.
Throughout this document, when MPTCP options get mentioned by symbolic name, like "MP_CAPABLE", that refers to a TCP option with the single MPTCP option type, using the subtype value for that symbolic name as defined in Section 8 💖. The subtype is a 4-bit field, which is the first 4 bits of the option payload, as shown in Figure 3 🌸. The MPTCP messages are defined in the sections that follow ✨.
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-----------------------+
| Kind | Length |Subtype| |
+---------------+---------------+-------+ |
| Subtype-specific data |
| (variable length) |
+---------------------------------------------------------------+Figure 3: MPTCP Option Format 🎀
Options tied to subflow initiation get used on packets with the SYN flag set 🚀. There's also one MPTCP option for signaling metadata, so segmented data can be recombined for delivery to the application 🧩.
The remaining options are signals that don't need to ride on any specific packet, like the ones for signaling additional addresses 🌈. Sure, an implementation might want to send MPTCP options ASAP, but it may not be possible to squeeze all the desired options onto one packet. That includes both MPTCP options and regular TCP options, like SACK (selective acknowledgment) [11] 😩. So an implementation may choose to send duplicate ACKs that carry the extra signaling info 🔁. This changes what a duplicate ACK means 🙃. In regular TCP, duplicate ACKs are usually only sent as a signal of a lost segment [12]. That's why an MPTCP implementation that receives a duplicate ACK containing an MPTCP option MUST NOT treat it as a signal of congestion 🚫. Also, an MPTCP implementation SHOULD NOT send more than two duplicate ACKs in a row just to carry MPTCP options, so no middleboxes misread it as congestion 🙈.
Warning: standard TCP validity checks, like making sure the sequence number and acknowledgment number are within window, MUST be undertaken before processing any MPTCP signals, as described in [13]. no skipping this, bestie 💀
3.1. Connection Initiation 🦋
okay bestie, so connection initiation kicks off with a SYN, SYN/ACK, ACK exchange on a single path 🌸 Every packet carries the MP_CAPABLE TCP option (Figure 4). It's basically the sender saying "hey, I'm multipath capable and I wanna do it on this particular connection" 💅
This option is also how the sender declares the 64-bit key it generated for this MPTCP connection 🔑. That key authenticates any future subflows that get added. Lowkey, this is the only time the key gets sent in clear on the wire (unless "fast close", Section 3.5, is used) 👀. After this, every future subflow identifies the connection with a 32-bit "token", which is a cryptographic hash of the key. How that hash works depends on the authentication algorithm that gets selected, and the way that selection happens is explained later in this section ✨.
The sender makes its own key, and how it does that is up to the implementation 🧚. The key MUST be hard to guess, and it MUST be unique for the sending host at any one time. Recommendations for generating random numbers for keys are given in [14]. Connections get indexed at each host by the token (a one-way hash of the key), so an implementation needs a mapping from each token to the matching connection, and from there to the keys for that connection 🗂️.
There's a risk that two different keys hash to the same token 😬. The risk of hash collisions is usually small, unless the host is handling many tens of thousands of connections. So an implementation SHOULD check its list of connection tokens for a collision before sending its key in the SYN/ACK. That check would, however, be costly for a server with thousands of connections 💀. The subflow handshake mechanism (Section 3.2) makes sure new subflows only join the correct connection, though. It does this through the cryptographic handshake, by checking the connection tokens in both directions, and by making sure sequence numbers are in-window. So in the worst case, if there was a token collision, the new subflow would not succeed, but the MPTCP connection would keep providing a regular TCP service 🫶.
The MP_CAPABLE option is carried on the SYN, SYN/ACK, and ACK packets that start the first subflow of an MPTCP connection. The data in each packet goes like this, where A = initiator and B = listener:
SYN (A->B): A's Key for this connection.SYN/ACK (B->A): B's Key for this connection.ACK (A->B): A's Key followed by B's Key.
The contents of the option are determined by the SYN and ACK flags of the packet, and the option's length field verifies that. For the diagram in Figure 4, "sender" and "receiver" mean the sender or receiver of the TCP packet (which can be either host). If the SYN flag is set, a single key is included. If only an ACK flag is set, both keys are present 🌈.
B's Key gets echoed in the ACK so the listener (Host B) can act statelessly until the TCP connection reaches the ESTABLISHED state 🧠. If the listener works this way, though, it MUST generate its key in a way that lets it verify that it generated the key when that key is echoed back in the ACK 🔍.
This exchange is how the safe passage of MPTCP options on SYN packets gets determined ✅. If any of these options get dropped, MPTCP will gracefully fall back to regular single-path TCP, as documented in Section 3.6. Also, new subflows MUST NOT be established (using the process documented in Section 3.2) until a Digital Signature Standard (DSS) option has been successfully received across the path (as documented in Section 3.3) 🚫.
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-------+---------------+
| Kind | Length |Subtype|Version|A|B|C|D|E|F|G|H|
+---------------+---------------+-------+-------+---------------+
| Option Sender's Key (64 bits) |
| |
| |
+---------------------------------------------------------------+
| Option Receiver's Key (64 bits) |
| (if option Length == 20) |
| |
+---------------------------------------------------------------+
Figure 4: Multipath Capable (MP_CAPABLE) OptionThe first 4 bits of the first octet in the MP_CAPABLE option (Figure 4) define the MPTCP option subtype (see Section 8; for MP_CAPABLE, this is 0). The remaining 4 bits of that octet specify the MPTCP version in use (for this specification, that's 0) 🎀.
The second octet is reserved for flags, and they're allocated like this 🚩:
A: The leftmost bit, labeled "A", SHOULD be set to 1 to indicate "Checksum Required", unless the system administrator has decided that checksums are not required (for example, if the environment is controlled and no middleboxes exist that might adjust the payload).B: The second bit, labeled "B", is an extensibility flag, and MUST be set to 0 for current implementations. This will be used for an extensibility mechanism in a future specification, and the impact of this flag will be defined at a later date. If a receiver gets a message with the 'B' flag set to 1 and doesn't understand it, then this SYN MUST be silently ignored. The sender is expected to retry with a format compatible with this legacy specification. Note that the length of the MP_CAPABLE option, and the meanings of bits "C" through "H", may be altered by setting B=1 ⚠️.CthroughH: The remaining bits, labeled "C" through "H", are used for crypto algorithm negotiation. Currently only the rightmost bit, labeled "H", is assigned. Bit "H" indicates the use of HMAC-SHA1 (as defined in Section 3.2). An implementation that only supports this method MUST set bit "H" to 1, and bits "C" through "G" to 0 💜.
A crypto algorithm MUST be specified. If flag bits C through H are all 0, the MP_CAPABLE option MUST be treated as invalid and ignored (that is, it must be treated as a regular TCP handshake) 🙅.
The selected authentication algorithm also affects the algorithm used to generate the token and the initial data sequence number (IDSN). In this specification, with only the SHA-1 algorithm (bit "H") specified and selected, the token MUST be a truncated (most significant 32 bits) SHA-1 hash ([4], [15]) of the key. A different 64-bit truncation (the least significant 64 bits) of the SHA-1 hash of the key MUST be used as the initial data sequence number. Note that the key MUST be hashed in network byte order. Also note that the "least significant" bits MUST be the rightmost bits of the SHA-1 digest, as per [4]. Future specifications of the use of the crypto bits may choose to specify different algorithms for token and IDSN generation 🔮.
The crypto and checksum bits negotiate capabilities in similar ways 👯. For the Checksum Required bit (labeled "A"), if either host requires checksums, checksums MUST be used. In other words, the only way checksums won't be used is if both hosts set A=0 in their SYNs. This decision gets confirmed by the setting of the "A" bit in the third packet (the ACK) of the handshake. For example, if the initiator sets A=0 in the SYN but the responder sets A=1 in the SYN/ACK, checksums MUST be used in both directions, and the initiator will set A=1 in the ACK. An implementation stores the decision on whether to use checksums in a per-connection binary state variable 📦.
For crypto negotiation, the responder gets the choice 👑. The initiator creates a proposal by setting a bit to 1 for each algorithm it supports (in this version of the specification, there's only one proposal, so bit "H" will always be set to 1). The responder responds with only 1 bit set, and that's the chosen algorithm. The reason for this behavior is that the responder will typically be a server with potentially many thousands of connections, so it may wish to choose an algorithm with minimal computational complexity, depending on the load. If a responder doesn't support (or doesn't want to support) any of the initiator's proposals, it can respond without an MP_CAPABLE option, which forces a fallback to regular TCP 🥲.
The MP_CAPABLE option is only used in the first subflow of a connection, in order to identify the connection. All following subflows use the "Join" option (see Section 3.2) to join the existing connection 🧩.
If a SYN contains an MP_CAPABLE option but the SYN/ACK doesn't, it's assumed that the passive opener isn't multipath capable. So the MPTCP session MUST operate as a regular, single-path TCP. If a SYN doesn't contain an MP_CAPABLE option, the SYN/ACK MUST NOT contain one in response. If the third packet (the ACK) doesn't contain the MP_CAPABLE option, then the session MUST fall back to operating as a regular, single-path TCP. This maintains compatibility with middleboxes on the path that drop some or all TCP options 🛡️. Note that an implementation MAY choose to attempt sending MPTCP options more than one time before making this decision to operate as regular TCP (see Section 3.8).
If the SYN packets go unacknowledged, it's up to local policy to decide how to respond 🤷♀️. A sender is expected to eventually fall back to single-path TCP (i.e., without the MP_CAPABLE option) to work around middleboxes that may drop packets with unknown options. However, the number of multipath-capable attempts made first is up to local policy. It's possible that MPTCP and non-MPTCP SYNs could get reordered in the network. That's why the final state is inferred from the presence or absence of the MP_CAPABLE option in the third packet of the TCP handshake. If this option isn't present, the connection SHOULD fall back to regular TCP, as documented in Section 3.6 🔁.
The initial data sequence number on an MPTCP connection is generated from the key. The algorithm for IDSN generation is also determined by the negotiated authentication algorithm. In this specification, with only the SHA-1 algorithm specified and selected, the IDSN of a host MUST be the least significant 64 bits of the SHA-1 hash of its key, i.e., IDSN-A = Hash(Key-A) and IDSN-B = Hash(Key-B). This deterministic generation of the IDSN lets a receiver make sure there are no gaps in sequence space at the start of the connection 🌟. The SYN with MP_CAPABLE occupies the first octet of data sequence space, although this doesn't need to be acknowledged at the connection level until the first data is sent (see Section 3.3) 🎉.
3.2. Starting a New Subflow ✨
Okay so once an MPTCP connection has kicked off with the MP_CAPABLE exchange, you can keep adding subflows to it 🦋 Hosts know their own addresses, and they can learn the other host's addresses through the signaling exchanges in Section 3.4 💅 So a host can start a new subflow over a currently unused pair of addresses ✨ Either host in a connection is allowed to start a new subflow, but it's expected to normally be the original connection initiator (check Section 3.8 for heuristics) 👑
A new subflow starts as a normal TCP SYN/ACK exchange 🎀 The Join Connection (MP_JOIN) TCP option is how you tell the other side which connection the new subflow belongs to 🧚 It uses the keying material exchanged in the initial MP_CAPABLE handshake (Section 3.1), and that handshake also negotiates the crypto algorithm for the MP_JOIN handshake 🔐
This section specifies MP_JOIN behavior using HMAC-SHA1 🍓 An MP_JOIN option shows up in the SYN, SYN/ACK, and ACK of the three-way handshake, but each one has a slightly different format, so pay attention 👀
In the first MP_JOIN on the SYN packet (see Figure 5), the initiator sends a token, a random number, and an address ID 🌸
The token identifies the MPTCP connection. It's a cryptographic hash of the receiver's key, as exchanged in the initial MP_CAPABLE handshake (Section 3.1) 🔑 In this specification, the tokens in this option are generated by the SHA-1 ([4], [15]) algorithm, truncated to the most significant 32 bits 💜 The token in the MP_JOIN option is the one the receiver of the packet uses to identify the connection. So Host A sends Token-B, which is generated from Key-B 🫶 Also, the hash generation algorithm can be overridden by the choice of cryptographic handshake algorithm, as defined in Section 3.1 🌈
The MP_JOIN SYN carries not just the token (which stays the same for a connection) but also random numbers (nonces) that help prevent replay attacks on the authentication method 🛡️ For advice on generating those random numbers, see [14] 📚
The MP_JOIN option includes an "Address ID" 🪪 This identifier only means something inside a single connection. It identifies the source address of the packet, even if a middlebox has changed the IP header in transit 🙈 The Address ID lets you remove an address (Section 3.4.2) without needing to know the source address at the receiver, so address removal works through NATs 🦋 The Address ID also lets new subflow setup attempts line up with address signaling (Section 3.4.1). That stops you from setting up duplicate subflows on the same path when an MP_JOIN and an ADD_ADDR go out at the same time 💅
The Address IDs for the subflow used in the initial SYN exchange of the first subflow are implicit, and their value is zero 0️⃣ A host MUST store the mappings between Address IDs and addresses, both for itself and for the remote host 🗂️ An implementation also needs to know which local and remote Address IDs go with which established subflows, for when addresses get removed from a local or remote host 🧠
The MP_JOIN option on packets with the SYN flag set also includes 4 bits of flags 🚩 Three of them are currently reserved and MUST be set to zero by the sender. The last bit, labeled "B", says whether the sender wants this subflow to be a backup path (B=1), used only if other paths fail, or to be used as part of the connection right away (B=0) ⚡ By setting B=1, the sender is asking the other host to only send data on this subflow if there are no available subflows where B=0 😌 Subflow policy gets more detail in Section 3.3.8 📖
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-----+-+---------------+
| Kind | Length = 12 |Subtype| |B| Address ID |
+---------------+---------------+-------+-----+-+---------------+
| Receiver's Token (32 bits) |
+---------------------------------------------------------------+
| Sender's Random Number (32 bits) |
+---------------------------------------------------------------+
Figure 5: Join Connection (MP_JOIN) Option (for Initial SYN)When a host gets a SYN with an MP_JOIN option that has a valid token for an existing MPTCP connection, the recipient SHOULD respond with a SYN/ACK that also contains an MP_JOIN option. That option has a random number and a truncated (leftmost 64 bits) Hash-based Message Authentication Code (HMAC) 🔐 This version is shown in Figure 6. If the token is unknown, or the host wants to refuse subflow establishment (for example, because it has a limit on how many subflows it will permit), the receiver sends back a reset (RST) signal, which is analogous to an unknown port in TCP 🚫 Calculating an HMAC does take cryptographic operations, but a 32-bit token in the MP_JOIN SYN is believed to give enough protection against blind state exhaustion attacks 🛡️ So there's no need for mechanisms that let a responder operate statelessly at the MP_JOIN stage 🙅♀️
An HMAC is sent by both hosts 🤝 The initiator (Host A) sends it in the third packet (the ACK), and the responder (Host B) sends it in the second packet (the SYN/ACK). Doing the HMAC exchange at this stage means both hosts have already traded random data (in the first two SYN packets) to use as the "message" 💌 This specification uses the HMAC defined in [10], along with the SHA-1 hash algorithm [4] (potentially implemented as in [15]), which generates a 160-bit / 20-octet HMAC 🧮 Option space is tight, so the HMAC in the SYN/ACK is truncated to the leftmost 64 bits. That's okay because random numbers are used, so an attacker only gets one chance to guess the HMAC correctly. If the HMAC is incorrect, the TCP connection is closed, and a new MP_JOIN negotiation with a new random number is required 😤
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-----+-+---------------+
| Kind | Length = 16 |Subtype| |B| Address ID |
+---------------+---------------+-------+-----+-+---------------+
| |
| Sender's Truncated HMAC (64 bits) |
| |
+---------------------------------------------------------------+
| Sender's Random Number (32 bits) |
+---------------------------------------------------------------+
Figure 6: Join Connection (MP_JOIN) Option (for Responding SYN/ACK) 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-----------------------+
| Kind | Length = 24 |Subtype| (reserved) |
+---------------+---------------+-------+-----------------------+
| |
| |
| Sender's HMAC (160 bits) |
| |
| |
+---------------------------------------------------------------+
Figure 7: Join Connection (MP_JOIN) Option (for Third ACK)The initiator's authentication info goes in its first ACK (the third packet of the handshake), as shown in Figure 7 🎁 This data has to be sent reliably, because it's the only time this HMAC is sent 😭 So receipt of this packet MUST trigger a regular TCP ACK in response, and the packet MUST be retransmitted if that ACK doesn't arrive. Basically, sending the ACK/MP_JOIN packet puts the subflow in the PRE_ESTABLISHED state, and it only moves to ESTABLISHED once an ACK comes back from the receiver ⏳ It's not permitted to send data while in PRE_ESTABLISHED 🙅♀️ The reserved bits in this option MUST be set to zero by the sender 🧊
The HMAC key, for the message sent by Host A, is Key-A followed by Key-B. For Host B, it's Key-B followed by Key-A 🗝️ These are the keys exchanged in the original MP_CAPABLE handshake. The "message" is the concatenation of each host's random number (denoted R). For Host A that's R-A followed by R-B, and for Host B it's R-B followed by R-A 💖
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1Wait, I need to correct that. Here's the full Figure 8 diagram, which I've copied exactly 🎀 It's the best way to see how all these TCP options fit together for authenticated subflow setup:
Host A Host B
------------------------ ----------
Address A1 Address A2 Address B1
---------- ---------- ----------
| | |
| SYN + MP_CAPABLE(Key-A) |
|--------------------------------------------->|
|<---------------------------------------------|
| SYN/ACK + MP_CAPABLE(Key-B) |
| | |
| ACK + MP_CAPABLE(Key-A, Key-B) |
|--------------------------------------------->|
| | |
| | SYN + MP_JOIN(Token-B, R-A) |
| |------------------------------->|
| |<-------------------------------|
| | SYN/ACK + MP_JOIN(HMAC-B, R-B) |
| | |
| | ACK + MP_JOIN(HMAC-A) |
| |------------------------------->|
| |<-------------------------------|
| | ACK |
HMAC-A = HMAC(Key=(Key-A+Key-B), Msg=(R-A+R-B))
HMAC-B = HMAC(Key=(Key-B+Key-A), Msg=(R-B+R-A))
Figure 8: Example Use of MPTCP AuthenticationIf the token Host B receives is unknown, or local policy prohibits accepting the new subflow, the recipient MUST respond with a TCP RST for the subflow 🚫
If the token is accepted at Host B, but the HMAC returned to Host A doesn't match the expected one, Host A MUST close the subflow with a TCP RST 💀
If Host B doesn't receive the expected HMAC, or the MP_JOIN option is missing from the ACK, it MUST close the subflow with a TCP RST 😬
If the HMACs check out, both hosts have authenticated each other as the same peers that existed at the start of the connection, and they've agreed on which connection this subflow will become part of 🥹👑
If the SYN/ACK Host A receives doesn't have an MP_JOIN option, Host A MUST close the subflow with a RST 🙈
Okay so the whole lost-MP_JOIN situation is a lot, but it's covered, bestie 💀. If MP_JOIN gets stripped from the SYN on the path from A to B, and Host B doesn't have a passive opener on that port, Host B just answers with a RST the normal way 🙈. And if Host A sends a SYN with an MP_JOIN option but gets back a SYN/ACK without MP_JOIN (either it got stripped on the way back, or it got stripped going out but B's passive opener responded like it was a brand new regular TCP session, lol 😭), then that subflow is unusable and Host A MUST close it with a RST. That's the rule, periodt 💅
Also, extra subflows can be made between any pair of ports (but check Section 3.8 for the heuristics 🧠). You do NOT need any explicit application-level accept or bind calls to open more subflows ✨ To hook a new subflow up to an existing connection, the token in the subflow's SYN exchange is what gets used for demultiplexing. That binds the 5-tuple of the TCP subflow to the local token of the connection, so it's giving "any port pairs can be used for a connection" energy 🦋
Demultiplexing subflow SYNs MUST be done using the token. This is different from traditional TCP, where the destination port is what's used to demux SYN packets, so it's a whole vibe shift 🌈 Once a subflow is set up, packets get demultiplexed by the 5-tuple, same as regular TCP. The 5-tuples get mapped to the local connection identifier (the token). Also, Host A knows its own local token for the subflow even though it never goes on the wire. Only the responder's token is sent 🎀
3.3. General MPTCP Operation ✨
This section is about how MPTCP moves data around. The big picture: an MPTCP implementation takes one input data stream from an application and splits it across one or more subflows, attaching enough control info that it can all be put back together and delivered reliably and in order to the receiving app 📦 The subsections that follow spell out the details 🧚
The data sequence mapping and the Data ACK both get signaled in the Data Sequence Signal (DSS) option (Figure 9) 💜 You can signal either one or both in a single DSS, depending on which flags are set. The data sequence mapping tells you how the subflow's sequence space maps onto the connection level, and the Data ACK acknowledges receipt of data at the connection level. There are more details on both in the next two subsections 🌸
Again, either or both the data sequence mapping and the Data ACK can be signaled in the DSS option, depending on the flags set. (Yes, I'm repeating myself, it's important 🙈)
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+----------------------+
| Kind | Length |Subtype| (reserved) |F|m|M|a|A|
+---------------+---------------+-------+----------------------+
| Data ACK (4 or 8 octets, depending on flags) |
+--------------------------------------------------------------+
| Data sequence number (4 or 8 octets, depending on flags) |
+--------------------------------------------------------------+
| Subflow Sequence Number (4 octets) |
+-------------------------------+------------------------------+
| Data-Level Length (2 octets) | Checksum (2 octets) |
+-------------------------------+------------------------------+Figure 9: Data Sequence Signal (DSS) Option
The flags, when set, define what's in this option 🍓
A= Data ACK presenta= Data ACK is 8 octets (if not set, Data ACK is 4 octets)M= Data Sequence Number (DSN), Subflow Sequence Number (SSN), Data-Level Length, and Checksum presentm= Data sequence number is 8 octets (if not set, DSN is 4 octets)
The flags a and m only mean something if the corresponding A or M flags are set. Otherwise they get ignored 🙃 The maximum length of this option, with all flags set, is 28 octets 📏
The F flag means "DATA_FIN" 🔥 If it's there, this mapping covers the final data from the sender. It's the connection-level version of the FIN flag in single-path TCP 👑 A connection doesn't close unless there's been a DATA_FIN exchange or a timeout. How DATA_FIN, the subflow-level FIN flag, and the data sequence mapping all interact is covered in Section 3.3.3 ✨ The remaining reserved bits MUST be set to zero by an implementation of this specification 💀
Heads up, the checksum is only in this option if MPTCP checksumming was negotiated during the MP_CAPABLE handshake (see Section 3.1). You can tell if it's there from the length of the option 🧐 If a checksum is present but its use wasn't negotiated in the MP_CAPABLE handshake, the checksum field MUST be ignored. And if a checksum isn't present when its use has been negotiated, the receiver MUST close the subflow with a RST, because it's considered broken 😭🫶
3.3.1. Data Sequence Mapping 🦋
Okay bestie, so the whole data stream gets stitched back together using the data sequence mapping parts of the DSS option (Figure 9) ✨. That mapping says which subflow sequence number lines up with which data sequence number, and the receiver uses it to make sure everything reaches the application in order 💅. The regular sequence numbers in the TCP header only matter inside a single subflow 🧚. It's expected (but not mandated) that SACK [11] gets used at the subflow level to make things more efficient 🌸
The mapping goes from subflow sequence space to data sequence space. You describe it with a starting subflow sequence number, a starting data-level sequence number, and how many bytes the mapping is valid for 🎀. Why explicit ranges and not per-packet signaling? Because segmentation or coalescing can happen somewhere other than the stack that made the data, like TCP segmentation offloading on network cards or middleboxes such as performance enhancing proxies 😭. Plus, one mapping can cover lots of packets, which is super handy for bulk transfers 🔥
Mappings are fixed: once they're processed, the subflow sequence number is bound to the data sequence number. A sender MUST NOT change this mapping after it has been declared 🙈. The same data sequence number can be mapped by different subflows for retransmission purposes (see Section 3.3.6) 🔁. This also means the same data can go out on multiple subflows at once for resilience or efficiency, especially on lossy links 🌈. The detailed behaviour of that is outside this document's scope, but an implementation SHOULD treat the first data received at a subflow for a data sequence space as the data to deliver to the application, and ignore any later data for that same space 👑.
The data sequence number is an absolute value, while the subflow sequence numbering is relative (the SYN at the start of the subflow has relative subflow sequence number 0) 💜. This lets middleboxes change a subflow's initial sequence number, like firewalls doing ISN randomization 🧸
The mapping also carries a checksum of the data it covers, if checksums were negotiated during the MP_CAPABLE exchange 🔍. The checksum exists to catch payloads that a non-MPTCP-aware middlebox has tweaked. If the checksum fails, it triggers a subflow failure or a fallback to regular TCP, as documented in Section 3.6, because MPTCP can no longer reliably know the subflow sequence space at the receiver to build data sequence mappings 😱
The checksum algorithm is the standard TCP checksum [1], run over the data covered by the mapping plus a pseudo-header, as shown in Figure 10 🦋
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+--------------------------------------------------------------+
| |
| Data Sequence Number (8 octets) |
| |
+--------------------------------------------------------------+
| Subflow Sequence Number (4 octets) |
+-------------------------------+------------------------------+
| Data-Level Length (2 octets) | Zeros (2 octets) |
+-------------------------------+------------------------------+
Figure 10: Pseudo-Header for DSS ChecksumHere's the tea ☕: the data sequence number in the pseudo-header is always the 64-bit value, no matter what length the DSS option itself uses. The standard TCP checksum was picked because it gets calculated for the TCP subflow anyway, and if you calculate it over the data before adding the pseudo-headers, you only have to do it once 🙌. Even better, since the TCP checksum is additive, the checksum for a DSN_MAP can be built by simply adding up the checksums for the data of each constituent TCP segment, then adding the checksum for the DSS pseudo-header 🧮
Checksumming relies on the TCP subflow containing contiguous data, so a TCP subflow MUST NOT use the Urgent Pointer to interrupt an existing mapping 🚫. Still, if Urgent data shows up on a subflow, it SHOULD be mapped to the data sequence space and delivered to the application, kind of like Urgent data in regular TCP 💅
To avoid possible deadlocks, subflow-level processing should be handled separately from connection-level processing 🔒. So even if there's no mapping from the subflow space to the data-level space, the data SHOULD still be ACKed at the subflow (if it's in-window) ✅. It can't be acknowledged at the data level (Section 3.3.2), though, because its data sequence numbers are unknown 🤷♀️. Implementations MAY hold onto that unmapped data for a short while, hoping a mapping shows up soon 🫶. Unmapped data doesn't count toward the connection-level receive window, since that window is relative to data sequence numbers, so if the receiver runs out of memory, the data gets discarded 💀. If a mapping for that subflow-level sequence space doesn't arrive within a receive window's worth of data, that subflow SHOULD be treated as broken, closed with a RST, and any unmapped data silently discarded 🙅♀️
Data sequence numbers are always 64-bit quantities and MUST be maintained as such in implementations 💖. If a connection is moving slowly, so wrap protection isn't needed, it's totally permissible to include just the lower 32 bits of the data sequence number in the mapping and/or Data ACK as an optimization. An implementation can make that call independently for each packet 🍓
An implementation MUST send the full 64-bit data sequence number if it's transmitting fast enough that the 32-bit value could wrap within the Maximum Segment Lifetime (MSL) [16] 🏎️. The lengths of the DSNs used in these values (which can differ) are declared with flags in the DSS option 🚩. Implementations MUST accept a 32-bit DSN and implicitly promote it to a 64-bit quantity by incrementing the upper 32 bits each time the lower 32 bits wrap 🔄. A sanity check MUST be implemented to make sure a wrap happens at an expected time (for example, the sequence number jumps from a very high number to a very low one) and isn't triggered by out-of-order packets 🧐
Just like standard TCP sequence numbers, the data sequence number should not start at zero, but at a random value, which makes blind session hijacking harder 🕵️♀️. This specification requires setting the initial data sequence number (IDSN) of each host to the least significant 64 bits of the SHA-1 hash of the host's key, as described in Section 3.1 🔐
A data sequence mapping doesn't need to be in every MPTCP packet, as long as the subflow sequence space in that packet is covered by a mapping the receiver already knows about 🧠. That cuts overhead when the mapping is known in advance. One example is a single subflow between the hosts. Another is when data is scheduled in chunks bigger than a packet 📦
An "infinite" mapping can be used to fall back to regular TCP by mapping the subflow-level data to the connection-level data for the rest of the connection (see Section 3.6) 🌀. You do this by setting the Data-Level Length field of the DSS option to the reserved value of 0. In that case, the checksum is also set to zero 0️⃣
3.3.2. Data Acknowledgments 💌
To give full end-to-end resilience, MPTCP provides a connection-level acknowledgment that works as a cumulative ACK for the whole connection 👑. This is the "Data ACK" field of the DSS option (Figure 9). It behaves like the standard TCP cumulative ACK, showing how much data has been received with no holes in it ✨. The subflow-level ACK is more like TCP SACK, since there may still be holes in the data stream at the connection level 🕳️. The Data ACK tells the other side the next data sequence number you're expecting 🎯
The Data ACK can be sent as the full 64-bit value or as the lower 32 bits, same as the DSN. If data arrives with a 64-bit DSN, it MUST be acknowledged with a 64-bit Data ACK 💯. If the DSN received is 32 bits, the implementation can choose whether to send a 32-bit or 64-bit Data ACK 🌸
The Data ACK proves that the data, and all required MPTCP signaling, has been received and accepted by the remote end 🥳. It's also how the left edge of the advertised receive window gets indicated. The receive window is shared by all subflows and is relative to the Data ACK (see Section 3.3.4), so an implementation MUST NOT use the RCV.WND field of a TCP segment at the connection level if that segment doesn't also carry a DSS option with a Data ACK field 🚧. Separating connection-level acknowledgments from the subflow level also lets each one be processed on its own, and a receiver is free to drop segments after acknowledging them at the subflow level, like when lots of segments arrive out of order and memory gets tight 🧺
An MPTCP sender MUST NOT free data from the send buffer until it has been acknowledged both by a Data ACK on any subflow and at the subflow level by every subflow the data was sent on 🛑. The first condition keeps the connection alive, and the second keeps each subflow alive and consistent when data needs retransmitting 💪. Heads up, though: if some data has to be retransmitted over and over on one subflow, it can block the sending window 😬. In that case, the MPTCP sender can decide to shut down the misbehaving subflow by sending a RST 🙅♀️
The Data ACK MAY be included in all segments 🙌. But in more advanced implementations, optimizations SHOULD be considered, where the Data ACK only shows up in segments when its value actually advances, and this behaviour MUST be treated as valid ✅. That way the sender's buffer still gets freed, and you save overhead when data is only moving one way 🦋
3.3.3. Closing a Connection ✨
okay so in regular TCP, a FIN basically says "hey, I'm done sending stuff" 💅 but in MPTCP a FIN only affects the subflow it's sent on 🦋 That's the whole trick! It lets each subflow do its own thing, so nodes get to pick which paths are in use at any moment. The FIN still works exactly like regular TCP, though. The subflow isn't fully closed until both sides have ACKed each other's FINs 🫶
When an app calls close() on a socket, it's saying it has no more data to send. In plain TCP that turns into a FIN on the connection. MPTCP needs its own version of that, and it's called the DATA_FIN 🎀
A DATA_FIN tells the other side the sender is finished, and it can be used to check that all the data actually arrived 🔥 Just like the FIN in regular TCP, it only goes one way, so it's a unidirectional signal.
The DATA_FIN gets signaled by setting the F flag in the Data Sequence Signal option (Figure 9) to 1. It takes up 1 octet, the final octet of the connection-level sequence space. The DATA_FIN gets counted in the Data-Level Length, but not at the subflow level. Here's an example: a segment with DSN 80 and Data-Level Length 11, with DATA_FIN set, maps 10 octets from the subflow into data sequence space 80-89. The DATA_FIN itself is DSN 90, so that segment gets acknowledged with a DATA_ACK of 91 🧚
Now, if the DATA_FIN isn't attached to a TCP segment that has data in it, the Data Sequence Signal MUST have a subflow sequence number of 0, a Data-Level Length of 1, and the data sequence number that matches the DATA_FIN itself. In that case the checksum only covers the pseudo-header 🙈
A DATA_FIN has the same semantics and behavior as a regular TCP FIN, just at the connection level 🌸 The big thing is that it only gets DATA_ACKed once all the data has been successfully received at the connection level. So a DATA_FIN is decoupled from a subflow FIN. You can only combine those two signals on one subflow if there's no data outstanding on the other subflows. If there is, you might have to retransmit data on different subflows 💜 Basically, a host MUST NOT close all of its working subflows unless it's actually safe to do so. That means either all outstanding data has been DATA_ACKed, or the segment with the DATA_FIN flag set is the only one still outstanding 👑
Once a DATA_FIN has been acknowledged, all remaining subflows MUST be closed with standard FIN exchanges. Both hosts SHOULD send FINs on all subflows, as a courtesy so middleboxes can clean up state even if one subflow has failed 🫶 Hosts are also encouraged to lower the timeouts (Maximum Segment Life) on subflows at end hosts. Any subflow that still has outstanding data queued (data that was already retransmitted on other subflows to get the DATA_FIN acknowledged) MAY be closed with a RST 💀
A connection counts as closed once both hosts' DATA_FINs have been acknowledged by DATA_ACKs ✨
Remember, a standard TCP FIN on one subflow only shuts down that subflow. If every subflow has closed with a FIN exchange but no DATA_FIN has been received and acknowledged, the MPTCP connection doesn't count as closed until a timeout passes ⏳ That means an implementation ends up with TIME_WAIT states at both the subflow and connection levels (see Appendix C). It's lowkey important because it allows "break-before-make" scenarios, where connectivity drops on every subflow before a new one can be re-established 🌈
3.3.4. Receiver Considerations ✨
Regular TCP advertises a receive window in every packet, which tells the sender how much data the receiver is willing to take past the cumulative ACK. That's how flow control works. It throttles down fast senders when receivers can't keep up 🍓
MPTCP has a unique receive window too, and it's shared between all the subflows 💖 The point is that any subflow can send data as long as the receiver is willing to accept it. The alternative would be keeping a receive window per subflow, and that could stall some subflows while others never use up their window. Nobody wants that, bestie 🙅♀️
The receive window is relative to the DATA_ACK. Just like in TCP, a receiver MUST NOT shrink the right edge of the receive window (that's DATA_ACK + receive window). The receiver uses the data sequence number to decide whether a packet should be accepted at the connection level 🔥
When it comes to accepting packets at the subflow level, regular TCP checks the packet's sequence number against the allowed receive window. With multipath, that check only uses the connection-level window. A sanity check SHOULD still be done at the subflow level, to make sure the subflow and mapped sequence numbers pass this test: SSN - SUBFLOW_ACK <= DSN - DATA_ACK. Here, SSN is the subflow sequence number of the received packet, and SUBFLOW_ACK is the subflow's RCV.NXT (the next expected sequence number). The connection-level versions of DSN and DATA_ACK work the same way 💅
In regular TCP, once a segment is in-window, it goes into either the in-order receive queue or the out-of-order queue. Multipath TCP does the same thing but at the connection level. A segment goes into the connection-level in-order or out-of-order queue only if it's in-window at both the connection and subflow levels. The stack still has to remember, for each subflow, which segments were received successfully so it can ACK them correctly at the subflow level 🦋 This is usually done with per-subflow out-of-order queues (holding only message headers, not the payloads) plus a record of the cumulative ACK value 🧚
Sizing the receiver buffer matters a lot, so implementers should really get this right ✨ The lower bound for full network utilization is the maximum bandwidth-delay product of any one of the paths. But that might not be enough when a packet gets lost on a slower subflow and has to be retransmitted (see Section 3.3.6). A tight upper bound is the maximum round-trip time (RTT) of any path multiplied by the total bandwidth available across all paths. That lets every subflow keep going at full speed while a packet gets fast-retransmitted on the maximum RTT path 🌸 Even that might fall short for full performance if a retransmit timeout happens on the maximum RTT path. How retransmission strategies and receive buffer sizing relate is something left for future study 🔮
3.3.5. Sender Considerations ✨
The sender keeps track of the receive window advertisements it gets from the receiver. It only updates its local receive window values when the largest sequence number allowed (DATA_ACK + receive window) goes up, on receipt of a DATA_ACK. This matters because it lets paths with different RTTs, and therefore different feedback loops, all work properly together 💜
MPTCP uses one receive window across all subflows. If that window were guaranteed to stay unchanged end-to-end, a host could always just read the most recent value. But some kinds of middleboxes can change the TCP-level receive window 🙈 They usually shrink the offered window. For short periods the window might get bigger, though, and that can't last for long, since the middlebox still has to keep up with delivering data to the receiver. So if the receive window sizes differ across subflows, MPTCP SHOULD use the largest of the most recent window sizes when it's doing its calculations when sending data. This rule is already implied by the requirement not to shrink the right edge of the window 👑
The sender MUST also remember the receive windows advertised by each subflow. The allowed window for subflow i is (ack_i, ack_i + rcv_wnd_i), where ack_i is subflow i's subflow-level cumulative ACK. This makes sure data never gets sent to a middlebox unless there's enough buffering for it 🫶
Put the two rules together and you get this: a sender can send data segments with data-level sequence numbers between (DATA_ACK, DATA_ACK + receive_window). Each of those segments gets mapped onto subflows, as long as the subflow sequence numbers are inside the allowed windows for those subflows. Subflow sequence numbers usually don't affect flow control if the same receive window is advertised on every subflow. They only do flow control for subflows with a smaller advertised receive window 🦋
The send buffer MUST, at a minimum, be as big as the receive buffer, so the sender can hit maximum throughput ✨
3.3.6. Reliability and Retransmissions ✨
The data sequence mapping lets senders resend data with the same data sequence number on a different subflow. When a host does this, it MUST still retransmit the original data on the original subflow 💅 That keeps subflow integrity intact, because middleboxes could replay old data or reject holes in subflows. The receiver will ignore these retransmissions. Yes, it's clearly suboptimal, but for compatibility reasons it's the sensible behavior for now. Optimizations could be negotiated in future versions of this protocol 🌈
This protocol spec doesn't mandate any particular retransmission mechanism. A lot depends on local policy (as discussed in Section 3.3.8) 🍓 You could imagine something aggressive, where every packet lost at subflow level gets retransmitted on a different subflow. That wastes bandwidth but might reduce application-to-application delays ⚡ Or something conservative, where connection-level retransmits only kick in after a few subflow-level retransmission timeouts happen 🧚
The envisioned standard connection-level retransmission mechanism sits around a connection-level data queue. All segments that haven't been DATA_ACKed get stored there. A timer starts when the head of the connection-level queue is ACKed at subflow level but its data isn't ACKed at data level. That timer guards against failures in retransmission caused by middleboxes that proactively ACK data 🙈
The sender MUST keep data in its send buffer for as long as the data hasn't been acknowledged at both the connection level and on all subflows where it was sent. That way the sender can always retransmit the data if needed, on the same subflow or a different one 💖 There's a special case when a subflow fails. The sender will usually resend the data on other working subflows after a timeout, and it'll keep trying to retransmit on the failed subflow too. The sender declares the subflow failed once a predefined upper bound on retransmissions is reached (which MAY be lower than the usual TCP limits of the Maximum Segment Life), or upon receipt of an ICMP error. Only then does it delete the outstanding data segments 💀
Multiple retransmissions are a signal that a subflow is performing badly, and they could lead a host to reset that subflow with a RST 👀 Honestly, more research is needed on the heuristics of how and when to reset underperforming subflows. For example, a highly asymmetric path might get misdiagnosed as underperforming 🫠
3.3.7. Congestion Control Considerations ✨
okay so each subflow in an MPTCP connection has its own congestion window, bestie 🎀 To be fair at bottlenecks and do resource pooling properly, the congestion windows on the subflows need to be coupled, so most of the traffic gets pushed onto the links that aren't congested 🦋 One algorithm for this is presented in [5]. It doesn't achieve perfect resource pooling, but it's "safe", meaning it's readily deployable on today's Internet. By safe we mean it doesn't take up more capacity on any one path than a single-path flow using only that route would, so it fairly coexists with single-path TCP at shared bottlenecks 💅
It's foreseeable that different congestion controllers will get implemented for MPTCP, each aiming for different properties in the resource pooling / fairness / stability design space, plus different properties in quality of service, reliability, and resilience ✨
No matter which algorithm is used, the MPTCP design aims to give the congestion control implementations enough information to make the right decisions. For each subflow, that includes which packets were lost and when 🧚
3.3.8. Subflow Policy 💜
Inside a local MPTCP implementation, a host can use whatever local policy it wants to decide how to split traffic across the available paths 👑
In the typical use case, where the goal is maximizing throughput, all available paths get used at the same time for data transfer, using coupled congestion control as described in [5]. More use cases are expected to show up, though 🌈
For instance, one option is an 'all-or-nothing' approach: have a second path ready in case the first one fails. Alternatives include fully saturating one path before using an additional path (the 'overflow' case) 🍓 Those choices would most likely be based on the monetary cost of links, but they could also depend on things like delay or jitter, where stability of delay or bandwidth matters more than throughput 🫶 Application requirements like these are discussed in detail in [6] 🔥
Making good choices at the sender needs full knowledge of the path "cost", which probably isn't going to be the case 😭 It'd be great for a receiver to signal its own path preferences, since it's often the multihomed party and might have to pay for metered incoming bandwidth 💀
Fine-grained control might be the most powerful solution, but it would need a mechanism like overloading the Explicit Congestion Notification (ECN) signal [17], which is undesirable, and it's felt there wouldn't be enough benefit to justify a whole new signal 🙈 So instead, the MP_JOIN option (see Section 3.2) contains the 'B' bit. It lets a host tell its peer that a path should be treated as a backup, used only when other working subflows fail (i.e., a subflow where the receiver has indicated B=1 SHOULD NOT be used to send data unless there are no usable subflows where B=0) ✨
If the set of available paths changes, a host may wish to signal a change in subflow priority to its peer (e.g., a subflow that was previously backup should now take priority over all remaining subflows) 👑 So the MP_PRIO option, shown in Figure 11, can be used to change the 'B' flag of the subflow it's sent on 🦋
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-----+-+--------------+
| Kind | Length |Subtype| |B| AddrID (opt) |
+---------------+---------------+-------+-----+-+--------------+Figure 11: Change Subflow Priority (MP_PRIO) Option
It should be noted that the backup flag is a request from a data receiver to a data sender only, and the data sender SHOULD adhere to these requests. A host can't assume the data sender will do so, though, since local policies (or technical difficulties) may override MP_PRIO requests 💅 Also note this signal applies to one direction only, so the sender of this option could keep using the subflow to send data even after signaling B=1 to the other host 🌸
This option can also apply to subflows other than the one it's sent on, by setting the optional Address ID field. That applies the given B setting to all subflows in this connection that use the address with the given Address ID 🧚 Whether the field is there depends on the option length: if Length==4 it's present, and if Length==3 it applies to the current subflow only 🎀 The use case is that a host can tell its peer an address is temporarily unavailable (say it has radio coverage issues), so the peer should drop to backup state on all subflows using that Address ID 🥺
3.4. Address Knowledge Exchange (Path Management) 🦋
"Path management" is our term for exchanging information about additional paths between hosts, which in this design is handled through multiple addresses at each host 💖 For more on the architectural thinking behind it, check the MPTCP Architecture document [2] 🧠
This design uses two ways of sharing that info, and both can be used on one connection ✨ The first is directly setting up new subflows, already covered in Section 3.2, where the initiator has an extra address. The second, in the subsections that follow, explicitly signals addresses to the other host so it can start new subflows itself 🫶 The two are complementary: the first is implicit and simple, while the explicit one is more complex but more robust 💜 Together they let addresses change in flight (so it works through NATs, since the source address doesn't need to be known), let you signal previously unknown addresses, and let you signal addresses from other address families (e.g., both IPv4 and IPv6) 🌈
Here's an example of typical operation 🍓
- An MPTCP connection is first set up between address/port A1 of Host A and address/port B1 of Host B 🎀 If Host A is multihomed and multiaddressed, it can start an extra subflow from its address A2 to B1 by sending a SYN with a Join option from A2 to B1, using B's previously declared token for this connection. Or, if B is multihomed, it can try to set up a new subflow from B2 to A1, using A's previously declared token. Either way, the SYN goes to the port already in use for the original subflow on the receiving host.
- At the same time (or after a timeout), an ADD_ADDR option (Section 3.4.1) is sent on an existing subflow, telling the receiver about the sender's alternative address(es). The recipient can use that to open a new subflow to the sender's additional address. In our example, A sends an ADD_ADDR option telling B about address/port A2. How the SYN-based option and ADD_ADDR get mixed, including timeouts, is implementation specific and can be tailored to local policy 🧚
- If subflow A2-B1 is set up successfully, Host B can use the Address ID in the Join option to match it with the ADD_ADDR option that also arrives on an existing subflow. Now B knows not to open A2-B1 and to ignore the ADD_ADDR 🙈 If B hasn't received the A2-B1 MP_JOIN SYN but did receive the ADD_ADDR, it can try to start a new subflow from one or more of its addresses to address A2. This lets new sessions open if one host is behind a NAT 🔥
Other ways of using the two signaling mechanisms are possible. For instance, signaling addresses in other address families can only be done explicitly, using the Add Address option 👑
3.4.1. Address Advertisement ✨
Okay so the Add Address (ADD_ADDR) TCP option is how you tell your bestie "hey, I can also be reached at these extra addresses (and maybe ports too)" 🦋 (see Figure 12). You can cram multiple instances into one message if there's enough TCP option space, and if there isn't, they just go out across multiple TCP messages. You can send it anytime during a connection, depending on when you want to turn on multiple paths or when new paths show up 💅. And like every MPTCP signal, the receiver MUST run the usual TCP validity checks before acting on it 🫶.
Every address gets an Address ID, which is how you uniquely name that address within a connection when it's time to remove it 🎀. The same ID also helps match up MP_JOIN options (see Section 3.2) that belong to the same address, even when address translators are in the picture. The Address ID MUST uniquely identify the address to the sender (within the scope of the connection), but how you allocate those IDs is totally up to your implementation 🧚.
All address IDs learned from either MP_JOIN or ADD_ADDR SHOULD be stored by the receiver in one data structure that gathers every Address ID to address mapping for a connection (identified by a token pair). That way you've got a stored mapping between Address ID, the observed source address, and the token pair, ready for handling future control info on that connection 📦. Also, an implementation MAY throw away incoming address advertisements whenever it wants. Maybe it wants to avoid holding that mapping state, or maybe the advertised addresses are no use to it (like IPv6 addresses when it only does IPv4). So a host MUST treat address advertisements as soft state, and it MAY choose to refresh them periodically 🌸.
This option is shown in Figure 12. The picture is sized for IPv4 (IPVer = 4). For IPv6, the IPVer field reads 6, and the address is 16 octets instead of 4 🌈.
The final 2 octets, which specify the TCP port number to use, are optional, and you can tell whether they're there just by looking at the option's length 💀. Most of the time the port pair stays the same as the initial subflow (port 80 stays port 80 on every subflow, and the client's ephemeral port stays put too). But sometimes you really need a different port, like with port-based load balancing 🍓. If no port is specified, MPTCP SHOULD attempt to connect to the specified address on the same port already in use by the subflow where the ADD_ADDR signal was sent. There's more on this in Section 3.8.
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-------+---------------+
| Kind | Length |Subtype| IPVer | Address ID |
+---------------+---------------+-------+-------+---------------+
| Address (IPv4 - 4 octets / IPv6 - 16 octets) |
+-------------------------------+-------------------------------+
| Port (2 octets, optional) |
+-------------------------------+
Figure 12: Add Address (ADD_ADDR) OptionThanks to NATs being everywhere, one host might totally try to advertise private addresses [18]. We don't want to ban that outright, since both hosts could have extra interfaces on the same private network, and a host MAY want to advertise those addresses 😭. The MP_JOIN handshake to create a new subflow (Section 3.2) has protections that keep security risks low. The MP_JOIN message carries a 32-bit token that uniquely identifies the connection to the receiving host. If the token is unknown, the host sends back a RST. In the unlikely event the token is known, subflow setup keeps going, but the HMAC exchange must happen for authentication. That exchange fails, which is enough protection against two unconnected hosts accidentally setting up a new subflow because of a private address signal 🔐. More security considerations about ADD_ADDR messages that accidentally misdirect, or maliciously direct, new MP_JOIN attempts are in Section 5 ⚠️.
Ideally, ADD_ADDR and REMOVE_ADDR options would get sent reliably and in order to the other end. That keeps address management from causing an outage when remove and add get processed in reverse order, and makes sure all possible paths get used 🚦. But losing reliability and ordering won't break the multipath connections. It just reduces your chances to open multipath paths and to survive different patterns of path failures 🥲.
So implementing reliability signals for these TCP options isn't necessary. To cut down the impact of losing these options, though, it is RECOMMENDED that a sender sends them on all available subflows. If these options need to be received in order, an implementation SHOULD only send one ADD_ADDR/REMOVE_ADDR option per RTT, to lower the risk of misordering 🧦.
A host can send an ADD_ADDR message with an Address ID that's already assigned. But the Address MUST be the same as the one previously assigned to that Address ID, and the Port MUST be different from any port already in use for that Address ID. If those conditions aren't met, the receiver SHOULD silently ignore the ADD_ADDR. A host that wants to replace an existing Address ID MUST first remove the existing one (Section 3.4.2) 🙅♀️.
A host that receives an ADD_ADDR but finds that a connection is already set up to that IP address and port number is unsuccessful. It SHOULD NOT make further connection attempts to that address/port combination for this connection. If a sender wants to trigger a new incoming connection attempt on a previously advertised address/port combination, it can refresh the ADD_ADDR information by sending the option again 🔁.
In normal MPTCP operation, it's unlikely there'll be enough TCP option space to fit ADD_ADDR alongside the options for data sequence numbering (Section 3.3.1). So it's expected that an MPTCP implementation will send the ADD_ADDR option on separate ACKs. As discussed earlier, though, an MPTCP implementation MUST NOT treat duplicate ACKs carrying any MPTCP option, with the exception of the DSS option, as indications of congestion [12]. And an MPTCP implementation SHOULD NOT send more than two duplicate ACKs in a row for signaling purposes 🫠.
3.4.2. Remove Address 👋
If a previously announced address becomes invalid during the lifetime of an MPTCP connection (say the interface disappears), the affected host SHOULD announce this, so the peer can remove the subflows related to that address 🗑️.
This happens through the Remove Address (REMOVE_ADDR) option (Figure 13). It removes a previously added address (or a list of addresses) from a connection and terminates any subflows currently using that address 💥.
For security purposes, if a host receives a REMOVE_ADDR option, it must make sure the affected path(s) are no longer in use before it starts closure. Receiving REMOVE_ADDR SHOULD first trigger a TCP keepalive [19] on the path, and if a response is received, the path SHOULD NOT be removed. The typical TCP validity tests on the subflow (like making sure sequence and ACK numbers are correct) MUST also be undertaken. An implementation can use indications of these test failures as part of intrusion detection or error logging 🕵️♀️.
Sending this message, and receiving it if no keepalive response came back, SHOULD trigger the sending of RSTs by both hosts on the affected subflow(s), if possible. It's a courtesy for cleaning up middlebox state, and it happens before any local state gets cleaned up 🧹.
Address removal is done by ID, so it still works with NATs and other middleboxes that rewrite source addresses 🙌. If there's no address at the requested ID, the receiver silently ignores the request.
A subflow that's still functioning MUST be closed with a FIN exchange, just like regular TCP, rather than by using this option. For more info, check out Section 3.3.3 🌈.
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-------+---------------+
| Kind | Length = 3+n |Subtype|(resvd)| Address ID | ...
+---------------+---------------+-------+-------+---------------+
(followed by n-1 Address IDs, if required)
Figure 13: Remove Address (REMOVE_ADDR) Option3.5. Fast Close 🔥
Okay so regular TCP has this big red button, the RST, that abruptly closes a connection 💀. With MPTCP, though, an RST only hits the one subflow it's sent on 🦋. The rest of the subflows keep going, and the whole MPTCP connection stays alive at the data level so you can do a break-before-make handover between subflows ✨. That's the whole point. But it also means we need an MPTCP-level "reset" for when you really do want the entire connection slammed shut, and that's the MP_FASTCLOSE option 👑.
MP_FASTCLOSE tells the peer that the connection is about to be abruptly closed and that no more data will be accepted 🫶. Why an MP_FASTCLOSE gets triggered is up to each implementation 🌸. Regular TCP doesn't let you send a RST while the connection is in a synchronized state [1], which is kind of a rule-following moment 🙈. Implementations do allow it in that state anyway, for example when the operating system is running out of resources 😭. In those cases, MPTCP should send the MP_FASTCLOSE instead 💅. Here's what the option looks like, in Figure 14 🧚:
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+-----------------------+
| Kind | Length |Subtype| (reserved) |
+---------------+---------------+-------+-----------------------+
| Option Receiver's Key |
| (64 bits) |
| |
+---------------------------------------------------------------+Figure 14: Fast Close (MP_FASTCLOSE) Option 🎀
If Host A wants to force an MPTCP connection closed, the Fast Close procedure goes like this 💖:
- Host A sends an ACK containing the MP_FASTCLOSE option on one subflow. That option carries Host B's key, the one declared during the initial connection handshake 🔑. On all the other subflows, Host A sends a regular TCP RST to close them and tears them down 💥. Host A now enters FASTCLOSE_WAIT state 🫠.
- When Host B gets an MP_FASTCLOSE with the valid key, it answers on the same subflow with a TCP RST and tears down all subflows 🙅♀️. Host B can now close the whole MPTCP connection, going directly to CLOSED state ✅.
- Once Host A gets the TCP RST on the remaining subflow, it can close that subflow and tear down the whole connection, moving from FASTCLOSE_WAIT to CLOSED states 🥹. If Host A gets an MP_FASTCLOSE back instead of a TCP RST, both hosts tried to fast close at the same time 👯. Host A should reply with a TCP RST and tear down the connection 🫡.
- If Host A doesn't get a TCP RST back for its MP_FASTCLOSE after one retransmission timeout (RTO), counted on the RTO of the subflow where the MPTCP_RST was sent, it SHOULD retransmit the MP_FASTCLOSE 🔁. The number of retransmissions SHOULD be limited so the connection doesn't hang around forever 🕰️, but what that limit is depends on the implementation 🤷♀️. A RECOMMENDED number is 3 💯.
3.6. Fallback ✨
okay so middleboxes are the chaotic gremlins of the internet 🙈 and sometimes they sit on a path and literally block MPTCP from working. MPTCP is built to handle a lot of middlebox messing (see Section 6), but there are still some situations where a subflow can't keep following the MPTCP rules. The big ones are when TCP options get dropped on a path and when the payload data gets modified. when that happens, we have to fall back to the last safe way of doing things 🫶 That might mean dropping back to plain regular TCP, or just removing the problem subflow.
at the very start of an MPTCP connection (aka the first subflow), it's super important to check that the path is fully MPTCP capable and that the needed TCP options actually reach each host 💅 The handshake in Section 3.1 SHOULD fall back to regular TCP if either of the SYN messages is missing the MPTCP options. This is the same, and totally wanted, behavior when a host isn't MPTCP capable or the path doesn't support the MPTCP options. when you're trying to join an existing MPTCP connection (Section 3.2) and the path isn't MPTCP capable, with the TCP options not getting through on the SYNs, the subflow gets closed according to the MP_JOIN logic 🦋
there's one more corner case we need to handle too 👀 this is when MPTCP options make it through on the SYN but NOT on the regular packets after it. this can be fixed if the subflow is the first subflow, since all the data in flight is contiguous, by following these rules:
A sender MUST include a DSS option with data sequence mapping in every segment until one of the sent segments has been acknowledged with a DSS option containing a Data ACK. Once the sender gets that ACK, it knows the DSS option gets through in both directions, and it can choose to send fewer DSS options than one per segment 🌸
but if an ACK comes back for data (not just for the SYN) without a DSS option containing a Data ACK, the sender decides the path is not MPTCP capable 😭 if this happens on an additional subflow (one started with MP_JOIN), the host MUST close the subflow with a RST. if it happens on the first subflow (the one started with MP_CAPABLE), it MUST drop out of MPTCP mode and go back to regular TCP. the sender sends one final data sequence mapping, with the Data-Level Length value of 0 meaning an infinite mapping (just in case the path drops options in one direction only), and then it goes back to sending data on the single subflow without any MPTCP options 🧚
heads up, this rule basically means you can't send data on the third packet of an MP_CAPABLE or MP_JOIN handshake, because that option and a DSS can't both fit in the TCP option space 🚫 if the initiator is supposed to send first, another segment has to be sent that carries the data and the DSS. also, an additional subflow can't be used until the initial path has been verified as MPTCP capable.
these rules should cover every case where this kind of failure could happen, whether it's on the forward or reverse path, and whether the server or the client sends data first 💖 if lost options on data packets happen on any subflow other than the initial one, it should be treated as a standard path failure. the data wouldn't get DATA_ACKed (since there's no mapping for it), and the subflow can be closed with a RST.
the case I just described is a special type of fallback, for when the missing MPTCP support is noticed before any data gets acknowledged at the connection level on a subflow. more generally, fallback (either closing a subflow or going back to regular TCP) can become necessary at any point in a connection if a non-MPTCP-aware middlebox changes the data stream 🔥
Warning, bestie: if a middlebox changes the data, the whole mapping can go haywire, so pay attention here 💀
as Section 3.3 explains, every chunk of data that has a mapping is protected by a checksum 🛡️ this is how we spot middleboxes that have touched the payload (added, removed, or changed data). a checksum fails if the data was changed in any way. it'll also catch the case where the length of data on the subflow went up or down, which means the data sequence mapping isn't valid anymore. the sender no longer knows what subflow-level sequence number the receiver is actually at (the middlebox is faking the ACKs back), so it can't signal any more mappings 😵 on top of that, besides payload changes that are valid at the application layer, false positives could happen across MPTCP segment boundaries and corrupt the data. so all data from the start of the segment that failed the checksum onward is not trustworthy.
when multiple subflows are in use, the data in flight on a subflow will probably include data that isn't contiguously part of the connection-level stream, since segments are spread across the subflows 🌈 because of the problems above, there's no way to figure out what the adjustment did to the data (especially any changes to the subflow sequence numbering). so the subflow can't be recovered, and the affected subflow MUST be closed right away with a RST, carrying an MP_FAIL option (Figure 15) that says which data sequence number is at the start of the segment (as defined by the data sequence mapping) that failed the checksum. heads up: MP_FAIL always needs the full 64-bit sequence number, even when 32-bit sequence numbers are normally used in the DSS signals on the path 🙈
1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+-------+----------------------+
| Kind | Length=12 |Subtype| (reserved) |
+---------------+---------------+-------+----------------------+
| |
| Data Sequence Number (8 octets) |
| |
+--------------------------------------------------------------+Figure 15: Fallback (MP_FAIL) Option ✨
the receiver MUST discard all data following the data sequence number specified. failed data MUST NOT be DATA_ACKed, so it'll get retransmitted on other subflows (Section 3.3.6) 🦋
there's a special case when there's a single subflow and it fails with a checksum error 🎀 if we know all the unacknowledged data in flight is contiguous (which is usually true with one subflow), an infinite mapping can be applied to the subflow without closing it first, which basically turns off all further MPTCP signaling 🌸 in this case, if a receiver spots a checksum failure when there's only one path, it sends back an MP_FAIL option on the subflow-level ACK, pointing at the data-level sequence number of the start of the segment where the checksum error happened. the sender gets this, and if all the unacknowledged data in flight is contiguous, it signals an infinite mapping. that infinite mapping is a DSS option (Section 3.3) on the first new packet, with a data sequence mapping that works retroactively, pointing at the start of the subflow sequence number of the last segment that was known to be delivered intact. from that point on, a middlebox can change the data without affecting MPTCP, because the data stream is now basically a regular, legacy TCP session 💅
in the rare case that the data isn't contiguous (which could happen when there's only one subflow but it's retransmitting data from a subflow that was recently closed uncleanly), the receiver MUST close the subflow with a RST with MP_FAIL. the receiver MUST discard all data that follows the data sequence number specified. the sender MAY try to create a new subflow that belongs to the same connection, and if it chooses to do that, it SHOULD put the single subflow right into single-path mode by setting an infinite data sequence mapping. that mapping begins from the data-level sequence number that was declared in the MP_FAIL 🧚
after a sender signals an infinite mapping, it MUST only use subflow ACKs to clear its send buffer. this is because Data ACKs can get out of sync with the subflow ACKs when middleboxes insert or delete data 😭 the receive SHOULD stop generating Data ACKs after it receives an infinite mapping.
when a connection has fallen back, only one subflow can send data, otherwise the receiver wouldn't know how to reorder the data 🫠 so in practice, all MPTCP subflows have to be terminated except one. once MPTCP falls back to regular TCP, it MUST NOT revert to MPTCP later in the connection 🚫 no take-backs, bestie.
and just to be clear, we are NOT trying to stop middleboxes that want to adjust the payload 💜 an MPTCP-aware middlebox could provide that functionality by also rewriting checksums ✨
3.7. Error Handling 🎀
on top of the fallback mechanism above, the standard classes of TCP errors might also need to be handled in an MPTCP-specific way. changing semantics (like what a RST means) are covered in Section 4. wherever possible, we don't want to stray from regular TCP behavior 💖
here's the list of possible errors and the right MPTCP behavior for each one:
- Unknown token in MP_JOIN (or HMAC failure in MP_JOIN ACK, or missing MP_JOIN in SYN/ACK response): send RST (analogous to TCP's behavior on an unknown port)
- DSN out of window (during normal operation): drop the data, do not send Data ACKs
- Remove request for unknown address ID: silently ignore
3.8. Heuristics 🌈
there are a bunch of heuristics that are needed for performance or deployment, but they aren't required for protocol correctness 🧠 this section walks through them. heads up that buffering and certain sender and receiver window behaviors are covered in Sections 3.3.4 and 3.3.5, and retransmission is in Section 3.3.6 🦋
3.8.1. Port Usage 🍓
under typical operation, an MPTCP implementation SHOULD use the same ports that are already in use. basically, the destination port of a SYN carrying an MP_JOIN option SHOULD be the same as the remote port of the first subflow in the connection. the local port for those SYNs SHOULD also match the first subflow's (so an implementation SHOULD reserve ephemeral ports across all local IP addresses), although there may be cases where that's not doable 🙈 this strategy helps maximize the chance that the SYN gets through a firewall or NAT at the recipient, and it avoids confusing any network monitoring software 💅
there may also be times when the passive opener wants to tell the other host that a specific port should be used. that's handled by the Add Address option, documented in Section 3.4.1 ✨ so it's totally possible to have multiple subflows between the same two addresses using different port pairs, and that could be used to load balance inside the network based on 5-tuples (for example, some ECMP implementations [7]) 🎀
3.8.2. Delayed Subflow Start ✨
Okay bestie, lots of TCP connections are super short, like only a few segments, and the overhead of MPTCP just outweighs the perks there 😭. So you need a heuristic to decide when to start using extra subflows in an MPTCP connection 🦋. We expect deployment experience to give more guidance on this, and it'll be shaped by particular application characteristics (which will probably keep changing over time) 💅. Still, here's a suggested general-purpose heuristic that an implementation MAY choose to use 🌸. Results from experimental deployments are needed to verify it's correct, so don't treat it as gospel yet ✨.
Here's the heuristic: if a host has data buffered for its peer (which basically means the application got a request for data), the host opens one subflow for each initial window's worth of data that's buffered 🎀.
Consideration should also be given to limiting the rate of adding new subflows, plus limiting the total number of subflows open for a particular connection 💜. A host may choose to vary these values based on its load or knowledge of traffic and path characteristics 🧚.
Note that this heuristic alone is probably insufficient 🙈. Traffic for lots of common apps, like downloads, is super asymmetric, and the multihomed host might be the client that never fills its buffers, so it never uses MPTCP 💀. Advanced APIs that let an app signal its traffic requirements would help with these decisions 👑.
An additional time-based heuristic could be applied too: open additional subflows after a given period of time has passed 🌈. That would fix the issue above and give resilience for low-bandwidth but long-lived apps 🫶.
This section showed some of the considerations an implementer should keep in mind when developing MPTCP heuristics, but it's not meant to be prescriptive 💖.
3.8.3. Failure Handling 🧯
Requirements for MPTCP's handling of unexpected signals were given in Section 3.7 🔥. There are other failure cases, though, where a host can choose appropriate behavior.
For example, Section 3.1 says that a host SHOULD fall back to trying regular TCP SYNs after one or more failures of MPTCP SYNs for a connection. A host may keep a system-wide cache of this kind of information, so it can back off from using MPTCP, first for that particular destination host, and eventually on a whole interface, if MPTCP connections keep failing 🙃.
Another failure could happen when the MP_JOIN handshake fails. Section 3.7 specifies that an incorrect handshake MUST lead to the subflow being closed with a RST. A host running an active intrusion detection system may choose to start blocking MP_JOIN packets from the source host if multiple failed MP_JOIN attempts are seen 🛡️. From the connection initiator's point of view, if an MP_JOIN fails, it SHOULD NOT attempt to connect to the same IP address and port during the lifetime of the connection, unless the other host refreshes the information with another ADD_ADDR option. Note that the ADD_ADDR option is informational only, and does not guarantee the other host will attempt a connection 🤷.
In addition, an implementation may learn, over a number of connections, that certain interfaces or destination addresses consistently fail, and it may default to not trying to use MPTCP for these. Behavior could also be learned for particularly badly performing subflows, or subflows that regularly fail during use, so it can temporarily choose not to use these paths 🧠.
4. Semantic Issues 🎀
In order to support multipath operation, the semantics of some TCP components have changed. To keep things clear, this section collects these semantic changes in one place as a reference 📚.
Sequence number: The (in-header) TCP sequence number is specific to the subflow. To let the receiver reorder application data, an additional data-level sequence space is used. In this data-level sequence space, the initial SYN and the final DATA_FIN occupy 1 octet of sequence space. There is an explicit mapping of data sequence space to subflow sequence space, which is signaled through TCP options in data packets 🔢.
ACK: The ACK field in the TCP header acknowledges only the subflow sequence number, not the data-level sequence space. Implementations SHOULD NOT attempt to infer a data-level acknowledgment from the subflow ACKs. This separates subflow- and connection-level processing at an end host ✂️.
Duplicate ACK: A duplicate ACK that includes any MPTCP signaling (with the exception of the DSS option) MUST NOT be treated as a signal of congestion. To limit the chances of non-MPTCP-aware entities mistakenly interpreting duplicate ACKs as a signal of congestion, MPTCP SHOULD NOT send more than two duplicate ACKs containing (non-DSS) MPTCP signals in a row 🚦.
Receive Window: The receive window in the TCP header indicates the amount of free buffer space for the whole data-level connection (as opposed to for this subflow) that is available at the receiver. This is the same semantics as regular TCP, but to maintain these semantics the receive window must be interpreted at the sender as relative to the sequence number given in the DATA_ACK rather than the subflow ACK in the TCP header. In this way, the original flow control role is preserved. Note that some middleboxes may change the receive window, so a host SHOULD use the maximum value of those recently seen on the constituent subflows for the connection-level receive window, and also needs to maintain a subflow-level window for subflow-level processing 🪟.
FIN: The FIN flag in the TCP header applies only to the subflow it is sent on, not to the whole connection. For connection-level FIN semantics, the DATA_FIN option is used 👋.
RST: The RST flag in the TCP header applies only to the subflow it is sent on, not to the whole connection. The MP_FASTCLOSE option provides the fast close functionality of a RST at the MPTCP connection level 💥.
Address List: Address list management (i.e., knowledge of the local and remote hosts' lists of available IP addresses) is handled on a per-connection basis (as opposed to per subflow, per host, or per pair of communicating hosts). This permits the application of per-connection local policy. Adding an address to one connection (either explicitly through an Add Address message, or implicitly through a Join) has no implication for other connections between the same pair of hosts 🏠.
5-tuple: The 5-tuple (protocol, local address, local port, remote address, remote port) presented by kernel APIs to the application layer in a non-multipath-aware application is that of the first subflow, even if the subflow has since been closed and removed from the connection. This decision, and other related API issues, are discussed in more detail in [6] 🎯.
5. Security Considerations ✨
okay bestie, this is the part where we talk about not getting hacked 💀🔒. As identified in [9], adding multipath to TCP is gonna bring a whole new set of threats, no cap. So [2] lays out requirements for a security solution for MPTCP 🫶. The big goal is for MPTCP to be "no worse" than regular TCP today, and the key security requirements are:
- Provide a mechanism to confirm that the parties in a subflow handshake are the same as in the original connection setup.
- Provide verification that the peer can receive traffic at a new address before using it as part of a connection.
- Provide replay protection, i.e., ensure that a request to add/remove a subflow is 'fresh'.
To hit those goals, MPTCP uses a hash-based handshake algorithm, documented in Sections 3.1 and 3.2 🔐✨.
The whole connection's security hangs on keys that get shared once at the start of the first subflow and are never sent over the network again (unless they're used in the fast close mechanism, Section 3.5) 🔑. To keep demultiplexing easy without giving away any crypto material, future subflows use a truncated cryptographic hash of this key as the connection identification "token" 🎀. The keys are concatenated and used as keys for Hash-based Message Authentication Codes (HMACs) on subflow setup, so the handshake can verify that the parties are the same as in the original connection setup. It also verifies that the peer can receive traffic at this new address. Replay attacks would still be possible when only keys are used, so the handshakes use single-use random numbers (nonces) at both ends. This means the HMAC will never be the same on two handshakes 🙈. Guidance on generating random numbers suitable for use as keys is given in [14] and discussed in Section 3.1.
The use of crypto capability bits in the initial connection handshake to negotiate a particular algorithm lets people deploy additional crypto mechanisms in the future 🦋. Note that this would only be susceptible to bid-down attacks if the attacker was on-path (and thus would be able to modify the data anyway). The security mechanism in this document should therefore protect against all forms of flooding and hijacking attacks discussed in [9] 👑.
During normal operation, regular TCP protection mechanisms (such as making sure sequence numbers are in-window) will provide the same level of protection against attacks on individual TCP subflows as exists for regular TCP today 🌸. Implementations will introduce additional buffers compared to regular TCP, to reassemble data at the connection level. The application of window sizing will minimize the risk of denial-of-service attacks consuming resources 🧚.
As discussed in Section 3.4.1, a host may advertise its private addresses, but these might point to different hosts in the receiver's network 👀. The MP_JOIN handshake (Section 3.2) will ensure that this does not succeed in setting up a subflow to the incorrect host. However, it could still create unwanted TCP handshake traffic. This feature of MPTCP could be a target for denial-of-service exploits, with malicious participants in MPTCP connections encouraging the recipient to target other hosts in the network 😭. Therefore, implementations should consider heuristics (Section 3.8) at both the sender and receiver to reduce the impact of this.
A small security risk could theoretically exist with key reuse, but in order to accomplish a replay attack, both the sender and receiver keys, and the sender and receiver random numbers, in the MP_JOIN handshake (Section 3.2) would have to match. 😬
Whilst this specification defines a "medium" security solution, meeting the criteria specified at the start of this section and the threat analysis ([9]), attacks only ever get worse 📈. So it is likely that a future Standards Track version of MPTCP would need to be able to support stronger security. There are several ways the security of MPTCP could potentially be improved. Some of these would be compatible with MPTCP as defined in this document, whilst others may not be. For now, the best approach is to get experience with the current approach, establish what might work, and check that the threat analysis is still accurate 🧠✨.
Possible ways of improving MPTCP security could include:
- defining a new MPCTP cryptographic algorithm, as negotiated in
MP_CAPABLE. A sub-case could be to include an additional deployment assumption, such as stateful servers, in order to allow a more powerful algorithm to be used. - defining how to secure data transfer with MPTCP, whilst not changing the signaling part of the protocol.
- defining security that requires more option space, perhaps in conjunction with a "long options" proposal for extending the TCP options space (such as those surveyed in [20]), or perhaps building on the current approach with a second stage of MPTCP-option-based security.
- revisiting the working group's decision to exclusively use TCP options for MPTCP signaling, and instead look at also making use of the TCP payloads 🌈.
MPTCP has been designed with several methods available to indicate a new security mechanism, including:
- available flags in
MP_CAPABLE(Figure 4); - available subtypes in the MPTCP option (Figure 3);
- the version field in
MP_CAPABLE(Figure 4);
6. Interactions with Middleboxes 🎀
okay so this section is literally about the messy real internet 🦋 MPTCP was designed to be deployable in the present world, which means it had to take into account "reasonable" existing middlebox behavior. Not the perfect internet, the actual one with all the boxes in it! 💅 In this section we walk through a few representative middlebox failure scenarios and show how MPTCP handles them, and then we list the design decisions that were made to accommodate the different middleboxes. ✨
6.1. Option Stripping and Dropping
Ngl, the big worry is our brand new TCP option. Middleboxes are supposed to forward packets with unknown options unchanged 🙈 but some of them don't. The ones we expect will either strip options and pass the data along, drop packets that carry new options, copy the same option into multiple segments (for example when doing segmentation), or drop options during segment coalescing. Yikes. 💀
MPTCP uses a single new TCP option "Kind", and every message type is defined by a "subtype" value (see Section 8). This should lower the chances of only some MPTCP options getting through, which is so smart 👑 The key differing characteristics end up being the different paths and whether the SYN flag is present.
MPTCP SYN packets on the first subflow of a connection contain the MP_CAPABLE option (Section 3.1). If this gets dropped, MPTCP SHOULD fall back to regular TCP. If packets with the MP_JOIN option (Section 3.2) get dropped, the paths just won't be used. Not the end of the world 🌸
If a middlebox strips options but otherwise passes the packets unchanged, MPTCP will behave safely 🫶 If an MP_CAPABLE option is dropped on either the outgoing or the return path, the initiating host can fall back to regular TCP, as shown in Figure 16 and discussed in Section 3.1.
Subflow SYNs contain the MP_JOIN option. If this option is stripped on the outgoing path, the SYN will look like a regular SYN to Host B. Depending on whether there's a listening socket on the target port, Host B will reply with either a SYN/ACK or an RST (and the subflow connection fails). When Host A gets the SYN/ACK, it sends an RST, because the SYN/ACK doesn't contain the MP_JOIN option and its token. Either way the subflow setup fails, but it doesn't affect the MPTCP connection as a whole. Main character energy for the connection, honestly 💖
Host A Host B
| Middlebox M |
| | |
| SYN(MP_CAPABLE) | SYN |
|-------------------|---------------->|
| SYN/ACK |
|<------------------------------------|
a) MP_CAPABLE option stripped on outgoing path
Host A Host B
| SYN(MP_CAPABLE) |
|------------------------------------>|
| Middlebox M |
| | |
| SYN/ACK |SYN/ACK(MP_CAPABLE)|
|<----------------|-------------------|
b) MP_CAPABLE option stripped on return pathFigure 16: Connection Setup with Middleboxes that Strip Options from Packets 🦋
We now look at data flow with MPTCP, assuming the flow was set up correctly, which means the options in the SYN packets got through the relevant middleboxes. If options are allowed through and there's no resegmentation or coalescing into TCP segments, MPTCP flows can proceed without problems ✨
The case where options get stripped on data packets was covered in the Fallback section. If a fraction of options gets stripped, the behavior is not deterministic 😭 If some data sequence mappings are lost, the connection can keep going as long as mappings exist for the subflow-level data (for example, if multiple maps were sent that reinforce each other). If some subflow-level space is left unmapped, though, the subflow is treated as broken and gets closed, through the process described in Section 3.6. MPTCP should survive losing some Data ACKs, but performance will degrade as the fraction of stripped options goes up. We don't expect this to show up in practice, though: most middleboxes will either strip all options or let them all through 🌈
We end this section with a list of middlebox classes, how they behave, and the parts of the MPTCP design that let things work through them. Issues about dropping packets with options or stripping options were discussed above, so they aren't repeated here:
- NATs [21] (Network Address (and Port) Translators) change the source address (and often the source port) of packets. This means a host won't know its public-facing address for signaling in MPTCP 🙈 So MPTCP permits implicit address addition via the MP_JOIN option, and the handshake mechanism makes sure that connection attempts to private addresses [18] don't cause problems. Explicit address removal is handled by an Address ID, so no knowledge of the source address is needed.
- Performance Enhancing Proxies (PEPs) [22] might proactively ACK data to boost performance. MPTCP, however, relies on accurate congestion control signals from the end host, and non-MPTCP-aware PEPs can't provide those signals. So MPTCP will fall back to single-path TCP, or close the problematic subflow (see Section 3.6). Bestie, it's giving "knowing your limits" 💅
- Traffic Normalizers [23] may not allow holes in sequence numbers, and may cache packets and retransmit the same data. MPTCP looks like standard TCP on the wire, and it will not retransmit different data on the same subflow sequence number. If a retransmission happens, the same data gets retransmitted on the original TCP subflow, even if it's also retransmitted at the connection level on a different subflow 🧚
- Firewalls [24] might perform initial sequence number randomization on TCP connections. MPTCP uses relative sequence numbers in the data sequence mapping to cope with this 💜 Like NATs, firewalls won't permit many incoming connections, so MPTCP supports address signaling (ADD_ADDR) so that a multiaddressed host can invite its peer behind the firewall/NAT to connect out to its additional interface. Inviting, not forcing, very polite 🫶
- Intrusion Detection Systems look out for traffic patterns and content that could threaten a network. Multipath means that such data is potentially spread out, so it's harder for an IDS to analyze the whole traffic, and it potentially increases the risk of false positives. However, for an MPTCP-aware IDS, tokens can be read by such systems to correlate multiple subflows and reassemble them for analysis. Smart, and iykyk 🔥
- Application-level middleboxes such as content-aware firewalls may alter the payload within a subflow, such as rewriting URIs in HTTP traffic. MPTCP will detect these using the checksum and close the affected subflow(s), if there are other subflows that can be used. If all subflows are affected, multipath will fall back to TCP, allowing such middleboxes to change the payload. MPTCP-aware middleboxes should be able to adjust the payload and MPTCP metadata so they don't break the connection 🌸
In addition, all classes of middleboxes may affect TCP traffic in the following ways:
- TCP options may be removed, or packets with unknown options dropped, by many classes of middleboxes. The initial SYN exchange, with a TCP option, is intended to be enough to identify the path capabilities. If such a packet doesn't get through, MPTCP will end up falling back to regular TCP 💀
- Segmentation/Coalescing (for example, TCP segmentation offloading) might copy options between packets and might strip some options. MPTCP's data sequence mapping includes the relative subflow sequence number instead of the sequence number in the segment. This way, the mapping is independent of the packets that carry it 🦋
- The receive window may be shrunk by some middleboxes at the subflow level. MPTCP will use the maximum window at the data level, but will also obey subflow-specific windows. Lowkey genius, no notes 👑
7. Acknowledgments 🌷
Okay, shoutout to everyone who made this possible 🫶 The authors were originally supported by Trilogy (http://www.trilogy-project.org), a research project (ICT-216372) partially funded by the European Community under its Seventh Framework Program.
Alan Ford was originally supported by Roke Manor Research ✨
The authors gratefully acknowledge significant input into this document from Sebastien Barre, Christoph Paasch, and Andrew McDonald 💖
The authors also wish to acknowledge reviews and contributions from Iljitsch van Beijnum, Lars Eggert, Marcelo Bagnulo, Robert Hancock, Pasi Sarolahti, Toby Moncaster, Philip Eardley, Sergio Lembo, Lawrence Conroy, Yoshifumi Nishida, Bob Briscoe, Stein Gjessing, Andrew McGregor, Georg Hampel, Anumita Biswas, Wes Eddy, Alexey Melnikov, Francis Dupont, Adrian Farrel, Barry Leiba, Robert Sparks, Sean Turner, Stephen Farrell, and Martin Stiemerling. A whole squad of legends 🌈👑
8. IANA Considerations ✨
Okay bestie, IANA is the registry keeper and this doc is giving them some homework 💅 First up, this document defines a new TCP option for MPTCP, and it gets the value 30 (decimal) from the TCP option space 🦋 That 30 is the "Kind" you'll see in every MPTCP option in here. Here's the entry:
+------+--------+-----------------------+-----------+
| Kind | Length | Meaning | Reference |
+------+--------+-----------------------+-----------+
| 30 | N | Multipath TCP (MPTCP) | RFC 6824 |
+------+--------+-----------------------+-----------+Table 1: TCP Option Kind Numbers
This document also defines a 4-bit subtype field 🎀 IANA has created a new sub-registry for it called "MPTCP Option Subtypes", living under the "Transmission Control Protocol (TCP) Parameters" registry, and IANA will maintain it. The initial values are below. Future assignments are to be defined by Standards Action as defined by [25]. Each entry has the subtype's symbolic name and its value, like so:
+-------+--------------+----------------------------+---------------+
| Value | Symbol | Name | Reference |
+-------+--------------+----------------------------+---------------+
| 0x0 | MP_CAPABLE | Multipath Capable | Section 3.1 |
| 0x1 | MP_JOIN | Join Connection | Section 3.2 |
| 0x2 | DSS | Data Sequence Signal (Data | Section 3.3 |
| | | ACK and data sequence | |
| | | mapping) | |
| 0x3 | ADD_ADDR | Add Address | Section 3.4.1 |
| 0x4 | REMOVE_ADDR | Remove Address | Section 3.4.2 |
| 0x5 | MP_PRIO | Change Subflow Priority | Section 3.3.8 |
| 0x6 | MP_FAIL | Fallback | Section 3.6 |
| 0x7 | MP_FASTCLOSE | Fast Close | Section 3.5 |
+-------+--------------+----------------------------+---------------+Table 2: MPTCP Option Subtypes
Values 0x8 through 0xe are currently unassigned 🌸 The value 0xf is reserved for Private Use within controlled testbeds.
IANA has also created another sub-registry, "MPTCP Handshake Algorithms", under the "Transmission Control Protocol (TCP) Parameters" registry. It's based on the flags in MP_CAPABLE (Section 3.1) 🧚 Those flags are 8 bits, labeled "A" through "H", and this document assigns the bits like this:
+----------+-------------------+-----------------------+
| Flag Bit | Meaning | Reference |
+----------+-------------------+-----------------------+
| A | Checksum required | RFC 6824, Section 3.1 |
| B | Extensibility | RFC 6824, Section 3.1 |
| C-G | Unassigned | |
| H | HMAC-SHA1 | RFC 6824, Section 3.2 |
+----------+-------------------+-----------------------+Table 3: MPTCP Handshake Algorithms
Note that the meanings of bits C through H can depend on bit B, depending on how Extensibility gets defined in future specifications. See Section 3.1 for more info 👑
Future assignments in this registry are also to be defined by Standards Action as defined by [25]. Each assignment consists of the value of the flags, a symbolic name for the algorithm, and a reference to its specification.
9. References
9.1. Normative References
[1] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.
[2] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. Iyengar, "Architectural Guidelines for Multipath TCP Development", RFC 6182, March 2011.
[3] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[4] National Institute of Science and Technology, "Secure Hash Standard", Federal Information Processing Standard (FIPS) 180-3, October 2008, http://csrc.nist.gov/publications/fips/fips180-3/fips180-3_final.pdf.
9.2. Informative References
[5] Raiciu, C., Handley, M., and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", RFC 6356, October 2011.
[6] Scharf, M. and A. Ford, "MPTCP Application Interface Considerations", Work in Progress, October 2012.
[7] Hopps, C., "Analysis of an Equal-Cost Multi-Path Algorithm", RFC 2992, November 2000.
[8] Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M., Duchene, F., Bonaventure, O., and M. Handley, "How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP", Usenix Symposium on Networked Systems Design and Implementation 012, 2012, https://www.usenix.org/conference/nsdi12/how-hard-can-it-be-designing-and-implementing-deployable-multipath-tcp.
[9] Bagnulo, M., "Threat Analysis for TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6181, March 2011.
[10] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, February 1997.
[11] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP Selective Acknowledgment Options", RFC 2018, October 1996.
[12] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681, September 2009.
[13] Gont, F., "Survey of Security Hardening Methods for Transmission Control Protocol (TCP) Implementations", Work in Progress, March 2012.
[14] Eastlake, D., Schiller, J., and S. Crocker, "Randomness Requirements for Security", BCP 106, RFC 4086, June 2005.
[15] Eastlake, D. and T. Hansen, "US Secure Hash Algorithms (SHA and SHA-based HMAC and HKDF)", RFC 6234, May 2011.
[16] Jacobson, V., Braden, B., and D. Borman, "TCP Extensions for High Performance", RFC 1323, May 1992.
[17] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001.
[18] Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G., and E. Lear, "Address Allocation for Private Internets", BCP 5, RFC 1918, February 1996.
[19] Braden, R., "Requirements for Internet Hosts - Communication Layers", STD 3, RFC 1122, October 1989.
[20] Ramaiah, A., "TCP option space extension", Work in Progress, March 2012.
[21] Srisuresh, P. and K. Egevang, "Traditional IP Network Address Translator (Traditional NAT)", RFC 3022, January 2001.
[22] Border, J., Kojo, M., Griner, J., Montenegro, G., and Z. Shelby, "Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations", RFC 3135, June 2001.
[23] Handley, M., Paxson, V., and C. Kreibich, "Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics", Usenix Security 2001, 2001, http://www.usenix.org/events/sec01/full_papers/handley/handley.pdf.
[24] Freed, N., "Behavior of and Requirements for Internet Firewalls", RFC 2979, October 2000.
[25] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008.
Appendix A. Notes on Use of TCP Options 🌈
Let's talk about TCP option space, which is seriously cramped 🙈 The Data Offset field in the TCP header is only 4 bits, and it sets the TCP header length in 32-bit words. The standard TCP header is 20 bytes, so that leaves a maximum of 40 bytes for options, and lots of those are already taken by stuff like timestamp and SACK.
We did a quick study on the commonly used TCP options in SYN, data, and pure ACK packets, and we found there's enough room to fit every option we propose in this document. Slay 💅
SYN packets usually include Maximum Segment Size (MSS) (4 bytes), window scale (3 bytes), SACK permitted (2 bytes), and timestamp (10 bytes) options. Together those add up to 19 bytes. Some operating systems seem to pad each option up to a word boundary, so they use 24 bytes instead. A brief survey suggests Windows XP and Mac OS X do this, whereas Linux does not. Optimistically, that leaves 21 bytes spare, or 16 if it has to be word-aligned. Either way, the SYN versions of Multipath Capable (12 bytes) and Join (12 or 16 bytes) options will fit in that leftover space ✨
TCP data packets usually carry timestamp options in every packet, which takes 10 bytes (or 12 with padding). That leaves 30 bytes, or 28 if it's word-aligned. The Data Sequence Signal (DSS) option changes length depending on whether the data sequence mapping and DATA_ACK are included, and whether the sequence numbers in use are 4 or 8 octets. The maximum size of the DSS option is 28 bytes, so even that will fit in the available space. Unless a connection is both bidirectional and high-bandwidth, though, it's unlikely that all that option space will be needed on each DSS option.
Inside the DSS option, the data sequence mapping and DATA_ACK don't have to appear in every packet. In many cases it's possible to alternate their presence, as long as the mapping covers the data being sent in the following packet. It would also be possible to alternate between 4- and 8-byte sequence numbers in each option 🦋
On subflow and connection setup, an MPTCP option is also set on the third packet (an ACK). These are 20 bytes (for Multipath Capable) and 24 bytes (for Join), and both fit in the available option space.
Pure ACKs in TCP usually contain only timestamps (10 bytes). Here, Multipath TCP typically only needs to encode the DATA_ACK, which is a maximum of 12 bytes. Sometimes ACKs contain SACK information, and depending on the number of lost packets, SACK may use up the entire option space. If a DATA_ACK has to be included too, then it's probably necessary to reduce the number of SACK blocks to make room for the DATA_ACK. However, the DATA_ACK is unlikely to be necessary when SACK is in use. Until at least some of the SACK blocks have been retransmitted, the cumulative data-level ACK won't move forward. (Or if it does, because of retransmissions on another path, then that path can also be used to transmit the new DATA_ACK.)
The ADD_ADDR option can be between 8 and 22 bytes, depending on whether IPv4 or IPv6 is used and whether or not the port number is present. It's unlikely that this kind of signaling would fit in a data packet, although if there's space, it's fine to include it. It is recommended to use duplicate ACKs with no other payload or options to send these rare signals. This is the reason for mandating that duplicate ACKs with MPTCP options are not taken as a signal of congestion 🙌
Finally, there are issues with reliable delivery of options. Options can also be sent on pure ACKs, and those aren't reliably sent. This isn't an issue for DATA_ACK, thanks to its cumulative nature, but it may be an issue for ADD_ADDR and REMOVE_ADDR options. For those, it is recommended to send the options redundantly. That can mean sending them on multiple paths, or on the same path across a number of ACKs, interspersed with data so they aren't read as congestion. The cases where options get stripped by middleboxes are discussed in Section 6 🫶
Appendix B. Control Blocks 🧩
Okay so, conceptually, an MPTCP connection can be represented as an MPTCP control block 💖 It holds several variables that track the progress and state of the connection, plus a set of linked TCP control blocks, one for each subflow that has been established. Think of it as the main folder with a tab for every subflow inside 📁✨
RFC 793 [1] specifies several state variables. Whenever possible, we reuse the same terminology as RFC 793 to describe the state variables that are maintained by MPTCP. Consistency is everything, bestie 🫶
B.1. MPTCP Control Block 👑
The MPTCP control block contains the following variable per connection.
B.1.1. Authentication and Metadata 🔐
Local.Token (32 bits): This is the token chosen by the local host on this MPTCP connection. The token MUST be unique among all established MPTCP connections, generated from the local key.
Local.Key (64 bits): This is the key sent by the local host on this MPTCP connection.
Remote.Token (32 bits): This is the token chosen by the remote host on this MPTCP connection, generated from the remote key.
Remote.Key (64 bits): This is the key chosen by the remote host on this MPTCP connection
MPTCP.Checksum (flag): This flag is set to true if at least one of the hosts has set the C bit in the MP_CAPABLE options exchanged during connection establishment, and is set to false otherwise. If this flag is set, the checksum must be computed in all DSS options. 🧾
B.1.2. Sending Side 📤
SND.UNA (64 bits): This is the data sequence number of the next byte to be acknowledged, at the MPTCP connection level. This variable is updated upon reception of a DSS option containing a DATA_ACK.
SND.NXT (64 bits): This is the data sequence number of the next byte to be sent. SND.NXT is used to determine the value of the DSN in the DSS option.
SND.WND (32 bits with RFC 1323, 16 bits otherwise): This is the sending window. MPTCP maintains the sending window at the MPTCP connection level and the same window is shared by all subflows. All subflows use the MPTCP connection level SND.WND to compute the SEQ.WND value that is sent in each transmitted segment. 🪟
B.1.3. Receiving Side 📥
RCV.NXT (64 bits): This is the data sequence number of the next byte that is expected on the MPTCP connection. This state variable is modified upon reception of in-order data. The value of RCV.NXT is used to specify the DATA_ACK that is sent in the DSS option on all subflows. 💌
RCV.WND (32 bits with RFC 1323, 16 bits otherwise): This is the connection-level receive window, which is the maximum of the RCV.WND on all the subflows.
B.2. TCP Control Blocks 🧱
The MPTCP control block also contains a list of the TCP control blocks that are associated to the MPTCP connection.
Note that the TCP control block on the TCP subflows does not contain the RCV.WND and SND.WND state variables, as these are maintained at the MPTCP connection level and not at the subflow level. Main character energy stays at the connection level 💅
Inside each TCP control block, the following state variables are defined.
B.2.1. Sending Side 📤
SND.UNA (32 bits): This is the sequence number of the next byte to be acknowledged on the subflow. This variable is updated upon reception of each TCP acknowledgment on the subflow.
SND.NXT (32 bits): This is the sequence number of the next byte to be sent on the subflow. SND.NXT is used to set the value of SEG.SEQ upon transmission of the next segment.
B.2.2. Receiving Side 📥
RCV.NXT (32 bits): This is the sequence number of the next byte that is expected on the subflow. This state variable is modified upon reception of in-order segments. The value of RCV.NXT is copied to the SEG.ACK field of the next segments transmitted on the subflow.
RCV.WND (32 bits with RFC 1323, 16 bits otherwise): This is the subflow-level receive window that is updated with the window field from the segments received on this subflow.
Appendix C. Finite State Machine 🌊
The diagram in Figure 17 shows the Finite State Machine for connection-level closure. It illustrates how the DATA_FIN connection-level signal (indicated as the DFIN flag on a DATA_ACK) interacts with subflow-level FINs, and permits "break-before-make" handover between subflows. Honestly, this diagram is so iconic 🦋
+---------+
| M_ESTAB |
+---------+
M_CLOSE | | rcv DATA_FIN
------- | | -------
+---------+ snd DATA_FIN / \ snd DATA_ACK[DFIN] +---------+
| M_FIN |<----------------- ------------------->| M_CLOSE |
| WAIT-1 |--------------------------- | WAIT |
+---------+ rcv DATA_FIN \ +---------+
| rcv DATA_ACK[DFIN] ------- | M_CLOSE |
| -------------- snd DATA_ACK | ------- |
| CLOSE all subflows | snd DATA_FIN |
V V V
+-----------+ +-----------+ +-----------+
|M_FINWAIT-2| | M_CLOSING | | M_LAST-ACK|
+-----------+ +-----------+ +-----------+
| rcv DATA_ACK[DFIN] | rcv DATA_ACK[DFIN] |
| rcv DATA_FIN -------------- | -------------- |
| ------- CLOSE all subflows | CLOSE all subflows |
| snd DATA_ACK[DFIN] V delete MPTCP PCB V
\ +-----------+ +---------+
------------------------>|M_TIME WAIT|----------------->| M_CLOSED|
+-----------+ +---------+
All subflows in CLOSED
------------
delete MPTCP PCB
Figure 17: Finite State Machine for Connection ClosureAuthors' Addresses 🎀
Here are the lovely humans behind this doc, with their addresses and emails so you can slide into their inboxes 💌
Alan Ford
Cisco
Ruscombe Business Park
Ruscombe, Berkshire RG10 9NN
UK
EMail: alanford@cisco.comCostin Raiciu
University Politehnica of Bucharest
Splaiul Independentei 313
Bucharest
Romania
EMail: costin.raiciu@cs.pub.roMark Handley
University College London
Gower Street
London WC1E 6BT
UK
EMail: m.handley@cs.ucl.ac.ukOlivier Bonaventure
Universite catholique de Louvain
Pl. Ste Barbe, 2
Louvain-la-Neuve 1348
Belgium
EMail: olivier.bonaventure@uclouvain.be📦 where this page comes from
- ietf: rfc6824 2013-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms