RFC 8041

Use Cases and Operational Experience with Multipath TCP

📅 published January 2017, from ietf

1. Introduction ✨

okay bestie, this RFC is literally the Multipath TCP (MPTCP) use-case and ops tea 🫶 it's giving "here's everything people actually did with it in real life" 💅

Heads up: the title block up top is the whole vibe check. This doc is RFC 8041, "Use Cases and Operational Experience with Multipath TCP," published in January 2017 as Informational (so it's not a Standards Track spec, just sharing the tea 🍵). The authors are O. Bonaventure (UCLouvain), C. Paasch (Apple, Inc.), and G. Detal (Tessares). It doesn't obsolete or update anything, and it's published under ISSN 2070-1721. 📚

Abstract

okay so the abstract is giving "quick preview" 🎀 it says this doc talks about both the use cases and the operational experience with Multipath TCP (MPTCP) in real networks. It lists a bunch of prominent use cases where MPTCP has been considered and is being used, and it also shares some heuristics and decisions that helped make those use cases happen, plus some possible improvements. slay 🦋

Status of This Memo

okay this part is the legal-ish intro but it's still important 🧚 This doc is not an Internet Standards Track spec. It's published for informational purposes only 🌸

It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 for that 💜

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc8041 🔥

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved 💖

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were on the publication date. Please read them carefully, because they explain your rights and restrictions for this doc 🙈 Code Components pulled out of this doc must include the Simplified BSD License text (as described in Section 4.e of the Trust Legal Provisions), and they come without warranty as described in that license 🍓

Table of Contents

okay the TOC is basically a map of the whole doc 🗺️ here it is, copied exactly so the numbers still line up:

1. Introduction ....................................................3
2. Use Cases .......................................................4
   2.1. Datacenters ................................................4
   2.2. Cellular/WiFi Offload ......................................5
   2.3. Multipath TCP Proxies ......................................8
3. Operational Experience ..........................................9
   3.1. Middlebox Interference .....................................9
   3.2. Congestion Control ........................................11
   3.3. Subflow Management ........................................12
   3.4. Implemented Subflow Managers ..............................13
   3.5. Subflow Destination Port ..................................15
   3.6. Closing Subflows ..........................................16
   3.7. Packet Schedulers .........................................17
   3.8. Segment Size Selection ....................................18
   3.9. Interactions with the Domain Name System ..................19
   3.10. Captive Portals ..........................................20
   3.11. Stateless Webservers .....................................20
   3.12. Load-Balanced Server Farms ...............................21
4. Security Considerations ........................................21
5. References .....................................................23
   5.1. Normative References ......................................23
   5.2. Informative References ....................................23
Acknowledgements ..................................................30
Authors' Addresses ................................................30

1. Introduction ✨

1.

okay so MPTCP got specified in [RFC6824] and five independent implementations have been built 🤯 As of November 2016, MPTCP was implemented (or being implemented) on these platforms:

  • Linux kernel [MultipathTCP-Linux]
  • Apple iOS and macOS
  • Citrix load balancers
  • FreeBSD [FreeBSD-MPTCP]
  • Oracle Solaris

The first three are known to interoperate 🫶 Three of these are open source (Linux kernel, FreeBSD, and Apple's iOS and macOS). And Apple's implementation is widely deployed 📱✨

Since [RFC6824] came out as an Experimental RFC, people (network researchers and regular users) have been gathering experience about the operational issues that pop up when MPTCP is used on today's Internet 👀

When the MPTCP working group was created, a few use cases were identified in [RFC6182] 🎀 Since then, other use cases have been proposed, and some have been tested and even deployed 🚀 We cover these in Section 2.

Section 3 is all about operational experience with MPTCP 💅 Most of it comes from the MPTCP implementation in the Linux kernel [MultipathTCP-Linux]. This open-source one has been downloaded and used by thousands of people all over the world 🌍 Lots of them shared feedback by writing up documents (scientific articles or blog posts) or posting to the mptcp-dev mailing list (see https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev). It's actively maintained and keeps getting better 💖 It runs on all kinds of hosts, from smartphones and embedded routers to high-end servers 🖥️

The Linux kernel one is not, by far, the most widespread MPTCP deployment 😭 Since September 2013, MPTCP has also been supported on smartphones and tablets, starting with iOS7 [IETFJ]. There are likely hundreds of millions of MPTCP-enabled devices 🤯 This implementation is currently only used to support the Siri voice recognition/control app 🎤 Some lessons from this deployment are described in [IETFJ] 📖

Okay, how Section 3 is laid out: supporting middleboxes was one of the hardest problems when designing MPTCP 🙈 Section 3.1 explains which types of middleboxes the Linux kernel MPTCP implementation supports and how it reacts when it runs into them 🧐 Section 3.2 sums up the MPTCP-specific congestion controls that got implemented 📊 Sections 3.3 to 3.7 talk about heuristics and issues around subflow management, plus scheduling across the subflows 🗂️ Section 3.8 covers problems that happened with subflows having different Maximum Segment Size (MSS) values 📏 Section 3.9 looks at issues with content delivery networks and suggests a fix 🌐 And finally, Section 3.10 documents a captive portal issue where MPTCP behaves suboptimally 🫠

2. Use Cases 🌸

okay so MPTCP has been tested in a bunch of use cases 🧪 There's already a huge pile of scientific literature on it [MPTCPBIB] 📚 Several of those papers identified possible improvements that are worth talking about here 💡

2.1. Datacenters 🏢

First up, and honestly kind of unexpected, is datacenters 😳 [HotNets][SIGCOMM11] documented MPTCP being used there. Today's datacenters are designed to offer several paths between single-homed servers 🛣️ Those multiple paths come from Equal-Cost Multipath (ECMP) and other load-balancing techniques inside the datacenter ⚖️ Most of the load-balancing techniques deployed in datacenters rely on hashes computed over the five tuple. So all packets from the same TCP connection follow the same path, and they don't get reordered 🙌

[HotNets] shows, through simulations, that MPTCP can make better use of the available network by using multiple subflows for each MPTCP session 📈 Now, [RFC6182] assumes that at least one of the communicating hosts has several IP addresses, but [HotNets] shows MPTCP is still beneficial when both hosts are single-homed 🤯 This idea is analyzed in more detail in [SIGCOMM11], where the Linux kernel MPTCP implementation was modified so it could use several subflows from the same IP address 🔧 Measurements in a public datacenter show the quantitative benefits of MPTCP [SIGCOMM11] in this environment 🏆

Okay, ECMP is widely used inside datacenters, but it's not the only place where a pair of hosts has different paths between them 👯 ECMP and other load-balancing techniques, like Link Aggregation Groups (LAGs), are everywhere in today's networks. Having multiple paths between a pair of single-homed hosts is becoming the norm instead of the exception 🫢 These paths often have the same cost (from an IGP metrics viewpoint), but they don't necessarily have the same performance 😬 For example, [IMC13c] reports a long measurement study showing that load-balanced Internet paths between that same pair of hosts can have huge delay differences 🐢💨

That's the intro and first use case done, bestie! 🌈 More parts are coming, so stay tuned 💜

2.2. Cellular/WiFi Offload 📱✨

okay bestie, next up is the cellular/WiFi offload use case 🦋 Smartphones and other mobile devices with two wireless interfaces are literally the most common place you'll spot Multipath TCP 💖 In September 2015 this was also the biggest deployment of MPTCP-enabled devices [IETFJ]. It got a quick chat at IETF 88 [IETF88], but no published paper or report actually analyses this deployment yet 🙈 So we're only looking at published papers that mostly used the Multipath TCP implementation in the Linux kernel for their experiments 🐧

the performance of Multipath TCP in wireless networks got a brief evaluation in [NSDI12] 👀 One experiment looked at Multipath TCP on a client with two wireless interfaces, and it showed that when the receive window is large, Multipath TCP can efficiently use both links. 🔥 BUT if the window gets smaller, packets sent on a slow path can block packets from going out on a faster path 😭 In some cases, Multipath TCP over two paths can end up performing worse than regular TCP over the best path. Yikes. That's why [NSDI12] proposes two heuristics, reinjection and penalization, to fix this problem 🛠️ Both of them have since been used in the Multipath TCP implementation in the Linux kernel, so they really did make it into the real world ✨ [CONEXT13] dug into the problem even more and found some other scenarios where Multipath TCP has trouble pooling the available paths efficiently. It proposes improvements to the Linux kernel implementation to cope with some of these problems 💅

the first experimental analysis of Multipath TCP in a public wireless environment was presented in [Cellnet12] 📶 These measurements check out how well Multipath TCP can use two wireless networks (real WiFi and 3G networks). They compare three modes of operation, so let's go through them 🎀

The first mode is using both wireless networks at the same time. Multipath TCP pools the available resources and uses both wireless interfaces, and this mode gives you fast handover from WiFi to cellular (or the other way around) when the user moves 🏃‍♀️ [CACM14] shows that handover between wireless networks is not an abrupt process. When a host moves, there are regions where one wireless network is weaker than the other, but the host still thinks that weaker network is up. When a mobile host enters one of those regions, being able to send packets over another wireless network is really important for a smooth handover. This is clearly illustrated by the packet trace discussed in [CACM14] 🫶

Many cellular networks use volume-based pricing, so users often prefer unmetered WiFi networks when they're available instead of metered cellular ones 💸 [Cellnet12] implements support for the MP_PRIO option to explore two other modes of operation 🔍

In the backup mode, Multipath TCP opens a TCP subflow over each interface, but the cellular interface is configured in backup mode. This means data only flows over the WiFi interface when both interfaces are considered active. If the WiFi interface fails, traffic switches quickly to the cellular interface, which gives the user a smooth handover [Cellnet12]. The catch is that the WiFi and cellular interfaces are likely to stay active all the time, since all subflows are established over the two interfaces 🔋

The single-path mode is a little different 🧚 It benefits from the break-before-make capability of Multipath TCP. When an MPTCP session is established, a subflow is created over the WiFi interface, and no packet is sent over the cellular interface as long as the WiFi interface stays up [Cellnet12]. So the cellular interface can sit idle and battery capacity is preserved 🔋💚 When the WiFi interface fails, a new subflow is established over the cellular interface so the established Multipath TCP sessions survive. Compared to the backup mode, [Cellnet12] measurements show this mode has a throughput drop while the cellular interface is brought up and the subflows are reestablished 📉

From a protocol viewpoint, [Cellnet12] talks about the problem posed by the unreliability of the REMOVE_ADDR option and proposes a small protocol extension so hosts can reliably exchange this option 🧐 It would be useful to analyze packet traces to figure out whether the unreliability of the REMOVE_ADDR option is actually an operational problem in real deployments 👩‍💻

Another study of Multipath TCP performance in wireless networks was reported in [IMC13b] 💻 This one uses laptops connected to various cellular ISPs and WiFi hotspots, and it compares various file transfer scenarios. [IMC13b] observes that 4-path MPTCP outperforms 2-path MPTCP, especially for larger files 🚀 However, for three congestion-control algorithms (LIA, OLIA, and Reno -- see Section 3.2), there is no significant performance difference for file sizes smaller than 4 MB 🤷‍♀️

A different study of Multipath TCP with two wireless networks is presented in [INFOCOM14] 🌈 In this one, the two networks had different qualities: a good network and a lossy network. When using two paths with different packet-loss ratios, the Multipath TCP congestion-control scheme moves traffic away from the lossy link, which it considers congested. But [INFOCOM14] documents an interesting scenario, which goes like this 👇

client ----------- path1 -------- server
  |                                  |
  +--------------- path2 ------------+

Figure 1: Simple network topology

Okay, so at the start, both paths in Figure 1 have the same quality and Multipath TCP spreads the load over both of them 🌸 During the transfer, path2 becomes lossy, maybe because the client moves 🏃‍♀️ Multipath TCP detects the packet losses, and those packets get retransmitted over path1, so the data transfer keeps going over that path. HOWEVER, the subflow over path2 is still up and sends one packet from time to time. The N packets have been acknowledged over the first subflow (at the MPTCP level), but they haven't been acknowledged at the TCP level over the second subflow. To keep the sequence numbers continuous over the second subflow, TCP keeps retransmitting these segments until they're acknowledged or the maximum number of retransmissions is reached. This is clearly inefficient, and it can lead to blocking, because the second subflow eats window space just to retransmit these packets 😩 [INFOCOM14] proposes a new Multipath TCP option to solve this problem. In practice, though, a new TCP option is probably not required 🙃 When the client sees that the data sent over the second subflow has been acknowledged over the first subflow, it could decide to terminate the second subflow by sending a RST segment 💅 If the interface tied to this subflow is still up, a new subflow could be reestablished right away. It would be immediately usable to send new data and wouldn't have to first retransmit the old data first 🦋 As of this writing, this dynamic management of subflows is not yet implemented in the Multipath TCP implementation in the Linux kernel 🙈

Some studies have started looking at Multipath TCP on smartphones with real applications 📲 Unlike the bulk transfers that a lot of publications use, many deployed apps don't exchange huge amounts of data and mainly use small connections. [COMMAG2016] proposes a software testing framework that automates Android applications so you can study how they interact with Multipath TCP 🤖 [PAM2016] analyses a one-month packet trace of all the packets exchanged by a dozen smartphones used by regular people. This analysis reveals that short connections are important on smartphones, and that the main benefit of Multipath TCP on smartphones is the ability to do seamless handovers between different wireless networks ✨ Long connections benefit from these handovers too 💜

2.3. Multipath TCP Proxies 🧚

Okay so, Multipath TCP isn't widely deployed on both clients and servers yet 🥺, so people have been getting creative with proxies. Two families of solutions are being used or tested right now 🎀.

The first use case is when an MPTCP-enabled client wants to use several interfaces to reach a regular TCP server. Picture a smartphone that needs both its WiFi and its cellular interface to move data 📱. There are a few proxy options here. An HTTP proxy deployed on an MPTCP-capable server would let the smartphone use Multipath TCP to access regular web servers. Obviously, this only works for applications that rely on HTTP. Another option is a proxy that can convert any Multipath TCP connection into a regular TCP connection. MPTCP-specific proxies have been proposed [HotMiddlebox13b] [HAMPEL].

Another option uses the SOCKS protocol [RFC1928] 💼. SOCKS is often used in enterprise networks so clients can reach external servers. The client opens a TCP connection to the SOCKS server, and the SOCKS server relays it to the final destination. If both the client and the SOCKS server use Multipath TCP, but the final destination doesn't, Multipath TCP can still be used on the path between the clients and the SOCKS server ✨. At IETF 93, Korea Telecom announced that they have deployed (in June 2015) a commercial service that uses Multipath TCP on smartphones. These smartphones access regular TCP servers through a SOCKS proxy. This lets them reach throughputs of up to 850 Mbps [KT] 🚀.

Measurements done with Android smartphones [Mobicom15] show that popular applications work correctly through a SOCKS proxy and MPTCP-enabled smartphones 📊. Thanks to Multipath TCP, long-lived connections can be spread across the two available interfaces 🦋. For short-lived connections, though, most of the data goes over the initial subflow, which is created over the interface for the default route, and the second subflow is almost not used [PAM2016] 🙈.

The second use case is when Multipath TCP is used by middleboxes, typically inside access networks. Various network operators are discussing and evaluating solutions for hybrid access networks [TR-348] 🏗️. These networks show up when a network operator controls two different access network technologies, e.g., wired and cellular, and wants to combine them to improve the bandwidth offered to end users [HYA-ARCH]. Several solutions are being investigated for such networks [TR-348]. Figure 2 shows how such a network is organized 🗺️. When a client creates a normal TCP connection, it is intercepted by the Hybrid CPE (HPCE), which converts it into a Multipath TCP connection so it can use the available access networks (DSL and LTE in the example). The Hybrid Access Gateway (HAG) does the opposite, so the regular server sees a normal TCP connection. Some of the solutions discussed for hybrid networks use Multipath TCP on the HCPE and the HAG. Other solutions rely on tunnels between the HCPE and the HAG [GRE-NOTIFY] 💜.

client --- HCPE ------ DSL ------- HAG --- internet --- server
            |                       |
            +------- LTE -----------+

Figure 2: Hybrid Access Network

3. Operational Experience ✨

3.1. Middlebox Interference 💅

Middlebox interference has been a big concern during the design of the Multipath TCP protocol 😭. Three studies on the interactions between Multipath TCP and middleboxes are worth talking about.

The first analysis appears in [IMC11]. This paper was the main motivation for Multipath TCP including various techniques to cope with middlebox interference 🔥. More specifically, Multipath TCP was designed to cope with middleboxes that:

  • change source or destination addresses
  • change source or destination port numbers
  • change TCP sequence numbers
  • split or coalesce segments
  • remove TCP options
  • modify the payload of TCP segments

All of these middlebox interferences have been included in the MBtest suite [MBTest]. This test suite is used in [HotMiddlebox13] to check how the Multipath TCP implementation in the Linux kernel [MultipathTCP-Linux] reacts when faced with middlebox interference 🧪. The test environment for this evaluation is a dual-homed client connected to a single-homed server. The middlebox behavior can be activated on any of the paths. The main results of this analysis are:

  • the Multipath TCP implementation in the Linux kernel is not affected by a middlebox that performs NAT or modifies TCP sequence numbers
  • when a middlebox removes the MP_CAPABLE option from the initial SYN segment, the Multipath TCP implementation in the Linux kernel falls back correctly to regular TCP
  • when a middlebox removes the DSS option from all data segments, the Multipath TCP implementation in the Linux kernel falls back correctly to regular TCP
  • when a middlebox performs segment coalescing, the Multipath TCP implementation in the Linux kernel is still able to accurately extract the data corresponding to the indicated mapping
  • when a middlebox performs segment splitting, the Multipath TCP implementation in the Linux kernel correctly reassembles the data corresponding to the indicated mapping. [HotMiddlebox13] shows, in Figure 4 in Section 3.3, a corner case with segment splitting that may lead to a desynchronization between the two hosts 👀

The interactions between Multipath TCP and real deployed middleboxes are also analyzed in [HotMiddlebox13]. It describes a particular scenario with the FTP Application Level Gateway running on a NAT 🎭.

Middlebox interference can also be detected by analyzing packet traces on MPTCP-enabled servers. A closer look at the packets received on the multipath-tcp.org server [TMA2015] shows that among the 184,000 Multipath TCP connections, only 125 of them were falling back to regular TCP. These connections came from 28 different client IP addresses. They include 91 HTTP connections and 34 FTP connections. The FTP interference is expected, since Application Level Gateways used for FTP modify the TCP payload and the DSS Checksum detects these modifications 🙃. The HTTP interference appeared only in the direction from server to client, and it could have been caused by transparent proxies deployed in cellular or enterprise networks. A longer trace is discussed in [COMCOM2016], and similar conclusions about middlebox interference are provided 📝.

From an operational viewpoint, knowing that Multipath TCP can cope with various types of middlebox interference is important 💖. However, there are situations where network operators need to gather information about where a particular middlebox interference occurs. The tracebox software [tracebox], described in [IMC13a], is an extension of the popular traceroute software. It lets network operators check at which hop a particular field of the TCP header (including options) is modified 🔍. Several network operators have used it to debug various middlebox interference problems. Experience with tracebox indicates that supporting the ICMP extension defined in [RFC1812] makes it easier to debug middlebox problems in IPv4 networks 🧠.

Users of the Multipath TCP implementation have reported some experiences with middlebox interference. The strangest scenario has been a middlebox that accepts the Multipath TCP options in the SYN segment but later replaces Multipath TCP options with a TCP EOL option [StrangeMbox] 🤯. This causes Multipath TCP to fall back to regular TCP without any impact on the application.

3.2. Congestion Control 🌈

Congestion control has been a big challenge for Multipath TCP 💪. The coupled congestion-control scheme defined in [RFC6356] is an adaptation of the NewReno algorithm. A detailed description of this coupled algorithm is provided in [NSDI11]. It is the default scheme in the Linux implementation of Multipath TCP, but Linux supports other schemes too 🐧.

The second congestion-control scheme is OLIA [CONEXT12]. It is also an adaptation of the NewReno single-path congestion-control scheme to support multiple paths. Simulations [CONEXT12] and measurements [CONEXT13] have shown that it provides some performance benefits compared to the default coupled congestion-control scheme 📈.

The delay-based scheme proposed in [ICNP12] has also been ported to the Multipath TCP implementation in the Linux kernel. It has been evaluated using simulations [ICNP12] and measurements [PaaschPhD] 🧑‍🔬.

BALIA, defined in [BALIA], provides a better balance between TCP friendliness, responsiveness, and window oscillation 🦄.

These different congestion-control schemes have been compared in several articles. [CONEXT13] and [PaaschPhD] compare these algorithms in an emulated environment. The evaluation showed that the delay-based congestion-control scheme is less able to efficiently use the available links than the other three schemes 🥲.

3.3. Subflow Management 🎀

The multipath capability of Multipath TCP comes from using one subflow per path. The Multipath TCP architecture [RFC6182] and the protocol specification [RFC6824] define the basic usage of subflows and the protocol mechanisms that are required to create and terminate them. However, there are no guidelines on how subflows are used during the lifetime of a Multipath TCP session 🤷‍♀️. Most of the published experiments with Multipath TCP were performed in controlled environments. Still, based on experience running them and discussions on the mptcp-dev mailing list, interesting lessons have been learned about managing these subflows 📬.

From a subflow viewpoint, the Multipath TCP protocol is completely symmetrical. Both the clients and the server have the capability to create subflows. However, in practice, existing Multipath TCP implementations have opted for a strategy where only the client creates new subflows. The main motivation for this strategy is that the client often resides behind a NAT or a firewall, which prevents passive subflow openings on the client 🛡️. Although there are environments such as datacenters where this problem does not occur, as of this writing, no precise requirement has emerged for allowing the server to create new subflows 🫶.

3.4. Implemented Subflow Managers ✨

okay so the Multipath TCP implementation in the Linux kernel has a few different strategies for managing the subflows that make up a Multipath TCP session 🫶 the basic one is the full-mesh. like the name says, it builds a full mesh of subflows between the communicating hosts 🕸️

the most common situation for this manager is a multihomed client talking to a single-homed server. here one subflow gets created for each interface on the client 💅 the full-mesh subflow manager is currently static. the subflows get created right after the initial subflow is made. if one subflow fails later in the session (say from excessive retransmissions or losing its interface), it is not always set up again. there's ongoing work to help the full-mesh path manager handle those moments better 🙈

now when the server is multihomed, the full-mesh manager can lead to a LOT of subflows getting established 😭 for example, picture a dual-homed client connected to a server with three interfaces. even though only the client creates subflows, six subflows end up being established 💀 that can be excessive in some environments, especially when the client and/or the server have lots of interfaces. implementations should limit the number of subflows that are used 🌸

creating subflows between multihomed clients and servers can sometimes cause operational issues. the mptcp-dev mailing list has had discussions about this 💬 sometimes network operators want better control over how Multipath TCP creates its subflows [MPTCP-MAX-SUB]. that might mean defining policy rules to steer the subflow manager. the two scenarios below show some of these needs 🎀

host1 ----------  switch1 ----- host2
  |                   |            |
  +--------------  switch2 --------+

Figure 3: Simple Switched Network Topology

look at the simple network topology in Figure 3 🧚 from an operational point of view, a network operator might want two subflows between the communicating hosts. from a bandwidth point of view, the most natural paths are host1-switch1-host2 and host1-switch2-host2. the catch is that a Multipath TCP implementation running on these two hosts may sometimes struggle to end up with that result 😩

so to understand the struggle, let's look at different strategies for allocating IP addresses. the first strategy is to assign two subnets. subnetA (or subnetB) holds the IP addresses of host1's interface to switch1 (or switch2) and host2's interface to switch1 (or switch2). in this setup, a Multipath TCP subflow manager should only create one subflow per subnet ✨ to enforce using these paths, the network operator would have to specify a policy that prefers subflows within the same subnet over subflows between addresses in different subnets. it should be noted that the policy should probably also specify how the subflow manager reacts when an interface or subflow fails 👑

the second strategy is to use a single subnet for all IP addresses. in that case it gets way harder to write a policy that says which subflows should be established 😵‍💫

the second subflow manager currently supported by the Multipath TCP implementation in the Linux kernel is the ndiffport subflow manager 🦋 it was originally built to exploit the path diversity that exists between single-homed hosts thanks to flow-based load-balancing techniques [SIGCOMM11]. this manager creates N subflows between the same pair of IP addresses. the N subflows are created by the client and differ only in the source port the client picks. it was not designed for multihomed hosts, so please don't reach for it there 🙅‍♀️

there's also a more flexible subflow manager that has been proposed, implemented, and evaluated in [CONEXT15] 🌈 it exposes various kernel events to a user space daemon, and that daemon decides when subflows need to be created and terminated based on various policies. so cool, right? 🤩

3.5. Subflow Destination Port 🎀

so the Multipath TCP protocol uses the token in the MP_JOIN option to link a subflow to an existing Multipath TCP session 🔗 that means there's no restriction on the source address, destination address, or source or destination ports for the new subflow. being able to use different source and destination addresses is key for supporting multihomed servers and clients 💖 being able to use different destination port numbers is worth talking about, though, because it has operational implications 👀

let's picture a dual-homed client that creates a second subflow to reach a single-homed server, like in Figure 4 🧁

client ------- r1 --- internet --- server
    |                   |
    +----------r2-------+

Figure 4: Multihomed-Client Connected to Single-Homed Server

when the Multipath TCP implementation in the Linux kernel creates that second subflow, it uses the same destination port as the initial subflow 🔒 the reason is that the server might sit behind a firewall that only accepts TCP connections (subflows included) on the official port number. using the same destination port for every subflow also helps operators who rely on port numbers to track application usage on their network 📊

now, Multipath TCP users have suggested changing the implementation so the client can use different destination ports to reach the server 🤔 the suggestion seems mainly motivated by traffic-shaping middleboxes in some wireless networks. in networks where different shaping rates are tied to different destination port numbers, this could let Multipath TCP reach higher performance 🚀 this behavior is valid according to the Multipath TCP specification [RFC6824]. an application could use an enhanced socket API [SOCKET] to behave this way 🧚‍♀️

BUT (lowercase energy, sorry bestie 😅) from an implementation point of view, supporting different destination ports for the same Multipath TCP connection can cause some issues 😬 a legacy TCP stack creates a listening socket to react to incoming SYN segments. that listening socket handles the SYN segments sent to one specific port number. demultiplexing incoming segments can therefore be done just by looking at the IP addresses and port numbers. with Multipath TCP, though, incoming SYN segments may carry an MP_JOIN option with a different destination port. that means all incoming segments that didn't match an existing listening socket or an already established socket must be parsed for a possible MP_JOIN option 🫠 this adds a cost on servers that didn't exist on legacy TCP implementations 💸

3.6. Closing Subflows 🦋

                    client                       server
                       |                           |
   MPTCP: ESTABLISHED  |                           | MPTCP: ESTABLISHED
   Sub: ESTABLISHED    |                           | Sub: ESTABLISHED
                       |                           |
                       |         DATA_FIN          |
   MPTCP: CLOSE-WAIT   | <------------------------ | close()   (step 1)
   Sub: ESTABLISHED    |         DATA_ACK          |
                       | ------------------------> | MPTCP: FIN-WAIT-2
                       |                           | Sub: ESTABLISHED
                       |                           |
                       |  DATA_FIN + subflow-FIN   |
   close()/shutdown()  | ------------------------> | MPTCP: TIME-WAIT
   (step 2)            |        DATA_ACK           | Sub: CLOSE-WAIT
   MPTCP: CLOSED       | <------------------------ |
   Sub: FIN-WAIT-2     |                           |
                       |                           |
                       |        subflow-FIN        |
   MPTCP: CLOSED       | <------------------------ | subflow-close()
   Sub: TIME-WAIT      |        subflow-ACK        |
   (step 3)            | ------------------------> | MPTCP: TIME-WAIT
                       |                           | Sub: CLOSED
                       |                           |

Figure 5: Multipath TCP may not be able to avoid time-wait state on the subflow (indicated as Sub in the drawing), even if enforced by the application on the client-side.

okay Figure 5 shows a very particular issue within Multipath TCP and it's giving chaos 🌀 lots of high-performance apps try to avoid TIME-WAIT state by deferring the closure of the connection until the peer has sent a FIN 🙌 so the client on the left of Figure 5 does a passive closure, moving from CLOSE-WAIT to Last-ACK and finally freeing its resources once it gets the ACK of the FIN. an application running on top of an MPTCP-enabled Linux kernel might use this approach too 💅 the difference is that the close() of the connection (step 1 in Figure 5) only triggers the sending of a DATA_FIN. nothing guarantees the kernel is ready to combine that DATA_FIN with a subflow-FIN 😳

the reception of the DATA_FIN makes the application trigger the closure of the connection (step 2), trying to dodge TIME-WAIT with this late closure. this time, the kernel might decide to combine the DATA_FIN with a subflow-FIN. and this decision is fatal 💀 the subflow's state machine won't move from CLOSE_WAIT to Last-ACK. instead it goes through FIN_WAIT-2 and into TIME-WAIT state. TIME-WAIT state consumes host resources for at least 2 MSL (Maximum Segment Lifetime) ⏳

so the smart app that tries to avoid TIME-WAIT by doing a late closure actually ends up with one of its subflows in TIME-WAIT anyway 🙃 a high-performance Multipath TCP kernel implementation should honor the application's wish to do a passive closure of the connection and successfully avoid TIME-WAIT state, even on the subflows 👑✨

the fix relies on an optimistic assumption: a host doing an active closure of a Multipath TCP connection by sending a DATA_FIN will soon also send a FIN on all of its subflows 🤞 so the passive closer of the connection can simply wait for the peer to send exactly that FIN, which enforces passive closure even on the subflows 🫶 of course, to avoid consuming resources forever, a timer must limit how long our implementation waits for the FIN ⏰🩷

3.7. Packet Schedulers 🎀

okay so in a Multipath TCP implementation, the packet scheduler is the algorithm that runs every time a packet is transmitted, and it decides which subflow that packet goes out on 🦋 the scheduler itself does not have any impact on the interoperability of Multipath TCP implementations, but it may clearly impact the performance of Multipath TCP sessions 💅 The Multipath TCP implementation in the Linux kernel has a pluggable architecture for the packet scheduler [PaaschPhD] ✨ As of this writing, two schedulers have been implemented: round-robin and lowest-rtt-first.

The second one, lowest-rtt-first, relies on the round-trip time (rtt) measured on each TCP subflow and sends first segments over the subflow with the lowest round-trip time 🚀 They are compared in [CSWS14]. The experiments and measurements described in [CSWS14] show that lowest-rtt-first appears to be the best compromise from a performance viewpoint, so it's giving main character energy 👑

Another study of the packet schedulers is presented in [PAMS2014] 🧚 This one relies on simulations with the Multipath TCP implementation in the Linux kernel. They compare lowest-rtt-first with round-robin and a random scheduler. They show some situations where lowest-rtt-first does not perform as well as the other schedulers, but there are many scenarios where the opposite is true 🙈 [PAMS2014] notes that "it is highly likely that the optimal scheduling strategy depends on the characteristics of the paths being used." ngl that's the real plot twist 💀

3.8. Segment Size Selection 🌸

When an application does a write/send system call, the kernel allocates a packet buffer (sk_buff in Linux) to store the data the app wants to send 📦 The kernel will store at most one MSS (Maximum Segment Size) of data per buffer. Here's the catch: the MSS can differ amongst subflows, so an MPTCP implementation has to select carefully the MSS used to generate application data 😭 The Linux kernel implementation had various ways of picking the MSS, like the minimum or the maximum amongst the different subflows. However, these MSS selection heuristics can cause significant performance issues in some environments 💅

Let's look at an example, bestie. An MPTCP connection has two established subflows that respectively use an MSS of 1420 and 1428 bytes. If MPTCP selects the maximum, then the application will generate segments of 1428 bytes of data. An MPTCP implementation will have to split that segment in two (1420-byte and 8-byte) segments when pushing on the subflow with the smallest MSS. The latter segment introduces a large overhead, because this single data segment uses 2 slots in the congestion window (in packets), which reduces by roughly twice the potential throughput (in bytes/s) of this subflow 💀 Taking the smallest MSS doesn't fix it either, since there might be a case where the subflow with the smallest MSS only sends a few packets, which reduces the potential throughput of the other subflows 🙈

The Linux implementation recently took another approach [DetalMSS] ✨ Instead of selecting the minimum and maximum values, it now dynamically adapts the MSS based on the contribution of all the subflows to the connection's throughput. For each subflow, it computes the potential throughput achieved by selecting each MSS value, and it takes into account the lost space in the congestion window. Then it selects the MSS that allows the highest potential throughput to be achieved 🔥

Given the prevalence of middleboxes that clamp the MSS, Multipath TCP implementations must be able to efficiently support subflows with different MSS values 🫶 The strategy described above is a possible solution to this problem.

3.9. Interactions with the Domain Name System 🌈

Multihomed clients such as smartphones can send DNS queries over any of their interfaces 📱 When a single-homed client performs a DNS query, it receives the best answer for its request from its local resolver. If the client is multihomed, though, the answer to the DNS query may vary with the interface it was sent over 👀

                   cdn1
                    |
        client -- cellular -- internet -- cdn3
           |                   |
           +----- wifi --------+
                    |
                  cdn2


                  Figure 6: Simple Network Topology

If the client sends a DNS query over the WiFi interface, the answer will point to the cdn2 server, while the same request sent over the cellular interface will point to the cdn1 server 🌸 This might cause problems for CDN providers that locate their servers inside ISP networks and have contracts that say the CDN server will only be accessed from within that particular ISP 😬 Now imagine both the client and the CDN servers support Multipath TCP. In this case, a Multipath TCP session from cdn1 or cdn2 would potentially use both the cellular network and the WiFi network. Serving the client from cdn2 over the cellular interface could violate the contract between the CDN provider and the network operators 🚨 A similar problem happens with regular TCP if the client caches DNS replies. For example, the client gets a DNS answer over the cellular interface, then stops that interface and starts using its WiFi interface. If the client retrieves data from cdn1 over its WiFi interface, this may also violate the contract between the CDN and the network operators 🙃

A possible solution to prevent this problem would be to modify the DNS resolution on the client 🧠 The client subnet Extension Mechanisms for DNS (EDNS) defined in [RFC7871] could be used for this purpose. When the client sends a DNS query from its WiFi interface, it should also send the client subnet corresponding to the cellular interface in this request. This would tell the resolver that the answer should be valid for both the WiFi and the cellular interfaces (e.g., the cdn3 server) ✨

3.10. Captive Portals 🦋

Multipath TCP lets a host use different interfaces to reach a server. In theory, this should keep connectivity going as long as at least one of the interfaces is active 🌷 In practice, though, there are some particular scenarios with captive portals that can cause operational problems 😤 The reference environment is shown in Figure 7.

        client -----  network1
             |
             +------- internet ------------- server

                 Figure 7: Issue with Captive Portal

The client is attached to two networks: network1, which provides limited connectivity, and the entire Internet through the second network interface. In practice, this scenario corresponds to an open WiFi network with a captive portal for network1, and a cellular service for the second interface 📶 On many smartphones, the WiFi interface is preferred over the cellular interface. If the smartphone learns a default route via both interfaces, it will typically prefer the WiFi interface to send its DNS request and create the first subflow 🙈 This is not optimal with Multipath TCP. A better approach would probably be to try a few attempts on the WiFi interface and then, upon failure of those attempts, use the second interface for the initial subflow as well 🫶

3.11. Stateless Webservers 🧚

MPTCP has been designed to interoperate with webservers that benefit from SYN-cookies to protect against SYN-flooding attacks [RFC4987] 🛡️ MPTCP does this by echoing the keys negotiated during the MP_CAPABLE handshake in the third ACK of the three-way handshake. Receiving that third ACK then lets the server reconstruct the state specific to MPTCP ✨

Okay, but there's one caveat, and it's the unreliable nature of the third ACK 😬 When the third ACK gets lost, the server can't reconstruct the MPTCP state, so MPTCP falls back to regular TCP in this case. This is different from regular TCP. When the client starts sending data, the first data segment also includes the SYN-cookie, which lets the server reconstruct the TCP state. Further, this data segment gets retransmitted by the client if it's lost, so it's resilient against loss 💪 MPTCP doesn't include the keys in this data segment, though, so the server cannot reconstruct the MPTCP state 💀

This issue might be considered a minor one for MPTCP. Losing the third ACK should only happen when packet loss is high, and in that case MPTCP provides a lot of benefits because it can move traffic away from the lossy link 🌊 It is undesirable for MPTCP to have a higher chance of falling back to regular TCP in those lossy environments 😭

[MPTCP-DEPLOY] discusses this issue and suggests a modified handshake mechanism that ensures reliable delivery of the MP_CAPABLE, following the three-way handshake 🔧 This modification would make MPTCP reliable, even in lossy environments where servers need to use SYN-cookies to protect against SYN-flooding attacks 🎀

3.12. Load-Balanced Server Farms 🍓

Large-scale server farms typically deploy thousands of servers behind a single virtual IP (VIP) 🏰 Traffic gets steered to these servers through Layer 4 load-balancers that make sure a TCP flow is always routed to the same server [Presto08] 💖

As Multipath TCP uses multiple different TCP subflows to steer traffic across the different paths, load-balancers need to make sure all these subflows get routed to the same server 🫠 This means the load-balancers need to track the MPTCP-related state, so they can parse the token in the MP_JOIN and assign those subflows to the appropriate server. But server farms typically deploy several load-balancers for reliability and capacity reasons, and since a TCP subflow might get routed to any of them, they would need to synchronize the MPTCP-related state. That is not feasible on a large scale 😵‍💫

The token (carried in the MP_JOIN) contains the information that indicates which MPTCP session the subflow belongs to. Because the token is a hash of the key, servers aren't able to generate the token in such a way that it can provide the necessary information to the load-balancers, which would let them route TCP subflows to the appropriate server 🔐 [MPTCP-LOAD] discusses this issue in detail and suggests two alternative MP_CAPABLE handshakes to overcome these 🌟

4. Security Considerations ✨

okay bestie, this section is SO important and I need you to read it carefully 🔥 this doc is informational, so it's sharing use cases and real-world ops experience for Multipath TCP. The full deep dive on the remaining security issues lives in [RFC7430], with some suggested fixes too 💅

Here's the big tea 🫶 Multipath TCP, just like SCTP, can send packets from one single connection over different paths. That's the whole point of it, and it's a design feature, not a bug. BUT it means middleboxes that were built assuming they'd see every packet of a connection, in both directions, might stop working correctly 😭 Think firewalls, IDS, and DPI boxes sitting in enterprise networks. Those guys expect to see all the packets from all the TCP connections, and with MPTCP they might miss some, because some packets take a different route.

Figure 8 is the first example 👀 Picture an MPTCP-enabled smartphone hooked up to both an enterprise network and a cellular network. If it opens a Multipath TCP connection to a server, some packets from the phone or the server might go over the cellular network, which means the enterprise middlebox just... can't see them 🙈

  smartphone +----- enterprise net --- MBox----+------ server
             |                                 |
             +----- cellular net  -------------+

           Figure 8: Enterprise Middlebox May Not Observe
                  All Packets from Multihomed Host

Figure 9 is the second one, and it's a little more chaotic 🌈 Multiple middleboxes live inside one network here. To keep it simple, the doc assumes network1 is the default IPv4 path and network2 is the default IPv6 path. A similar thing could happen with per-flow load balancing like ECMP [RFC2992]. With regular TCP, every packet of each connection goes through either Mbox1 or Mbox2. With MPTCP, though, the client can easily set up one subflow over network1 and another over network2, so each middlebox only ever sees part of the end-to-end MPTCP connection 😩

  client ----R-- network1  --- MBox1 -----R------------- server
             |                            |
             +-- network2  --- MBox2 -----+

                   Figure 9: Interactions between
               Load-Balancing and Security Middleboxes

In both of these cases, an attacker could dodge security measures that work on the TCP byte stream and run on the middleboxes. They do this by controlling which bytes actually go over each subflow, and there are tools that make this kind of evasion easier [PZ15] [PT14]. Now, this isn't an MPTCP problem, because MPTCP is behaving correctly 🦋 But it does show how hard it is to enforce security policy by relying only on on-path middleboxes, instead of enforcing it directly on the endpoints 💀

5. References 📚

5.1. Normative References

[RFC6182] Ford, A., Raiciu, C., Handley, M., Barre, S., and J. Iyengar, "Architectural Guidelines for Multipath TCP Development", RFC 6182, DOI 10.17487/RFC6182, March 2011, <http://www.rfc-editor.org/info/rfc6182>.

[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, <http://www.rfc-editor.org/info/rfc6824>.

5.2. Informative References

[BALIA] Peng, Q., Walid, A., Hwang, J., and S. Low, "Multipath TCP: analysis, design, and implementation", IEEE/ACM Trans. on Networking (TON), Volume 24, Issue 1, February 2016.

[CACM14] Paasch, C. and O. Bonaventure, "Multipath TCP", Communications of the ACM, 57(4):51-57, April 2014, <http://inl.info.ucl.ac.be/publications/multipath-tcp>.

[Cellnet12] Paasch, C., Detal, G., Duchene, F., Raiciu, C., and O. Bonaventure, "Exploring Mobile/WiFi Handover with Multipath TCP", ACM SIGCOMM workshop on Cellular Networks (Cellnet12), August 2012, <http://inl.info.ucl.ac.be/publications/exploring-mobilewifi-handover-multipath-tcp>.

[COMCOM2016] Tran, V., De Coninck, Q., Hesmans, B., Sadre, R., and O. Bonaventure, "Observing real Multipath TCP traffic", Computer Communications, DOI 10.1016/j.comcom.2016.01.014, April 2016, <http://inl.info.ucl.ac.be/publications/observing-real-multipath-tcp-traffic>.

[COMMAG2016] De Coninck, Q., Baerts, M., Hesmans, B., and O. Bonaventure, "Observing Real Smartphone Applications over Multipath TCP", IEEE Communications Magazine Network Testing Series, 54(3), March 2016, <http://inl.info.ucl.ac.be/publications/observing-real-smartphone-applications-over-multipath-tcp>.

[CONEXT12] Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. Leboudec, "MPTCP is not Pareto-Optimal: Performance Issues and a Possible Solution", CoNEXT '12: Proceedings of the 8th international conference on Emerging networking experiments and technologies, DOI 10.1145/2413176.2413178, December 2012.

[CONEXT13] Paasch, C., Khalili, R., and O. Bonaventure, "On the Benefits of Applying Experimental Design to Improve Multipath TCP", Conference on emerging Networking EXperiments and Technologies (CoNEXT), DOI 10.1145/2535372.2535403, December 2013, <http://inl.info.ucl.ac.be/publications/benefits-applying-experimental-design-improve-multipath-tcp>.

[CONEXT15] Hesmans, B., Detal, G., Barre, S., Bauduin, R., and O. Bonaventure, "SMAPP: Towards Smart Multipath TCP-enabled APPlications", Proc. Conext 2015, Heidelberg, Germany, December 2015, <http://inl.info.ucl.ac.be/publications/smapp-towards-smart-multipath-tcp-enabled-applications>.

[CSWS14] Paasch, C., Ferlin, S., Alay, O., and O. Bonaventure, "Experimental evaluation of multipath TCP schedulers", CSWS '14: Proceedings of the 2014 ACM SIGCOMM workshop on Capacity sharing workshop, DOI 10.1145/2630088.2631977, August 2014.

[DetalMSS] Detal, G., "dynamically adapt mss value", Post on the mptcp-dev mailing list, September 2014, <https://listes-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/2014-09/msg00130.html>.

[FreeBSD-MPTCP] Williams, N., "Multipath TCP For FreeBSD Kernel Patch v0.5", <http://caia.swin.edu.au/urp/newtcp/mptcp>.

[GRE-NOTIFY] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. Zhang, "GRE Notifications for Hybrid Access", Work in Progress, draft-lhwxz-gre-notifications-hybrid-access-01, January 2015.

[HAMPEL] Hampel, G., Rana, A., and T. Klein, "Seamless TCP mobility using lightweight MPTCP proxy", MobiWac '13: Proceedings of the 11th ACM international symposium on Mobility management and wireless access, DOI 10.1145/2508222.2508226, November 2013.

[HotMiddlebox13] Hesmans, B., Duchene, F., Paasch, C., Detal, G., and O. Bonaventure, "Are TCP Extensions Middlebox-proof?", CoNEXT workshop Hot Middlebox, December 2013, <http://inl.info.ucl.ac.be/publications/are-tcp-extensions-middlebox-proof>.

[HotMiddlebox13b] Detal, G., Paasch, C., and O. Bonaventure, "Multipath in the Middle(Box)", HotMiddlebox '13, December 2013, <http://inl.info.ucl.ac.be/publications/multipath-middlebox>.

[HotNets] Raiciu, C., Pluntke, C., Barre, S., Greenhalgh, A., Wischik, D., and M. Handley, "Data center networking with multipath TCP", Hotnetx-IX: Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks Article No. 10, DOI 10.1145/1868447.1868457, October 2010, <http://doi.acm.org/10.1145/1868447.1868457>.

[HYA-ARCH] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. Zhang, "Hybrid Access Network Architecture", Work in Progress, draft-lhwxz-hybrid-access-network-architecture-02, January 2015.

[ICNP12] Cao, Y., Xu, M., and X. Fu, "Delay-based congestion control for multipath TCP", 20th IEEE International Conference on Network Protocols (INCP), DOI 10.1109/ICNP.2012.6459978, October 2012.

[IETF88] Stewart, L., "IETF 88 Meeting minutes of the MPTCP working group", November 2013, <https://www.ietf.org/proceedings/88/minutes/minutes-88-mptcp>.

[IETFJ] Bonaventure, O. and S. Seo, "Multipath TCP Deployments", IETF Journal, Vol. 12, Issue 2, November 2016.

[IMC11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A., Handley, M., and H. Tokuda, "Is it still possible to extend TCP?", IMC '11: Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference, DOI 10.1145/2068816.2068834, November 2011, <http://doi.acm.org/10.1145/2068816.2068834>.

[IMC13a] Detal, G., Hesmans, B., Bonaventure, O., Vanaubel, Y., and B. Donnet, "Revealing Middlebox Interference with Tracebox", Proceedings of the 2013 ACM SIGCOMM conference on Internet measurement conference, DOI 10.1145/2504730.2504757, October 2013, <http://inl.info.ucl.ac.be/publications/revealing-middlebox-interference-tracebox>.

[IMC13b] Chen, Y., Lim, Y., Gibbens, R., Nahum, E., Khalili, R., and D. Towsley, "A measurement-based study of MultiPath TCP performance over wireless network", ICM '13: Proceedings of the 2013 conference on Internet measurement conference, DOI 10.1145/2504730.2504751, October 2013, <http://doi.acm.org/10.1145/2504730.2504751>.

[IMC13c] Pelsser, C., Cittadini, L., Vissicchio, S., and R. Bush, "From Paris to Tokyo: on the suitability of ping to measure latency", IMC '13: Proceedings of the 2013 conference on Internet measurement Conference, DOI 10.1145/2504730.2504765, October 2013, <http://doi.acm.org/10.1145/2504730.2504765>.

[INFOCOM14] Lim, Y., Chen, Y., Nahum, E., Towsley, D., and K. Lee, "Cross-layer path management in multi-path transport protocol for mobile devices", IEEE INFOCOM'14, DOI 10.1109/INFOCOM.2014.6848120, April 2014.

[KT] Seo, S., "KT's GiGA LTE", July 2015, <https://www.ietf.org/proceedings/93/slides/slides-93-mptcp-3.pdf>.

[MBTest] Hesmans, B., "MBTest", October 2013, <https://bitbucket.org/bhesmans/mbtest>.

Acknowledgements ✨

This work was partially supported by the FP7-Trilogy2 project. 💖 We would like to thank all the implementers and users of the Multipath TCP implementation in the Linux kernel. 🫶 This document has benefited from the comments of John Ronan, Yoshifumi Nishida, Phil Eardley, Jaehyun Hwang, Mirja Kuehlewind, Benoit Claise, Jari Arkko, Qin Wu, Spencer Dawkins, and Ben Campbell. 👑

Authors' Addresses 🦋

  • Olivier Bonaventure, UCLouvain, Email: Olivier.Bonaventure@uclouvain.be
  • Christoph Paasch, Apple, Inc., Email: cpaasch@apple.com
  • Gregory Detal, Tessares, Email: Gregory.Detal@tessares.net
📦 where this page comes from