RFC 7098
Using the IPv6 Flow Label for Load Balancing in Server Farms
1. Introduction ✨
Hiii bestie 🎀 Let's talk about this RFC! It's RFC 7098, titled Using the IPv6 Flow Label for Load Balancing in Server Farms, and it's an Informational doc, so it's not a standards-track spec 💅 It was published in January 2014 under ISSN 2070-1721. The authors are B. Carpenter (Univ. of Auckland), S. Jiang (Huawei Technologies Co., Ltd), and W. Tarreau (HAProxy Technologies, Inc.) 🌸 The header doesn't list anything it obsoletes or updates, so there's no drama there 🙈
Anyway, the whole point is that the IPv6 flow label can help with layer 3/4 load distribution and balancing for big server farms 🦋
Abstract
This doc explains how the IPv6 flow label, the way it's currently specified, can be used to level up layer 3/4 (L3/4) load distribution and balancing for large server farms 💖
Status of This Memo
Okay, the status stuff, but make it cute 🧚 This document is not an Internet Standards Track specification. It's just published for informational purposes, so no pressure 🌈
It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, though. See Section 2 of RFC 5741 for that tea ☕
Want the current status, any errata, or a way to give feedback? Check http://www.rfc-editor.org/info/rfc7098 ✨
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read these, because they spell out your rights and restrictions for this doc 🫶 Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 🔥
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Summary of Flow Label Specification . . . . . . . . . . . . . 2
3. Summary of Server Farm Load-Balancing Techniques . . . . . . 4
4. Applying the Flow Label to Layer 3/4 Load Balancing . . . . . 8
5. Security Considerations . . . . . . . . . . . . . . . . . . . 10
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 11
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 12
7.1. Normative References . . . . . . . . . . . . . . . . . . 12
7.2. Informative References . . . . . . . . . . . . . . . . . 121. Introduction ✨
Okay so the IPv6 flow label got redefined in [RFC6437], and it's now a recommended IPv6 node requirement per [RFC6434] 💅 Its use for load sharing in multipath routing is specified in [RFC6438]. Another scenario where the flow label could be useful is load distribution for large server farms 🦋
Quick vocab check: load distribution is a slightly more general term than load balancing, but "load balancing" is what people say more often 🌸 In a server farm, both terms mean mechanisms that spread the workload across different servers to optimize performance. Server load balancing usually deals with HTTP traffic, but most of the techniques would work for other upper-layer apps too 💖
This doc starts with quick intros to the flow label and to server load-balancing techniques. Then it shows how the flow label can enhance load balancers that operate on IP packets and TCP sessions, which are commonly known as layer 3/4 load balancers 🎀
So why bother? The motivation is to improve the performance of most types of layer 3/4 load balancers. It's especially helpful for traffic with multiple IPv6 extension headers, and especially for fragmented packets 😭 Fragmented packets are often the result of customers reaching the load balancer through a VPN with a limited MTU, and they're a common performance problem 💀
2. Summary of Flow Label Specification 🌈
The IPv6 flow label [RFC6437] is a 20-bit field included in every IPv6 header [RFC2460]. [RFC6434] says it's recommended to be supported in all IPv6 nodes. There's extra background in [RFC6436] and [RFC6294] if you're a nerd like me 🧚
Per its definition, the flow label should be set to a constant value for a given traffic flow (like one HTTP connection). That value is supposed to belong to a uniform statistical distribution, which makes it potentially valuable for load-balancing purposes ✨
Any device that can see the IPv6 header can see the flow label, and it sits at a fixed position in every IPv6 packet 🦋 Transport-layer info, like port numbers, is different. It isn't always in a fixed position, because it comes after any IPv6 extension headers that might be there. Honestly, finding the transport header is always more complicated for IPv6 than for IPv4, since IPv6 has no Internet Header Length field 😩
It gets worse with fragments. If packets are fragmented, the flow label is in all the fragments, but the transport header only shows up in one packet 💅 So for the lifetime of a transport-layer connection, the flow label can be a more convenient "handle" than the port number for identifying that particular connection. Main character energy 👑
Now the rules. According to RFC 6437, source hosts should set the flow label. If they don't (meaning its value is zero), forwarding nodes, like the first-hop router, may set it instead 🌸 Either way, the flow label value must stay constant for a given transport session, which is normally identified by the IPv6 and Transport header 5-tuple. By default, the flow label value should be calculated by a stateless algorithm. The result should form part of a statistically uniform distribution, no matter which node set it 💖
Real talk: at the time of writing, very few traffic flows include a non-zero flow label value 💀 The mechanism described below is designed to be added to existing load-balancing mechanisms, so it becomes effective as more and more flows carry a non-zero label 🦋 Even if the flow label comes from an imperfectly uniform distribution, it still increases the information entropy of the IPv6 header as a whole. That's what allows load balancing based on the flow label to be rolled out gradually 🎀
If the recommendations in Section 3 of RFC 6437 are followed for traffic from a given source going to a well-known TCP port at a given destination, the flow label can act as a substitute for the port numbers as far as a load balancer is concerned 🫶 It can also be found at a fixed position in the layer 3 header, even when extension headers are present. Iconic 🔥
The flow label is defined as an end-to-end component of the IPv6 header, but there are three qualifications to that:
- Until the IPv6 flow label specification in RFC 6437 is widely implemented, as recommended by RFC 6434, the flow label will often be set to the default value of zero 🙈
- Because of the recommendation to use a stateless algorithm to calculate the label, there is a low (but non-zero) probability that two simultaneous flows from the same source to the same destination have the same flow label value, even though they have different transport-protocol port numbers 👀
- The Flow Label field sits in an unprotected part of the IPv6 header. That means intentional or unintentional changes to its value cannot be easily detected by a receiver 😱
The first two points are addressed below in Section 4, and the third is addressed in Section 5 🌈
3. Summary of Server Farm Load-Balancing Techniques ✨
okay so load balancing for server farms is lowkey a mix-and-match situation, with a bunch of methods often stacked together 🎀 [Tarreau]. this section is a quick tour of the common ones 💅. heads up that the flow label isn't relevant to all of them 🙈. also, the actual load-balancing algorithm (the choice of which server to use for a new client session) is irrelevant to this discussion. we give examples for HTTP, but analogous techniques may be used for other application protocols 🦋.
- The simplest method is using the DNS to return different server addresses for a single name such as
www.example.comto different users 🍓. This is typically done by rotating the order in which different addresses within the server site are listed by the relevant authoritative DNS server, on the assumption that the client will pick the first one. Routing may be configured such that the different addresses are handled by different ingress routers. Several variants of this load-balancing mechanism exist, such as expecting some clients to use all the advertised addresses when multiple connections are involved, or directing the traffic to multiple sites, also known as global load balancing. None of these mechanisms are in the scope of this document, and the proposal in this document does not affect their usability nor aim to replace them, so they will not be discussed further 💖. - Another method, for HTTP servers, is to operate a layer 7 reverse proxy in front of the server farm 👑. The reverse proxy will present a single IP address to the world, communicated to clients by a single
AAAArecord. For each new client session (an incoming TCP connection and HTTP request), it will pick a particular server and proxy the session to it. The act of proxying should be more efficient and less resource-intensive than the act of serving the required content. The proxy must retain TCP state and proxy state for the duration of the session. This TCP state could, potentially, include the incoming flow label value 🧚. - A component of some load-balancing systems is an SSL reverse proxy farm 🌈. The individual SSL proxies handle all cryptographic aspects and exchange unencrypted HTTP with the actual servers. Thus, from the load-balancing point of view, this really looks just like a server farm, except that it's specialized for HTTPS. Each proxy will retain SSL and TCP and maybe HTTP state for the duration of the session, and the TCP state could potentially include the flow label ✨.
- Finally, the "front end" of many load-balancing systems is a layer 3/4 load balancer 🔥. While it can be a dedicated device, it is also a standard function of some network switches or routers (e.g. using Equal-Cost Multipath Routing (ECMP) [RFC2991]). In this case, it is the layer 3/4 load balancer whose IP address is published as the primary
AAAArecord for the service. All client sessions will pass through this device. Depending on the specific scenario, the balancer will assign new sessions among the actual application servers, across an SSL proxy farm, or among a set of layer 7 proxies. In all cases, the layer 3/4 load balancer has to classify incoming packets very quickly and choose the target server or proxy so as to ensure persistence. 'Persistence' is defined as the guarantee that a given client session will run to completion on a single server. The layer 3/4 load balancer therefore needs to inspect each incoming packet to classify it. There are two common types of layer 3/4 load balancers: the totally stateless ones, which only act on single packets, generally involving a per-packet hashing of easy-to-find information such as the source address and/or port into a server number, and the stateful ones, that take the routing decision on the very first packets of a session and maintain the same direction for all packets belonging to the same session. Clearly, both types of layer 3/4 balancers could inspect and make use of the flow label value 🫶.
Our focus is on how the balancer identifies a particular flow. For clarity, note that two aspects of layer 3/4 load balancers are not affected by use of the flow label to identify sessions 👀:
- Balancers use various techniques to redirect traffic to a specific target server.
- All servers are configured with the same IP address, they are all on the same LAN, and the load balancer sends directly to their individual MAC addresses. In this case, return packets from the server to the client are sent back without passing through the balancer, a technique known as direct server return, but we are not concerned here with the return packets.
- All servers are configured with the same IP address, treated locally as an anycast address by layer 3 ECMP routing.
- Each server has its own IP address, and the balancer uses an IP-in-IP tunnel to reach it.
- Each server has its own IP address, and the balancer performs NAPT (Network Address and Port Translation) to deliver the client's packets to that address.
- The choice between these methods is not affected by use of the flow label 💜.
- A layer 3/4 balancer must correctly handle Path MTU Discovery by forwarding relevant ICMPv6 packets in both directions. This too is not directly affected by use of the flow label. It should be noted that there may be difficulty correlating an ICMPv6 "Packet too big" response with the session it refers to, but that is out of the scope of the present document 🙃.
The following diagram, inspired by [Tarreau], shows a layout with various methods in use together. (Below, "ASIC" stands for "Application-Specific Integrated Circuit".) 🌸
___________________________________________
( )
( Clients in the Internet )
(___________________________________________)
| |
------------ DNS-based ------------
| Ingress | load splitting | Ingress |
| router | affects | router |
------------ routing ------------
___|____________________________|___
| |
| |
| |
------------ ------------
| L3/4 ASIC| | L3/4 ASIC|
| balancer | | balancer |
------------ ------------
| load |
| spreading |
__________|________________________|___________
| | | |
------------ ------------ -------- --------
|HTTP proxy|...|HTTP proxy| | SSL |...| SSL |
| balancer | | balancer | | proxy| | proxy|
------------ ------------ -------- --------
____|_____________|_____________|_________|_____
| | | | |
-------- -------- -------- -------- --------
|HTTP | |HTTP | |HTTP | |HTTP | |HTTP |
|server| |server| |server| |server| |server|
-------- -------- -------- -------- --------From the previous paragraphs, we can identify several points in this diagram where the flow label might be relevant 🔍:
- Layer 3/4 load balancers.
- SSL proxies.
- HTTP proxies.
However, usage by the proxies seems unlikely to affect performance, because they must in any case process the application-layer header, so in this document we focus only on layer 3/4 balancers 🦋.
4. Applying the Flow Label to Layer 3/4 Load Balancing ✨
okay so here's the model for using the flow label to level up a layer 3/4 load balancer 💖 it's giving efficiency, bestie 🧚
- We only care about IPv6 traffic where the flow label has been set according to [RFC6437] 🎀 If an incoming packet's flow label is zero, the load balancer just keeps reading the transport header the old-school way. As the flow label gets more popular per RFC 6434, load balancers (and therefore users) get a growing performance boost 📈✨
- If the flow label is non-zero, a layer 3/4 load balancer can use the 2-tuple
{source address, flow label}as the session key for whatever load distribution algorithm it runs 🦋 Or it can go with the 3-tuple{dest address, source address, flow label}, especially if the server farm has multiple server IP addresses 🌸 Using the 3-tuple is way quicker than digging for the transport port numbers later in the packet, no cap 💅 Also, transport info like the source port isn't repeated in fragments, which is why stateless load balancers usually can't handle fragmented traffic. They generally can't reassemble fragments, so it's not it 💀
A stateless layer 3/4 balancer just runs a hash algorithm on the 2-tuple or 3-tuple for every packet, so the same flow always lands on the same target server 🍓 Needless to say, the hash algorithm has to be well chosen for its job, but this is a problem for lots of stateless load balancing, not just this one 🙈 The discussion in [RFC6438] applies here too.
A stateful layer 3/4 balancer runs its usual distribution algorithm on the first packet of a session, then stores the {tuple, server} association in a table so every later packet in that session goes to the same server 💜 For all the subsequent packets, it can ignore all the IPv6 extension headers, which should lead to a performance benefit ✨ Whether that benefit actually matters depends on the engineering details of the specific load balancer, so it's not automatic 🫶
Heads up: this kind of balancer won't spot new transport sessions from the same source that use the same flow label. They all get delivered to the same server 👑 That's like how existing hash-based layer 4 balancers always send similarly hashed packets to the same destination. Also, a global state table in a flow label balancer can't be shared between multiple services that rely on transport-layer info, since the whole point of the flow label is to skip looking up that info 🔥
Another thing: the balancer won't detect FIN/ACK sequences at the end of sessions, so it has to rely on inactivity timers to delete session state 😭 But every existing balancer already needs timers to deal with hung sessions, so the impact on memory use is unlikely to be significant 🌈
- Layer 3/4 balancers that redirect incoming packets by NAPT aren't expected to save any time from the flow label 🙃 They have no choice but to follow the extension header chain so they can find and modify the port number and the transport checksum. Same goes for balancers that do TCP state tracking for any reason ⭐
- Correct ICMPv6 handling for Path MTU Discovery requires the layer 3/4 balancer to keep state for the client source address, regardless of the port numbers or the flow label 🧚♀️
- SSL and HTTP proxies, if present, should forward the flow label value towards the server 🎀 This usually gives no performance benefit, but it's consistent with the general model for the flow label described in RFC 6437 💖
The performance benefit, if any, depends entirely on the engineering trade-offs in the layer 3/4 balancer's design, so don't expect magic 🪄 An extra test is needed to check whether the label is non-zero. If it is, all the extension header logic can be skipped except for the first packet of a new flow ✨ Since the state to store is only the tuple and the server identifier, storage requirements go down 🦋 The method also works for fragmented traffic, and for flows where transport info is missing (unknown transport protocol) or obfuscated (e.g., IPsec) 🌸 Traffic reaching the load balancer through a VPN is especially prone to the fragmentation issue because of MTU size problems 😩 For some load balancer designs, these are very significant advantages 👑
In the unlikely event that two simultaneous flows from the same source address have the same flow label value, both flows end up on the same server. There, they're told apart as usual by their port numbers 💅 There are roughly one million possible flow label values, and if the rules for flow label generation in [RFC6437] are followed, this is a statistically rare event that won't wreck the overall load balancing 🍓 Plus, with a million possible label values, most sites have way more label values than servers, so it's already expected that multiple flow label values will land on the same server for a given client IP address 🫶
Now, if many thousands of clients are hidden behind the same large-scale NAPT with one shared IP address, the assumption of low collision probability might stop being true, unless the flow label values are random enough to avoid following similar sequences for all those clients 😬 This isn't expected to be a factor for IPv6 anyway, since there's no need to run large-scale NAPT with address sharing [RFC4864] 🙈 The collision probability is low for sites that use network prefix translation [RFC6296], since that technique gives each client a different address 💜
5. Security Considerations 🛡️
Security aspects of the flow label are discussed in [RFC6437] 📖 As noted there, a malicious source or man-in-the-middle could mess with load balancing by manipulating flow labels 😤 This risk already exists today wherever the source address and port are used as a hashing key in layer 3/4 load balancers, and wherever a persistence cookie is used in HTTP to pick a server 🍪 It even exists on layer 3 components that only rely on the source address to choose a destination, which makes them more DDoS-prone 💀 Still, all these methods are in use right now because the benefits for load balancing and persistence hugely outweigh the risks ✨ The flow label doesn't significantly change this situation 💅
Specifically, the IPv6 flow label specification [RFC6437] states that "stateless classifiers should not use the flow label alone to control load distribution, and stateful classifiers should include explicit methods to detect and ignore suspect flow label values." 🎀 The first point is handled by also using the source address 🫶 The second point is more complicated 🌀 If the risk is considered serious, the site ingress router or the layer 3/4 balancer should use a suitable heuristic to verify incoming flows with non-zero flow label values 🔍 If a flow from a given source address and port number doesn't have a constant flow label value, it's suspect and should be dropped 🚫 This deals with both intentional and accidental changes to the flow label 👑
⚠️ A malicious source or man-in-the-middle could generate a flow where the flow label is constant but the transport port numbers in some packets are invalid. If those packets get load balanced only on the basis of the flow label, they could reach the target server and create a single-source DoS attack on its TCP engine 😱
RFC 6437 notes in its Security Considerations that if the covert channel risk is considered significant, a firewall might rewrite non-zero flow labels 🧱 As long as this is done as described in RFC 6437, it won't invalidate the mechanisms described above 🌸
The flow label may be useful for protecting servers against DDoS attacks 🛡️ As noted in RFC 6437, a source should generate flow label values that are hard to predict, most likely by including a secret nonce in the hash used to generate each label 🤫 The attacker doesn't know the nonce, so they have no way to invent flow labels that all target the same server, even if they know both the hash algorithm and the load balancing algorithm 🧠 Still, it's important to understand that it's always trivial to force a load balancer to stick to the same server during an attack 😮💨 So the security of the whole solution must not rely on the unpredictability of the flow label values alone 🙅♀️ It should also include defensive measures, like the ones most load balancers already have against abnormal use of source addresses or session cookies 🫶
New flows are assigned to a server according to any of the usual algorithms available on the load balancer (e.g., least connections, round robin, etc.) 🎲 The association between the 2-tuple {source address, flow label} and the server is stored in a table (often called stick table) so that future traffic from the same source using the same flow label can go to the same server 📋 This method is more robust against a loss of server, and it also makes it harder for an attacker to target a specific server, because the association between a flow label value and a server isn't known externally 🦋
If a stateless hash function is used to assign client packets to specific servers, it may be advisable to use a cryptographic hash function of some kind 🔐 That way, an attacker can't predict how the load balancer will behave 🌈
6. Acknowledgements 🌸
Valuable comments and contributions were made by Fred Baker, Olivier Bonaventure, Ben Campbell, Lorenzo Colitti, Linda Dunbar, Donald Eastlake, Joel Jaeggli, Gurudeep Kamat, Warren Kumari, Julia Renouard, Julius Volz, and others 💖 Thank you, legends 🫶
7. References 📚
7.1. Normative References
[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998.
[RFC6434] Jankiewicz, E., Loughney, J., and T. Narten, "IPv6 Node Requirements", RFC 6434, December 2011.
[RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, "IPv6 Flow Label Specification", RFC 6437, November 2011.
7.2. Informative References 🦋
okay so these are the "for your reading list" ones, bestie 📚 they're not required to implement the flow label, but they're the receipts behind the whole vibe 💅
[RFC2991] Thaler, D. and C. Hopps, "Multipath Issues in Unicast and
Multicast Next-Hop Selection", RFC 2991, November 2000.
[RFC4864] Van de Velde, G., Hain, T., Droms, R., Carpenter, B., and
E. Klein, "Local Network Protection for IPv6", RFC 4864,
May 2007.
[RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for
the IPv6 Flow Label", RFC 6294, June 2011.
[RFC6296] Wasserman, M. and F. Baker, "IPv6-to-IPv6 Network Prefix
Translation", RFC 6296, June 2011.
[RFC6436] Amante, S., Carpenter, B., and S. Jiang, "Rationale for
Update to the IPv6 Flow Label Specification", RFC 6436,
November 2011.
[RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label
for Equal Cost Multipath Routing and Link Aggregation in
Tunnels", RFC 6438, November 2011.
[Tarreau] Tarreau, W., "Making applications scalable with load
balancing", 2006, <http://1wt.eu/articles/2006_lb/>.Authors' Addresses 🎀
meet the squad who wrote this whole thing 🫶 here's where you can find them, babe 💌
Brian Carpenter
Department of Computer Science
University of Auckland
PB 92019
Auckland 1142
New Zealand
EMail: brian.e.carpenter@gmail.com
Sheng Jiang
Huawei Technologies Co., Ltd
Q14, Huawei Campus
No.156 Beiqing Road
Hai-Dian District, Beijing 100095
P.R. China
EMail: jiangsheng@huawei.com
Willy Tarreau
HAProxy Technologies, Inc.
R&D Network Products
3 rue du petit Robinson
78350 Jouy-en-Josas
France
EMail: willy@haproxy.com📦 where this page comes from
- ietf: rfc7098 2014-01
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms