RFC 6438

Using the IPv6 Flow Label for Equal Cost Multipath Routing and Link Aggregation in Tunnels

📅 published November 2011, from ietf
  • Organization: Internet Engineering Task Force (IETF) 🎀
  • Request for Comments: 6438
  • Authors: B. Carpenter (Univ. of Auckland) and S. Amante (Level 3) 💖
  • Category: Standards Track
  • ISSN: 2070-1721
  • Date: November 2011
  • Obsoletes / Updates: this header doesn't list anything it obsoletes or updates 🦋

Abstract

okay bestie, the IPv6 flow label comes with some rules about how you're allowed to use it 🧚 this doc explains how those rules play out when you use the flow label to load balance with equal cost multipath routing and link aggregation 🔥 the big focus is IP-in-IPv6 tunneled traffic 💅

Status of This Memo

this is an Internet Standards Track document 🌸

it's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community ✨ it got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 👑 if you want more on Internet Standards, check out Section 2 of RFC 5741 📚

for the current status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6438 💜

Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. 🎀

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date of publication 🦋 please actually read these, because they spell out your rights and restrictions for this document 🙈 code components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 💅

Table of Contents

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . . 2
  1.1.  Choice of IP Header Fields for Hash Input . . . . . . . . . 3
  1.2.  Flow Label Rules  . . . . . . . . . . . . . . . . . . . . . 4
2.  Normative Notation  . . . . . . . . . . . . . . . . . . . . . . 5
3.  Guidelines  . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.  Security Considerations . . . . . . . . . . . . . . . . . . . . 7
5.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . 7
6.  References  . . . . . . . . . . . . . . . . . . . . . . . . . . 8
  6.1.  Normative References  . . . . . . . . . . . . . . . . . . . 8
  6.2.  Informative References  . . . . . . . . . . . . . . . . . . 8

1. Introduction ✨

okay so picture this: several network paths between the same two nodes, and the routing system says they're all equally good (capacity and latency both) 🌈 it's giving "share the traffic" energy 💖 the two techniques for that are equal cost multipath (ECMP) routing and link aggregation (LAG) [IEEE802.1AX] 🔥 there are tons of ways to pull this off, but certain goals need to be met:

  • Maintain roughly equal share of traffic on each path. (In some cases, the multiple paths might not all have the same capacity, and the goal might be appropriately weighted traffic shares rather than equal shares. This would affect the load-sharing algorithm but would not otherwise change the argument.) 🍓
  • Minimize or avoid out-of-order delivery for individual traffic flows. 🦋
  • Minimize idle time on any path when queue is non-empty. ⭐

the thing is, these goals kinda fight each other 💀 for example, if you're super strict about avoiding idle time, a tiny packet sent down an idle path could overtake a bigger packet from the same flow, and boom, out-of-order delivery 🙈

here's one lightweight way to do ECMP or LAG: if there are N equally good paths to pick from, form a modulo(N) hash [RFC2991] over a defined set of fields in each packet header that are certain to keep the same values for the whole life of a flow 🧚 then use the hash output to pick a path 💅 if the hash function gives output values with a uniform statistical distribution, this method shares traffic roughly equally across the N paths 👑 and if the header fields in the hash input are consistent, every packet from a given flow generates the same hash output, so out-of-order delivery won't happen 🫶 with a large number of unique flows, the method will probably avoid idle time too, since the queue on each link stays non-empty 🌸

1.1. Choice of IP Header Fields for Hash Input 🎀

in the rest of this doc, "flow" means a sequence of packets that can be identified either by the source and destination IP addresses alone {2-tuple}, or by the source IP address, destination IP address, protocol number, source port number, and destination port number {5-tuple} 💜 the latter is what [RFC2474] calls a "microflow," but this term isn't used with the flow label in [RFC3697] 🦋

so which header fields identify a flow and act as input keys to a modulo(N) hash algorithm? 🤔 when routing general traffic, a super common pick is just a hash of the source and destination IP addresses, aka the 2-tuple 🍓 that's necessary and sufficient to avoid out-of-order delivery, and in the core of the network, with a wide variety of sources and destinations, it's often statistically enough to spread the load evenly ✨ in practice, lots of implementations use the 5-tuple {dest addr, source addr, protocol, dest port, source port} as hash input keys, to boost the chance of evenly sharing traffic across the equal cost paths 💖 but putting transport-layer info into a hash can be a problem for IP fragments [RFC2991] or for encrypted traffic 🙈 including the protocol and port numbers, which total 40 bits, makes the hash a little more expensive to compute, but it does improve the distribution, thanks to the variable nature of ephemeral ports 🌈 ephemeral port numbers are pretty well distributed [Lee10] and typically contribute 16 variable bits 🔥

BUT for IPv6, transport-layer info is a pain to extract 😭 the placement and length of next-headers varies, and every implementation must be able to skip over next-headers, even though they're rarely present in real traffic 💅 [RFC2460] basically implies that next-headers, except hop-by-hop options, aren't normally inspected by intermediate nodes in the network 🧚 this could be a challenge for some hardware implementations, and it raises the possibility that network equipment vendors might cut down the length of the fields they extract from an IPv6 header 💀

the Generic Routing Encapsulation (GRE) header [RFC2784] has a similar problem: the possible presence of a GRE header, and of a GRE key inside it, makes things as tricky as the possible presence of IPv6 extension headers 🌸 anything that complicates header analysis is a no from me 🙅‍♀️

IP-in-IP tunnels are a whole different story 🦋 figuring out the flow inside the tunnel is more complicated, mostly because nearly all hardware can only identify flows from the outermost IP header 😭 say traffic from lots of sources to lots of destinations gets aggregated into one IP-in-IP tunnel from tunnel endpoint (TEP) A to TEP B (see figure) 🎀 then every packet in the tunnel has outer source address A and outer destination address B 💜 and they probably also share the same port and protocol numbers 🍓 if there are multiple paths between routers R1 and R2, and ECMP or LAG picks a path, then the 2-tuple or 5-tuple (and its hash) stays constant, so no load sharing happens, which is called polarization 💀 if one or a few tunnels make up a big chunk of the traffic, the traffic won't spread across the R1-to-R2 paths the way you wanted, because of partial polarization 🙈 (related issues come up with MPLS [MPLS-LABEL] 🔥)

     _____           _____               _____           _____
    | TEP |_________| R1  |-------------| R2  |_________| TEP |
    |__A__|         |_____|-------------|_____|         |__B__|
            tunnel          ECMP or LAG         tunnel
                                here

as we said, for IPv6 the 5-tuple is a pain to extract because of where next-headers sit 😤 so the real question is whether the 20-bit flow label in IPv6 packets could work as input to an ECMP or LAG hash algorithm 🧚 especially for tunnels where the inner packet header is unreachable 🌈 if the flow label could stand in for the port numbers and protocol number in the 5-tuple, the implementation would get way simpler 👑 🫶

that's the setup, bestie. Section 1.2 covers the flow label rules, which is the next part of this doc 💜

1.2. Flow Label Rules ✨

okay so the flow label was left Experimental by [RFC2460] 🧪 but [RFC3697] came along and defined it way better 💅. there are three rules from that RFC we need to quote, bestie:

  1. "The Flow Label value set by the source MUST be delivered unchanged to the destination node(s)."
  2. "IPv6 nodes MUST NOT assume any mathematical or other properties of the Flow Label values assigned by source nodes."
  3. "Router performance SHOULD NOT be dependent on the distribution of the Flow Label values. Especially, the Flow Label bits alone make poor material for a hash key."

these rules, especially the last one, have made designers hesitant to use the flow label for ECMP or LAG 😭. the tea is that most nodes today set a zero value in the flow label, and the first rule absolutely forbids the routing system from changing the flow label once a packet has left the source node 🙅‍♀️. so for normal IPv6 traffic, a flow label that's typically zero adds no value to an ECMP or LAG hash, but it also doesn't hurt the distribution of the hash values. neutral vibes 🌸.

BUT in an IP-in-IPv6 tunnel, the TEP is itself the source node of the outer packets 🫶. that means a TEP may freely set a flow label in the outer IPv6 header of the packets it sends into the tunnel. main character energy for the TEP 👑.

the second two rules need to be read in the context of [RFC3697], which assumes that routers using the flow label in some way will be involved in some sort of method of establishing flow state: "To enable flow-specific treatment, flow state needs to be established on all or a subset of the IPv6 nodes on the path from the source to the destination(s)." the RFC should perhaps have been clearer that a router that has taken part in flow state establishment can rely on properties of the resulting flow label values without further signaling 🧠. if a router knows these properties, rule 2 is irrelevant, and it can choose to deviate from rule 3 ✨.

in the tunneling situation we sketched above, routers R1 and R2 can rely on the flow labels set by TEP A and TEP B being assigned by a known method 🦋. this lets an ECMP or LAG method be based on the flow label consistently with [RFC3697], regardless of whether the non-tunnel traffic carries non-zero flow label values. it's giving organized 🎀.

the IETF has recently revised RFC 3697 [RFC6437] 🌈. that revision is fully compatible with the present document and obviates the concerns resulting from the above three rules, so the present specification applies both to RFC 3697 and to RFC 6437. no notes, just vibes 💜.

2. Normative Notation 📝

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. this is the spec's own dictionary, so read these exactly as written 🤓.

3. Guidelines 🧚

okay this is the juicy part 🍓. we assume that the routers supporting ECMP or LAG (R1 and R2 in the above figure) are unaware that they are handling tunneled traffic 🙈. if you want the IPv6 flow label in an ECMP or LAG hash in the tunneled scenario shown above, these guidelines apply:

  • Inner packets MUST be encapsulated in an outer IPv6 packet whose source and destination addresses are those of the tunnel endpoints (TEPs). no skipping this step, bestie 🚫.
  • The flow label in the outer packet SHOULD be set by the sending TEP to a 20-bit value in accordance with [RFC6437]. The same flow label value MUST be used for all packets in a single user flow, as determined by the IP header fields of the inner packet. consistency is everything 💖.
  • To achieve this, the sending TEP MUST classify all packets into flows once it has determined that they should enter a given tunnel, and then write the relevant flow label into the outer IPv6 header ✨. a user flow could be identified by the sending TEP most simply by its {destination, source} address 2-tuple, or by its 5-tuple {dest addr, source addr, protocol, dest port, source port} 🎀. at present there would be little point in using the {dest addr, source addr, flow label} 3-tuple of the inner packet, but doing so would be a future-proof option 🔮. the choice of n-tuple is an implementation choice in the sending TEP 💅.
    • As specified in [RFC6437], the flow label values should be chosen from a uniform distribution. Such values will be suitable as input to a load-balancing hash function and will be hard for a malicious third party to predict 🔐.
    • The sending TEP MAY perform stateless flow label assignment by using a suitable 20-bit hash of the inner IP header's 2-tuple or 5-tuple as the flow label value 🧮.
    • If the inner packet is an IPv6 packet, its flow label value could also be included in this hash 🌟.
    • This stateless method creates a small probability of two different user flows hashing to the same flow label. Since [RFC6437] allows a source (the TEP in this case) to define any set of packets that it wishes as a single flow, occasionally labeling two user flows as a single flow through the tunnel is acceptable 🫧.
  • At intermediate routers that perform load distribution, the hash algorithm used to determine the outgoing component-link in an ECMP and/or LAG toward the next hop MUST minimally include the 3-tuple {dest addr, source addr, flow label} and MAY also include the remaining components of the 5-tuple 🦋. this applies whether the traffic is tunneled traffic only or a mixture of normal traffic and tunneled traffic 🌈.
    • Intermediate IPv6 router(s) will presumably encounter a mixture of tunneled traffic and normal IPv6 traffic. Because of this, the design should also include {protocol, dest port, source port} as input keys to the ECMP and/or LAG hash algorithms, to provide additional entropy for flows whose flow label is set to zero, including non-tunneled traffic flows 🧃.
  • Individual nodes in a network are free to implement different algorithms that conform to this specification without impacting the interoperability or function of the network 🤝.
  • Operations, Administration, and Maintenance (OAM) techniques will need to be adapted to manage ECMP and LAG based on the flow label. The issues will be similar to those that arise for MPLS [RFC4379] and pseudowires [RFC6391] 🛠️.

4. Security Considerations 🚨

the flow label is not protected in any way and can be forged by an on-path attacker 😱. this is serious, bestie.

however, it is expected that tunnel endpoints and the ECMP or LAG paths will be part of a managed infrastructure that is well protected against on-path attacks (e.g., by using IPsec between the two tunnel endpoints) 🔒. off-path attackers are unlikely to guess a valid flow label if an apparently pseudo-random and unpredictable value is used 🎲. in either case, the worst an attacker could do against ECMP or LAG is attempt to selectively overload a particular path. for further discussion, see [RFC6437] 💀.

5. Acknowledgements 💜

this document was suggested by corridor discussions at IETF 76 🫶. Joel Halpern made crucial comments on an early version 👑. we are grateful to Qinwen Hu for general discussion about the flow label 🌸. valuable comments and contributions were made by Miguel Garcia, Brian Haberman, Sheng Jiang, Thomas Narten, Jarno Rajahalme, Brian Weis, and others 🧚.

6. References 📚

6.1. Normative References

[RFC2119]      Bradner, S., "Key words for use in RFCs to Indicate
               Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2460]      Deering, S. and R. Hinden, "Internet Protocol, Version
               6 (IPv6) Specification", RFC 2460, December 1998.

[RFC3697]      Rajahalme, J., Conta, A., Carpenter, B., and S.
               Deering, "IPv6 Flow Label Specification", RFC 3697,
               March 2004.

[RFC6437]      Amante, S., Carpenter, B., Jiang, S., and J.
               Rajahalme, "IPv6 Flow Label Specification", RFC 6437,
               November 2011.

6.2. Informative References

[IEEE802.1AX]  Institute of Electrical and Electronics Engineers,
               "Link Aggregation", IEEE Standard 802.1AX-2008, 2008.

[Lee10]        Lee, D., Carpenter, B., and N. Brownlee, "Observations
               of UDP to TCP Ratio and Port Numbers", Fifth
               International Conference on Internet Monitoring and
               Protection ICIMP 2010, May 2010.

[MPLS-LABEL]   Kompella, K., Drake, J., Amante, S., Henderickx, W.,
               and L. Yong, "The Use of Entropy Labels in MPLS
               Forwarding", Work in Progress, May 2011.

[RFC2474]      Nichols, K., Blake, S., Baker, F., and D. Black,
               "Definition of the Differentiated Services Field (DS
               Field) in the IPv4 and IPv6 Headers", RFC 2474,
               December 1998.

[RFC2784]      Farinacci, D., Li, T., Hanks, S., Meyer, D., and P.
               Traina, "Generic Routing Encapsulation (GRE)",
               RFC 2784, March 2000.

[RFC2991]      Thaler, D. and C. Hopps, "Multipath Issues in Unicast
               and Multicast Next-Hop Selection", RFC 2991,
               November 2000.

[RFC4379]      Kompella, K. and G. Swallow, "Detecting Multi-Protocol
               Label Switched (MPLS) Data Plane Failures", RFC 4379,
               February 2006.

[RFC6391]      Bryant, S., Filsfils, C., Drafz, U., Kompella, V.,
               Regan, J., and S. Amante, "Flow-Aware Transport of
               Pseudowires over an MPLS Packet Switched Network",
               RFC 6391, November 2011.

Authors' Addresses 🎀

Brian Carpenter
Department of Computer Science
University of Auckland
PB 92019
Auckland  1142
New Zealand

EMail: brian.e.carpenter@gmail.com


Shane Amante
Level 3 Communications, LLC
1025 Eldorado Blvd
Broomfield, CO  80021
USA

EMail: shane@level3.net
📦 where this page comes from