RFC 4928
Avoiding Equal Cost Multipath Treatment in MPLS Networks
Avoiding Equal Cost Multipath Treatment in MPLS Networks ✨
Part 1 of 2 🎀 more to come!
Title block 📋 This RFC is RFC 4928, and it's a Best Current Practice (it's officially BCP 128) 💅. It was written by G. Swallow and S. Bryant from Cisco Systems, Inc., plus L. Andersson from Acreo AB. It came out in June 2007 🌸. There are no "Obsoletes" or "Updates" lines in the header, so it doesn't replace or change any other RFC. It's a standalone vibe 🦋.
Status of This Memo 🎀
Okay, so this document is an Internet Best Current Practice for the Internet Community, and the authors are literally asking for discussion and suggestions for improvements 🫶. Anyone can share this memo and it has unlimited distribution 🌈.
Copyright Notice 🔒
Copyright (C) The IETF Trust (2007).
Abstract ✨
Okay bestie, the big topic here is Equal Cost Multipath (ECMP) behavior in MPLS networks that are already out in the wild 🌍. The doc gives best practice recommendations for anyone building an application that runs over MPLS and doesn't want packets from the same flow getting reordered because they took different equal cost paths 😭. The sneaky part is that these recommendations rely on peeking at the IP version number field in packets 👀. The authors admit it's heuristic, but it's still a relatively safe way to run MPLS networks, even if someone later allocates new IP version numbers for some purpose 💖.
Table of Contents 📚
1. Introduction ....................................................2
1.1. Terminology ................................................2
2. Current ECMP Practices ..........................................2
3. Recommendations for Avoiding ECMP Treatment .....................4
4. Security Considerations .........................................5
5. IANA Considerations .............................................5
6. References ......................................................6
6.1. Normative References .......................................6
6.2. Informative References .....................................61. Introduction 🦋
Okay so this section is the setup 🧚. It describes ECMP behavior in MPLS networks that are deployed right now 💅. The scenario is that several packets from the same top-level LSP might get sent down different equal cost paths, which means packets from that same top-level LSP can come out of order 😬. The section also says it's a best practice guide for anyone building an application on MPLS who wants to dodge that out-of-order risk 🫶.
Here's the tea ☕. Turning off ECMP is technically an option most operators have, but basically nobody (if anyone at all) has actually done it 💀. And app designers don't control how the networks their app runs over behave. So ECMP is just a reality you have to deal with, bestie 👑.
1.1. Terminology 💜
Here's the glossary, lowkey the best part of this section 🍓:
ECMP Equal Cost Multipath
FEC Forwarding Equivalence Class
IP ECMP A forwarding behavior in which the selection of the
next-hop between equal cost routes is based on the
header(s) of an IP packet
Label ECMP A forwarding behavior in which the selection of the
next-hop between equal cost routes is based on the label
stack of an MPLS packet
LSP Label Switched Path
LSR Label Switching Router2. Current ECMP Practices 🌸
Okay, this section is where it gets juicy 🔥. The MPLS label stack and Forwarding Equivalence Classes are defined in [RFC3031] 📖. The label stack doesn't carry a Protocol Identifier, which is kinda wild 😭. Instead, the payload of an MPLS packet is identified by the FEC of the bottom-most label. So if you don't know the label binding for that bottom label, you literally can't tell what type of payload you've got 🙈. And since an LSR only needs to know the binding for the labels it actually has to process, LSRs along an LSP very often can't figure out what kind of payload they're carrying 💀.
Now, about ECMP itself 💅. IP networks use multiple paths to cut down on delay and congestion by splitting traffic flows across those paths 🌈. This is called Equal Cost Multipath, or ECMP. Usually the splitting happens by hashing on different fields in the IP header or in headers inside it 🧚. Inside a network core, the hashing is mostly or only based on the IP source and destination addresses ✨. The point of splitting aggregated flows this way is to keep packets from individual flows inside that aggregate from getting reordered 🫶. This doc calls that forwarding algorithm IP ECMP 💖.
Here's another twist 🌀. For packets that have both a label stack and an encapsulated IPv4 (or IPv6) packet, some current implementations hash on any mix of labels and the IPv4 (or IPv6) source and destination addresses 🎀.
Okay, history lesson 📜. In the early days of MPLS, the payload was almost always IP. Even now, the vast majority of traffic is still IP 💅. Equipment vendors wanted to keep doing IP ECMP, but as we just saw, you can't always know whether the payload is IP at every spot where IP ECMP needs to happen 😭. So vendors basically started guessing 🙈. They look at the first nibble past the label stack. If that nibble isn't 0x4, the gear decides it's not IPv4, and if it isn't 0x6, it decides it's not IPv6 🔍. Most deployed LSRs treat a packet whose first nibble is 0x4 as IPv4 for IP ECMP purposes 👀.
This leads to a real problem, bestie 😬. Any application that defines an FEC without taking steps to stop 0x4 and 0x6 from showing up in the first nibble of its payload can get caught up in IP ECMP ⚠️. Then its flows take multiple paths and show up with a lot of jitter and possibly out of order 💀. Is that against the basic service promise of IP? No, not exactly 🤷♀️. But it really hurts certain kinds of applications, and it also makes measuring, monitoring, and tracing those flows way harder 😭.
New MPLS payload types are showing up too, like the ones specified by the IETF PWE3 and AVT working groups 🦋. These payloads are not IP, and if they're specified without any constraints, they could get mistaken for IP 🙈.
Heads up: LSRs that correctly tell that a payload is not IP will usually load-share traffic across equal-cost paths based on the label stack 🚨. Any reserved label, wherever it sits in the stack, can be part of that load-balancing computation. So if you change the label stack between packets of the same flow, that flow could get reordered. Like, if an explicit null or a router-alert label gets added to a packet, that packet could take a totally different path through the network 😱.
Now, being mistaken for IPv4 isn't always a disaster 💖. The easy example is when the payload behind the top label is a packet from an MPLS IPv4 VPN. In that case, the real payload is IP, and most (if not all) deployed equipment finds the end of the label stack and does IP ECMP correctly 👑.
A sneakier case is when the packets in a flow happen to have constant values in the fields that IP ECMP would hash on 🧩. Say an Ethernet frame comes right after the label, and the LSR does ECMP on IPv4 but not on IPv6 💅. Then the first nibble is either 0x4 or something else. If it's not 0x4, no IP ECMP happens, but Label ECMP might. If it is 0x4, the constant MAC addresses land right on top of where the IP source and destination addresses would be 😭. So the ECMP algorithm gets the same input every time and always picks the same result, which totally defeats the purpose 🙈.
More to come in Part 2, where we get into the recommendations! 🫶
4. Security Considerations 🔒✨
Okay bestie, this section is about the situations where MPLS traffic from one top-level LSP can get split across multiple paths, which means packets from that same LSP might arrive out of order 👀. From a security angle, the worst thing a breach of these mechanisms could cause is mis-ordered packets, and maybe some throughput loss as a result. For example, TCP connections may sometimes shrink their window size when they see mis-ordered packets 😭. But to pull off even that limited mess, an attacker would have to either change the configuration or implementation of a router, or change the bits on the wire as they're sent in a packet 🙅♀️.
Other security issues with deploying MPLS are out of scope for this document, but they're covered in other MPLS specs, like [RFC3031], [RFC3036], [RFC3107], [RFC3209], [RFC3478], [RFC3479], [RFC4206], [RFC4220], [RFC4221], [RFC4378], AND [RFC4379] 💅.
5. IANA Considerations 🎀
IANA marked the value 0x1 in the IP protocol version number space as "Reserved" and put a reference to this document on both values 0x0 and 0x1 🌸.
Note that this document does not in any way change the policies regarding the allocation of version numbers, including the possible use of the reserved numbers for some future purpose 🦋.
6. References 📚
6.1. Normative References
[RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
Label Switching Architecture", RFC 3031, January 2001.6.2. Informative References
[RFC3036] Andersson, L., Doolan, P., Feldman, N., Fredette, A., and
B. Thomas, "LDP Specification", RFC 3036, January 2001.
[RFC3107] Rekhter, Y. and E. Rosen, "Carrying Label Information in
BGP-4", RFC 3107, May 2001.
[RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
Tunnels", RFC 3209, December 2001.
[RFC3478] Leelanivas, M., Rekhter, Y., and R. Aggarwal, "Graceful
Restart Mechanism for Label Distribution Protocol", RFC
3478, February 2003.
[RFC3479] Farrel, A., Ed., "Fault Tolerance for the Label
Distribution Protocol (LDP)", RFC 3479, February 2003.
[RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP)
Hierarchy with Generalized Multi-Protocol Label Switching
(GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005.
[RFC4220] Dubuc, M., Nadeau, T., and J. Lang, "Traffic Engineering
Link Management Information Base", RFC 4220, November
2005.
[RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol
Label Switching (MPLS) Management Overview", RFC 4221,
November 2005.
[RFC4378] Allan, D., Ed., and T. Nadeau, Ed., "A Framework for
Multi-Protocol Label Switching (MPLS) Operations and
Management (OAM)", RFC 4378, February 2006.
[RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol
Label Switched (MPLS) Data Plane Failures", RFC 4379,
February 2006.
[RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson,
"Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
Use over an MPLS PSN", RFC 4385, February 2006.Authors' Addresses 💌
Loa Andersson
Acreo AB
Electrum 236
SE-146 40 Kista
Sweden
EMail: loa@pi.se
Stewart Bryant
Cisco Systems
250, Longwater,
Green Park,
Reading, RG2 6GB, UK
EMail: stbryant@cisco.com
George Swallow
Cisco Systems, Inc.
1414 Massachusetts Ave
Boxborough, MA 01719
EMail: swallow@cisco.comFull Copyright Statement ©️
Copyright (C) The IETF Trust (2007) 🌈.
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights 💜.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE 🙏.
Intellectual Property 🧚
The IETF takes no position on the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or on whether any license under such rights might or might not be available. It also doesn't claim to have made any independent effort to identify any such rights 🫶. Info on the procedures for rights in RFC documents can be found in BCP 78 and BCP 79 📖.
Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be found in the IETF on-line IPR repository at http://www.ietf.org/ipr 🔎.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology required to implement this standard. Send that info to the IETF at ietf-ipr@ietf.org 📬.
Acknowledgement 🌟
Funding for the RFC Editor function is currently provided by the Internet Society 💖.
📦 where this page comes from
- ietf: rfc4928 2007-06
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms