RFC 8469

Recommendation to Use the Ethernet Control Word

📅 published November 2018, from ietf

Recommendation to Use the Ethernet Control Word 💖

okay bestie, so this is RFC 8469 🎀. It's written by S. Bryant, A. Malis (Huawei), and I. Bagdonas (Equinix). It's a Standards Track doc, published November 2018 (ISSN 2070-1721), and it updates RFC 4448 🫶. It doesn't obsolete anything 💅.

Abstract ✨

the pseudowire (PW) encapsulation of Ethernet, as defined in RFC 4448, says the control word (CW) is optional 🙈. Without the CW, an Ethernet PW packet can get mistaken for an IP packet by a label switching router (LSR). That can push it onto the wrong equal-cost multipath (ECMP) path, and then packets get misordered 😭. It's gotten worse because of gear with Ethernet Media Access Control (MAC) addresses that start with 0x4 or 0x6 🔥. The Ethernet PW CW fixes this 💅. This document RECOMMENDS the use of the Ethernet PW CW in all but exceptional circumstances 👑.

this doc updates RFC 4448 🫶.

Status of This Memo 🦋

this one is an Internet Standards Track document 🌸.

it's a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, received public review, and was approved for publication by the Internet Engineering Steering Group (IESG). More info on Internet Standards is in Section 2 of RFC 7841 📚.

info about the current status of this document, any errata, and how to give feedback on it lives at https://www.rfc-editor.org/info/rfc8469 ✨.

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 🙏. Code Components extracted from this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as described in the Simplified BSD License ⚠️.

Table of Contents 📑

   1. Introduction ....................................................3
   2. Specification of Requirements ...................................3
   3. Background ......................................................4
   4. Recommendation ..................................................5
   5. Equal-Cost Multipath (ECMP) .....................................5
   6. Mitigations .....................................................6
   7. Operational Considerations ......................................6
   8. Security Considerations .........................................7
   9. IANA Considerations .............................................7
   10. References .....................................................7
      10.1. Normative References ......................................7
      10.2. Informative References ....................................8
   Acknowledgments ....................................................9
   Authors' Addresses .................................................9

1. Introduction 🌸

okay so the PW encapsulation of Ethernet, from [RFC4448], says the control word (CW) is optional. Here's the lore: it's common for label switching routers (LSRs) to peek past the end of the label stack and check whether the payload is an IP packet. If it is, they pick the next hop based on the so-called "five-tuple" (IP source address, IP destination address, protocol/next-header, transport-layer source port, and transport-layer destination port) 👀. Without a PW CW, an Ethernet PW packet can get misidentified as an IP packet by an LSR that's picking an ECMP path based on that five-tuple. That can send the packet down the wrong ECMP path, which leads to misordered packets 😭. More on this is in [RFC4928] 📖.

flow misordering can also happen on a single path when traffic classification and differential forwarding treatment are in use. These errors happen when a forwarder wrongly assumes the packet is IP and applies a forwarding policy based on fields in the PW payload 🙈.

IPv4 and IPv6 packets start with 0x4 and 0x6, respectively 🧚. Misidentification can happen if an Ethernet PW packet without a CW is carrying an Ethernet packet whose destination address starts with either of those values.

this problem has gotten way more serious lately, for a couple of reasons 🔥. First, the IEEE Registration Authority Committee (RAC) has assigned Ethernet MAC addresses that start with 0x4 or 0x6, and equipment using MAC addresses from those series has been deployed in networks. Second, privacy concerns have led to MAC address randomization, which assigns local MAC addresses randomly for privacy. Random assignment gives addresses starting with one of those two values roughly one time in eight 🎲.

the use of the Ethernet PW CW addresses this problem 💅.

This document RECOMMENDS the use of the Ethernet PW CW in all but exceptional circumstances.

2. Specification of Requirements 📣

okay this part is the rules of the game 🎮. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 💜.

3. Background 📚

Ethernet PW encapsulation is specified in [RFC4448] 🫶. The bit that matters most here is Section 4.6, and part of it is quoted below so you don't have to go digging. Heads up: RFC 4448 uses the citation [PWE3-CW] to point at [RFC4385], and the citation [VCCV] to point at the doc that was eventually published as [RFC5085] 🔍.

The control word defined in this section is based on the Generic PW MPLS Control Word as defined in [PWE3-CW]. It provides the ability to sequence individual frames on the PW, avoidance of equal-cost multiple-path load-balancing (ECMP) [RFC2992], and Operations and Management (OAM) mechanisms including VCCV [VCCV]. [PWE3-CW] states, "If a PW is sensitive to packet misordering and is being carried over an MPLS PSN that uses the contents of the MPLS payload to select the ECMP path, it MUST employ a mechanism which prevents packet misordering." This is necessary because ECMP implementations may examine the first nibble after the MPLS label stack to determine whether the labeled packet is IP or not. Thus, if the source MAC address of an Ethernet frame carried over the PW without a control word present begins with 0x4 or 0x6, it could be mistaken for an IPv4 or IPv6 packet. This could, depending on the configuration and topology of the MPLS network, lead to a situation where all packets for a given PW do not follow the same path. This may increase out-of-order frames on a given PW, or cause OAM packets to follow a different path than actual traffic (see Section 4.4.3, "Frame Ordering"). The features that the control word provides may not be needed for a given Ethernet PW. For example, ECMP may not be present or active on a given MPLS network, strict frame sequencing may not be required, etc. If this is the case, the control word provides little value and is therefore optional. Early Ethernet PW implementations have been deployed that do not include a control word or the ability to process one if present. To aid in backwards compatibility, future implementations MUST be able to send and receive frames without the control word present.

when PWs first got deployed, some commercially significant equipment couldn't process the Ethernet CW 🙃. Back then, people also believed the RAC hadn't issued any Ethernet MAC address starting with 0x4 or 0x6. So it seemed safe to deploy Ethernet PWs without the CW 🤷‍♀️.

but the RAC has since issued Ethernet MAC addresses starting with 0x4 and with 0x6. So the assumption that, in real networks, there would be no confusion between an Ethernet PW packet without the CW and an IP packet is no longer correct 💀.

possibly because of unauthorized Ethernet MAC addresses, that assumption has been unsafe for a while. That led some equipment vendors to build more complex, proprietary methods to tell Ethernet PW packets apart from IP packets 🧩. These methods rely on heuristics that examine transit packets to guess the exact payload type, and they can't be reliable because the payload inside these packets is random 🎰.

a posting on the NANOG email list highlighted this problem 📬:

https://mailman.nanog.org/pipermail/nanog/2016-December/089395.html

4. Recommendation 🏆

the ambiguity between an MPLS payload that's an Ethernet PW and one that's an IP packet goes away when the Ethernet PW CW is used ✨. This document updates [RFC4448] to say that both the ingress provider edge (PE) and the egress PE SHOULD support the Ethernet PW CW, and that, if supported, the CW MUST be used 👑.

when ECMP needs to apply to Ethernet PW traffic, and both the ingress and egress PEs support Entropy Label Indicator / Entropy Label (ELI/EL) [RFC6790] and Flow-Aware Transport of Pseudowires (FAT PW) [RFC6391], then either method may be used 🌊. Using both methods on the same PW isn't normally necessary and should be avoided unless circumstances require it 🙅‍♀️. For multi-segment PWs, if ELI/EL is used, then it SHOULD be used on every segment of the PW 🧵. How you guarantee ELI/EL gets used on every segment is out of scope for this document 🤐.

to be continued 🦋

5. Equal-Cost Multipath (ECMP) ✨

okay bestie, so when an Ethernet PW carries so much traffic that ECMP is needed 🌊, there are two methods you can use:

  • Flow-Aware Transport of Pseudowires over an MPLS Packet Switched Network, specified in [RFC6391], or
  • Label Switched Path (LSP) entropy labels, specified in [RFC6790].

RFC 6391 works by increasing the entropy of the bottom-of-stack label 🧚. For it to work, both the ingress and egress PEs have to support this feature. Sufficient LSRs on the LSP between the ingress and egress PE also need to be able to pick an ECMP path on an MPLS packet with the resulting stack depth. It's a whole chain, babe 💅.

RFC 6790 works by putting an entropy value into the LSP part of the label stack 🎀. The ingress and egress PEs need to support inserting and removing the EL and the ELI, and enough LSRs on the LSP need to be able to do ECMP based on the EL. Obsessed with how this is layered 😭.

In both cases, there's some stuff to think about when you want OAM packets to follow the same path as a data packet 🦋. This is described in detail in Section 7 of [RFC6391] and Section 6 of [RFC6790]. Honestly though, in both cases it's way better than the ECMP situation when the Ethernet PW CW wasn't used, because right now there's no known method of getting a PW OAM packet to follow the same path as a PW data packet that gets ECMP'd based on the five-tuple of the IP payload. Not gonna lie, that's a whole vibe check 🙈.

The PW label gets pushed before the LSP label. The ELI/EL labels live in the LSP layer rather than the PW layer, so they get pushed after the PW label has already been pushed 🌸.

6. Mitigations 🛡️

If you can't use the Ethernet PW CW, you can turn off the effects of ECMP by carrying the PW over a traffic-engineered path that doesn't let load balancing touch the payload. RSVP-TE [RFC3209] is an example of this 💜. Lowkey though, these paths might still get hit by link-bundle load balancing, and of course the single LSP has to carry the full PW load. So it's not a perfect fix, just a smart workaround ✨.

7. Operational Considerations 📋

Sometimes whether the CW gets included in the PW is decided by equipment configuration ⚙️. It's also possible that the default configuration in those cases is to turn the CW off. So be careful: software that implements this recommendation shouldn't depend on existing configuration settings that stop the CW from being used. It is recommended that platform software emit a rate-limited message saying the CW can be used but is disabled because of existing configuration 📣.

Here's the tea 🍵: instead of including a payload type in the packet, MPLS relies on the control plane to signal which payload type follows the bottom of the label stack. Some LSRs try to guess the packet type by inspecting the MPLS payload, sometimes looking right past the PW CW 👀. If the payload seems to be IP, or IP carried in an Ethernet header, they do an ECMP calculation based on what they assume the five-tuple fields are. But guessing the payload type like this is not an exact science 🙃. When a packet that isn't IP gets mistaken for an IP packet, the result can be packets delivered out of order, and that kind of misordering can be really hard for an operator to diagnose 😵‍💫. So if you enable a capability that lets information gleaned from packet inspection past the PW CW be used in any ECMP calculation, operators should know that Ethernet frames may be delivered out of order even though the CW is present. Major warning energy 🚨.

8. Security Considerations 🔐

This document expresses a preference for one existing and widely deployed Ethernet PW encapsulation over another 💖. These methods have identical security considerations, which are discussed in [RFC4448]. This document introduces no additional security issues ✅.

9. IANA Considerations 🌷

This document has no IANA actions.

10. References 📚

10.1. Normative References

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC4385]  Bryant, S., Swallow, G., Martini, L., and D. McPherson,
           "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
           Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385,
           February 2006, <https://www.rfc-editor.org/info/rfc4385>.

[RFC4448]  Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron,
           "Encapsulation Methods for Transport of Ethernet over MPLS
           Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006,
           <https://www.rfc-editor.org/info/rfc4448>.

[RFC4928]  Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal
           Cost Multipath Treatment in MPLS Networks", BCP 128,
           RFC 4928, DOI 10.17487/RFC4928, June 2007,
           <https://www.rfc-editor.org/info/rfc4928>.

[RFC6391]  Bryant, S., Ed., Filsfils, C., Drafz, U., Kompella, V.,
           Regan, J., and S. Amante, "Flow-Aware Transport of
           Pseudowires over an MPLS Packet Switched Network",
           RFC 6391, DOI 10.17487/RFC6391, November 2011,
           <https://www.rfc-editor.org/info/rfc6391>.

[RFC6790]  Kompella, K., Drake, J., Amante, S., Henderickx, W., and
           L. Yong, "The Use of Entropy Labels in MPLS Forwarding",
           RFC 6790, DOI 10.17487/RFC6790, November 2012,
           <https://www.rfc-editor.org/info/rfc6790>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

10.2. Informative References

[RFC2992]  Hopps, C., "Analysis of an Equal-Cost Multi-Path
           Algorithm", RFC 2992, DOI 10.17487/RFC2992, November 2000,
           <https://www.rfc-editor.org/info/rfc2992>.

[RFC3209]  Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
           and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
           Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
           <https://www.rfc-editor.org/info/rfc3209>.

[RFC5085]  Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual
           Circuit Connectivity Verification (VCCV): A Control
           Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085,
           December 2007, <https://www.rfc-editor.org/info/rfc5085>.

Acknowledgments 🫶

The authors thank Job Snijders for drawing attention to this problem 👏. The authors also thank Pat Thaler for clarifying the matter of local MAC address assignment 💜. We thank Sasha Vainshtein for his valuable review comments 🌈.

Authors' Addresses 🦋

Stewart Bryant
Huawei

Email: stewart.bryant@gmail.com


Andrew G. Malis
Huawei

Email: agmalis@gmail.com


Ignas Bagdonas
Equinix

Email: ibagdona.ietf@gmail.com>
📦 where this page comes from