RFC 9784

Virtual Ethernet Segments for EVPN and Provider Backbone Bridge EVPN

📅 published June 2025, from ietf

RFC 9784: Virtual Ethernet Segments for EVPN and Provider Backbone Bridge EVPN 🦋

  • Number and series: RFC 9784 from the Internet Engineering Task Force (IETF) 💖
  • Authors: A. Sajassi and P. Brissette (Cisco Systems), R. Schell and J. Drake (Independent), and J. Rabadan (Nokia) 🫶
  • Category: Standards Track ✨
  • ISSN: 2070-1721
  • Date: June 2025 🌸
  • Obsoletes/Updates: The title block doesn't list any "Obsoletes" or "Updates" lines, so this RFC doesn't mark any other document as obsolete or updated 💅

Abstract ✨

Okay so, Ethernet VPN (EVPN) and Provider Backbone Bridge EVPN (PBB-EVPN) are a whole suite of solutions for delivering Ethernet services over MPLS/IP networks, and they are giving multihoming realness 👑. They support Single-Active and All-Active redundancy modes for an Ethernet Segment (ES). An ES is a collection of physical links connecting a multihomed device or network to a set of Provider Edge (PE) devices 🔥.

This document makes the ES concept go even bigger 🦋. An ES can now be associated with a set of Ethernet Virtual Circuits (EVCs), like VLANs, or with other things, like MPLS Label Switched Paths (LSPs) or pseudowires (PWs) 💜. This upgraded concept is called virtual Ethernet Segments (vESes) 🌈. The document lists the requirements and specifies the extensions needed to support vES in both EVPN and PBB-EVPN 💅.

Status of This Memo 🎀

This is an Internet Standards Track document 💖.

It's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community 🫶. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG) ✨. For more on Internet Standards, check Section 2 of RFC 7841 📚.

Info about the current status of this document, any errata, and how to send feedback is at https://www.rfc-editor.org/info/rfc9784 🌸.

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved 🔒.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication 📜. Please read these carefully, because they describe your rights and restrictions for this document 🙈. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Revised BSD License ⭐.

Table of Contents 🧚

1.  Introduction
  1.1.  Requirements Language
  1.2.  vESes in Access Ethernet Networks
  1.3.  vESes in Access MPLS Networks
2.  Terminology
3.  Requirements
  3.1.  Single-Homed and Multihomed vES
  3.2.  Local Switching
  3.3.  EVC Service Types
  3.4.  Designated Forwarder (DF) Election
  3.5.  EVC Monitoring
  3.6.  Failure and Recovery
  3.7.  Fast Convergence
4.  Solution Overview
  4.1.  EVPN DF Election for vES
  4.2.  Grouping and Route Coloring for vES
    4.2.1.  EVPN Route Coloring for vES
    4.2.2.  PBB-EVPN Route Coloring for vES
5.  Failure Handling and Recovery
  5.1.  EVC Failure Handling for Single-Active vES in EVPN
  5.2.  EVC Failure Handling for a Single-Active vES in PBB-EVPN
  5.3.  Port Failure Handling for Single-Active vESes in EVPN
  5.4.  Port Failure Handling for Single-Active vESes in PBB-EVPN
  5.5.  Fast Convergence in EVPN and PBB-EVPN
6.  Security Considerations
7.  IANA Considerations
8.  References
  8.1.  Normative References
  8.2.  Informative References
Acknowledgements
Authors' Addresses

1. Introduction ✨

Ethernet VPN (EVPN) [RFC7432] and Provider Backbone Bridge EVPN (PBB-EVPN) [RFC7623] are a comprehensive suite of solutions for delivering Ethernet services over MPLS/IP networks 💖. They offer advanced multihoming capabilities 👑. They support Single-Active and All-Active redundancy modes for an Ethernet Segment (ES). As defined in [RFC7432], an ES is a collection of Ethernet links that connect a customer site to one or more Provider Edge (PE) devices 🫶.

This document stretches the ES concept by letting an ES be associated with a set of Ethernet Virtual Circuits (EVCs), such as VLANs, or with other entities, including MPLS Label Switched Paths (LSPs) or pseudowires (PWs) 🦋. This extended concept is called virtual Ethernet Segments (vESes) 🌈. The document lists the requirements and specifies the necessary extensions to support vES in both EVPN and PBB-EVPN 💅. The scope includes PBB-EVPN [RFC7623], EVPN over MPLS [RFC7432], and EVPN over IP [RFC8365]. It does not include EVPN over SRv6 [RFC9252] 🙈.

1.1. Requirements Language 📣

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 💜.

1.2. vESes in Access Ethernet Networks 🌸

Okay so, some service providers (SPs) want to stretch the idea of physical Ethernet links in an ES so it covers EVCs instead 🧚. Multiple EVCs, like VLANs, can get aggregated onto a single physical External Network-Network Interface (ENNI) ✨. An ES made of EVCs rather than physical links is called a vES 💖.

Figure 1 shows two PE devices (PE1 and PE2), each with an ENNI aggregating several EVCs. Some of those EVCs on a given ENNI can be associated with vESes 🎀. For example, the multihomed vES in Figure 1 is made of EVC4 on ENNI1 and EVC5 on ENNI2 🦋.

                 Third-Party
   +-----+       EAP
   | CE11|EVC1  +---------+
   +-----+   \  |         |       +---+
   Cust. A    \-0=========0--ENNI1|   |
   +-----+      |         |  ENNI1|   |   +-------+   +---+
   | CE12|EVC2--0=========0--ENNI1|PE1|---|       |   |   |
   +-----+      |         |  ENNI1|   |   |SP     |---|PE3|-
                |       ==0--ENNI1|   |   |IP/MPLS|   |   | \  +---+
   +-----+      |      /  |       +---+   |Core   |   +---+  \-|   |
   | CE22|EVC3--0==== /   |               |Network|            |CE4|
   +-----+      |    X    |               |       |   +---+    |   |
   Cust. B      |   / \   |       +---+   |       |   |   |  /-|   |
   +-----+     -0===   ===0--ENNI2|   |   |       |---|PE4|-/  +---+
   | CE3 |EVC4/ |         |  ENNI2|PE2|---|       |   |   |
   |     |EVC5--0=========0--ENNI2|   |   +-------+   +---+
   +-----+      |         |       +---+
   Cust. C      +---------+   /\
          /\                  ||
          ||                  ENNI
          EVCs             Interface
   <--------802.1Q---------->  <---- EVPN Network -----> <-802.1Q->

 Figure 1: Single-Homed Devices and a Dual-Homed Device/Network on
                           the Same ENNI

ENNIs are commonly used to reach remote customer sites over independent Ethernet access networks or third-party Ethernet Access Providers (EAPs) 🌍. An ENNI can aggregate traffic from many vESes (e.g., hundreds to thousands), and each vES is represented by its associated EVC on that ENNI 💅. That makes ENNIs and their EVCs key parts of SP external boundaries, which are carefully designed and closely monitored 🔥. As a reminder, the ENNI is the demarcation between the SP (IP/MPLS core network) and the third-party Ethernet Access Provider 📌.

To meet customers' Service Level Agreements (SLAs), SPs build redundancy with multiple EVPN PEs and across multiple ENNIs (as in Figure 1) 👑. A given vES can be multihomed to two or more EVPN PE devices, on two or more ENNIs, through its associated EVCs 🫶. Just like the physical ESs in [RFC7432] and [RFC7623], these vESes can be single-homed or multihomed ESs. When multihomed, they can operate in either Single-Active or All-Active redundancy modes 💖. In a typical SP external-boundary scenario (e.g., with an EAP), an ENNI can be associated with several thousands of single-homed vESes, several hundreds of Single-Active vESes, and tens or hundreds of All-Active vESes 🌈. The specific figures in this document reflect the relative quantities (hundreds, thousands, etc.) of various elements as understood at the time of writing 🧚.

1.3. vESes in Access MPLS Networks ✨

okay so bestie, now we're going somewhere spicy 🌈 some SPs want to take the "physical link in an ES" idea and apply it to individual PWs or to MPLS LSPs in Access MPLS networks. so a vES is a bundle of PWs or a bundle of LSPs 🫶 Figure 2 is basically the visual for this, so look at it real quick:

                MPLS Aggregation
                Network
  +-----+      +-----------------+
  | CE11|EVC1  |                 |
  +-----+   \ +AG1-+  PW1      +-+---+
  Cust. A    -0----|===========|     |
  +-----+     | ---+===========|     |   +-------+   +---+
  | CE12|EVC2-0/   |  PW2   /\ | PE1 +---+       |   |   |
  +-----+     ++---+      /=||=|     |   |       +---+PE3+-
               |         //=||=|     |   |IP/MPLS|   |   | \  +---+
               |        //  \/ +-+---+   |Core   |   +---+  \-+   |
  +-----+EVC3  |    PW3//  LSP1  |       |Network|            |CE4|
  | CE13|    \+AG2-+==//         |       |       |   +---+    |   |
  +-----+     0    |==/PW4  /\ +-+---+   |       |   |   |  /-+   |
              0    |==PW5===||=|     |   |       +---+PE4+-/  +---+
  +-----+    /++---+==PW6===||=| PE2 +---+       |   |   |
  | CE14|EVC4  |            \/ |     |   +-------+   +---+
  +-----+      |           LSP2+-+---+
  Cust. C      +-----------------+
         /\
         ||
         EVCs
  <--802.1Q--> <-----MPLS Agg----> <--- EVPN Network ---> <-802.1Q->

      Figure 2: A Dual-Homed and Single-Homed Network on MPLS
                        Aggregation Networks

so in certain setups, SPs lean on MPLS Aggregation Networks that are run by separate admin entities or third-party orgs just to reach their own IP/MPLS core network 💅 that's exactly what Figure 2 is showing.

in those setups, a vES is defined as a set of individual PWs when aggregation isn't feasible 🦋 if aggregation is possible, the vES can hang out with a group of PWs that share the same unidirectional LSP pair, where that LSP pair is the ingress and egress LSPs between the same endpoints. iconic 👑

now, in Figure 2, EVC3 hooks up to a VPWS instance in AG2, which connects to PE1 and PE2 through PW3 and PW5, respectively. EVC4 hooks up to another VPWS instance on AG2, which connects to PE1 and PE2 through PW4 and PW6, respectively. 😭 since the PWs for both VPWS instances can get aggregated into the same LSP pair going to and coming from the MPLS network, a common vES can be defined for all four PWs. also in Figure 2, LSP1 and LSP2 are the two LSP pairs, one between PE1 and AG2 and one between PE2 and AG2. the vES is made of those two LSP pairs (LSP1 and LSP2), and each LSP pair has two PWs. this vES gets shared by two separate EVPN instances (like EVI-1 and EVI-2) in the EVPN network. PW3 and PW4 are tied to EVI-1 and EVI-2 on PE1, and PW5 and PW6 are tied to EVI-1 and EVI-2 on PE2. main character energy 🎀

but sometimes aggregating PWs that share the same LSP pair just isn't possible 🙈 for example, if PW3 ended up terminating on a third PE, like PE3, instead of PE1, then the vES would have to be defined on each PW on each PE. no shortcuts there, bestie 💀

for MPLS/IP access networks where a vES stands for a set of LSP pairs or a set of PWs, this document extends the Single-Active multihoming procedures from [RFC7432] and [RFC7623] so they work with vES 🌸 extending vES to All-Active multihoming in MPLS/IP access networks is beyond the scope of this document, so that's a later problem 🧚

here's the game plan 📋 this document defines the vES concept and specifies the extra extensions needed to support a vES in line with [RFC7432] and [RFC7623]. Section 3 lists the requirements for a vES. Section 4 breaks down the extensions for vES that apply to EVPN solutions, including the ones in [RFC7432] and [RFC7209]. those extensions are built to meet the requirements from Section 3. Section 4 also gives an overview of the solution, and Section 5 covers failure handling, recovery, scalability, and fast convergence of [RFC7432] and [RFC7623] for vESes. the whole thing is lowkey genius ✨

2. Terminology 🎀

okay the abbreviations, bestie 📖

AC:       Attachment Circuit

B-MAC:    Backbone MAC Address

CE:       Customer Edge

C-MAC:    Customer/Client MAC Address

DF:       Designated Forwarder

ENNI:     External Network-Network Interface

ES:       Ethernet Segment

ESI:      Ethernet Segment Identifier

Ethernet A-D:  Ethernet Auto-Discovery

EVC:      Ethernet Virtual Circuit [MEF63]

EVI:      EVPN Instance

EVPN:     Ethernet VPN

I-SID:    Service Instance Identifier (24 bits and global within a
          PBB network; see [RFC7080]).

MAC:      Media Access Control

PBB:      Provider Backbone Bridge

PBB-EVPN:  Provider Backbone Bridge EVPN

PE:       Provider Edge

VPWS:     Virtual Private Wire Service

Single-Active (SA) Redundancy Mode:  When only a single PE, among a
          group of PEs attached to an ES, is allowed to forward
          traffic to/from that ES, the ES is defined as operating in
          Single-Active redundancy mode.

All-Active (AA) Redundancy Mode:  When all PEs attached to an ES are
          allowed to forward traffic to/from that ES, the ES is
          defined as operating in All-Active redundancy mode.

3. Requirements 📋

this section is the to-do list for vES in EVPN and PBB-EVPN solutions 💖 these requirements sit on top of the ones in [RFC8214], [RFC7432], and [RFC7623], so don't skip those, bestie.

3.1. Single-Homed and Multihomed vES 🦋

a PE device MUST support the following types of vESes:

(R1a) The PE MUST handle single-homed vESes on a single physical port, such as a single ENNI.

(R1b) The PE MUST support a combination of single-homed vESes and Single-Active multihomed vESes simultaneously on a single physical port, such as a single ENNI. throughout this document, Single-Active multihomed vESes get called "Single-Active vESes" 🫶

(R1c) The PE MAY support All-Active multihomed vESes on a single physical port. throughout this document, All-Active multihomed vESes get called "All-Active vESes" ✨

(R1d) The PE MAY support a combination of All-Active vESes along with other types of vESes on a single physical port. 💅

(R1e) A multihomed vES, whether Single-Active or All-Active, can span across two or more ENNIs on any two or more PEs. 🌈

3.2. Local Switching 🔥

lots of different types of vESes can be bundled on one physical port of a PE device, and some of them can belong to the same service instance (like an EVI) 👑 that means you need local switching among the vESes for the same service instance on the same physical port (like an ENNI) of the PE. obviously 🙄

(R2a) A PE device that supports the vES function MUST support local switching among different vESes associated with the same service instance on a single physical port. for example, in Figure 1, PE1 has to support local switching between CE11 and CE12, which are mapped to two single-homed vESes on ENNI1. when it comes to Single-Active vESes, local switching happens among active EVCs associated with the same service instance on the same ENNI. 🎀

3.3. EVC Service Types 🧚

a physical port, like an ENNI on a PE device, can aggregate loads of EVCs, each tied to a vES 🌸 an EVC can carry one or more VLANs. usually it carries a single VLAN and so it maps to a single broadcast domain, but there's no rule stopping an EVC from carrying multiple VLANs, so it's giving flexibility 💖

(R3a) An EVC can be associated with a single broadcast domain, like in a VLAN-based service or a VLAN bundle service. 🦋

(R3b) An EVC MAY be associated with several broadcast domains, like in a VLAN-aware bundle service. ✨

similarly, a PE can aggregate multiple LSPs and PWs. with individual PWs per vES, a PW is usually tied to a single broadcast domain, though nothing stops a PW from carrying multiple VLANs if it's configured in Raw mode 🍓

(R3c) A PW can be associated with a single broadcast domain, like in a VLAN-based service or a VLAN bundle service. 💜

(R3d) A PW MAY be associated with several broadcast domains, like in a VLAN-aware bundle service. 🌸

3.4. Designated Forwarder (DF) Election 👑

Section 8.5 of [RFC7432] specifies the default DF election procedure for EVPN, and [RFC7623] and [RFC8214] apply it too. [RFC8584] adds more DF election procedures for EVPN. these DF election procedures run at the granularity of (ESI, Ethernet Tag). for a vES, the same EVPN default procedure still applies, just at the granularity of (vESI, Ethernet Tag). here the Ethernet Tag is an I-SID in PBB-EVPN and a VLAN ID (VID) in EVPN. as [RFC7432] describes, this default DF election procedure at the granularity of (vESI, Ethernet Tag) is also called "service carving." the goal of service carving is to spread the DFs for different vESes evenly across PEs, so traffic is evenly distributed too 🫶 the following requirements apply to DF election for vESes in EVPN and PBB-EVPN. 🔥

(R4a) A PE that supports vES function MUST support a vES with m EVCs among n ENNIs belonging to p PEs in any arbitrary order, where n >= p >= m >=2. for example, if there's a vES with 2 EVCs and 5 ENNIs on 5 PEs (PE1 through PE5), then the vES can be dual-homed to PE2 and PE4, and DF election must happen between PE2 and PE4. 💅

(R4b) Each vES MUST be identified by its own virtual ESI (vESI). 🌈

3.5. EVC Monitoring 🙈

to detect the failure of an individual EVC and then run DF election for its associated vES because of that failure, each EVC should be monitored independently. 💀

(R5a) Each EVC SHOULD be independently monitored for its operational health. ⭐

(R5b) A failure in a single EVC, among many aggregated on a single physical port or ENNI, MUST trigger a DF election for its associated vES. 😭🫶

3.6. Failure and Recovery 🌧️

(R6a) okay so failure and failure recovery of an EVC for a single-homed vES SHALL NOT impact any other EVCs within its service instance or any other service instances 🙅‍♀️ in other words, for PBB-EVPN, it SHALL NOT trigger any MAC flushing within both its own I-SID and other I-SIDs. no drama across the whole network, just the one lane 🛣️

(R6b) for All-Active vES, failure and failure recovery of an EVC for that vES SHALL NOT impact any other EVCs within its service instance or any other service instances 💖 in other words, for PBB-EVPN, it SHALL NOT trigger any MAC flushing within both its own I-SID and other I-SIDs. the All-Active crew stays chill even when one EVC has a moment 🫶

(R6c) failure and failure recovery of an EVC for a Single-Active vES SHALL impact only its own service instance 💅 in other words, for PBB-EVPN, MAC flushing SHALL be limited to the associated I-SID only and SHALL NOT impact any other I-SIDs. it stays in its own lane, periodt ✨

(R6d) failure and failure recovery of an EVC for a Single-Active vES MUST only impact C-MACs associated with a multihomed device/network for that service instance 🎀 in other words, MAC flushing MUST be limited to a single service instance (I-SID in the case of PBB-EVPN) and only C-MACs for a Single-Active multihomed device/network. tiny blast radius, big respect 🌸

3.7. Fast Convergence ⚡

okay this one is so relatable 😭 many EVCs (and their associated vESes) are aggregated via a single physical port (e.g., ENNI), so when that physical port fails, it impacts many vESes and equally triggers many ES route withdrawals. formulating, sending, receiving, and processing that many BGP messages can add delay to DF election and convergence time 🐢 so it's highly desirable to have a mass-withdraw mechanism similar to the one in [RFC7432] for withdrawing many Ethernet A-D per ES routes. one big group hug instead of a hundred tiny texts 🫂

(R7a) there SHOULD be a mechanism equivalent to EVPN mass withdraw such that upon an ENNI failure, only a single BGP message to the PEs is needed to trigger DF election for all impacted vESes associated with that ENNI 🌈 one message, and the whole squad re-elects their DF. iconic efficiency 👑

4. Solution Overview ✨

the solutions described in [RFC7432] and [RFC7623] are leveraged as is 💁‍♀️ the one twist is that the ESI assignment is performed for an EVC or a group of EVCs or LSPs/PWs instead of a link or a group of physical links. so the ESI is associated with a vES (hereby referred to as the "vESI") 🎀

in the EVPN solution, the overall procedures stay consistent 🧚 the main difference is how physical port failures that can affect multiple vESes get handled. Sections 5.1 and 5.3 describe the procedures for managing physical port or link failures in the context of EVPN. in a typical multihomed setup, MAC addresses learned behind a vES are advertised using the ESI associated with the vES (i.e., the vESI). EVPN aliasing and mass-withdraw operations are conducted with respect to the vES identifier. specifically, the Ethernet A-D routes for these operations are advertised using the vESI instead of the ESI 💜

for the PBB-EVPN solution, the main change is with respect to the B-MAC address assignment 🦋 it's done in a similar way to what is described in Section 6.2.1.1 of [RFC7623], with these refinements:

  • one shared B-MAC address SHOULD be used per PE for the single-homed vESes. in other words, a single B-MAC is shared for all single-homed vESes on that PE.
  • one shared B-MAC address SHOULD be used per PE, per physical port (e.g., ENNI) for the Single-Active vESes. in other words, a single B-MAC is shared for all Single-Active vESes that share the same ENNI.
  • one shared B-MAC address MAY be used for all Single-Active vESes on that PE.
  • one B-MAC address SHOULD be used per set of EVCs representing an All-Active vES. in other words, a single B-MAC address is used per vES for All-Active scenarios.
  • a single B-MAC address MAY also be used per vES, per PE for Single-Active scenarios.

4.1. EVPN DF Election for vES 👑

the service carving procedures for vESes are almost the same as the procedures in Section 8.5 of [RFC7432] and in [RFC8584], except that ES is replaced with vES 🔄

for the sake of clarity and completeness, the default DF election procedure of [RFC7432] is repeated below with the necessary changes 📝

  1. when a PE discovers the vESI or is configured with the vESI associated with its attached vES, it advertises an ES route with the associated ES-Import Route Target extended community attribute.
  2. the PE then starts a timer (default value = 3 seconds) to allow the reception of ES routes from other PE nodes connected to the same vES ⏲️ this timer value MUST be the same across all PEs connected to the same vES.
  3. when the timer expires, each PE builds an ordered list of the IP addresses of all the PE nodes connected to the vES (including itself), in increasing numeric value. each IP address in this list is extracted from the Originator Router's IP address field of the advertised ES route. every PE is then given an ordinal indicating its position in the ordered list, starting with 0 as the ordinal for the PE with the numerically lowest IP address. the ordinals are used to determine which PE node will be the DF for a given EVPN instance on the vES using the following rule: assuming a redundancy group of N PE nodes, the PE with ordinal i is the DF for an EVPN instance with an associated Ethernet Tag value of V when (V mod N) = i. it's worth noting that using the Originator Router's IP address field in the ES route to get the PE IP address needed for the ordered list allows a CE to be multihomed across different ASes, if such need ever arises 🌍
  4. the PE that is elected as a DF for a given EVPN instance will unblock traffic for that EVPN instance 🚪 note that the DF PE unblocks all traffic in both ingress and egress directions for Single-Active vESes and unblocks multi-destination in the egress direction for All-Active multihomed vESes. all non-DF PEs block all traffic in both ingress and egress directions for Single-Active vESes and block multi-destination traffic in the egress direction for All-Active vESes.

in case of an EVC failure, the affected PE withdraws its corresponding ES route if there are no more EVCs associated to the vES in the PE 🙈 this re-triggers the DF election procedure on all the PEs in the redundancy group. for PE node failure, or upon PE commissioning or decommissioning, the PEs re-trigger the DF election procedure across all affected vESes 🔁 in a Single-Active scenario, when a service moves from one PE in the redundancy group to another PE because of DF re-election, the PE (which ends up being the elected DF for the service) MUST trigger a MAC address flush notification towards the associated vES if the multihoming device is a bridge or the multihoming network is an Ethernet bridged network 🌉

for LSP-based and PW-based vES, the non-DF PE SHOULD signal PW-status 'standby' to the Aggregation PE (e.g., AG1 and AG2 in Figure 2) 📡 and a new DF PE MAY send a Label Distribution Protocol (LDP) MAC withdraw message as a MAC address flush notification. it's worth noting that the PW-status is signaled for the scenarios where there is a one-to-one mapping between EVI (EVPN instance) and the PW 💜

4.2. Grouping and Route Coloring for vES 🌈

physical ports (e.g., ENNI) that aggregate many EVCs get 'colored' to enable the grouping schemes described below 🎨

by default, the MAC address of the corresponding port (e.g., ENNI) is used to represent the 'color' of the port, and the EVPN Router's MAC Extended Community defined in [RFC9135] is used to signal this color 💄

the difference between coloring mechanisms for EVPN and PBB-EVPN is that the extended community is advertised with the Ethernet A-D per ES route for EVPN, whereas the extended community is advertised with the B-MAC route for PBB-EVPN 🧐

the subsequent sections detailing Grouping of Ethernet A-D per ES routes and Grouping of B-MAC addresses are essential for addressing port failure handling, as discussed in Sections 5.3, 5.4, and 5.5 🛟

4.2.1. EVPN Route Coloring for vES 🎨

when a PE discovers the vESI or is configured with the vESI associated with its attached vES, an ES route and an Ethernet A-D per ES route are generated using the vESI identifier ✨

these ES and Ethernet A-D per ES routes specific to each vES are colored with an attribute representing their association to a physical port (e.g., ENNI) 🌸

the corresponding port 'color' is encoded in the EVPN Router's MAC Extended Community defined in [RFC9135] and advertised along with the ES and Ethernet A-D per ES routes for this vES. the color (which is the MAC address of the port) MUST be unique 🔑

the PE also constructs a special Grouping Ethernet A-D per ES route that represents all the vESes associated with the port (e.g., ENNI) 🧺 the corresponding port 'color' is encoded in the ESI field. for this encoding, Type 3 ESI (Section 5 of [RFC7432]) is used with the MAC field set to the color (MAC address) of the port and the 3-octet local discriminator field set to 0xFFFFFF 🔢

the ESI label extended community (Section 7.5 of [RFC7432]) is not relevant to the Grouping Ethernet A-D per ES route 🙅‍♀️ the label value is not used for encapsulating Broadcast, Unknown Unicast, and Multicast (BUM) packets for any split-horizon function. the ESI label extended community MUST NOT be added to the Grouping Ethernet A-D per ES route and MUST be ignored on receiving the PE 🚫

the Grouping Ethernet A-D per ES route is advertised with a list of Route Targets corresponding to the affected service instances. if the number of associated Route Targets exceeds the capacity of a single route, multiple Grouping Ethernet A-D per ES routes are advertised accordingly as specified in Section 8.2 of [RFC7432] 📦

4.2.2. PBB-EVPN Route Coloring for vES 💎

in PBB-EVPN, particularly when there are large numbers of service instances (i.e., I-SIDs) associated with each EVC, the PE device MAY assign a color attribute to each vES B-MAC route, indicating their association with a physical port (e.g., an ENNI) 🪩

the corresponding port 'color' is encoded in the EVPN Router's MAC Extended Community defined in [RFC9135] and advertised along with the B-MAC for this vES in PBB-EVPN 🎀

the PE MAY also construct a special Grouping B-MAC route that represents all the vESes associated with the port (e.g., ENNI). the corresponding port 'color' is encoded directly into this special Grouping B-MAC route 🧁

5. Failure Handling and Recovery ✨

okay so there are a bunch of failure scenarios we need to think about, bestie 🫶

A: CE uplink port failure B: Ethernet Access Network failure C: PE access-facing port or link failure D: PE node failure E: PE isolation from IP/MPLS network

the solutions in [RFC7432], [RFC7623], and [RFC8214] already protect you from these failures, as described in those references 💅. but here's the twist: once you add vESes, there's one extra failure that didn't exist before, which is a single EVC going down 😭. to handle vES failures, you have to monitor EVCs or PWs independently. and when something fails or comes back, you have to run the right DF election and recovery steps, no skipping 🔥.

[RFC7023] is what we use to monitor EVCs. when an EVC fails, the DF election procedure from Section 4.1 kicks in ⭐. for PBB-EVPN, you need some extra stuff to make the failure and recovery procedures from [RFC7623] meet these requirements. we'll get into those extensions in the next section 🦋.

[RFC4377] and [RFC6310] are used to monitor the status of the LSPs and/or PWs tied to a vES 🌸.

                         B            D
                         ||           ||
                         \/           \/
                       +-----+
          +-----+      |     |       +---+
          | CE1 |EVC2--0=====0--ENNI1|   |   +-------+
          +-----+      |    =0--ENNI1|PE1|---|       |  +---+  +---+
          Cust. A      |   / |       |   |   |IP/MPLS|--|PE3|--|CE4|
          +-----+      |  /  |       +---+   |Network|  |   |  +---+
          |     |EVC2--0==   |               |       |  +---+
          | CE2 |      |     |       +---+   |       |
          |     |EVC3--0=====0--ENNI2|PE2|---|       |
          +-----+      |     |       |   |   +-------+
                       +-----+       +---+
                 /\                /\     /\
                 ||                ||     ||
                 A                 C      E

Figure 3: Failure Scenarios A, B, C, D, and E 🎀

5.1. EVC Failure Handling for Single-Active vES in EVPN

okay so in [RFC7432], when a DF PE that's connected to a Single-Active multihomed ES loses connectivity to the segment (link or port failure), it tells the remote PEs to invalidate all the MAC addresses tied to that ES 💀. it does this with a mass-withdraw message, by withdrawing the Ethernet A-D per ES route 🧚. worth knowing: for dual-homing where there's only a single backup path, the remote PEs can skip MAC invalidating, because they can just update the next hop for the affected MAC entries to the backup path, following the procedure in Section 8.2 of [RFC7432] 🌈.

now if an EVC failure hits a single vES, you use this same EVPN procedure 💜. the mass withdraw is carried by withdrawing the Ethernet A-D per vES route that has the vESI for the failed EVC in it ✨. when the remote PEs get this, they do the same things outlined in Section 8.2 of [RFC7432] 🫶.

5.2. EVC Failure Handling for a Single-Active vES in PBB-EVPN

in [RFC7432], when a PE connected to a Single-Active ES loses connectivity to the segment (link or port failure), it tells the remote PE to flush all the C-MAC addresses tied to that ES 🙈. it does this by updating the advertised B-MAC route's MAC Mobility extended community 🦋.

now, if an EVC failure hits a single vES and you use the PBB-EVPN procedure above, it's giving overkill 😭. one physical port can carry a ton of EVCs (and their vESes), so updating the B-MAC for that port with the MAC Mobility extended community would flush C-MACs not just for the broken EVC but for every other EVC on that port too 💀. yikes.

so to shrink the C-MAC flushing down to only the impacted service instances (the ones hit by the EVC failure), PBB-EVPN C-MAC flushing has to work per service instance, meaning per I-SID 💖. [RFC9541] introduces a B-MAC/I-SID route, where the existing PBB-EVPN B-MAC route gets changed to carry an I-SID in the "Ethernet Tag ID" field instead of a NULL value 🌸. to the receiving PE, that field means "flush all C-MAC addresses tied to this I-SID for this B-MAC" 🧚. this per-I-SID C-MAC flushing mechanism SHOULD be used when an EVC failure impacts a vES 🔥. since an EVC usually maps to a single broadcast domain (and so a single service instance), the affected PE only needs to advertise one B-MAC/I-SID route 🎀. but if the failed EVC carries multiple VLANs, each with its own broadcast domain, then the affected PE has to advertise multiple B-MAC/I-SID routes, meaning one route per VLAN (broadcast domain), which is one route per I-SID 👑. each B-MAC/I-SID route basically tells the remote PEs to flush C-MACs for the advertised B-MAC, but only for the advertised I-SID ✨.

the B-MAC/I-SID C-MAC flushing works fine when there are only a few VLANs (I-SIDs) per EVC 💅. but if the number of I-SIDs tied to a failed EVC is large, then it is RECOMMENDED to assign a B-MAC per vES, and when an EVC fails, the affected PE just withdraws that B-MAC message to the other PEs 🌈.

5.3. Port Failure Handling for Single-Active vESes in EVPN

when lots of EVCs are aggregated on one physical port on a PE, and each EVC corresponds to a vES, then a port failure hits all of those EVCs and their vESes at once 😭. if the number of EVCs tied to the Single-Active vESes for that port is in the thousands, then thousands of service instances are impacted 💀. so the failure propagation in BGP has to handle all of those impacted service instances 🔥. to make that happen, the following extensions get added to the baseline EVPN mechanism ✨:

  1. The PE MAY color each Ethernet A-D per ES route for a given vES, as described in Section 4.2.1 💖. the PE SHOULD use the MAC physical port by default 🌸. the receiving PEs take note of this color and create a list of vESes for that color 🧚.
  2. The PE MAY advertise a special Grouping Ethernet A-D per ES route for that color, which represents all the vESes associated with the port 🦋.
  3. Upon a port failure (e.g., an ENNI failure), the PE MAY send a mass-withdraw message by withdrawing the Grouping Ethernet A-D per ES route 🫶.
  4. When this message is received, the remote PE MAY detect the special vES mass-withdraw message by identifying the Grouping Ethernet A-D per ES route 🌈. the remote PEs MAY then access the list of vESes created in item 1 for the specified color, and locally start the MAC address invalidating procedures for each vES in that list 👑.

in scenarios where a logical ENNI is used, the above procedure works the exact same way 🎀. the logical ENNI is represented by a Grouping Ethernet A-D per ES route, where the Type 3 ESI and the 6 bytes used in the ENNI's ESI MAC address field are used as a color for the vESes, as described above and in Section 4.2.1 💜.

5.4. Port Failure Handling for Single-Active vESes in PBB-EVPN

same vibe as before: when lots of EVCs are aggregated on one physical port on a PE, and each EVC corresponds to a vES, a port failure takes out all of those EVCs and their vESes 😭. if the number of EVCs tied to the Single-Active vESes for that port is in the thousands, then thousands of service instances (I-SIDs) are impacted 💀. in those failure scenarios, there are two MAC flushing mechanisms from [RFC7623] you can use 🦋.

  1. If the MAC address of the physical port is used for PBB encapsulation as the B-MAC SA, then upon the port failure, the PE MUST use the EVPN MAC route withdrawal message to signal the flush 🔥.
  2. If the PE's shared MAC address is used for PBB encapsulation as the B-MAC SA, then upon the port failure, the PE MUST re-advertise this MAC route with the MAC Mobility extended community to signal the flush 🌸.

the first method is recommended because it shrinks the scope of flushing the most 💖.

as mentioned above, advertising the extended community along with the B-MAC route for coloring purposes is optional, and it's only recommended when there are many vESes per physical port and each vES is tied to a very large number of service instances (aka a large number of I-SIDs) 🧚.

if there are large numbers of service instances (I-SIDs) tied to each EVC, and there's a B-MAC assigned per vES like the section above recommends, then to handle port failure efficiently, the following extensions get added to the baseline PBB-EVPN mechanism ✨:

  1. Each vES MAY be colored with a MAC address representing the physical port, just like the coloring mechanism for EVPN 🌈. in other words, each B-MAC representing a vES is advertised with the 'color' of the physical port per Section 4.2.2 🫶. the receiving PEs take note of this color being advertised along with the B-MAC route, and for each color they create a list of vESes associated with it 👑.
  2. The PE MAY advertise a special Grouping B-MAC route for that color (consisting of a port MAC address by default), which represents all the vESes associated with the port 🎀.
  3. Upon a port failure (e.g., ENNI failure), the PE MAY send a mass-withdraw message by withdrawing the Grouping B-MAC route 💜.
  4. When this message is received, the remote PE MAY detect the special vES mass-withdraw message by identifying the Grouping B-MAC route 🦋. the remote PEs MAY then access the list created in (1) above for the specified color and flush all C-MACs associated with the failed physical port 💅.

5.5. Fast Convergence in EVPN and PBB-EVPN ✨

okay so picture this 🎀: a bunch of EVCs are all squished through one physical port on a PE, and every single EVC corresponds to its own vES. when that port fails 💀, it takes out ALL the EVCs and ALL their vESes at the same time. two things have to happen:

  • For EVPN, start the mass-withdraw procedure for all vESes tied to the failed port so the MACs get invalidated 🫧. For PBB-EVPN, flush all C-MACs tied to the failed port across all vESes and the impacted I-SIDs.
  • Run DF election for all impacted vESes tied to the failed port 👑.

Section 5.3 already explains how to do a mass withdraw for all the affected vESes and invalidate MACs using a single BGP withdrawal of the Grouping Ethernet A-D per ES route 💖. Section 5.4 explains how to flush only the C-MAC addresses tied to the failed physical port (aka optimum C-MAC flushing) and, optionally, withdraw a Grouping B-MAC route 🦋.

This section is all about doing DF election in the most optimal way 🧠. You trigger DF election for all the impacted vESes (which can be a LOT!) among the participating PEs with one BGP message, instead of blasting out a huge pile of BGP messages, one per vES 🙅‍♀️. This section assumes the MAC flushing mechanism from Section 5.4 and route coloring are being used.

                 +-----+
      +----+     |     |       +---+
      | CE1|AC1--0=====0--ENNI1|   |  +-------+
      |    |AC2--0     |       |PE1|--|       |
      +----+     |\  ==0--ENNI2|   |  |       |
                 | \/  |       +---+  |       |
                 | /\  |              |IP/MPLS|
      +----+     |/  \ |       +---+  |Network|   +---+  +---+
      | CE2|AC4--0    =0--ENNI3|   |  |       |---|PE4|--|CE4|
      |    |AC4--0=====0--ENNI3|PE2|--|       |   +---+  +---+
      +----+     | ====0--ENNI3|   |  |       |
                 |/    |       +---+  |       |
                 0     |              |       |
      +----+    /|     |       +---+  |       |
      | CE3|AC5- |     |       |PE3|--|       |
      |    |AC6--0=====0--ENNI4|   |  +-------+
      +----+     |     |       +---+
                 +-----+

            Figure 4: Fast Convergence Upon ENNI Failure

As discussed in Section 4.2, it is highly desirable to have a mass-withdraw mechanism similar to the one in [RFC7432]. Although such an optimization is desirable, it is OPTIONAL 🌈. If the optimization is implemented, the following procedures are used:

  1. When a vES is configured, the PE advertises the ES route for this vES with a color that corresponds to the associated physical port 🎨.
  2. All receiving PEs within the redundancy group record this color and compile a list of vESes associated with it 📝.
  3. Additionally, the PE advertises a Grouping Ethernet A-D per ES route for EVPN, and a Grouping B-MAC route for PBB-EVPN, which corresponds to the color and vES grouping 🍓.
  4. In the event of a port failure, such as an ENNI failure, the PE withdraws the previously advertised Grouping Ethernet A-D per ES route or Grouping B-MAC route associated with the failed port. The PE should prioritize sending these Grouping route withdrawal messages over the withdrawal of individual vES routes affected by the failure 🏃‍♀️. For instance, as depicted in Figure 4, when the physical port associated with ENNI3 fails on PE2, it withdraws the previously advertised Grouping Ethernet A-D per ES route. Upon receiving this withdrawal message, other multihoming PEs (such as PE1 and PE3) recognize that the vESes associated with CE1 and CE3 are impacted, based on the associated color, and thus initiate the DF election procedure for these vESes. Furthermore, upon receiving this withdrawal message, remote PEs (such as PE4) initiate the failover procedure for the vESes associated with CE1 and CE3 and switch to the other PE for each vES redundancy group 🔄.
  5. On reception of a Grouping Ethernet A-D per ES route or Grouping B-MAC route withdrawal, other PEs in the redundancy group initiate DF election procedures across all their affected vESes 💅.
  6. The PE with the physical port failure (ENNI failure) sends a vES route withdrawal for every impacted vES. Upon receiving these messages, the other PEs clear up their BGP tables 🧹. It should be noted that the vES route withdrawal messages are not used for executing DF election procedures by the receiving PEs when a Grouping Ethernet A-D per ES route or Grouping B-MAC route withdrawal has been previously received 📌.

6. Security Considerations 🔐

All the security considerations in [RFC7432] and [RFC7623] apply directly to this document, because this document leverages the control and data plane procedures described in those documents. Please take these seriously, bestie 😭.

This document does not introduce any new security considerations beyond that of [RFC7432] and [RFC7623], because advertisements and the processing of ES routes for vES in this document follow that of physical ES in those RFCs 🛡️.

7. IANA Considerations 🌸

This document has no IANA actions. Nothing to register here, no cap 🙌.

8. References 📚

8.1. Normative References

8.2. Informative References

Acknowledgements 🫶

The authors would like to thank Mei Zhang, Jose Liste, and Luc André Burdet for their reviews of this document and their feedback. We love a supportive squad 💜.

Authors' Addresses 💌

Ali Sajassi Cisco Systems Email: sajassi@cisco.com

Patrice Brissette Cisco Systems Email: pbrisset@cisco.com

Rick Schell Independent Email: rick_schell@outlook.com

John E. Drake Independent Email: je_drake@yahoo.com

Jorge Rabadan Nokia Email: jorge.rabadan@nokia.com

📦 where this page comes from