RFC 8365

A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)

📅 published March 2018, from ietf

RFC 8365: A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN) 💖

Okay bestie, here's the quick title-block tea 🫶 This one is from the IETF (Internet Engineering Task Force), it's a Standards Track doc (so it's an actual Internet standard, no cap 🔥), and it came out in March 2018. The ISSN is 2070-1721 ✨. The authors are A. Sajassi, Ed. (Cisco), J. Drake, Ed. (Juniper), N. Bitar (Nokia), R. Shekhar (Juniper), J. Uttaro (AT&T), and W. Henderickx (Nokia), and the "Ed." means they're the editors 💅. The title block doesn't list anything this RFC obsoletes or updates, so it's not replacing or patching any older doc 🦋

Abstract

Okay so this doc is all about how Ethernet VPN (EVPN) can be used as a Network Virtualization Overlay (NVO) solution 🌸 It also checks out the different tunnel encapsulation options over IP and how each one changes the EVPN control plane and procedures. The encapsulations it looks at are Virtual Extensible LAN (VXLAN), Network Virtualization using Generic Routing Encapsulation (NVGRE), and MPLS over GRE 💜

It's also applicable to Generic Network Virtualization Encapsulation (GENEVE), but ngl that needs some extra work, and that work lives in a separate doc 🙈 The doc also brings new multihoming procedures for split-horizon filtering and mass withdrawal, plus EVPN route constructions for VXLAN/NVGRE encapsulations. Last but not least, it covers Autonomous System Border Router (ASBR) procedures for multihoming Network Virtualization Edge (NVE) devices. It's giving full main character energy 👑

Status of This Memo

This is an Internet Standards Track document. ✨

It's a product of the Internet Engineering Task Force (IETF) 🌈 It reflects the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). If you want more on Internet Standards, check out Section 2 of RFC 7841 🎀

Info about the current status of this doc, any errata, and how to send feedback is at https://www.rfc-editor.org/info/rfc8365 🍓

Copyright Notice

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info), as they stand on the publication date. Please read these carefully, bestie, because they spell out your rights and restrictions for this doc 🫶 Any Code Components pulled out of this doc have to include the Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as the Simplified BSD License describes 🔥

Table of Contents 📜

   1. Introduction ....................................................4
   2. Requirements Notation and Conventions ...........................5
   3. Terminology .....................................................5
   4. EVPN Features ...................................................7
   5. Encapsulation Options for EVPN Overlays .........................8
      5.1. VXLAN/NVGRE Encapsulation ..................................8
           5.1.1. Virtual Identifiers Scope ...........................9
           5.1.2. Virtual Identifiers to EVI Mapping .................11
           5.1.3. Constructing EVPN BGP Routes .......................13
      5.2. MPLS over GRE .............................................15
   6. EVPN with Multiple Data-Plane Encapsulations ...................15
   7. Single-Homing NVEs - NVE Residing in Hypervisor ................16
      7.1. Impact on EVPN BGP Routes & Attributes for VXLAN/NVGRE ....16
      7.2. Impact on EVPN Procedures for VXLAN/NVGRE Encapsulations ..17
   8. Multihoming NVEs - NVE Residing in ToR Switch ..................18
      8.1. EVPN Multihoming Features .................................18
           8.1.1. Multihomed ES Auto-Discovery .......................18
           8.1.2. Fast Convergence and Mass Withdrawal ...............18
           8.1.3. Split-Horizon ......................................19
           8.1.4. Aliasing and Backup Path ...........................19
           8.1.5. DF Election ........................................20
      8.2. Impact on EVPN BGP Routes and Attributes ..................20
      8.3. Impact on EVPN Procedures .................................20
           8.3.1. Split Horizon ......................................21
           8.3.2. Aliasing and Backup Path ...........................22
           8.3.3. Unknown Unicast Traffic Designation ................22
   9. Support for Multicast ..........................................23
   10. Data-Center Interconnections (DCIs) ...........................24
      10.1. DCI Using GWs ............................................24
      10.2. DCI Using ASBRs ..........................................24
           10.2.1. ASBR Functionality with Single-Homing NVEs ........25
           10.2.2. ASBR Functionality with Multihoming NVEs ..........26
   11. Security Considerations .......................................28
   12. IANA Considerations ...........................................29
   13. References ....................................................29
      13.1. Normative References .....................................29
      13.2. Informative References ...................................30
   Acknowledgements ..................................................32
   Contributors ......................................................32
   Authors' Addresses ................................................33

1. Introduction ✨

Okay so this doc spells out how Ethernet VPN (EVPN) [RFC7432] can work as a Network Virtualization Overlay (NVO) solution 💖 It also explores the different tunnel encapsulation options over IP and how they affect the EVPN control plane and procedures. The encapsulations analyzed here are Virtual Extensible LAN (VXLAN) [RFC7348], Network Virtualization using Generic Routing Encapsulation (NVGRE) [RFC7637], and MPLS over Generic Routing Encapsulation (GRE) [RFC4023] 🦋

GENEVE is also in scope [GENEVE], but it needs some incremental work, which is covered in a separate doc [EVPN-GENEVE] 🙈 The doc additionally brings new multihoming procedures for split-horizon filtering and mass withdrawal. It also defines EVPN route constructions for VXLAN/NVGRE encapsulations, plus Autonomous System Border Router (ASBR) procedures for multihoming Network Virtualization Edge (NVE) devices 💅

So what even is an NVO? Bestie, it's a solution for multi-tenant data centers, especially ones with virtualized hosts like Virtual Machines (VMs) or other virtual workloads 🌸 According to [RFC7364], the key requirements for an NVO solution are these:

Isolation of network traffic per tenant
Support for a large number of tenants (tens or hundreds of thousands)
Extension of Layer 2 (L2) connectivity among different VMs belonging to a given tenant segment (subnet) across different Points of Delivery (PoDs) within a data center or between different data centers
Allowing a given VM to move between different physical points of attachment within a given L2 segment

The underlay network for NVO solutions is assumed to provide IP connectivity between NVO endpoints 🌈

This doc describes how EVPN can be used as an NVO solution and looks at how EVPN functions and procedures apply to it 🍓 Specifically, it covers the tunnel encapsulation options for EVPN over IP and how they affect the EVPN control plane, plus procedures for two main scenarios:

(a) single-homing NVEs - when an NVE resides in the hypervisor, and
(b) multihoming NVEs - when an NVE resides in a Top-of-Rack (ToR) device.

The encapsulation options for EVPN overlays analyzed in this doc are:

VXLAN and NVGRE
MPLS over GRE

Before we get into the different encapsulation options for EVPN over IP, it's important to know the EVPN solution's main features, how those features are currently supported, and any impact the encapsulation has on them 🎀

2. Requirements Notation and Conventions 📝

Okay this part is the rulebook for the capital letters 👑 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, just like shown here 💖 So if you see them in ALL CAPS, that's the official BCP 14 vibe, and if they're lowercase, they're just regular words, iykyk 🙈

3. Terminology 💜

Most of the terminology used in this doc comes from [RFC7432] and [RFC7365] 🦋 Here's the glossary, bestie:

VXLAN: Virtual Extensible LAN
GRE: Generic Routing Encapsulation
NVGRE: Network Virtualization using Generic Routing Encapsulation
GENEVE: Generic Network Virtualization Encapsulation
PoD: Point of Delivery
NV: Network Virtualization
NVO: Network Virtualization Overlay
NVE: Network Virtualization Edge
VNI: VXLAN Network Identifier
VSID: Virtual Subnet Identifier (for NVGRE)
I-SID: Service Instance Identifier
EVPN: Ethernet VPN
EVI: EVPN Instance. An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN 🌸
MAC-VRF: A Virtual Routing and Forwarding table for Media Access Control (MAC) addresses on a PE 🧚
IP-VRF: A Virtual Routing and Forwarding table for Internet Protocol (IP) addresses on a PE 🧚
ES: Ethernet Segment. When a customer site (device or network) is connected to one or more PEs via a set of Ethernet links, then that set of links is referred to as an 'Ethernet segment' 🔥
Ethernet Segment Identifier (ESI): A unique non-zero identifier that identifies an Ethernet segment is called an 'Ethernet Segment Identifier' ✨
Ethernet Tag: An Ethernet tag identifies a particular broadcast domain, e.g., a VLAN. An EVPN instance consists of one or more broadcast domains 🌈
PE: Provider Edge
Single-Active Redundancy Mode: When only a single PE, among all the PEs attached to an ES, is allowed to forward traffic to/from that ES for a given VLAN, then the Ethernet segment is defined to be operating in Single-Active redundancy mode 🦋
All-Active Redundancy Mode: When all PEs attached to an Ethernet segment are allowed to forward known unicast traffic to/from that ES for a given VLAN, then the ES is defined to be operating in All-Active redundancy mode 👑
PIM-SM: Protocol Independent Multicast - Sparse-Mode
PIM-SSM: Protocol Independent Multicast - Source-Specific Multicast
BIDIR-PIM: Bidirectional PIM

Part 1 of 7 is done, bestie 🦋 More parts are coming with the rest of the doc, so stay tuned 💖

4. EVPN Features ✨

okay bestie, let's talk about why EVPN [RFC7432] is such a main character 💅. It was originally built to meet the requirements in [RFC7209], and it has a bunch of attributes that go straight at control-plane scaling and ease of deployment. literally iconic 😭🫶

Control-plane information gets distributed with BGP, and broadcast and multicast traffic goes out over a shared multicast tree or with ingress replication. 🦋
Control-plane learning is used for MAC (and IP) addresses instead of data-plane learning. The latter needs flooding of unknown unicast and Address Resolution Protocol (ARP) frames, whereas the former doesn't need any flooding at all. no cap, that's a huge win 🌸
A Route Reflector (RR) turns a full mesh of BGP sessions among PE devices into a single BGP session between a PE and the RR. 🧚 And RR hierarchy can be leveraged to scale the number of BGP routes on the RR. so scalable 💖
Auto-discovery via BGP is used to find PE devices in a given VPN, PE devices in a given redundancy group, tunnel encapsulation types, multicast tunnel types, multicast members, etc. it's giving organized ✨
All-Active multihoming is used. This lets a given Customer Edge (CE) device have multiple links to multiple PEs, and traffic to and from that CE fully utilizes all of those links. 🔥
When a link between a CE and a PE fails, the PEs for that EVI get notified through the withdrawal of a single EVPN route. That lets those PEs remove the withdrawing PE as a next hop for every MAC address tied to the failed link. This is termed "mass withdrawal". 💀 one route, whole cleanup, periodt
BGP route filtering and constrained route distribution are leveraged so the control-plane traffic for a given EVI only goes to the PEs in that EVI. 🎀
When an IEEE 802.1Q [IEEE.802.1Q] interface is used between a CE and a PE, each VLAN ID (VID) on that interface can be mapped onto a bridge table (for up to 4094 such bridge tables). All of these bridge tables may be mapped onto a single MAC-VRF in the case of VLAN-aware bundle service. 🌈
VM Mobility mechanisms make sure all PEs in a given EVI know which ES a given VM is currently associated with, identified by its MAC and IP addresses. 🙈
RTs let the operator (or customer) define a spectrum of logical network topologies, including mesh, hub and spoke, and extranets (e.g., a VPN whose sites are owned by different enterprises), without needing proprietary software or the help of other virtual or physical devices. 👑

Because the design goal for NVO is millions of instances per common physical infrastructure, the scaling properties of the control plane for NVO are extremely important. EVPN and the extensions described herein are designed with this level of scalability in mind. ✨🌸

5. Encapsulation Options for EVPN Overlays 🦋

5.1. VXLAN/NVGRE Encapsulation 💖

Both VXLAN and NVGRE are technologies that provide a data plane encapsulation, used to carry a packet across the common physical IP infrastructure between Network Virtualization Edges (NVEs), e.g., VXLAN Tunnel End Points (VTEPs) in a VXLAN network. Both include the identifier of the specific NVO instance in each packet: VNI in VXLAN and VSID in NVGRE. For the rest of this document we use VNI to represent the NVO instance, and VSID can equally be used if the encapsulation is NVGRE, unless stated otherwise. 🧚

Note that a PE is equivalent to an NVE/VTEP. 💅

VXLAN encapsulation is based on UDP, with an 8-byte header following the UDP header. VXLAN provides a 24-bit VNI, which typically maps one-to-one to the tenant VID, as described in [RFC7348]. In this scenario, the ingress VTEP does not include an inner VLAN tag on the encapsulated frame, and the egress VTEP discards frames that have an inner VLAN tag. This mode of operation in [RFC7348] maps to VLAN-Based Service in [RFC7432], where a tenant VID gets mapped to an EVI. 🎀

VXLAN also lets you include an inner VLAN tag in the encapsulated frame, if it's explicitly configured at the VTEP. This mode can map to VLAN Bundle Service in [RFC7432] because all of the tenant's tagged frames map to a single bridge table / MAC-VRF, and the inner VLAN tag is not used for lookup by the disposition PE when doing VXLAN decapsulation, as described in Section 6 of [RFC7348]. 🌈

[RFC7637] encapsulation is based on GRE encapsulation, and it mandates the inclusion of the optional GRE Key field, which carries the VSID. There is a one-to-one mapping between the VSID and the tenant VID, as described in [RFC7637]. Including an inner VLAN tag is prohibited. This mode of operation in [RFC7637] maps to VLAN Based Service in [RFC7432]. 🙈

As described in the next section, there is no change to the encoding of EVPN routes to support VXLAN or NVGRE encapsulation, except for using the BGP Encapsulation Extended Community to indicate the encapsulation type (e.g., VXLAN or NVGRE). However, there is potential impact to the EVPN procedures depending on where the NVE is located (i.e., in a hypervisor or ToR) and whether multihoming capabilities are required. 👑

5.1.1. Virtual Identifiers Scope 🌸

Although VNIs are defined as 24-bit globally unique values, there are scenarios where it's desirable to use a locally significant value for the VNI, especially in the context of a data-center interconnect. ✨

5.1.1.1. Data-Center Interconnect with Gateway 🦋

In the case where NVEs in different data centers need to be interconnected, and the NVEs need to use VNIs as globally unique identifiers within a data center, then a Gateway (GW) needs to be employed at the edge of the data-center network (DCN). This is because the Gateway provides the functionality of translating the VNI when crossing network boundaries, which may align with operator span-of-control boundaries. As an example, consider the network of Figure 1. Assume there are three network operators: one for each of the DC1, DC2, and WAN networks. The Gateways at the edge of the data centers are responsible for translating the VNIs between the values used in each DCN and the values used in the WAN. 💅

                         +--------------+
                         |              |
       +---------+       |     WAN      |       +---------+
+----+  |        +---+  +----+        +----+  +---+        |  +----+
|NVE1|--|        |   |  |WAN |        |WAN |  |   |        |--|NVE3|
+----+  |IP      |GW |--|Edge|        |Edge|--|GW | IP     |  +----+
+----+  |Fabric  +---+  +----+        +----+  +---+ Fabric |  +----+
|NVE2|--|         |       |              |       |         |--|NVE4|
+----+  +---------+       +--------------+       +---------+  +----+

|<------ DC 1 ------>                          <------ DC2  ------>|

          Figure 1: Data-Center Interconnect with Gateway

5.1.1.2. Data-Center Interconnect without Gateway 💜

In the case where NVEs in different data centers need to be interconnected, and the NVEs need to use locally assigned VNIs (e.g., similar to MPLS labels), there may be no need to employ Gateways at the edge of the DCN. More specifically, the VNI value used by the transmitting NVE is allocated by the NVE that is receiving the traffic (in other words, this is similar to a "downstream-assigned" MPLS label). This lets the VNI space be decoupled between different DCNs without needing a dedicated Gateway at the edge of the data centers. This topic is covered in Section 10.2. 🧚

                          +--------------+
                          |              |
          +---------+     |     WAN      |    +---------+
  +----+  |         |   +----+        +----+  |         |  +----+
  |NVE1|--|         |   |ASBR|        |ASBR|  |         |--|NVE3|
  +----+  |IP Fabric|---|    |        |    |--|IP Fabric|  +----+
  +----+  |         |   +----+        +----+  |         |  +----+
  |NVE2|--|         |     |              |    |         |--|NVE4|
  +----+  +---------+     +--------------+    +---------+  +----+

  |<------ DC 1 ----->                        <---- DC2  ------>|

           Figure 2: Data-Center Interconnect with ASBR

5.1.2. Virtual Identifiers to EVI Mapping 🎀

Just like in [RFC7432], where two options existed for mapping broadcast domains (represented by VLAN IDs) to an EVI, when the EVPN control plane is used with VXLAN (or NVGRE encapsulation), there are also two options for mapping broadcast domains represented by VXLAN VNIs (or NVGRE VSIDs) to an EVI: 🌈

   Option 1: A Single Broadcast Domain per EVI

In this option, a single Ethernet broadcast domain (e.g., subnet) represented by a VNI is mapped to a unique EVI. This corresponds to the VLAN-Based Service in [RFC7432], where a tenant-facing interface, logical interface (e.g., represented by a VID), or physical interface gets mapped to an EVI. As such, a BGP Route Distinguisher (RD) and Route Target (RT) are needed per VNI on every NVE. The advantage of this model is that it allows the BGP RT constraint mechanisms to be used to limit the propagation and import of routes to only the NVEs that are interested in a given VNI. The disadvantage of this model may be the provisioning overhead if the RD and RT are not derived automatically from the VNI. 🦋

In this option, the MAC-VRF table is identified by the RT in the control plane and by the VNI in the data plane. In this option, the specific MAC-VRF table corresponds to only a single bridge table. 💖

   Option 2: Multiple Broadcast Domains per EVI

In this option, multiple subnets, each represented by a unique VNI, are mapped to a single EVI. For example, if a tenant has multiple segments/subnets each represented by a VNI, then all the VNIs for that tenant are mapped to a single EVI; the EVI in this case represents the tenant and not a subnet. This corresponds to the VLAN-aware bundle service in [RFC7432]. The advantage of this model is that it doesn't require provisioning an RD/RT per VNI. However, this is a moot point compared to Option 1 where auto-derivation is used. The disadvantage of this model is that routes would be imported by NVEs that may not be interested in a given VNI. 🫶

In this option, the MAC-VRF table is identified by the RT in the control plane; a specific bridge table for that MAC-VRF is identified by the <RT, Ethernet Tag ID> in the control plane. In this option, the VNI in the data plane is sufficient to identify a specific bridge table. 🌸

5.1.2.1. Auto-Derivation of RT

Okay bestie, so to keep config simple 🎀, when you go with a single VNI per EVI, the RT used for EVPN can be auto-derived ✨. The RD can be auto-generated like [RFC7432] describes, and the RT can be auto-derived as we're about to see 💅.

A Gateway PE (the one in Figure 1) sits in both the DCN and WAN BGP sessions, so it's super important that the auto-derived RT values don't clash between DCNs and WANs, assuming both live in the same Autonomous System (AS) 😭. There are also times when both VXLAN and NVGRE encapsulations are needed inside the same DCN, and their VNIs are administered independently, which means the VNI spaces can overlap 🙈. To avoid RT space conflicts, the 6-byte RT values with a 2-octet AS number for DCNs can be auto-derived like this:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Global Administrator      |    Local Administrator        |
+-----------------------------------------------+---------------+
| Local Administrator (Cont.)   |
+-------------------------------+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Global Administrator      |A| TYPE| D-ID  | Service ID    |
+-----------------------------------------------+---------------+
|       Service ID (Cont.)      |
+-------------------------------+

The 6-octet RT field is made up of two sub-fields 🧚:

Global Administrator sub-field: 2 octets. This sub-field holds an AS number assigned by IANA <https://www.iana.org/assignments/as-numbers/>.
Local Administrator sub-field: 4 octets
- A: a single-bit field that tells you whether this RT is auto-derived:

0: auto-derived
1: manually derived

Type: a 3-bit field that identifies the space the other 3 bytes are defined in. These spaces are defined:

0 : VID (802.1Q VLAN ID)
1 : VXLAN
2 : NVGRE
3 : I-SID
4 : EVI
5 : dual-VID (QinQ VLAN ID)

D-ID: a 4-bit field that identifies the domain-id 🌸. The default domain-id is zero, which means only a single numbering space exists for a given technology. But if more than one number space exists for a given technology (for example, overlapping VXLAN spaces), then each of those number spaces needs to be identified by its own domain-id, starting from 1.
Service ID: this 3-octet field is set to VNI, VSID, I-SID, or VID.

Real talk: RT auto-derivation only works for 2-octet AS numbers 🙅‍♀️. For 4-octet AS numbers, the RT needs to be manually configured, because the 3-octet VNI fields can't fit inside the 2-octet local administrator field 😤.

5.1.3. Constructing EVPN BGP Routes

Okay so in EVPN, an MPLS label (for instance, one that identifies the forwarding table) gets distributed by the egress PE over the EVPN control plane 🦋. The ingress PE puts it in the MPLS header of a packet, and when the egress PE receives that packet, it uses the label for disposition. This is really similar to how an egress NVE uses the VNI, with one difference: an MPLS label has local significance, while a VNI typically has global significance 💜. So, to specifically support locally assigned VNIs, the MPLS Label1 field in the MAC/IP Advertisement route, the MPLS label field in the Ethernet A-D per EVI route, and the MPLS label field in the P-Multicast Service Interface (PMSI) Tunnel attribute of the Inclusive Multicast Ethernet Tag (IMET) route are used to carry the VNI 🫶. For the rest of this memo, those MPLS label fields will be called the VNI field. The VNI field is used for both local and global VNIs, and in either case the entire 24-bit field is used to encode the VNI value ✨.

For the VLAN-Based Service (a single VNI per MAC-VRF), the Ethernet Tag field in the MAC/IP Advertisement, Ethernet A-D per EVI, and IMET route MUST be set to zero, just like in the VLAN-Based Service in [RFC7432] 🍓.

For the VLAN-Aware Bundle Service (multiple VNIs per MAC-VRF, with each VNI tied to its own bridge table), the Ethernet Tag field in the MAC Advertisement, Ethernet A-D per EVI, and IMET route MUST identify a bridge table within a MAC-VRF. The set of Ethernet Tags for that EVI needs to be configured consistently on all PEs within that EVI 👑. For locally assigned VNIs, the value advertised in the Ethernet Tag field MUST be set to a VID, just like in the VLAN-aware bundle service in [RFC7432]. That setting must be done consistently on all PE devices participating in that EVI within a given domain 💅. For global VNIs, the value advertised in the Ethernet Tag field SHOULD be set to a VNI, as long as it matches the existing semantics of the Ethernet Tag. In other words, it identifies a bridge table within a MAC-VRF, and the set of VNIs is configured consistently on each PE in that EVI 🌈.

Heads up bestie: these Ethernet Tag rules are different for locally vs. globally assigned VNIs, so double-check which one you're running before you configure anything 😭.

To show which data-plane encapsulation type (VXLAN, NVGRE, MPLS, or MPLS in GRE) should be used, the BGP Encapsulation Extended Community defined in [RFC5512] gets included with all EVPN routes advertised by an egress PE. Those routes are MAC Advertisement, Ethernet A-D per EVI, Ethernet A-D per ESI, IMET, and Ethernet Segment 🦋. Five new values were assigned by IANA to extend the list of encapsulation types defined in [RFC5512]. They're listed in Section 11 ⭐.

The MPLS encapsulation tunnel type (also listed in Section 11) is needed so you can tell apart an advertising node that only supports non-MPLS encapsulations from one that supports both MPLS and non-MPLS encapsulations 🔥. An advertising node that only supports MPLS encapsulation doesn't need to advertise any encapsulation tunnel types. So if the BGP Encapsulation Extended Community isn't present, then either MPLS encapsulation or a statically configured encapsulation is assumed 🙈.

The Next Hop field of the MP_REACH_NLRI attribute of the route MUST be set to the IPv4 or IPv6 address of the NVE 🎀. The remaining fields in each route are set as per [RFC7432] 💖.

Note that this procedure (using the MPLS Label field to carry the VNI when a Tunnel Encapsulation Extended Community says a VNI is in use) lines up with what Section 8.2.2.2 of [TUNNEL-ENCAP] says ("When a Valid VNI has not been Signaled") 🧚.

5.2. MPLS over GRE

Okay so the EVPN data plane is modeled as an EVPN MPLS client layer sitting on top of an MPLS PSN tunnel server layer 🌸. Some EVPN functions (split-horizon, Aliasing, and Backup Path) are tied to the MPLS client layer. If MPLS over GRE encapsulation is used, the EVPN MPLS client layer can be carried over an IP PSN tunnel transparently 🦋. That means there's no impact on the EVPN procedures or the associated data-plane operation, which is so clean 💅.

[RFC4023] defines the standard for using MPLS over GRE encapsulation, and it can be used for this purpose ✨. However, when MPLS over GRE is used together with EVPN, it is recommended that the GRE key field be present and used to carry a 32-bit entropy value, but only if the P nodes can perform Equal-Cost Multipath (ECMP) hashing based on the GRE key 🔥. Otherwise, the GRE header SHOULD NOT include the GRE key field. The Checksum and Sequence Number fields MUST NOT be included, and the corresponding C and S bits in the GRE header MUST be set to zero 🙅‍♀️. A PE that can support this encapsulation SHOULD advertise its EVPN routes along with the Tunnel Encapsulation Extended Community indicating MPLS over GRE encapsulation, as described in the previous section 💜.

6. EVPN with Multiple Data-Plane Encapsulations

Bestie, the BGP Encapsulation Extended Community from [RFC5512] lets each NVE in a given EVI know which encapsulations every other NVE in that EVI supports 👑. So each NVE in an EVI can support multiple data-plane encapsulations ✨. An ingress NVE can only send a frame to an egress NVE if the set of encapsulations the egress NVE advertises has a non-empty intersection with the set the ingress NVE supports. Then it's up to the ingress NVE which encapsulation to pick from that intersection 🫶. (As noted in Section 5.1.3, if the BGP Encapsulation Extended Community isn't present, then the default MPLS encapsulation or a locally configured encapsulation is assumed 🌈.)

When a PE advertises multiple supported encapsulations, it MUST advertise encapsulations that use the same EVPN procedures, including the procedures for split-horizon filtering described in Section 8.3.1 💅. For example, VXLAN and NVGRE (or MPLS and MPLS over GRE) use the same EVPN procedures, so a PE can advertise both and support either one or both at the same time 🦋. However, a PE MUST NOT advertise VXLAN and MPLS encapsulations together, because (a) the MPLS field of EVPN routes is set to either an MPLS label or a VNI, but not both, and (b) some EVPN procedures (such as split-horizon filtering) are different for VXLAN/NVGRE and MPLS encapsulations 😭.

An ingress node that uses shared multicast trees for sending broadcast or multicast frames MAY maintain distinct trees for each different encapsulation type 🧚.

It's the responsibility of the operator of a given EVI to make sure that all of the NVEs in that EVI support at least one common encapsulation 🎀. If this condition is violated, it could result in service disruption or failure 💀. The BGP Encapsulation Extended Community gives you a way to detect when this condition is violated, but the actions to take are up to the operator and are outside the scope of this document 🙈.

7. Single-Homing NVEs - NVE Residing in Hypervisor

So when an NVE and its hosts/VMs are co-located in the same physical device (say, when they live on a server), the links between them are virtual and they typically share fate 🌸. That means the hosts/VMs are usually not multihomed. And if they are multihomed, that multihoming is purely a local matter for the server hosting the VMs and the NVEs, and it doesn't need to be "visible" to any other NVEs on other servers ✨. So it doesn't require any specific protocol mechanisms 💅. The most common case of this is when the NVE resides on the hypervisor 👑.

In the subsections that follow, we'll look at how EVPN procedures are impacted when the NVE resides on the hypervisor and VXLAN (or NVGRE) encapsulation is used 🦋.

7.1. Impact on EVPN BGP Routes & Attributes for VXLAN/NVGRE Encapsulations

okay so picture this 🎀 you've got groups of data centers under different administrative domains, and they're linked up through one or more backbone core providers (see [RFC7365]). in that setup the RD must be a unique value per EVI or per NVE, as [RFC7432] explains 💅. basically, whenever there's more than one administrative domain for a global VNI, you need a unique RD, and whenever the VNI only has local significance, you still need a unique RD too 🫶. so the iconic move is to just use a unique RD as described in [RFC7432] at all times ✨

when the NVEs live on the hypervisor, the EVPN BGP routes and attributes tied to multihoming are no longer required 🙈. that trims the list down to a subset of four out of the total of eight listed in Section 7 of [RFC7432]:

MAC/IP Advertisement Route
Inclusive Multicast Ethernet Tag Route
MAC Mobility Extended Community
Default Gateway Extended Community

BUT (lowkey the plot twist 😭) as noted in Section 8.6 of [RFC7432], if you want a single-homing ingress NVE to get fast convergence, Aliasing, and Backup Path when it's talking to multihomed egress NVEs attached to a given ES, that ingress NVE should be able to receive and process routes that are Ethernet A-D per ES and Ethernet A-D per EVI 🦋

7.2. Impact on EVPN Procedures for VXLAN/NVGRE Encapsulations

when the NVEs reside on the hypervisors, the EVPN procedures tied to multihoming are no longer required 💖 so the NVE's to-do list shrinks to this cute little subset 🌸

Local learning of MAC addresses received from the VMs per Section 10.1 of [RFC7432].
Advertising locally learned MAC addresses in BGP using the MAC/IP Advertisement routes.
Performing remote learning using BGP per Section 9.2 of [RFC7432].
Discovering other NVEs and constructing the multicast tunnels using the IMET routes.
Handling MAC address mobility events per the procedures of Section 15 in [RFC7432].

same plot twist as before though 😭 as noted in Section 8.6 of [RFC7432], for a single-homing ingress NVE to get fast convergence, Aliasing, and Backup Path when it's dealing with multihomed egress NVEs on a given ES, that ingress NVE should implement the ingress node processing of routes that are Ethernet A-D per ES and Ethernet A-D per EVI, as defined in Sections 8.2 ("Fast Convergence") and 8.4 ("Aliasing and Backup Path") of [RFC7432] 🔥

8. Multihoming NVEs - NVE Residing in ToR Switch ✨

okay now we're getting into the main-character stuff 👑 this section is about the case where the NVEs sit in the ToR switches AND the servers (where the VMs live) are multihomed to those ToR switches 🍓 the multihoming NVE runs in All-Active or Single-Active redundancy mode. if the servers are single-homed to the ToR switches instead, the whole thing gets really similar to the NVE-on-the-hypervisor case from Section 7, at least as far as the required EVPN functionality goes 🧚

[RFC7432] defines a set of BGP routes, attributes, and procedures to support multihoming 💜 so we'll recap those first, then talk about which ones get affected by the VXLAN (or NVGRE) encapsulation and what changes are needed 🎀 spoiler, as you'll see later in this section: the only EVPN procedure hit by a non-MPLS overlay encapsulation (like VXLAN or NVGRE) is split-horizon filtering for multihomed ESs, because these encapsulations give you room for one ID instead of a stack of labels. see Section 8.3.1 for the details 🦋

8.1. EVPN Multihoming Features

in this section we're doing a quick recap of the EVPN multihoming features so you can see where the encapsulation matters 💅 it only covers the features and functions at a high level, so if you want the deep dive, go read [RFC7432] 📖

8.1.1. Multihomed ES Auto-Discovery

EVPN NVEs (or PEs) connected to the same ES (for example, the same server via a Link Aggregation Group (LAG)) can find each other automatically through the exchange of BGP routes, with minimal to no configuration 🫶✨

8.1.2. Fast Convergence and Mass Withdrawal

EVPN has a way to quickly and efficiently tell remote NVEs to update their forwarding tables when connectivity to an ES fails (say, a link or port goes down) 🚨 each NVE advertises an Ethernet A-D route per ES for every locally attached segment. if connectivity to that attached segment breaks, the NVE withdraws the matching Ethernet A-D route. that triggers every NVE that gets the withdrawal to update its next-hop adjacencies for all the MAC addresses tied to that ES 🔥 if no other NVE advertised an Ethernet A-D route for the same segment, the NVE that got the withdrawal just invalidates the MAC entries for that segment. otherwise, it updates the next-hop adjacency list accordingly 💖

8.1.3. Split-Horizon

so say a server is multihomed to two or more NVEs (represented by an ES ES1) and it's running All-Active redundancy mode 🌈 if it sends a BUM (Broadcast, Unknown unicast, or Multicast) packet to one of those NVEs, it's really important that the packet doesn't get looped back to the server through another NVE connected to that same server 😭 the filtering mechanism on the NVE that prevents this loop and packet duplication is called "split-horizon filtering" 🙅

8.1.4. Aliasing and Backup Path

picture a station that's multihomed to several NVEs. it's totally possible that only one NVE learns a set of the MAC addresses tied to traffic sent by that station 🥺 which means remote NVEs only hear MAC Advertisement routes for those addresses from that single NVE, even though several NVEs are connected to the multihomed station. so the remote NVEs can't load-balance traffic well across the NVEs on the multihomed ES 💀 this could happen when the NVEs do data-path learning on the access and the station's load-balancing function hashes traffic from a given source MAC address to a single NVE. it also happens when the NVEs rely on control-plane learning on the access (for example, using ARP), since ARP traffic gets hashed to a single link in the LAG 🙈

to fix this, EVPN brings in the concept of "Aliasing" 🦋 this is the ability of an NVE to signal that it has reachability to a given locally attached ES, even when it hasn't learned any MAC addresses from that segment. the Ethernet A-D route per EVI is used for this 💅 remote NVEs that receive MAC Advertisement routes with non-zero ESIs should treat the MAC address as reachable via all the NVEs that advertise reachability to the relevant Segment using Ethernet A-D routes with the same ESI and with the Single-Active flag reset ✨

Backup Path is a closely related function, just for the case where the redundancy mode is Single-Active 🎀 here the NVE signals that it has reachability to a given locally attached ES using the Ethernet A-D route too. remote NVEs that receive the MAC Advertisement routes (with non-zero ESI) should treat the MAC address as reachable via the advertising NVE. on top of that, the remote NVEs should install a Backup Path for that MAC, pointing at the NVE that advertised reachability to the relevant segment using an Ethernet A-D route with the same ESI and with the Single-Active flag set 🔥

8.1.5. DF Election

if a host is multihomed to two or more NVEs on an ES running All-Active redundancy mode, then for a given EVI, only one of those NVEs, called the "Designated Forwarder" (DF), is responsible for sending it broadcast, multicast, and (if it's configured for that EVI) unknown unicast frames 👑

this is required so that multi-destination frames don't get delivered to a multihomed host or VM more than once in the All-Active case 🫶

in NVEs where frames tagged as IEEE 802.1Q [IEEE.802.1Q] come in from hosts, the DF election should be done based on host VIDs per Section 8.5 of [RFC7432] 💅 also, multihoming PEs of a given ES MAY perform DF election using configured IDs such as VNI, EVI, normalized VIDs, and etc., as long as those IDs are configured consistently across the multihoming PEs 🌸

in GWs where VXLAN-encapsulated frames are received, the DF election is performed on VNIs 🦋 again, it's assumed that for a given Ethernet segment, VNIs are unique and consistent (so no duplicate VNIs exist) 🙏

8.2. Impact on EVPN BGP Routes and Attributes

since multihoming is supported in this scenario, the entire set of BGP routes and attributes defined in [RFC7432] gets used 🧚 the Ethernet Tag field in the MAC Advertisement, Ethernet A-D per EVI, and IMET) routes is set the same way as in Section 5.1.3 💖 and the VNI field in the MAC Advertisement and Ethernet A-D per EVI routes is also set the same way as in Section 5.1.3 ✨

8.3. Impact on EVPN Procedures

there are two cases to look at here, depending on whether the NVEs are running Single-Active or All-Active redundancy mode 🌈

first up, the Single-Active redundancy mode 🎀 here the hosts are multihomed to a set of NVEs, but only one NVE is active at any given time for a given VNI. in this case Aliasing isn't required, and split-horizon filtering may not be required either 🙌 but other functions like multihomed ES auto-discovery, fast convergence and mass withdrawal, Backup Path, and DF election are required 💅

second, the All-Active redundancy mode 🔥 out of all the EVPN multihoming features listed in Section 8.1, using VXLAN or NVGRE encapsulation impacts the split-horizon and Aliasing features. that's because those two rely on the MPLS client layer, and since that layer is absent with these encapsulations, alternative procedures and mechanisms are needed to provide the required functions 😭 those are covered in detail next 🦋

8.3.1. Split Horizon 🦋

okay so split horizon is basically how we stop packets from doing endless laps 🔄💀. In EVPN, an MPLS label handles this for All-Active multihoming. The ingress NVE adds a label that says which site the packet came from (aka an ESI label) when it encapsulates. Then the egress NVE checks that label when it tries to forward a multi-destination frame out an interface. If the label matches the same site identifier (ESI) tied to that interface, the packet gets dropped. No loops, no drama 💅

Since VXLAN and NVGRE encapsulations don't carry the ESI label, we need a different way to do split-horizon filtering for those. The recommended approach when VXLAN (or NVGRE) encapsulation is used goes like this:

Every NVE keeps track of the IP address(es) of the other NVE(s) it shares multihomed ESs with. When it gets a multi-destination frame from the overlay network, it looks at the source IP address in the tunnel header (that's the ingress NVE) and filters the frame out of every local interface connected to an ES shared with that ingress NVE. 🫶 With this approach, the ingress NVE is required to replicate locally to all directly attached Ethernet segments (regardless of the DF election state) for all flooded traffic coming in from the access interfaces (i.e., from the hosts). This is called "Local Bias" 🍓, and the perk is that each NVE only needs a single IP address for split-horizon filtering, instead of one IP address per Ethernet segment per NVE. Iconic efficiency, honestly ✨

Okay, now the important part, so listen up 👑 In order to get split-horizon filtering working properly among the same group of multihoming PE devices, a MIX of PE devices is not allowed on a given Ethernet segment. You MUST NOT configure MPLS over GRE PEs running the [RFC7432] split-horizon procedures alongside VXLAN/NVGRE PEs running local-bias procedures on that same segment. Not me being bossy, just the rules 🙈

8.3.2. Aliasing and Backup Path 🌸

The Aliasing and Backup Path procedures for VXLAN/NVGRE encapsulation are super similar to the ones for MPLS, so this should feel familiar 🧚. With MPLS, the Ethernet A-D route per EVI is used for Aliasing when the ES is operating in All-Active multihoming, and that same route is used for Backup Path when the ES is operating in Single-Active multihoming. With VXLAN/NVGRE, it's the same route doing both jobs for Aliasing and Backup Path. The only difference is that the Ethernet Tag and VNI fields in the Ethernet A-D per EVI route get set as described in Section 5.1.3 💖

8.3.3. Unknown Unicast Traffic Designation 🔥

Okay, this one is a little spicy, so buckle up 🎢. In EVPN, when an ingress PE uses ingress replication to flood unknown unicast traffic to egress PEs, it uses a different EVPN MPLS label (than the one for known unicast traffic) to identify that BUM traffic. The egress PEs use this label to recognize the BUM traffic and apply DF filtering for All-Active multihomed sites.

Now, if there's no unknown unicast traffic designation, but unknown unicast flooding is enabled, you can get transient duplicate traffic going to All-Active multihomed sites. 😭 Here's how it happens: the host MAC address gets learned by the egress PE(s) and advertised to the ingress PE, but that MAC Advertisement hasn't been received or processed by the ingress PE yet. So the host MAC is unknown on the ingress PE but known on the egress PE(s). When a packet for that host MAC shows up at the ingress PE, it floods it via ingress replication to all the egress PE(s). Since the MAC is known to those egress PEs, multiple copies end up going to the All-Active multihomed site. Chaotic, but it's real.

Just so we're clear, this transient duplication ONLY happens when all four of these are true: a) the destination host is multihomed via All-Active redundancy mode, b) flooding of unknown unicast is enabled in the network, c) ingress replication is used, and d) traffic for the destination host arrives at the ingress PE before it learns the host MAC address via BGP EVPN advertisement. Wild that it's rare, but still possible 🙈

If you want to avoid that transient packet duplication (however low the probability may be), then VXLAN-GPE encapsulation needs to be used between these PEs, and the ingress PE needs to set the BUM Traffic Bit (B bit) [VXLAN-GPE] to indicate that this is ingress-replicated BUM traffic. Main character energy, the fix is literally one bit 💅

9. Support for Multicast ✨

okay so the EVPN IMET route is what we use to discover the multicast tunnels between the endpoints in a given EVI (e.g., given VNI) for VLAN-Based Service, and for a given <EVI, VLAN> for VLAN-Aware Bundle Service 🌈. All fields of this route get set as described in Section 5.1.3. The originating router's IP address field is set to the NVE's IP address 🫶

This route gets tagged with the PMSI Tunnel attribute, which encodes the type of multicast tunnel to use, plus the multicast tunnel identifier. The tunnel encapsulation gets encoded by adding the BGP Encapsulation Extended Community as per Section 5.1.1. For example, the PMSI Tunnel attribute might say the multicast tunnel is PIM-SM (PIM-SM, lowkey iconic), while the BGP Encapsulation Extended Community says the encapsulation for that tunnel is VXLAN 🦋. The following tunnel types from [RFC6514] can be used in the PMSI Tunnel attribute for VXLAN/NVGRE:

         + 3 - PIM-SSM Tree
         + 4 - PIM-SM Tree
         + 5 - BIDIR-PIM Tree
         + 6 - Ingress Replication

Now for the requirements, which are kinda the whole point 👑. In case of VXLAN and NVGRE encapsulations with locally assigned VNIs, just like in [RFC7432], each PE MUST advertise an IMET route to other PEs in an EVPN instance for the multicast tunnel type it uses (i.e., ingress replication, PIM-SM, PIM-SSM, or BIDIR-PIM tunnel). However, for globally assigned VNIs, each PE MUST advertise an IMET route to other PEs in an EVPN instance for ingress replication or a PIM-SSM tunnel, and they MAY advertise an IMET route for a PIM-SM or BIDIR-PIM tunnel. For PIM-SM or BIDIR-PIM tunnels, the PE doesn't need any info from the IMET route to set those tunnels up 💜

In the scenario where the multicast tunnel is a tree, both the Inclusive and the Aggregate Inclusive variants may be used 🌸. With the former, a multicast tree is dedicated to one VNI. With the latter, a multicast tree is shared among multiple VNIs. For VNI-Based Service, Aggregate Inclusive mode works by having the NVEs advertise multiple IMET routes with different RTs (one per VNI), but with the same tunnel identifier encoded in the PMSI Tunnel attribute. For VNI-Aware Bundle Service, Aggregate Inclusive mode works by having the NVEs advertise multiple IMET routes with different VNIs encoded in the Ethernet Tag field, but with the same tunnel identifier encoded in the PMSI Tunnel attribute ✨

10. Data-Center Interconnections (DCIs) 🌐

For DCIs, two main scenarios are considered when connecting data centers running evpn-overlay (as described here) over an MPLS/IP core network 🍓:

Scenario 1: DCI using GWs
Scenario 2: DCI using ASBRs

The following two subsections describe the operations for each of these scenarios 💅

10.1. DCI Using GWs 🧚

This is the typical scenario for interconnecting data centers over WAN 🌍. In this setup, EVPN routes get terminated and processed in each GW, and MAC/IP routes are always re-advertised from DC to WAN. But from WAN to DC, they are NOT re-advertised if unknown MAC addresses (and default IP address) are used in the NVEs. Each GW maintains a MAC-VRF (and/or IP-VRF) for each EVI, so this is basically the GW doing all the heavy lifting 💪

The main advantage is that NVEs don't need to maintain MAC and IP addresses from any remote data centers when default IP routes and unknown MAC routes are used. They only need to maintain routes that are local to their own DC, which is so relieving 😭🫶. When default IP routes and unknown MAC routes are used, any unknown IP and MAC packets from NVEs get forwarded to the GWs, where all the VPN MAC and IP routes are maintained. This shrinks the size of the MAC-VRF and IP-VRF significantly at the NVEs 🔥.

It also gives faster convergence when a link or NVE fails in a multihomed network or a device redundancy scenario, because failure-related BGP routes (like a mass withdrawal message) don't have to get propagated all the way to the remote NVEs in the remote DCs. Honestly, that's so smart ✨. This approach is described in detail in Section 3.4 of [DCI-EVPN-OVERLAY] 🦋

10.2. DCI Using ASBRs 🦋

okay so this one is basically the opposite of the first approach, bestie 💅 It's all about keeping the DCI devices simple. The NVEs have to carry bigger MAC-VRF (and IP-VRF) tables, but the DCI devices don't need to keep any MAC (or IP) forwarding tables at all 🫶 And the DCI devices don't have to terminate and process the multihoming routes either. They just relay those messages so an end-to-end Label Switched Path (LSP) can get set up 🌈 So the DCI devices act a lot like ASBRs for inter-AS Option B (see Section 10 of [RFC4364]) ✨ This means the locally assigned VNIs get used just like downstream-assigned MPLS VPN labels, where the VNIs basically work like 24-bit VPN labels 🎀 This approach works for data centers (or Carrier Ethernet networks) with MPLS encapsulation too, no cap 💜

Here's the tea on inter-AS Option B 🍓 When an ASBR gets an EVPN route from its DC over internal BGP (iBGP) and passes it along to other ASBRs, it rewrites the BGP next hops to itself. So the identity of the PE that originated the advertisement gets lost 💀 That rewrite hurts the EVPN mass withdrawal route (Ethernet A-D per ES) and how its procedure works, and it is not great 😭 The Aliasing mechanism/procedure is fine though, iykyk 🙈 When Aliasing routes (Ethernet A-D per EVI) get advertised, the receiving PE first resolves a MAC address for a given EVI into its corresponding <ES, EVI>. Then it resolves that <ES, EVI> into multiple paths (and their associated next hops) it can be reached through ✨ Since Aliasing and MAC routes both get advertised per EVI and use the same RD and RT (per EVI), the receiving PE can tie them together on a per-BGP-path basis (for example, per originating PE) 💅 That means it can do recursive route resolution: a MAC is reachable via an <ES, EVI>, which is reachable via a set of BGP paths, so the MAC is reachable via that set of BGP paths 🦋 Because everything is per EVI, the link between MAC routes and their Aliasing route is fixed by the same RD and RT, so there's zero ambiguity even when the BGP next hop gets rewritten as routes pass through ASBRs 🔥 The receiving PE might get several Aliasing routes for the same EVI from one next hop (a single ASBR), and it can still create multiple paths toward that <ES, EVI> 🫶

BUT, when the BGP next-hop address of the originating PE gets rewritten, the mass withdrawal route (Ethernet A-D per ES) can't be tied to its MAC routes using RDs and RTs, because the RD for the mass withdrawal route is different from the one used for the MAC routes 😭 So the functionality needed at the ASBRs and the receiving PEs depends on whether the Mass Withdrawal route is originated and whether there is a need to handle route resolution ambiguity for this route 🙈 The next two subsections cover what ASBRs and receiving PEs need to do, depending on whether the NVEs live in hypervisors or in ToR switches 💖

10.2.1. ASBR Functionality with Single-Homing NVEs 🌸

When NVEs live in hypervisors (as described in Section 7.1), there's no multihoming, so the originating NVE doesn't need to send Ethernet A-D per ES or Ethernet A-D per EVI routes ✨ But, as noted in Section 7, a single-homing ingress NVE can still get fast convergence, Aliasing, and Backup Path when it's talking to multihoming egress NVEs attached to a given ES 💅 To make that work, the single-homing NVE SHOULD be able to receive and process Ethernet A-D per ES and Ethernet A-D per EVI routes 🦋 How those routes get handled is covered in the next section 🎀

10.2.2. ASBR Functionality with Multihoming NVEs 🌈

When NVEs live in ToR switches and run in multihoming redundancy mode, the originating multihoming NVE needs to send Ethernet A-D per ES route(s) (used for mass withdrawal) and Ethernet A-D per EVI routes (used for Aliasing), as described in Section 8 🫶 Like we said above, when ASBRs rewrite the BGP next hop, things get ambiguous when Ethernet A-D per ES routes show up at a remote NVE through a different ASBR 💀 The receiving NVE can't tie that route to the MAC/IP routes of that ES advertised by the same originating NVE, so mass withdrawal per ES stops working for the receiving NVE in a different AS 😭

Let's look at an example 🧚 Say a CE is multihomed to PE1 and PE2, and those PEs are connected through ASBR1 and then ASBR2 to the remote PE3 💜 Now, PE1 learns M1 from CE1 but PE2 doesn't ✨ So PE1 advertises Ethernet A-D per ES1, Ethernet A-D per EVI1, and M1, while PE2 only advertises Ethernet A-D per ES1 and Ethernet A-D per EVI1 🎀 ASBR1 gets all five advertisements and passes them to ASBR2 (with itself as the BGP next hop) 🌸 Then ASBR2 passes them to the remote PE3, with itself as the BGP next hop 🦋 PE3 gets all five routes, and they all have the same BGP next hop (ASBR2) 💅 The two Ethernet A-D per ES routes PE3 receives have the same info, meaning the same ESI and the same BGP next hop 🙈 Both routes stay in the BGP process on PE3 (since they have different RDs, so they're treated as different BGP routes), but only one of them actually gets used in the L2 routing table (L2 RIB) 🔥

                  PE1
                 /   \
                CE     ASBR1---ASBR2---PE3
                 \   /
                  PE2

                    Figure 3: Inter-AS Option B

Now, when the AC between PE2 and the CE fails, PE2 sends a Network Layer Reachability Information (NLRI) withdrawal for the Ethernet A-D per ES route 💀 That withdrawal gets passed along and received by PE3 ✨ The BGP process on PE3 removes the matching BGP route, but it doesn't remove the associated info (namely ESI and BGP next hop) from the L2 routing table (L2 RIB) 😭 Why? Because it still has the other Ethernet A-D per ES route (from PE1) with the exact same info 🙈 That's why the mass withdrawal mechanism doesn't work when you're doing DCI with inter-AS Option B 💖 But the Aliasing function still works, and so does "mass withdrawal per EVI" (which is tied to withdrawing the EVPN route used for Aliasing, i.e., the Ethernet A-D per EVI route) 🌈

In the example above, PE3 gets two Aliasing routes with the same BGP next hop (ASBR2) but different RDs 🫶 One of those Aliasing routes has the same RD as the MAC route (M1) that was advertised 🎀 PE3 follows the route resolution procedure from [RFC7432] when it gets the two Aliasing routes 🦋 So it resolves M1 to <ES, EVI1>, then resolves <ES, EVI1> to a BGP path list with two paths, along with the matching VNIs/MPLS labels (one for PE1 and one for PE2) 💅 Even though both paths are advertised by the same BGP next hop (ASRB2), PE3 can handle them properly 🔥 So M1 is reachable via two paths 💜 That creates two end-to-end LSPs for M1, from PE3 to PE1 and from PE3 to PE2, so when PE3 wants to forward traffic to M1, it can load-balance across the two LSPs ✨ Route resolution for Aliasing routes with the same BGP next hop isn't explicitly spelled out in [RFC7432], but this is the expected behavior, so we're elaborating on it here 🌸

When the AC between PE2 and the CE fails and PE2 sends NLRI withdrawal for Ethernet A-D per EVI routes, and those withdrawals get passed along and received by PE3, PE3 removes the Aliasing route and updates the path list, meaning it removes the path for PE2 🧚 So all the MAC routes for that <ES, EVI> that point to that path list now have an updated path list with only one path, the one for PE1 🎀 This counts as mass withdrawal at the per-EVI level 💖 Per-EVI mass withdrawal takes longer to converge than per-ES mass withdrawal, but it's still way faster than withdrawing on a per-MAC basis 🌈

If a PE gets detached from a given ES, then, on top of withdrawing its previously advertised Ethernet A-D per ES routes, it MUST also withdraw its previously advertised Ethernet A-D per EVI routes for that ES 🫶 For a remote PE that's separated from the withdrawing PE by one or more EVPN inter-AS Option B ASBRs, the withdrawal of Ethernet A-D per ES routes is not actionable 💀 However, the remote PE can correlate a previously advertised Ethernet A-D per EVI route with any MAC/IP Advertisement routes that the withdrawing PE also advertised for that <ES, EVI, BD> 😭 So when it gets the withdrawal of an Ethernet A-D per EVI route, it SHOULD remove the withdrawing PE as a next hop for all MAC addresses associated with that <ES, EVI, BD> 🙈

Going back to our example: when the AC between PE2 and the CE fails, PE2 withdraws its Ethernet A-D per ES and per EVI routes 💅 When PE3 gets the withdrawal of an Ethernet A-D per EVI route, it removes PE2 as a valid next hop for all MAC addresses associated with the matching <ES, EVI, BD> 🔥 So all the MAC next hops for that <ES, EVI, BD> now have just one next hop, the LSP to PE1 ✨

To wrap this section up 💖 Aliasing (and Backup Path) functionality should work as-is for inter-AS Option B without needing any extra functionality in ASBRs or PEs 🌈 The mass withdrawal functionality, though, falls back from per-ES mode to per-EVI mode for inter-AS Option B 🦋 Basically, PEs that get a mass withdrawal route from the same AS act on the Ethernet A-D per ES route, while PEs that get mass withdrawal routes from different ASes act on the Ethernet A-D per EVI route 🫶

Security Considerations 🔐

🚨 Heads up, bestie: this part is serious. Please read every line.

This document uses IP-based tunnel technologies to support data-plane transport 🌊 So the security considerations of those tunnel technologies apply too. This document defines support for VXLAN [RFC7348] and NVGRE encapsulations [RFC7637]. The security considerations from those RFCs apply to the data-plane aspects of this document 💀

As with [RFC5512], any modification of the information used to form encapsulation headers, to choose a tunnel type, or to choose a particular tunnel for a particular payload type may lead to user data packets getting misrouted, misdelivered, and/or dropped 😭 That is a whole disaster, so be careful 🙈

More broadly, the security considerations for transporting IP reachability information using BGP are discussed in [RFC4271] and [RFC4272], and they apply equally to the extensions described in this document 🎀

IANA Considerations 📋

This document registers the following in the "BGP Tunnel Encapsulation Attribute Tunnel Types" registry 💅

Value    Name
-----    ------------------------
8        VXLAN Encapsulation
9        NVGRE Encapsulation
10       MPLS Encapsulation
11       MPLS in GRE Encapsulation
12       VXLAN GPE Encapsulation

References 📚

13.1. Normative References

okay so here are the must-read docs, bestie 📚✨ the ones the whole spec is built on. you need these before anything else makes sense 💅

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC7432]  Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
           Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
           Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
           2015, <https://www.rfc-editor.org/info/rfc7432>.

[RFC7348]  Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
           L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
           eXtensible Local Area Network (VXLAN): A Framework for
           Overlaying Virtualized Layer 2 Networks over Layer 3
           Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
           <https://www.rfc-editor.org/info/rfc7348>.

[RFC5512]  Mohapatra, P. and E. Rosen, "The BGP Encapsulation
           Subsequent Address Family Identifier (SAFI) and the BGP
           Tunnel Encapsulation Attribute", RFC 5512,
           DOI 10.17487/RFC5512, April 2009,
           <https://www.rfc-editor.org/info/rfc5512>.

[RFC4023]  Worster, T., Rekhter, Y., and E. Rosen, Ed.,
           "Encapsulating MPLS in IP or Generic Routing Encapsulation
           (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005,
           <https://www.rfc-editor.org/info/rfc4023>.

[RFC7637]  Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
           Virtualization Using Generic Routing Encapsulation",
           RFC 7637, DOI 10.17487/RFC7637, September 2015,
           <https://www.rfc-editor.org/info/rfc7637>.

13.2. Informative References

these ones are the extra reading 🧠 super helpful for context, but the spec doesn't rely on them to work, so no panic if you skip ahead 🌸

[RFC7209]  Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N.,
           Henderickx, W., and A. Isaac, "Requirements for Ethernet
           VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014,
           <https://www.rfc-editor.org/info/rfc7209>.

[RFC4272]  Murphy, S., "BGP Security Vulnerabilities Analysis",
           RFC 4272, DOI 10.17487/RFC4272, January 2006,
           <https://www.rfc-editor.org/info/rfc4272>.

[RFC7364]  Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
           Kreeger, L., and M. Napierala, "Problem Statement:
           Overlays for Network Virtualization", RFC 7364,
           DOI 10.17487/RFC7364, October 2014,
           <https://www.rfc-editor.org/info/rfc7364>.

[RFC7365]  Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
           Rekhter, "Framework for Data Center (DC) Network
           Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
           2014, <https://www.rfc-editor.org/info/rfc7365>.

[RFC6514]  Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
           Encodings and Procedures for Multicast in MPLS/BGP IP
           VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,
           <https://www.rfc-editor.org/info/rfc6514>.

[RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
           Border Gateway Protocol 4 (BGP-4)", RFC 4271,
           DOI 10.17487/RFC4271, January 2006,
           <https://www.rfc-editor.org/info/rfc4271>.

[RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
           Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
           2006, <https://www.rfc-editor.org/info/rfc4364>.

[TUNNEL-ENCAP]
           Rosen, E., Ed., Patel, K., and G. Velde, "The BGP Tunnel
           Encapsulation Attribute", Work in Progress draft-ietf-idr-
           tunnel-encaps-09, February 2018.

[DCI-EVPN-OVERLAY]
           Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
           A., and J. Drake, "Interconnect Solution for EVPN Overlay
           networks", Work in Progress, draft-ietf-bess-dci-evpn-
           overlay-10, March 2018.

[EVPN-GENEVE]
           Boutros, S., Sajassi, A., Drake, J., and J. Rabadan, "EVPN
           control plane for Geneve", Work in Progress,
           draft-boutros-bess-evpn-geneve-02, March 2018.

[VXLAN-GPE]
           Maino, F., Kreeger, L., Ed., and U. Elzur, Ed., "Generic
           Protocol Extension for VXLAN", Work in Progress,
           draft-ietf-nvo3-vxlan-gpe-05, October 2017.

[GENEVE]   Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
           "Geneve: Generic Network Virtualization Encapsulation",
           Work in Progress, draft-ietf-nvo3-geneve-06, March 2018.

[IEEE.802.1Q]
           IEEE, "IEEE Standard for Local and metropolitan area
           networks - Bridges and Bridged Networks - Media Access
           Control (MAC) Bridges and Virtual Bridged Local Area
           Networks", IEEE Std 802.1Q.

Acknowledgements 🫶

the authors wanna give a big shoutout to Aldrin Isaac, David Smith, John Mullooly, Thomas Nadeau, Samir Thoria, and Jorge Rabadan for their super valuable comments and feedback 💖 they really made this better, no cap. they also thanked Jakob Heitz for his contribution on Section 10.2, which is so sweet 🎀

Contributors 💜

these are the people who helped write the doc 👑 the list is grouped by company:

S. Salam
K. Patel
D. Rao
S. Thoria
D. Cai
Cisco

Y. Rekhter
A. Issac
W. Lin
N. Sheth
Juniper

L. Yong
Huawei

Authors' Addresses 🌈

and here are the editors and authors, with how to reach them 📬 ali and john are the editors, so they're the main girlies on this one ✨

Ali Sajassi (editor)
Cisco
United States of America

Email: sajassi@cisco.com


John Drake (editor)
Juniper Networks
United States of America

Email: jdrake@juniper.net


Nabil Bitar
Nokia
United States of America

Email: nabil.bitar@nokia.com


R. Shekhar
Juniper
United States of America

Email: rshekhar@juniper.net


James Uttaro
AT&T
United States of America

Email: uttaro@att.com


Wim Henderickx
Nokia
Copernicuslaan 50
2018 Antwerp
Belgium

Email: wim.henderickx@nokia.com

📦 where this page comes from

ietf: rfc8365 2018-03
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms