RFC 9469

Applicability of Ethernet Virtual Private Network (EVPN) to Network Virtualization over Layer 3 (NVO3) Networks

📅 published September 2023, from ietf

Applicability of Ethernet Virtual Private Network (EVPN) to Network Virtualization over Layer 3 (NVO3) Networks 🎀

  • Document: RFC 9469
  • Authors: J. Rabadan, Ed. (Nokia), M. Bocci (Nokia), S. Boutros (Ciena), A. Sajassi (Cisco)
  • Category: Informational 🦋
  • ISSN: 2070-1721
  • Date: September 2023 🌸
  • Obsoletes / Updates: the header has no Obsoletes or Updates line, so this doc doesn't replace or update any other RFC listed there ✨

Abstract ✨

okay so EVPN gives you a unified control plane that handles NVE auto-discovery, tenant MAC/IP dissemination, and a bunch of advanced features, and it does it in a scalable way, which is exactly what NVO3 networks need 💅 It's a scalable solution for NVO3 and it keeps the underlay IP Fabric independent. That means you don't have to turn on Protocol Independent Multicast (PIM) in the underlay or keep multicast state for tenant Broadcast Domains. Main character energy 👑

This doc explains how EVPN gets used for NVO3 networks. It covers basic Layer 2 and Layer 3 connectivity, plus the advanced stuff like MAC Mobility, MAC Protection and Loop Protection, multihoming, Data Center Interconnect (DCI), and more 🧚 The big headline is that no new EVPN procedures are introduced 🫶

Status of This Memo ✨

This doc is not an Internet Standards Track specification. It's published for informational purposes, so no pressure 🌈

It's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so check out Section 2 of RFC 7841 if you want the details 🔥

For the current status, any errata, and how to send feedback, head to https://www.rfc-editor.org/info/rfc9469 💖

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. 💜

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they were in effect on the publication date. Read those carefully, because they spell out your rights and restrictions for this doc 📜 Code Components pulled out of this doc must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Revised BSD License 🫶

Table of Contents 🌸

   1.  Introduction
   2.  EVPN and NVO3 Terminology
   3.  Why is EVPN Needed in NVO3 Networks?
   4.  Applicability of EVPN to NVO3 Networks
     4.1.  EVPN Route Types Used in NVO3 Networks
     4.2.  EVPN Basic Applicability for Layer 2 Services
       4.2.1.  Auto-Discovery and Auto-Provisioning
       4.2.2.  Remote NVE Auto-Discovery
       4.2.3.  Distribution of Tenant MAC and IP Information
     4.3.  EVPN Basic Applicability for Layer 3 Services
     4.4.  EVPN as Control Plane for NVO3 Encapsulations and Geneve
     4.5.  EVPN OAM and Application to NVO3
     4.6.  EVPN as the Control Plane for NVO3 Security
     4.7.  Advanced EVPN Features for NVO3 Networks
       4.7.1.  Virtual Machine (VM) Mobility
       4.7.2.  MAC Protection, Duplication Detection, and Loop
               Protection
       4.7.3.  Reduction/Optimization of BUM Traffic in Layer 2
               Services
       4.7.4.  Ingress Replication (IR) Optimization for BUM Traffic
       4.7.5.  EVPN Multihoming
       4.7.6.  EVPN Recursive Resolution for Inter-subnet Unicast
               Forwarding
       4.7.7.  EVPN Optimized Inter-subnet Multicast Forwarding
       4.7.8.  Data Center Interconnect (DCI)
   5.  Security Considerations
   6.  IANA Considerations
   7.  References
     7.1.  Normative References
     7.2.  Informative References
   Acknowledgments
   Authors' Addresses

1. Introduction ✨

okay bestie, here's the setup 🦋 In Network Virtualization over Layer 3 (NVO3) networks, Network Virtualization Edge (NVE) devices sit at the edge of the underlay network. They provide Layer 2 and Layer 3 connectivity among Tenant Systems (TSes) that belong to the same tenant 💅 The NVEs need to build and maintain mapping tables so they can deliver encapsulated packets to their intended destination NVE(s). There are different ways to create and spread those mapping table entries, but NVEs may swap that info directly with each other over a control plane protocol, like Ethernet Virtual Private Network (EVPN) 🌈 EVPN is an efficient, flexible, and unified control plane option that works for Layer 2 and Layer 3 Virtual Network (VN) service connectivity. And this doc does not introduce any new EVPN procedures 🫶

In this doc, we assume the EVPN control plane module lives in the NVEs 🧚 The NVEs can be virtual switches inside hypervisors, Top-of-Rack (ToR) switches, Leaf switches, or Data Center Gateways. As described in [RFC7365], Network Virtualization Authorities (NVAs) may be used to provide forwarding information to the NVEs. In that case, EVPN could be used to spread the information across multiple federated NVAs. The applicability of EVPN would then be similar to what's described in this doc 💜 For simplicity though, the description assumes control plane communication among NVE(s) ✨

2. EVPN and NVO3 Terminology ✨

okay so this section is the glossary bestie, and we need it before anything else makes sense 🫶 this doc uses all the terms from [RFC7365] too, plus the extra ones below.

  • AC: Attachment Circuit, which is the logical interface tied to a given BT 🔌 to figure out which AC a packet came in on, the NVE looks at the physical or logical port and/or the VLAN tags (those can be individual c-tags, s-tags, or ranges of both).
  • ARP and NDP: Address Resolution Protocol (IPv4) and Neighbor Discovery Protocol (IPv6), respectively 🌸
  • BD: Broadcast Domain, which lines up with a tenant IP subnet 🏠 if no suppression techniques are in use, a BUM frame injected into a Broadcast Domain reaches every NVE attached to that Broadcast Domain 📣 an EVI can hold one or several Broadcast Domains, depending on the service model [RFC7432]. this doc just says "Broadcast Domain" when it means a tenant subnet, so that's the vibe 💅
  • BT: Bridge Table, as defined in [RFC7432] 🌉 a BT is what a Broadcast Domain looks like inside one NVE. when a given EVI has a single Broadcast Domain, the MAC-VRF is the same thing as the BT on that NVE. even though a Broadcast Domain stretches across multiple NVEs and a BT is really its per-NVE piece, this doc uses BT and Broadcast Domain interchangeably 🦋
  • BUM: Broadcast, Unknown Unicast, and Multicast frames 📡
  • Clos: a multistage network topology described in [CLOS1953], where every edge switch (a Leaf) connects to every core switch (a Spine) 🕸️ it's typically used in Data Centers 🏢
  • DF and NDF: Designated Forwarder and Non-Designated Forwarder, respectively 👑 these are the roles a given PE can hold in a given ES.
  • ECMP: Equal-Cost Multipath 🛣️
  • ES: Ethernet Segment 🔗 when a Tenant System (TS) hooks up to one or more NVEs through a set of Ethernet links, that set is called an "Ethernet Segment". every ES gets its own unique Ethernet Segment Identifier (ESI) in the NVO3 network, and that ESI shows up in the EVPN routes specific to that ES 🫧
  • Ethernet Tag: this stands in for a Broadcast Domain that's configured on a given ES, and it's what Designated Forwarder election uses 🗳️ basically any of these can represent a Broadcast Domain: VIDs (Q-in-Q tags included), configured IDs, VNIs, normalized VIDs, Service Instance Identifiers (I-SIDs), etc. the one catch is that whatever you pick has to be configured consistently across all the multihomed PEs attached to that ES ⚠️
  • EVI or EVPN Instance: a Layer 2 Virtual Network that uses an EVPN control plane to swap reachability info among its member NVEs 🌐 it corresponds to a set of MAC-VRFs belonging to the same tenant. see MAC-VRF below 💜
  • EVPN: Ethernet Virtual Private Network, as described in [RFC7432] ✨
  • EVPN VLAN-Aware Bundle Service Interface: like the VLAN-bundle interface, except each individual VLAN value maps to its own Broadcast Domain 🎀 so this interface has multiple Broadcast Domains per EVI for a given tenant. each one is identified by an "Ethernet Tag", a control plane value that picks out the routes for that Broadcast Domain inside the EVI 🧩
  • EVPN VLAN-Based Service Interface: one of the three service interfaces defined in [RFC7432] 🍓 it's a Broadcast Domain that uses a single VLAN per physical access port to attach tenant traffic. with this one there's only one Broadcast Domain per EVI.
  • EVPN VLAN-Bundle Service Interface: similar to the VLAN-based interface, but it uses a bundle of VLANs per physical port to attach tenant traffic to the Broadcast Domain 🧺 just like the VLAN-based one, there's only one Broadcast Domain per EVI.
  • Geneve: Generic Network Virtualization Encapsulation, an NVO3 encapsulation defined in [RFC8926] 🌀
  • IP-VRF: IP Virtual Routing and Forwarding table, as defined in [RFC4364] 🗺️ it stores IP Prefixes that belong to the tenant's IP space, and EVPN distributes them among the NVEs of the same tenant. a Route Distinguisher (RD) and one or more Route Targets (RTs) are required properties of an IP-VRF 🔑 an NVE gets an IP-VRF for a given tenant when it's attached to multiple subnets of that tenant and needs local inter-subnet forwarding across them.
  • IRB: Integrated Routing and Bridging 🌉 this is the logical interface that connects a Broadcast Domain instance (or a BT) to an IP-VRF, and it forwards packets whose destination sits in a different subnet 🏃‍♀️
  • MAC-VRF: MAC Virtual Routing and Forwarding table, as defined in [RFC7432] 💖 it's the instantiation of an EVI (EVPN Instance) in an NVE. a Route Distinguisher (RD) and one or more RTs are required properties of a MAC-VRF, and they're normally different from the ones on the associated IP-VRF (when the MAC-VRF has an IRB interface) 🙈
  • NVE: Network Virtualization Edge 🏰 a network entity sitting at the edge of an underlay network that implements Layer 2 and/or Layer 3 network virtualization functions. the network-facing side uses the underlying Layer 3 network to tunnel tenant frames to and from other NVEs, while the tenant-facing side sends and receives Ethernet frames to and from individual Tenant Systems 📦 in this document, an NVE could be a virtual switch inside a hypervisor, a switch, or a router, and it runs EVPN in the control plane 🧠
  • NVO3 tunnels: Network Virtualization over Layer 3 tunnels 🚇 in this document these are a way to encapsulate tenant frames or packets inside IP packets. the IP Source Addresses (SAs) or Destination Addresses (DAs) belong to the underlay IP address space, and they identify NVEs attached to the same underlay network 🌍 examples of NVO3 tunnel encapsulations are VXLAN [RFC7348], Geneve [RFC8926], or MPLSoUDP [RFC7510] 🦄
  • PE: Provider Edge 🛎️
  • PMSI: Provider Multicast Service Interface 📣
  • PTA: PMSI Tunnel Attribute 🏷️
  • RT and RD: Route Target and Route Distinguisher, respectively 💅
  • RT-1, RT-2, RT-3, etc.: these refer to the Route Types, followed by the type numbers, as defined in the "EVPN Route Types" IANA registry (see <https://www.iana.org/assignments/evpn/>) 📒
  • SA and DA: Source Address and Destination Address, respectively 📮 they're used with MAC or IP, like IP SA or MAC DA.
  • SBD: Supplementary Broadcast Domain, as defined in [RFC9136] 🛰️ it's a Broadcast Domain with no Attachment Circuits, only IRB interfaces, and it provides connectivity among all the IP-VRFs of a tenant in the Interface-ful IP-VRF-to-IP-VRF models 🫂
  • TS: Tenant System 🧑‍💻 a physical or virtual system that can act as a host or a forwarding element, such as a router, switch, or firewall. it belongs to a single tenant and connects to one or more Broadcast Domains of that tenant.
  • VID: Virtual Local Area Network Identifier 🏷️
  • VNI: Virtual Network Identifier 🪪 no matter which NVO3 encapsulation is used, the tunnel header always carries a VNI. the ingress NVE adds it (based on the mapping table lookup), and it identifies the BT at the egress NVE 🎯 it's called VNI in VXLAN or Geneve, Virtual Subnet ID (VSID) in nvGRE, or Label in MPLSoGRE or MPLSoUDP. this doc just calls it a generic VNI for any NVO3 encapsulation 🌈
  • VXLAN: Virtual eXtensible Local Area Network, an NVO3 encapsulation defined in [RFC7348] 🌸

3. Why is EVPN Needed in NVO3 Networks? ✨

okay bestie, so why did data centers go all-in on NVO3? 🦋 mostly because of the issues covered in [RFC7364]. 💖 Nowadays a Data Center is built on a Clos design, where every Leaf hooks up to a layer of Spines, and there are a bunch of ECMPs between any two Leaf nodes. 🌸 Every Leaf-to-Spine link is a routed link, and together they make what we call an underlay IP Fabric. 💅 The underlay IP Fabric doesn't have loop or flooding drama (unlike the old Spanning Tree Data Center designs, ugh 💀), convergence is fast, and ECMP spreads utilization really nicely across all the links. ✨

On this architecture, and as [RFC7364] explains, multi-tenant intra-subnet and inter-subnet connectivity is handled by NVO3 tunnels. 🧚 VXLAN [RFC7348] and Geneve [RFC8926] are two examples of these tunnels.

So why is a control plane protocol alongside NVO3 tunnels so helpful? 👑 There are three main reasons:

  • (a) Auto-discovery of the remote NVEs that are attached to the same VPN instance (Layer 2 and/or Layer 3) as the ingress NVE is. 🔍
  • (b) Dissemination of the MAC/IP host information so that mapping tables can be populated on the remote NVEs. 📒
  • (c) Advanced features such as MAC Mobility, MAC Protection, BUM and ARP/ND traffic reduction/suppression, multihoming, functionality similar to Prefix Independent Convergence (PIC) [BGP-PIC], fast convergence, etc. 🔥

"Flood and learn" is a possible approach to get (a) and (b) above for multipoint Ethernet services. 🌊 It means "flooding" BUM traffic from the ingress NVE to all the egress NVEs attached to the same Broadcast Domain, instead of using a specific control plane on the NVEs. 🙈 The egress NVEs may then use data path source MAC address "learning" on the frames received over the NVO3 tunnels. When the destination host replies and the frames arrive at the NVE that originally flooded the BUM frames, that NVE will also "learn" the source MAC address of the frame encapsulated on the NVO3 tunnel. 🫶 Here's the catch, though. This approach has the following drawbacks:

  • Flood and learn needs the ingress NVE to know the IP addresses of the remote NVEs attached to the same Broadcast Domain, in order to flood a given BUM frame. This may be done as follows:
    • The remote tunnel IP addresses can be statically provisioned on the ingress NVE. If the ingress NVE receives a BUM frame for the Broadcast Domain on an ingress Attachment Circuit, it will do ingress replication and send the frame to all the configured egress NVE destination IP addresses in the Broadcast Domain. 🍓
    • All the NVEs attached to the same Broadcast Domain can subscribe to an underlay IP multicast group that is dedicated to that Broadcast Domain. When an ingress NVE receives a BUM frame on an ingress Attachment Circuit, it sends a single copy of the frame encapsulated into an NVO3 tunnel, using the multicast address as the destination IP address of the tunnel. This solution requires PIM in the underlay network and the association of individual Broadcast Domains to underlay IP multicast groups. 😬
  • "Flood and learn" solves the issues of auto-discovery and of learning the MAC to VNI/tunnel IP mapping on the NVEs for a given Broadcast Domain. 🎉 However, it does not provide a solution for advanced features, and it does not scale well (mostly because of the constant flooding and the underlay PIM states that must be maintained). 💀

EVPN brings a unified control plane that solves NVE auto-discovery, tenant MAC/IP dissemination, and advanced features in a scalable way, and it keeps the independence of the underlay IP Fabric. 💜 In other words, there's no need to enable PIM in the underlay network or maintain multicast states for tenant Broadcast Domains. Iconic, honestly. 👑

Section 4 describes how EVPN can be used to meet the control plane requirements in an NVO3 network. 🦋

4. Applicability of EVPN to NVO3 Networks 🌈

This section talks about how EVPN applies to NVO3 networks. 🧚 It's not meant to be a full, comprehensive explanation of the protocol itself. It's more of an introduction that points you at the right reference document, so you can dig deeper whenever you want. 📚

4.1. EVPN Route Types Used in NVO3 Networks

EVPN supports multiple Route Types, and each type has its own function. 💅 For convenience, Table 1 gives a summary of all the existing EVPN Route Types and what they're used for. In this document, we may refer to these route types as RT-x routes, where x is the type number included in the first column of Table 1. 🎀

+======+================+=======================================+
| Type | Description    | Usage                                 |
+======+================+=======================================+
| 1    | Ethernet Auto- | Multihoming: Used for MAC mass-       |
|      | Discovery      | withdraw when advertised per Ethernet |
|      |                | Segment and for aliasing/backup       |
|      |                | functions when advertised per EVI.    |
+------+----------------+---------------------------------------+
| 2    | MAC/IP         | Host MAC/IP dissemination; supports   |
|      | Advertisement  | MAC Mobility and protection.          |
+------+----------------+---------------------------------------+
| 3    | Inclusive      | NVE discovery and BUM flooding tree   |
|      | Multicast      | setup.                                |
|      | Ethernet Tag   |                                       |
+------+----------------+---------------------------------------+
| 4    | Ethernet       | Multihoming: ES auto-discovery and DF |
|      | Segment        | election.                             |
+------+----------------+---------------------------------------+
| 5    | IP Prefix      | IP Prefix dissemination.              |
+------+----------------+---------------------------------------+
| 6    | Selective      | Indicate interest for a multicast S,G |
|      | Multicast      | or *,G.                               |
|      | Ethernet Tag   |                                       |
+------+----------------+---------------------------------------+
| 7    | Multicast Join | Multihoming: S,G or *,G state synch.  |
|      | Synch          |                                       |
+------+----------------+---------------------------------------+
| 8    | Multicast      | Multihoming: S,G or *,G leave synch.  |
|      | Leave Synch    |                                       |
+------+----------------+---------------------------------------+
| 9    | Per-Region     | BUM tree creation across regions.     |
|      | I-PMSI A-D     |                                       |
+------+----------------+---------------------------------------+
| 10   | S-PMSI A-D     | Multicast tree for S,G or *,G states. |
+------+----------------+---------------------------------------+
| 11   | Leaf A-D       | Used for responses to explicit        |
|      |                | tracking.                             |
+------+----------------+---------------------------------------+

                     Table 1: EVPN Route Types

4.2. EVPN Basic Applicability for Layer 2 Services

Even though EVPN's applicability to NVO3 networks spans multiple documents, its baseline specification is [RFC7432]. 🌸 [RFC7432] allows multipoint Layer 2 VPNs to be operated as IP VPNs [RFC4364], where MACs and the information needed to set up flooding trees are distributed by Multiprotocol BGP (MP-BGP) [RFC4760]. Building on [RFC7432], [RFC8365] describes how to use EVPN to deliver Layer 2 services specifically in NVO3 networks. 💖

Figure 1 shows a Layer 2 service deployed with an EVPN Broadcast Domain in an NVO3 network. 🔍

                            +--TS2---+
                            *        | Single-Active
                            *        |   ESI-1
                          +----+  +----+
                          |BD1 |  |BD1 |
            +-------------|    |--|    |-----------+
            |             +----+  +----+           |
            |              NVE2    NVE3          NVE4
            |           EVPN NVO3 Network       +----+
       NVE1(IP-A)                               | BD1|-----+
      +-------------+      RT-2                 |    |     |
      |             |    +-------+              +----+     |
      |   +----+    |    |MAC1   |               NVE5     TS3
TS1--------|BD1 |    |    |IP1    |              +----+     |
MAC1   |   +----+    |    |Label L|--->          | BD1|-----+
IP1    |             |    |NH IP-A|              |    | All-Active
      | Hypervisor  |    +-------+              +----+  ESI-2
      +-------------+                              |
            +--------------------------------------+

         Figure 1: EVPN for L2 in an NVO3 Network - Example

In a simple NVO3 network, like the example in Figure 1, these are the basic constructs EVPN uses for Layer 2 services (or Layer 2 Virtual Networks): 🧩

  • BD1 is an EVPN Broadcast Domain for a given tenant, and TS1, TS2, and TS3 are connected to it. The five NVEs shown are attached to BD1 and connected to the same underlay IP network. That means each NVE learns the remote NVEs' loopback addresses via the underlay routing protocol. 🌐
  • NVE1 is deployed as a virtual switch in a hypervisor, with IP-A as its underlay loopback IP address. The rest of the NVEs in Figure 1 are physical switches, and TS2/TS3 are multihomed to them. TS1 is a virtual machine, identified by MAC1 and IP1. TS2 and TS3 are physically dual-connected to NVEs, so they're normally not considered virtual machines. 🖥️
  • The terms Single-Active and All-Active in Figure 1 refer to the mode in which TS2 and TS3 are multihomed to the NVEs in BD1. In All-Active mode, all the multihoming links are active and can send or receive traffic. In Single-Active mode, only one link (out of the set of links connected to the NVEs) is active. 🦋

4.2.1. Auto-Discovery and Auto-Provisioning ✨

Okay bestie, auto-discovery is literally one of EVPN's core powers 💅 The NVE setup is super automated, which means less clicking around and fewer chances for a human to typo something 🙈

Here are some of the auto-discovery and auto-provisioning vibes EVPN brings:

  • Automation on Ethernet Segments (ESes) 🌸: An Ethernet Segment is a group of NVEs that are attached to the same Tenant System or network. It gets identified by an Ethernet Segment Identifier (ESI) in the control plane, but neither the ESI nor the NVEs that share the same Ethernet Segment are required to be manually provisioned in the local NVE. Iconic 😭
    • If the multihomed Tenant System or network runs protocols like the Link Aggregation Control Protocol (LACP) [IEEE.802.1AX_2014], the Multiple Spanning Tree Protocol (MSTP), G.8032, etc., and all the NVEs in the Ethernet Segment can listen to the protocol PDUs to uniquely identify the multihomed Tenant System/network, then the ESI can be "auto-sensed" or "auto-provisioned" following the guidelines in Section 5 of [RFC7432]. The ESI can also be auto-derived out of other parameters that are common to all NVEs attached to the same Ethernet Segment. 🦋
    • As described in [RFC7432], EVPN can also auto-derive the BGP parameters required to advertise the presence of a local Ethernet Segment in the control plane (RT and RD). Local Ethernet Segments get advertised using Ethernet Segment routes, and the ESI-import Route Target used by Ethernet Segment routes can be auto-derived based on the procedures of Section 7.6 of [RFC7432]. 💜
    • By listening to other Ethernet Segment routes that match the local ESI and import Route Target, an NVE can also auto-discover the other NVEs participating in the multihoming for the Ethernet Segment. 👀
    • Once the NVE has auto-discovered all the NVEs attached to the same Ethernet Segment, the NVE can automatically perform the Designated Forwarder election algorithm (which determines the NVE that will forward traffic to the multihomed Tenant System/network). EVPN guarantees that all the NVEs in the Ethernet Segment have a consistent Designated Forwarder election. 👑
  • Auto-provisioning of services 🧚: When deploying a Layer 2 service for a tenant in an NVO3 network, all the NVEs attached to the same subnet must be configured with a MAC-VRF and the Broadcast Domain for the subnet, as well as certain parameters for them. Note that if the EVPN service interfaces are VLAN-based or VLAN-bundle, implementations do not normally have a specific provisioning for the Broadcast Domain since, in this case, it is the same construct as the MAC-VRF. EVPN allows auto-deriving as many MAC-VRF parameters as possible. As an example, the MAC-VRF's Route Target and Route Distinguisher for the EVPN routes may be auto-derived. Section 5.1.2.1 of [RFC8365] specifies how to auto-derive a MAC-VRF's Route Target as long as a VLAN-based service interface is implemented. [RFC7432] specifies how to auto-derive the Route Distinguisher. 🎀

4.2.2. Remote NVE Auto-Discovery 🦋

Auto-discovery via MP-BGP [RFC4760] is used to discover the remote NVEs attached to a given Broadcast Domain, the NVEs participating in a given redundancy group, the tunnel encapsulation types supported by an NVE, etc. 🌈

In particular, when a new MAC-VRF and Broadcast Domain are enabled, the NVE will advertise a new Inclusive Multicast Ethernet Tag route. Besides other fields, the Inclusive Multicast Ethernet Tag route will encode the IP address of the advertising NVE, the Ethernet Tag (which is zero in the case of VLAN-based and VLAN-bundle interfaces), and a PMSI Tunnel Attribute (PTA) that indicates the information about the intended way to deliver BUM traffic for the Broadcast Domain. 📡

When BD1 is enabled in the example of Figure 1, NVE1 will send an Inclusive Multicast Ethernet Tag route including its own IP address, an Ethernet-Tag for BD1, and the PMSI Tunnel Attribute to the remote NVEs. Assuming Ingress Replication (IR) is used, the Inclusive Multicast Ethernet Tag route will include an identification for Ingress Replication in the PMSI Tunnel Attribute and the VNI that the other NVEs in the Broadcast Domain must use to send BUM traffic to the advertising NVE. The other NVEs in the Broadcast Domain will import the Inclusive Multicast Ethernet Tag route and will add NVE1's IP address to the flooding list for BD1. Note that the Inclusive Multicast Ethernet Tag route is also sent with a BGP encapsulation attribute [RFC9012] that indicates what NVO3 encapsulation the remote NVEs should use when sending BUM traffic to NVE1. 🫶

Refer to [RFC7432] for more information about the Inclusive Multicast Ethernet Tag route and forwarding of BUM traffic. See [RFC8365] for its considerations on NVO3 networks. 💖

4.2.3. Distribution of Tenant MAC and IP Information 💅

Tenant MAC/IP information is advertised to remote NVEs using MAC/IP Advertisement routes. Following the example of Figure 1:

  • In a given EVPN Broadcast Domain, the MAC addresses of TSes are first learned at the NVE they are attached to via data path or management plane learning. In Figure 1, we assume NVE1 learns MAC1/IP1 in the management plane (for instance, via Cloud Management System) since the NVE is a virtual switch. NVE2, NVE3, NVE4, and NVE5 are ToR/Leaf switches, and they normally learn MAC addresses via data path. 🍓
  • Once NVE1's BD1 learns MAC1/IP1, NVE1 advertises that information along with a VNI and Next-Hop IP-A in a MAC/IP Advertisement route. The EVPN routes are advertised using the Route Distinguisher / Route Targets of the MAC-VRF where the Broadcast Domain belongs. Similarly, all the NVEs in BD1 learn local MAC/IP addresses and advertise them in MAC/IP Advertisement routes. ✨
  • The remote NVEs can then add MAC1 to their mapping table for BD1 (BT). For instance, when TS3 sends frames to NVE4 with the destination MAC address = MAC1, NVE4 does a MAC lookup on the Bridge Table that yields IP-A and Label L. NVE4 can then encapsulate the frame into an NVO3 tunnel with IP-A as the tunnel destination IP address and L as the VNI. Note that the MAC/IP Advertisement route may also contain the host's IP address (as shown in the example of Figure 1). While the MAC of the received MAC/IP Advertisement route is installed in the Bridge Table, the IP address may be installed in the Proxy ARP/ND table (if enabled) or in the ARP/IP-VRF tables if the Broadcast Domain has an IRB. See Section 4.7.3 for more information about Proxy ARP/ND and Section 4.3 for more details about IRB and Layer 3 services. 🌸

Refer to [RFC7432] and [RFC8365] for more information about the MAC/IP Advertisement route and the forwarding of known unicast traffic. 🧚

4.3. EVPN Basic Applicability for Layer 3 Services ✨

Okay so for Layer 3 stuff, the reference docs are [RFC9136] and [RFC9135] 📚 They explain how EVPN handles inter-subnet traffic 🦋 The way it works is that IRB interfaces sit between Broadcast Domains and IP-VRFs. Each EVPN Broadcast Domain maps to one IP subnet. When IP packets from one Broadcast Domain are headed for a different subnet (a different Broadcast Domain) of the same tenant, they get sent to the IRB attached to the local Broadcast Domain on the source NVE. 💖 As [RFC9135] covers, there are two forwarding models based on how packets move between the ingress NVE and the egress NVE: Asymmetric and Symmetric. 💅

The Asymmetric model is shown in Figure 2 🌸 Every Broadcast Domain of the tenant has to be configured on every NVE attached to that tenant. Because every NVE is attached to every subnet, there's no need to advertise IP Prefixes between NVEs. 🫶 It's called "Asymmetric" because the ingress and egress NVEs don't do the same number of lookups in the data plane. 🙈 In Figure 2, say TS1 and TS2 are in different subnets and TS1 sends IP packets to TS2. The data path needs a MAC lookup at BD1's table, an IP lookup at the IP-VRF, a MAC lookup at BD2's table on the ingress NVE1, and then only a MAC lookup at the egress NVE. The two IP-VRFs in Figure 2 aren't connected by tunnels. All the connectivity between NVEs is done over tunnels between the Broadcast Domains. 🧚

                 +-------------------------------------+
                 |             EVPN NVO3               |
                 |                                     |
               NVE1                                 NVE2
         +--------------------+            +--------------------+
         | +---+IRB +------+  |            |  +------+IRB +---+ |
   TS1-----|BD1|----|IP-VRF|  |            |  |IP-VRF|----|BD1| |
         | +---+    |      |  |            |  |      |    +---+ |
         | +---+    |      |  |            |  |      |    +---+ |
         | |BD2|----|      |  |            |  |      |----|BD2|----TS2
         | +---+IRB +------+  |            |  +------+IRB +---+ |
         +--------------------+            +--------------------+
                 |                                     |
                 +-------------------------------------+

Figure 2: EVPN for L3 in an NVO3 Network - Asymmetric Model

In the Symmetric model, shown in Figure 3, the ingress and egress NVEs need the same number of data path lookups. 🔥 For example, if TS1 sends IP packets to TS3, the data path needs a MAC lookup at NVE1's BD1 table, an IP lookup at NVE1's IP-VRF, and then an IP lookup and a MAC lookup at NVE2's IP-VRF and BD3, respectively. In the Symmetric model, inter-subnet connectivity between NVEs runs over tunnels between the IP-VRFs. 👑

                 +-------------------------------------+
                 |             EVPN NVO3               |
                 |                                     |
               NVE1                                 NVE2
         +--------------------+            +--------------------+
         | +---+IRB +------+  |            |  +------+IRB +---+ |
   TS1-----|BD1|----|IP-VRF|  |            |  |IP-VRF|----|BD3|-----TS3
         | +---+    |      |  |            |  |      |    +---+ |
         | +---+IRB |      |  |            |  +------+          |
   TS2-----|BD2|----|      |  |            +--------------------+
         | +---+    +------+  |                        |
         +--------------------+                        |
                 |                                     |
                 +-------------------------------------+

Figure 3: EVPN for L3 in an NVO3 Network - Symmetric Model

The Symmetric model scales better than Asymmetric because NVEs don't need to be attached to all of the tenant's subnets. 🌈 The catch is that it needs NVO3 tunnels on the IP-VRFs, plus IP Prefixes getting exchanged between NVEs in the control plane. EVPN uses MAC/IP Advertisement routes to exchange host IP routes, and IP Prefix routes to exchange prefixes of any length, host routes included. 💜 Using Figure 3 as the example, NVE2 has to advertise TS3's host route and/or TS3's subnet so the IP lookup on NVE1's IP-VRF actually succeeds. 🎀

[RFC9135] specifies using MAC/IP Advertisement routes to advertise host routes. Section 4.4.1 of [RFC9136] specifies using IP Prefix routes to advertise IP Prefixes in an "Interface-less IP-VRF-to-IP-VRF Model". The Symmetric model for host routes can be implemented with either approach:

a. [RFC9135] uses MAC/IP Advertisement routes to pass along the information needed to fill the Layer 2, ARP/ND, and Layer 3 Forwarding Information Base tables on the remote NVE. For instance, in Figure 3, NVE2 would advertise a MAC/IP Advertisement route carrying TS3's IP and MAC addresses, along with two labels / VNIs: a label-3/VNI-3 that identifies BD3 for MAC lookup (used for Layer 2 traffic if NVE1 were attached to BD3 too), and a label-1/VNI-1 that identifies the IP-VRF for IP lookup (used for Layer 3 traffic). NVE1 imports the MAC/IP Advertisement route and installs TS3's IP in the IP-VRF route table with label-1/VNI-1. Traffic, for example from TS2 to TS3, gets encapsulated with label-1/VNI-1 and forwarded to NVE2. 🦋

b. [RFC9136] uses MAC/IP Advertisement routes to pass along the information for the Layer 2 Forwarding Information Base and ARP/ND tables, and IP Prefix routes to fill the IP-VRF Layer 3 Forwarding Information Base table. For instance, in Figure 3, NVE2 would advertise a MAC/IP Advertisement route with TS3's MAC and IP addresses and a single label-3/VNI-3. In this example, NVE1 wouldn't import that MAC/IP Advertisement route because NVE1 isn't attached to BD3. NVE2 would also advertise an IP Prefix route with TS3's IP address and label-1/VNI-1. NVE1's IP-VRF imports that IP Prefix route, and the host route gets installed in the Layer 3 Forwarding Information Base associated with label-1/VNI-1. Traffic from TS2 to TS3 gets encapsulated with label-1/VNI-1. 💅

4.4. EVPN as Control Plane for NVO3 Encapsulations and Geneve 🧁

[RFC8365] explains how to use EVPN for NVO3 encapsulations like VXLAN, nvGRE, or MPLSoGRE. 🌸 The procedures can easily be applied to any other NVO3 encapsulation, especially Geneve.

Geneve [RFC8926] is the proposed standard encapsulation specified in the IETF Network Virtualization Overlays Working Group. 🫶 The EVPN control plane can signal the Geneve encapsulation type in the BGP Tunnel Encapsulation Extended Community (see [RFC9012]).

Geneve needs a control plane [NVO3-ENCAP] to do the following:

  • Negotiate a subset of Geneve option TLVs that can be carried on a Geneve tunnel,
  • Enforce an order for Geneve option TLVs, and
  • Limit the total number of options that could be carried on a Geneve tunnel.

The EVPN control plane can easily extend the BGP Tunnel Encapsulation attribute sub-TLV [RFC9012] to specify the Geneve tunnel options that a given NVE can receive or transmit over a Geneve tunnel. 🧚 [EVPN-GENEVE] describes the EVPN control plane extensions that support Geneve.

4.5. EVPN OAM and Application to NVO3 🔍

EVPN Operations, Administration, and Maintenance (OAM), as described in [EVPN-LSP-PING], defines mechanisms for detecting data plane failures in an EVPN deployment over an MPLS network. 💖 These mechanisms detect failures related to point-to-point (P2P) and Point-to-Multipoint (P2MP) connectivity, for multi-tenant unicast and multicast Layer 2 traffic, between multi-tenant access nodes connected to EVPN PE(s). They cover single-homed, Single-Active, and All-Active redundancy models.

In general, EVPN OAM mechanisms defined for EVPN deployed in MPLS networks are equally applicable to EVPN in NVO3 networks. ✨

4.6. EVPN as the Control Plane for NVO3 Security 🔐

EVPN can signal the security protection capabilities of a sender NVE, plus which portion of an NVO3 packet (a Geneve packet being the example) the sender NVE can protect. This helps protect the privacy and integrity of tenant traffic carried over NVO3 tunnels [SECURE-EVPN]. 🛡️💜

4.7. Advanced EVPN Features for NVO3 Networks 👑

This section covers how EVPN can deliver advanced capabilities in NVO3 networks. 🌈

4.7.1. Virtual Machine (VM) Mobility 🚀

[RFC7432] swaps the classic Ethernet "flood and learn" behavior among NVEs for BGP-based MAC learning. 🙈 In return, you get more control over where MAC addresses live in the Broadcast Domain, and that unlocks advanced features like MAC Mobility. If we assume Virtual Machine (VM) Mobility means the VM's MAC and IP addresses move with the VM, then EVPN's MAC Mobility is the required procedure that makes VM Mobility happen. 🦋

According to Section 15 of [RFC7432], when a MAC is advertised for the first time in a Broadcast Domain, all the NVEs attached to that Broadcast Domain store Sequence Number zero for that MAC. When the MAC "moves" to a remote NVE within the same Broadcast Domain, the NVE that just learned the MAC locally increases the Sequence Number in the MAC/IP Advertisement route's MAC Mobility extended community to show that it owns the MAC now. 💅 That makes every NVE in the Broadcast Domain update its tables immediately, with no need to wait for any aging timer. ⭐ EVPN guarantees fast MAC Mobility without flooding or packet drops in the Broadcast Domain. 🎀

4.7.2. MAC Protection, Duplication Detection, and Loop Protection 🔐

okay so bestie, advertising MACs in the control plane unlocks some seriously iconic features: MAC Protection, Duplication Detection, and Loop Protection 💅✨

First up, MAC Protection. In a MAC/IP Advertisement route, this is EVPN's way of telling the receiving NVE that a MAC must be protected [RFC7432]. The protection gets flagged in the "Sticky bit" of the MAC Mobility extended community that rides along with the MAC/IP Advertisement route for that MAC. NVEs whose Attachment Circuits connect to servers or VMs that need protecting can set the Sticky bit on the MAC/IP Advertisement routes they send for those MACs 🛡️ Also, statically configured MAC addresses should be advertised as Protected MAC addresses, since they don't go through MAC Mobility procedures. Main character energy for those static MACs 👑

Next, MAC Duplication Detection, which is EVPN's ability to catch duplicate MAC addresses [RFC7432] 🕵️‍♀️ A "MAC move" is a relearn event. It happens either at an access Attachment Circuit or when a MAC/IP Advertisement route shows up with a Sequence Number higher than the one stored for that MAC. So if a MAC keeps bouncing between two NVEs (N times) inside an M-second window, it gets declared a duplicate, and the detecting NVE stops re-advertising it. Wild, right? 🙈

[RFC7432] gives us MAC Duplication Detection, and with an extension it can also protect the Broadcast Domain from loops caused by backdoor links between NVEs 🔁 The same Sequence Number principle can be stretched to protect the Broadcast Domain against loops. When a MAC gets flagged as a duplicate, the NVE may install it as a drop-MAC and discard received frames whose source MAC address or destination MAC address matches that duplicate MAC. The MAC Duplication extension for Loop Protection is described in Section 15.3 of [RFC7432BIS]. It's giving security guard energy 💜

4.7.3. Reduction/Optimization of BUM Traffic in Layer 2 Services 📉

Okay, so in Broadcast Domains that get flooded a lot by Unknown Unicast and broadcast frames, EVPN can help cut that down and sometimes even shut it off completely 🌊➡️🫧

In Broadcast Domains where most of the broadcast traffic comes from the Address Resolution Protocol (ARP) and the Neighbor Discovery Protocol (NDP) on the Tenant Systems, EVPN's Proxy ARP and Proxy ND capabilities can reduce the flooding drastically 💖 The use of Proxy ARP/ND is specified in [RFC9161].

Proxy ARP/ND procedures come with an assumption: Tenant Systems always send a Gratuitous ARP (GARP) or an unsolicited Neighbor Advertisement message when they come up in the Broadcast Domain. If that holds, Proxy ARP/ND can drastically reduce Unknown Unicast flooding in the Broadcast Domain 🧚

The flooding caused by Tenant Systems' IGMP / Multicast Listener Discovery (MLD) or PIM messages in the Broadcast Domain can also be suppressed with IGMP/MLD and PIM Proxy functions, as specified in [RFC9251] and [EVPN-PIM-PROXY]. Those two documents also explain how to forward IP multicast traffic efficiently within the same Broadcast Domain, turn soft state IGMP/MLD/PIM messages into hard state BGP routes, and give fast convergence redundancy for IP multicast on multihomed ESes 🎀 Soft state into hard state. Iconic glow-up. ✨

4.7.4. Ingress Replication (IR) Optimization for BUM Traffic 🚀

So when an NVE attached to a Broadcast Domain needs to send BUM traffic to the remote NVEs on that same Broadcast Domain, Ingress Replication is a super common pick in NVO3 networks 🙌 It's completely independent of the underlay's multicast capabilities, which is so reliable. If the flooding reduction and suppression procedures are turned on (Section 4.7.3), Ingress Replication might be good enough, even though it creates multiple copies of the same frame at the ingress NVE. But in Broadcast Domains where Multicast (or Broadcast) traffic is heavy, Ingress Replication can be very inefficient and cause performance issues on virtual switch-based NVEs 😭

[EVPN-OPT-IR] specifies the use of Assisted Replication (AR) NVO3 tunnels in EVPN Broadcast Domains. AR keeps the underlay independent while still forwarding Broadcast and multicast traffic efficiently 🦋 It uses AR-REPLICATORs that replicate the broadcast/multicast traffic on behalf of the AR-LEAF NVEs. AR-LEAF NVEs are usually virtual switches or NVEs with limited replication capabilities. AR can run in a single-stage replication mode (Non-Selective Mode) or a dual-stage replication mode (Selective Mode). Both modes are detailed in [EVPN-OPT-IR].

[EVPN-OPT-IR] also describes a way to avoid sending BUM traffic to NVEs that don't need that type of traffic. You do this by enabling Pruned Flood Lists (PFLs) on a given Broadcast Domain 🌸 For example, a virtual switch NVE that learns all its local MAC addresses for a Broadcast Domain through a Cloud Management System doesn't need to receive that Broadcast Domain's Unknown Unicast traffic. PFLs help optimize BUM flooding in the Broadcast Domain. Not everyone needs to be at the party 🎉

4.7.5. EVPN Multihoming 🫶

Another fundamental EVPN concept is multihoming. A given Tenant System can be multihomed to two or more NVEs for a given Broadcast Domain, and the set of links connected to that same Tenant System is defined as an ES. EVPN supports Single-Active and All-Active multihoming. In Single-Active multihoming, only one link in the Ethernet Segment is active. In All-Active multihoming, all the links in the Ethernet Segment are active for unicast traffic. Both modes support load-balancing:

  • Single-Active multihoming means per-service load-balancing to and from the Tenant System. For example, in Figure 1 for BD1, only one of the NVEs can forward traffic from and to TS2. For a different Broadcast Domain, the other NVE may forward traffic.
  • All-Active multihoming means per-flow load-balancing for unicast frames to and from the Tenant System. That is, in Figure 1 and for BD1, both NVE4 and NVE5 can forward known unicast traffic to and from TS3. For BUM traffic, only one of the two NVEs can forward traffic to TS3, and both can forward traffic from TS3.

There are two key aspects to EVPN multihoming procedures 👀

Designated Forwarder (DF) election: The Designated Forwarder is the NVE that forwards traffic to the Ethernet Segment in Single-Active mode. In All-Active mode, the Designated Forwarder is the NVE that forwards BUM traffic to the Ethernet Segment.

Split-horizon function: This prevents the Tenant System from receiving echoed BUM frames that it sent itself to the Ethernet Segment. It matters most in All-Active ESes, where the TS may forward BUM frames to a Non-Designated Forwarder NVE that floods them back to the Designated Forwarder NVE, which then sends them back to the TS 🙃 As an example, assume NVE4 is the Designated Forwarder for ESI-2 in BD1. Figure 1 shows that BUM frames sent from TS3 to NVE5 will be received at NVE4. NVE4 will forward them back to TS3 since NVE4 is the Designated Forwarder for BD1. Split-horizon lets NVE4 (and any multihomed NVE, for that matter) tell whether an EVPN BUM frame comes from the same Ethernet Segment or a different one. If the frame belongs to the same ESI-2, NVE4 will not forward the BUM frame to TS3, even though it is the Designated Forwarder ✋

While [RFC7432] describes the default algorithm for Designated Forwarder election, [RFC8584] and [EVPN-PREF-DF] specify other algorithms and procedures that optimize DF election 💫

The split-horizon function is specified in [RFC7432]. It's carried out with a special ESI-label, which identifies in the data path all the BUM frames originating from a given NVE and Ethernet Segment. Since the ESI-label is an MPLS label, it can't be used in all the non-MPLS NVO3 encapsulations. That's why [RFC8365] defines a modified split-horizon procedure based on the source IP address of the NVO3 tunnel, known as "Local-Bias" 🦄 Worth knowing: Local-Bias only works for All-Active multihoming, not for Single-Active multihoming ⚠️

Local-Bias is All-Active only, bestie. Don't try to use it with Single-Active multihoming 🚫

4.7.6. EVPN Recursive Resolution for Inter-subnet Unicast Forwarding 🧩

Section 4.3 describes how EVPN can be used for inter-subnet forwarding among subnets of the same tenant. MAC/IP Advertisement routes and IP Prefix routes let you advertise host routes and IP Prefixes of any length 🌈 The procedures in Section 4.3 are similar to the ones in [RFC4364], but they only apply to NVO3 tunnels. [RFC9136] goes further with advanced inter-subnet forwarding procedures. These allow IP Prefix routes to resolve not only to BGP next hops but also to "overlay indexes" that can be a MAC, a Gateway IP (GW-IP), or an ESI, all of them in the tenant space 💅

Figure 4 shows an example that uses Recursive Resolution to a GW-IP, as described in Section 4.4.2 of [RFC9136]. In this example, IP-VRFs in NVE1 and NVE2 are connected by a Supplementary Broadcast Domain (SBD). An SBD is a Broadcast Domain that connects all the IP-VRFs of the same tenant via IRB and has no Attachment Circuits 🧵 NVE1 advertises the host route TS2-IP/L (IP address and Prefix Length of TS2) in an IP Prefix route with overlay index GW-IP=IP1. IP1 is also advertised in a MAC/IP Advertisement route associated with M1, VNI-S, and BGP next-hop NVE1. When NVE2 imports the two routes, it installs TS2-IP/L in the IP-VRF with a next hop of the GW-IP IP1. NVE2 also installs M1 in the Supplementary Broadcast Domain, with VNI-S and NVE1 as the next hop. If TS3 sends a packet with IP DA=TS2, NVE2 performs a Recursive Resolution of the IP Prefix route prefix information to the forwarding information of the correlated MAC/IP Advertisement route 🔍

The IP Prefix route's Recursive Resolution has several advantages. It gives better convergence in scaled networks, since multiple IP Prefix routes can be invalidated with a single withdrawal of the overlay index route 📉 It also lets you advertise multiple IP Prefix routes from an overlay index that can move or change dynamically 🏃‍♀️ [RFC9136] describes a few use cases 📚

                 +-------------------------------------+
                 |             EVPN NVO3               |
                 |                                     +
               NVE1                                 NVE2
         +--------------------+            +--------------------+
         | +---+IRB +------+  |            |  +------+IRB +---+ |
   TS1-----|BD1|----|IP-VRF|  |            |  |IP-VRF|----|BD3|-----TS3
         | +---+    |      |-(SBD)------(SBD)-|      |    +---+ |
         | +---+IRB |      |IRB(IP1/M1)    IRB+------+          |
   TS2-----|BD2|----|      |  |            +-----------+--------+
         | +---+    +------+  |                        |
         +--------------------+                        |
                 |   RT-2(M1,IP1,VNI-S,NVE1)-->        |
                 |     RT-5(TS2-IP/L,GW-IP=IP1)-->     |
                 +-------------------------------------+

            Figure 4: EVPN for L3 - Recursive Resolution Example

That's the whole vibe: one GW-IP, many prefixes, zero drama. 🌈💖

4.7.7. EVPN Optimized Inter-subnet Multicast Forwarding ✨

Okay bestie, remember the Supplementary Broadcast Domain from Section 4.7.6? 💖 [EVPN-IRB-MCAST] uses that same concept for inter-subnet multicast across Broadcast Domains of the same tenant 🦋. It gives efficient forwarding of IP multicast traffic from any Broadcast Domain to any other Broadcast Domain, or even back to the same Broadcast Domain where the source lives. Iconic 👑. The [EVPN-IRB-MCAST] procedures work alongside EVPN multihoming and any tree allowed on NVO3 networks, including IR or AR 🌈. It also explains how EVPN and other multicast technologies, like Multicast VPN (MVPN) and PIM, work together for inter-subnet multicast 🫶.

[EVPN-MVPN-SEAMLESS] is another potential solution for EVPN to MVPN interoperability 🧚.

4.7.8. Data Center Interconnect (DCI) 🌸

Tenant Layer 2 and Layer 3 services on NVO3 networks often need to stretch to remote NVO3 networks that are connected through non-NOV3 Wide Area Networks (WANs), mostly MPLS-based WANs 🌍. [RFC9014] defines some architectural models you can use to interconnect NVO3 networks over MPLS WANs 💅.

When NVO3 networks are connected by MPLS WANs, [RFC9014] specifies how EVPN can be used end to end, even though the WAN uses a different encapsulation ✨. [RFC9014] also supports NVO3 or Segment Routing transport tunnels in the WAN. Segment Routing encodes 32-bit or 128-bit Segment Identifiers into labels or IPv6 addresses, respectively 🎀.

EVPN can also run in the WAN for Layer 2 and Layer 3 services, but there may be a need for a Gateway function between EVPN for NVO3 encapsulations and IP VPN for MPLS tunnels if the operator uses IP VPN in the WAN 🦋. [EVPN-IPVPN-INTERWORK] specifies the interworking function between EVPN and IP VPN for unicast inter-subnet forwarding 💖. If inter-subnet multicast forwarding is also needed across an IP VPN WAN, [EVPN-IRB-MCAST] describes the required interworking between EVPN and MVPNs 🔥.

5. Security Considerations 🔐

This document does not introduce any new procedure or additional signaling in EVPN. It relies on the security considerations of the individual specifications used as references throughout the document. In particular, and as mentioned in [RFC7432], control plane and forwarding path protection are aspects to secure in any EVPN domain when applied to NVO3 networks 🚨.

[RFC7432] mentions security techniques such as those discussed in [RFC5925] for authenticating BGP messages. The techniques included in [RFC4271], [RFC4272], and [RFC6952] for securing BGP are also relevant for EVPN in NVO3 networks 💀.

6. IANA Considerations 🌷

This document has no IANA actions.

7. References 📚

7.1. Normative References

7.2. Informative References 📚✨

Okay bestie, these are the extra reads that back up the main text. Here's the full list, copied exactly so nothing gets lost 🫶

[BGP-PIC]  Bashandy, A., Ed., Filsfils, C., and P. Mohapatra, "BGP
           Prefix Independent Convergence", Work in Progress,
           Internet-Draft, draft-ietf-rtgwg-bgp-pic-19, 1 April 2023,
           <https://datatracker.ietf.org/doc/html/draft-ietf-rtgwg-
           bgp-pic-19>.

[CLOS1953] Clos, C., "A study of non-blocking switching networks",
           The Bell System Technical Journal, Vol. 32, Issue 2,
           DOI 10.1002/j.1538-7305.1953.tb01433.x, March 1953,
           <https://ieeexplore.ieee.org/document/6770468>.

[EVPN-GENEVE]
           Boutros, S., Ed., Sajassi, A., Drake, J., Rabadan, J., and
           S. Aldrin, "EVPN control plane for Geneve", Work in
           Progress, Internet-Draft, draft-ietf-bess-evpn-geneve-06,
           26 May 2023, <https://datatracker.ietf.org/doc/html/draft-
           ietf-bess-evpn-geneve-06>.

[EVPN-IPVPN-INTERWORK]
           Rabadan, J., Ed., Sajassi, A., Ed., Rosen, E., Drake, J.,
           Lin, W., Uttaro, J., and A. Simpson, "EVPN Interworking
           with IPVPN", Work in Progress, Internet-Draft, draft-ietf-
           bess-evpn-ipvpn-interworking-08, 5 July 2023,
           <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
           evpn-ipvpn-interworking-08>.

[EVPN-IRB-MCAST]
           Lin, W., Zhang, Z., Drake, J., Rosen, E., Ed., Rabadan,
           J., and A. Sajassi, "EVPN Optimized Inter-Subnet Multicast
           (OISM) Forwarding", Work in Progress, Internet-Draft,
           draft-ietf-bess-evpn-irb-mcast-09, 21 February 2023,
           <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
           evpn-irb-mcast-09>.

[EVPN-LSP-PING]
           Jain, P., Sajassi, A., Salam, S., Boutros, S., and G.
           Mirsky, "LSP-Ping Mechanisms for EVPN and PBB-EVPN", Work
           in Progress, Internet-Draft, draft-ietf-bess-evpn-lsp-
           ping-11, 29 May 2023,
           <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
           evpn-lsp-ping-11>.

[EVPN-MVPN-SEAMLESS]
           Sajassi, A., Thiruvenkatasamy, K., Thoria, S., Gupta, A.,
           and L. Jalil, "Seamless Multicast Interoperability between
           EVPN and MVPN PEs", Work in Progress, Internet-Draft,
           draft-ietf-bess-evpn-mvpn-seamless-interop-05, 13 March
           2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
           bess-evpn-mvpn-seamless-interop-05>.

[EVPN-OPT-IR]
           Rabadan, J., Ed., Sathappan, S., Lin, W., Katiyar, M., and
           A. Sajassi, "Optimized Ingress Replication Solution for
           Ethernet VPN (EVPN)", Work in Progress, Internet-Draft,
           draft-ietf-bess-evpn-optimized-ir-12, 25 January 2022,
           <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
           evpn-optimized-ir-12>.

[EVPN-PIM-PROXY]
           Rabadan, J., Ed., Kotalwar, J., Sathappan, S., Zhang, Z.,
           and A. Sajassi, "PIM Proxy in EVPN Networks", Work in
           Progress, Internet-Draft, draft-skr-bess-evpn-pim-proxy-
           01, 30 October 2017,
           <https://datatracker.ietf.org/doc/html/draft-skr-bess-
           evpn-pim-proxy-01>.

[EVPN-PREF-DF]
           Rabadan, J., Ed., Sathappan, S., Lin, W., Drake, J., and
           A. Sajassi, "Preference-based EVPN DF Election", Work in
           Progress, Internet-Draft, draft-ietf-bess-evpn-pref-df-11,
           6 July 2023, <https://datatracker.ietf.org/doc/html/draft-
           ietf-bess-evpn-pref-df-11>.

[IEEE.802.1AX_2014]
           IEEE, "IEEE Standard for Local and metropolitan area
           networks -- Link Aggregation", IEEE Std 802.1AX-2014,
           DOI 10.1109/IEEESTD.2014.7055197, December 2014,
           <https://doi.org/10.1109/IEEESTD.2014.7055197>.

[NVO3-ENCAP]
           Boutros, S., Ed. and D. Eastlake 3rd, Ed., "Network
           Virtualization Overlays (NVO3) Encapsulation
           Considerations", Work in Progress, Internet-Draft, draft-
           ietf-nvo3-encap-09, 7 October 2022,
           <https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-
           encap-09>.

[RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
           Border Gateway Protocol 4 (BGP-4)", RFC 4271,
           DOI 10.17487/RFC4271, January 2006,
           <https://www.rfc-editor.org/info/rfc4271>.

[RFC4272]  Murphy, S., "BGP Security Vulnerabilities Analysis",
           RFC 4272, DOI 10.17487/RFC4272, January 2006,
           <https://www.rfc-editor.org/info/rfc4272>.

[RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
           Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
           2006, <https://www.rfc-editor.org/info/rfc4364>.

[RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
           "Multiprotocol Extensions for BGP-4", RFC 4760,
           DOI 10.17487/RFC4760, January 2007,
           <https://www.rfc-editor.org/info/rfc4760>.

[RFC5925]  Touch, J., Mankin, A., and R. Bonica, "The TCP
           Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
           June 2010, <https://www.rfc-editor.org/info/rfc5925>.

[RFC6952]  Jethanandani, M., Patel, K., and L. Zheng, "Analysis of
           BGP, LDP, PCEP, and MSDP Issues According to the Keying
           and Authentication for Routing Protocols (KARP) Design
           Guide", RFC 6952, DOI 10.17487/RFC6952, May 2013,
           <https://www.rfc-editor.org/info/rfc6952>.

[RFC7348]  Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
           L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
           eXtensible Local Area Network (VXLAN): A Framework for
           Overlaying Virtualized Layer 2 Networks over Layer 3
           Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
           <https://www.rfc-editor.org/info/rfc7348>.

[RFC7432BIS]
           Sajassi, A., Burdet, L., Drake, J., and J. Rabadan, "BGP
           MPLS-Based Ethernet VPN", Work in Progress, Internet-
           Draft, draft-ietf-bess-rfc7432bis-07, 13 March 2023,
           <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
           rfc7432bis-07>.

[RFC7510]  Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black,
           "Encapsulating MPLS in UDP", RFC 7510,
           DOI 10.17487/RFC7510, April 2015,
           <https://www.rfc-editor.org/info/rfc7510>.

[RFC8365]  Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
           Uttaro, J., and W. Henderickx, "A Network Virtualization
           Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
           DOI 10.17487/RFC8365, March 2018,
           <https://www.rfc-editor.org/info/rfc8365>.

[RFC8584]  Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake,
           J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet
           VPN Designated Forwarder Election Extensibility",
           RFC 8584, DOI 10.17487/RFC8584, April 2019,
           <https://www.rfc-editor.org/info/rfc8584>.

[RFC8926]  Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
           "Geneve: Generic Network Virtualization Encapsulation",
           RFC 8926, DOI 10.17487/RFC8926, November 2020,
           <https://www.rfc-editor.org/info/rfc8926>.

[RFC9012]  Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
           "The BGP Tunnel Encapsulation Attribute", RFC 9012,
           DOI 10.17487/RFC9012, April 2021,
           <https://www.rfc-editor.org/info/rfc9012>.

[RFC9014]  Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
           A., and J. Drake, "Interconnect Solution for Ethernet VPN
           (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014,
           May 2021, <https://www.rfc-editor.org/info/rfc9014>.

[RFC9135]  Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
           Rabadan, "Integrated Routing and Bridging in Ethernet VPN
           (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
           <https://www.rfc-editor.org/info/rfc9135>.

[RFC9136]  Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
           A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
           (EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
           <https://www.rfc-editor.org/info/rfc9136>.

[RFC9161]  Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Hankins, G.,
           and T. King, "Operational Aspects of Proxy ARP/ND in
           Ethernet Virtual Private Networks", RFC 9161,
           DOI 10.17487/RFC9161, January 2022,
           <https://www.rfc-editor.org/info/rfc9161>.

[RFC9251]  Sajassi, A., Thoria, S., Mishra, M., Patel, K., Drake, J.,
           and W. Lin, "Internet Group Management Protocol (IGMP) and
           Multicast Listener Discovery (MLD) Proxies for Ethernet
           VPN (EVPN)", RFC 9251, DOI 10.17487/RFC9251, June 2022,
           <https://www.rfc-editor.org/info/rfc9251>.

[SECURE-EVPN]
           Sajassi, A., Banerjee, A., Thoria, S., Carrel, D., Weis,
           B., and J. Drake, "Secure EVPN", Work in Progress,
           Internet-Draft, draft-ietf-bess-secure-evpn-00, 20 June
           2023, <https://datatracker.ietf.org/doc/html/draft-ietf-
           bess-secure-evpn-00>.

Acknowledgments 💖

Huge thanks to Aldrin Isaac, who left comments that the authors appreciated 🫶

Authors' Addresses 🏡

Jorge Rabadan (editor)
Nokia
520 Almanor Ave
Sunnyvale, CA 94085
United States of America
Email: jorge.rabadan@nokia.com


Matthew Bocci
Nokia
Email: matthew.bocci@nokia.com


Sami Boutros
Ciena
Email: sboutros@ciena.com


Ali Sajassi
Cisco Systems, Inc.
Email: sajassi@cisco.com
📦 where this page comes from