RFC 9136
IP Prefix Advertisement in Ethernet VPN (EVPN)
RFC 9136: IP Prefix Advertisement in Ethernet VPN (EVPN) 🦋
Okay bestie, let's start with the title block 💖. This is RFC 9136 from the IETF (Internet Engineering Task Force), and it's called "IP Prefix Advertisement in Ethernet VPN (EVPN)". It's a Standards Track document, ISSN 2070-1721, dated October 2021. J. Rabadan is the editor, and W. Henderickx (Nokia), J. Drake and W. Lin (Juniper), and A. Sajassi (Cisco) are the other authors. The title block doesn't list anything this one obsoletes or updates, so there's nothing to chase there 🙈
Abstract
Okay so the BGP MPLS-based Ethernet VPN (EVPN) mechanism, which is RFC 7432 🎀, gives you a flexible control plane for intra-subnet connectivity in an MPLS and/or Network Virtualization Overlay (NVO) network, which is RFC 7365 ✨. But some networks also need dynamic and efficient inter-subnet connectivity between Tenant Systems and end devices. Those can be physical or virtual, and they don't have to take part in dynamic routing protocols. This document defines a brand-new EVPN route type for advertising IP prefixes, and it walks through some use-case examples showing where it's used. It's giving main character energy 💅
Status of This Memo
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841 🧚
Information about the current status of this document, any errata, and how to give feedback on it can be found at https://www.rfc-editor.org/info/rfc9136 🌸
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date this document was published. Please review these documents carefully, because they spell out your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 📜
Table of Contents
1. Introduction
1.1. Terminology
2. Problem Statement
2.1. Inter-Subnet Connectivity Requirements in Data Centers
2.2. The Need for the EVPN IP Prefix Route
3. The BGP EVPN IP Prefix Route
3.1. IP Prefix Route Encoding
3.2. Overlay Indexes and Recursive Lookup Resolution
4. Overlay Index Use Cases
4.1. TS IP Address Overlay Index Use Case
4.2. Floating IP Overlay Index Use Case
4.3. Bump-in-the-Wire Use Case
4.4. IP-VRF-to-IP-VRF Model
4.4.1. Interface-less IP-VRF-to-IP-VRF Model
4.4.2. Interface-ful IP-VRF-to-IP-VRF with SBD IRB
4.4.3. Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB
5. Security Considerations
6. IANA Considerations
7. References
7.1. Normative References
7.2. Informative References
Acknowledgments
Contributors
Authors' Addresses1. Introduction ✨
Okay, so [RFC7365] is the framework for Data Center (DC) Network Virtualization over Layer 3 🌈. It says Network Virtualization Edge (NVE) devices have to deliver Layer 2 and Layer 3 virtualized network services in multi-tenant DCs. [RFC8365] talks about using EVPN as the go-to tech for Layer 2 or intra-subnet services in these DCs. This document, together with [RFC9135], covers using EVPN for Layer 3 or inter-subnet connectivity services 💖
[RFC9135] describes some pretty common inter-subnet forwarding scenarios, where Tenant Systems (TSs) swap packets with TSs sitting in remote subnets. To make that work, [RFC9135] explains how Media Access Control (MAC) addresses and IPs encoded in TS RT-2 routes don't just fill up the MAC Virtual Routing and Forwarding (MAC-VRF) and overlay Address Resolution Protocol (ARP) tables. They also fill up the IP-VRF tables with the encoded TS host routes (/32 or /128) 🦋. Sometimes EVPN can advertise IP prefixes instead, which gives you aggregation in the IP-VRF tables rather than passing along every single host route 🫶. This document builds on the scenarios in [RFC9135] and defines how EVPN can be used to advertise IP prefixes. Interoperability between EVPN and Layer 3 Virtual Private Network (VPN) [RFC4364] IP Prefix routes is out of scope here 🙈
Section 2.1 covers the inter-subnet connectivity requirements in DCs. Section 2.2 explains why a new EVPN route type is needed for IP prefix advertisements. Sections 3, 4, and 5 describe this route type and how it gets used in some specific use cases ✨
1.1. Terminology
Okay, this part is the key-word decoder ring 💍. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in BCP 14 [RFC2119] [RFC8174], but only when they show up in all capitals, exactly like that 🔥
Here's the glossary, bestie 📖
- AC: Attachment Circuit
- ARP: Address Resolution Protocol
- BD: Broadcast Domain. As per [RFC7432], an EVI is made of a single BD or multiple BDs. In the VLAN-bundle and VLAN-based service models (see [RFC7432]), a BD is the same thing as an EVI. In the VLAN-aware bundle service model, an EVI holds multiple BDs. Also, in this document, "BD" and "subnet" mean the same thing 🌸
- BD Route Target: This refers to the broadcast-domain-assigned Route Target [RFC4364]. In the VLAN-aware bundle service model, every BD instance in the MAC-VRF shares the same Route Target 💜
- BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per [RFC7432].
- CE: Customer Edge
- DA: Destination Address
- DGW: Data Center Gateway
- Ethernet A-D Route: Ethernet Auto-Discovery (A-D) route, as per [RFC7432].
- Ethernet NVO Tunnel: This refers to Network Virtualization Overlay tunnels that carry an Ethernet payload. VXLAN and GENEVE are examples of this kind of tunnel 🍓
- EVI: EVPN Instance, spanning the NVE/PE devices that take part in that EVPN, as per [RFC7432].
- EVPN: Ethernet VPN, as per [RFC7432].
- GENEVE: Generic Network Virtualization Encapsulation, as per [RFC8926].
- GRE: Generic Routing Encapsulation
- GW IP: Gateway IP address
- IPL: IP Prefix Length
- IP NVO Tunnel: This refers to Network Virtualization Overlay tunnels that carry an IP payload (no MAC header in the payload) 🧚
- IP-VRF: A Virtual Routing and Forwarding table for IP routes on an NVE/PE. The IP routes could be populated by the EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a Layer 3 VPN in an NVE/PE 👑
- IRB: Integrated Routing and Bridging interface. It connects an IP-VRF to a BD (or subnet).
- MAC: Media Access Control
- MAC-VRF: A Virtual Routing and Forwarding table for MAC addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is also an instantiation of an EVI in an NVE/PE.
- ML: MAC Address Length
- ND: Neighbor Discovery
- NVE: Network Virtualization Edge
- NVO: Network Virtualization Overlay
- PE: Provider Edge
- RT-2: EVPN Route Type 2, i.e., MAC/IP Advertisement route, as defined in [RFC7432].
- RT-5: EVPN Route Type 5, i.e., IP Prefix route, as defined in Section 3. This is the new one, bestie 🎀
- SBD: Supplementary Broadcast Domain. A BD that has no ACs, only IRB interfaces, and is used to provide connectivity among all the IP-VRFs of the tenant. The SBD is only required in the IP-VRF-to-IP-VRF use cases (see Section 4.4) 💅
- SN: Subnet
- TS: Tenant System
- VA: Virtual Appliance
- VM: Virtual Machine
- VNI: Virtual Network Identifier. As in [RFC8365], the term is used as a representation of a 24-bit NVO instance identifier, with the understanding that "VNI" refers to a VXLAN Network Identifier in VXLAN, or a Virtual Network Identifier in GENEVE, etc., unless it is stated otherwise 🌈
- VSID: Virtual Subnet Identifier
- VTEP: VXLAN Termination End Point, as per [RFC7348].
- VXLAN: Virtual eXtensible Local Area Network, as per [RFC7348].
This document also assumes you're familiar with the terminology of [RFC7365], [RFC7432], and [RFC8365] 🤓
2. Problem Statement 🌸
This section is where we get into the actual problem 😭. It describes the inter-subnet connectivity requirements in DCs, and explains why a specific route type is needed to advertise IP prefixes.
2.1. Inter-Subnet Connectivity Requirements in Data Centers 🏢✨
Okay bestie, so [RFC7432] is the control plane for an NVO solution in DCs 💅. NVE devices can live in hypervisors or Top-of-Rack (ToR) switches, as described in [RFC8365] 🌸.
These are the considerations for TSs (physical or virtual systems identified by MAC, and possibly IP addresses) that are connected to BDs by Attachment Circuits 🦋:
- The Tenant Systems may be VMs that generate traffic from their own MAC and IP 💖.
- The Tenant Systems may be VA entities that forward traffic to/from IP addresses of different end devices sitting behind them 🧚.
- These VAs can be firewalls, load balancers, NAT devices, other appliances, or virtual gateways with virtual routing instances 🎀.
- These VAs don't necessarily join dynamic routing protocols, so they rely on the EVPN NVEs to advertise the routes on their behalf 🫶.
- In all these cases, the VA forwards traffic to other TSs using its own source MAC. The source IP will be the one tied to the end device behind the VA, or a translated IP address (part of a public NAT pool) if the VA is doing NAT 🌈.
- Note that the same IP address and endpoint could exist behind two of these TSs 👀. One example is certain appliance resiliency mechanisms, where a virtual IP or floating IP can be owned by one of the two VAs running the resiliency protocol (the Master VA). The Virtual Router Redundancy Protocol (VRRP) [RFC5798] is one particular example of this. Another example is multihomed subnets, i.e., the same subnet is connected to two VAs 😱.
- Although these VAs provide IP connectivity to VMs and the subnets behind them, they don't always have their own IP interface connected to the EVPN NVE. Layer 2 firewalls are examples of VAs that don't support IP interfaces 🙈.
Figure 1 illustrates some of the examples above 🖼️.
NVE1
+-----------+
TS1(VM)--| (BD-10) |-----+
M1/IP1 +-----------+ | DGW1
+---------+ +-------------+
| |----| (BD-10) |
SN1---+ NVE2 | | | IRB1\ |
| +-----------+ | | | (IP-VRF)|---+
SN2---TS2(VA)--| (BD-10) |-| | +-------------+ _|_
| M2/IP2 +-----------+ | VXLAN/ | ( )
IP4---+ <-+ | GENEVE | DGW2 ( WAN )
| | | +-------------+ (___)
vIP23 (floating) | |----| (BD-10) | |
| +---------+ | IRB2\ | |
SN1---+ <-+ NVE3 | | | | (IP-VRF)|---+
| M3/IP3 +-----------+ | | | +-------------+
SN3---TS3(VA)--| (BD-10) |---+ | |
| +-----------+ | |
IP5---+ | |
| |
NVE4 | | NVE5 +--SN5
+---------------------+ | | +-----------+ |
IP6------| (BD-1) | | +-| (BD-10) |--TS4(VA)--SN6
| \ | | +-----------+ |
| (IP-VRF) |--+ ESI4 +--SN7
| / \IRB3 |
|---| (BD-2) (BD-10) |
SN4| +---------------------+
Note:
ESI4 = Ethernet Segment Identifier 4
Figure 1: DC Inter-subnet Use CasesHere's the setup 🗺️. NVE1, NVE2, NVE3, NVE4, NVE5, DGW1, and DGW2 share the same BD for a particular tenant. BD-10 is made up of the collection of BD instances defined in all the NVEs. All the hosts on BD-10 belong to the same IP subnet. The hosts connected to BD-10 are:
- TS1 is a VM that generates/receives traffic to/from IP1, where IP1 belongs to the BD-10 subnet 💁.
- TS2 and TS3 are VAs that send/receive traffic to/from the subnets and hosts sitting behind them (SN1, SN2, SN3, IP4, and IP5) 🛸. Their IP addresses (IP2 and IP3) belong to the BD-10 subnet, and they can also generate/receive traffic. When these VAs receive packets destined to their own MAC addresses (M2 and M3), they route the packets to the proper subnet or host. These VAs don't support routing protocols to advertise the subnets connected to them, and they can move to a different server and NVE when the cloud management system decides to do so 🚚. These VAs may also support redundancy mechanisms for some subnets, similar to VRRP, where a floating IP is owned by the Master VA and only the Master VA forwards traffic to a given subnet. For example, vIP23 in Figure 1 is a floating IP that can be owned by TS2 or TS3, depending on which system is the Master. Only the Master will forward traffic to SN1 👑.
- Integrated Routing and Bridging interfaces IRB1, IRB2, and IRB3 have their own IP addresses that belong to the BD-10 subnet too 🌷. These IRB interfaces connect the BD-10 subnet to Virtual Routing and Forwarding (IP-VRF) instances that can route traffic to other subnets for the same tenant (within the DC or at the other end of the WAN) 🌍.
- TS4 is a Layer 2 VA that provides connectivity to subnets SN5, SN6, and SN7, but it doesn't have an IP address of its own in BD-10 🍓. TS4 is connected to a port on NVE5 that is assigned to Ethernet Segment Identifier 4 (ESI4).
For a BD that an ingress NVE is attached to, "Overlay Index" is defined as an identifier that the ingress EVPN NVE requires in order to forward packets to a subnet or host in a remote subnet 🔑. For example, vIP23 (Figure 1) is an Overlay Index that any NVE attached to BD-10 needs to know in order to forward packets to SN1. The IRB3 IP address is an Overlay Index required to get to SN4, and ESI4 is an Overlay Index needed to forward traffic to SN5. In other words, the Overlay Index is a next hop in the overlay address space that can be an IP address, a MAC address, or an ESI 💜. When advertised along with an IP prefix, the Overlay Index requires a recursive resolution to find out the egress NVE that the EVPN packets need to be sent to 🔁.
All the DC use cases in Figure 1 require inter-subnet forwarding 🚀. Therefore, the individual host routes and subnets:
a) must be advertised from the NVEs (since VAs and VMs don't participate in dynamic routing protocols) and
b) may be associated with an Overlay Index that can be a VA IP address, a floating IP address, a MAC address, or an ESI. The Overlay Index is further discussed in Section 3.2 ✨.
2.2. The Need for the EVPN IP Prefix Route 🌟
[RFC7432] defines a MAC/IP Advertisement route (also referred to as "RT-2") where a MAC address can be advertised together with an IP address length and IP address (IP). A variable IP address length might have been used to signal an IP prefix in a route type 2, but there are several specific use cases where using this route type to deliver IP prefixes isn't suitable 🙅♀️.
One example is the "floating IP" example described in Section 2.1 🎈. In this example, it's necessary to decouple the advertisement of the prefixes from the advertisement of a MAC address of either M2 or M3. Otherwise, the solution gets highly inefficient and doesn't scale 😩.
For example, say 1,000 prefixes are advertised from M2 (using RT-2) and the floating IP owner changes from M2 to M3. Then 1,000 routes would be withdrawn by M2 and readvertised by M3 💀. But if a separate route type is used, 1,000 routes can be advertised as associated with the floating IP address (vIP23), and only one RT-2 is needed to advertise the ownership of the floating IP, i.e., vIP23 and M2 in the route type 2 🧁. When the floating IP owner changes from M2 to M3, a single RT-2 withdrawal/update is required to indicate the change. The remote DGW won't change any of the 1,000 prefixes associated with vIP23. It only updates the ARP resolution entry for vIP23 (now pointing at M3) 🥳.
An EVPN route (type 5) for advertising IP prefixes is described in this document 📣. This new route type has a different role from the RT-2 route and addresses the inter-subnet connectivity scenarios for DCs (or NVO-based networks in general) described in this document 🌐. Using this new RT-5, an IP prefix may be advertised along with an Overlay Index, which can be a GW IP address, a MAC, or an ESI. The IP prefix may also be advertised without an Overlay Index. In that case, the BGP next hop points at the egress NVE, Area Border Router (ABR), or ASBR, and the MAC in the EVPN Router's MAC Extended Community provides the inner MAC destination address to be used 📦. As discussed throughout the document, the EVPN RT-2 doesn't meet the requirements for all the DC use cases, so this EVPN route type 5 is required 💅.
The EVPN route type 5 decouples the IP prefix advertisements from the MAC/IP Advertisement routes in EVPN 🧩. Hence:
a) Clean and clear advertisements of IPv4 or IPv6 prefixes in a Network Layer Reachability Information (NLRI) message without MAC addresses are allowed 🌸.
b) Since the route type is different from the MAC/IP Advertisement route, the current procedures described in [RFC7432] don't need to be modified 🙌.
c) A flexible implementation is allowed where the prefix can be linked to different types of Overlay/Underlay Indexes: overlay IP addresses, overlay MAC addresses, overlay ESIs, underlay BGP next hops, etc. 🦋
d) An EVPN implementation that doesn't require IP prefixes can simply discard them by looking at the route type value 🗑️.
The following sections describe how EVPN is extended with a route type for advertising IP prefixes, and how this route is used to address the inter-subnet connectivity requirements that exist in the DC 🎀.
3. The BGP EVPN IP Prefix Route 🧚
The BGP EVPN NLRI as defined in [RFC7432] is shown below 🖍️:
+-----------------------------------+
| Route Type (1 octet) |
+-----------------------------------+
| Length (1 octet) |
+-----------------------------------+
| Route Type specific (variable) |
+-----------------------------------+
Figure 2: BGP EVPN NLRIThis document defines an additional route type (RT-5) in the IANA "EVPN Route Types" registry [EVPNRouteTypes] to be used for advertising EVPN routes that use IP prefixes 📋:
Value: 5
Description: IP PrefixAccording to Section 5.4 of [RFC7606], a node that doesn't recognize route type 5 (RT-5) will ignore it. So an NVE following this document can still be attached to a BD where an NVE ignoring RT-5s is attached. The regular procedures described in [RFC7432] would apply in that case for both NVEs 🤝. If two or more NVEs are attached to different BDs of the same tenant, they MUST support the RT-5 for the proper inter-subnet forwarding operation of the tenant 🚨.
The detailed encoding of this route and its associated procedures are described in the following sections 📖✨.
3.1. IP Prefix Route Encoding ✨
okay bestie, the IP Prefix route type for IPv4 has its Length field set to 34 💖 and it's built out of these fields, in this exact order:
+---------------------------------------+
| RD (8 octets) |
+---------------------------------------+
|Ethernet Segment Identifier (10 octets)|
+---------------------------------------+
| Ethernet Tag ID (4 octets) |
+---------------------------------------+
| IP Prefix Length (1 octet, 0 to 32) |
+---------------------------------------+
| IP Prefix (4 octets) |
+---------------------------------------+
| GW IP Address (4 octets) |
+---------------------------------------+
| MPLS Label (3 octets) |
+---------------------------------------+Figure 3: EVPN IP Prefix Route NLRI for IPv4 🎀
And for IPv6, the Length field is set to 58 🌈 with this layout:
+---------------------------------------+
| RD (8 octets) |
+---------------------------------------+
|Ethernet Segment Identifier (10 octets)|
+---------------------------------------+
| Ethernet Tag ID (4 octets) |
+---------------------------------------+
| IP Prefix Length (1 octet, 0 to 128) |
+---------------------------------------+
| IP Prefix (16 octets) |
+---------------------------------------+
| GW IP Address (16 octets) |
+---------------------------------------+
| MPLS Label (3 octets) |
+---------------------------------------+Figure 4: EVPN IP Prefix Route NLRI for IPv6 🦋
Here's the breakdown, no cap 💅
- The Length field of the BGP EVPN NLRI for an EVPN IP Prefix route MUST be either 34 (if IPv4 addresses are carried) or 58 (if IPv6 addresses are carried). The IP prefix and gateway IP address MUST be from the same IP address family. 🔥
- The Route Distinguisher (RD) and Ethernet Tag ID MUST be used as defined in [RFC7432] and [RFC8365]. In particular, the RD is unique per MAC-VRF (or IP-VRF). The MPLS Label field is set to either an MPLS label or a VNI, as described in [RFC8365] for other EVPN route types. 💜
- The Ethernet Segment Identifier MUST be a non-zero 10-octet identifier if the ESI is used as an Overlay Index (see the definition of "Overlay Index" in Section 3.2). It MUST be all bytes zero otherwise. The ESI format is described in [RFC7432]. 🌸
- The IP prefix length can be set to a value between 0 and 32 (bits) for IPv4 and between 0 and 128 for IPv6, and it specifies the number of bits in the prefix. The value MUST NOT be greater than 128. 🙈
- The IP prefix is a 4- or 16-octet field (IPv4 or IPv6). 🍓
- The GW IP Address field is a 4- or 16-octet field (IPv4 or IPv6) and will encode a valid IP address as an Overlay Index for the IP prefixes. The GW IP field MUST be all bytes zero if it is not used as an Overlay Index. Refer to Section 3.2 for the definition and use of the Overlay Index. ✨
- The MPLS Label field is encoded as 3 octets, where the high-order 20 bits contain the label value, as per [RFC7432]. When sending, the label value SHOULD be zero if a recursive resolution based on an Overlay Index is used. If the received MPLS label value is zero, the route MUST contain an Overlay Index, and the ingress NVE/PE MUST perform a recursive resolution to find the egress NVE/PE. If the received label is zero and the route does not contain an Overlay Index, it MUST be "treat as withdraw" [RFC7606]. 👑
The RD, Ethernet Tag ID, IP prefix length, and IP prefix are part of the route key used by BGP to compare routes. The rest of the fields are not part of the route key. 🧚
An IP Prefix route MAY be sent along with an EVPN Router's MAC Extended Community (defined in [RFC9135]) to carry the MAC address that is used as the Overlay Index. Note that the MAC address may be that of a TS. 💀
As described in Section 3.2, certain data combinations in a received route would imply a treat-as-withdraw handling of the route [RFC7606]. 😭
3.2. Overlay Indexes and Recursive Lookup Resolution ✨
okay bestie, RT-5 routes get recursive lookup resolution through Overlay Indexes 🦋 here's the tea:
- An Overlay Index can be an ESI, an IP address in the tenant's address space, or a MAC address, and an NVE uses it as the next hop for a given IP prefix. An Overlay Index always needs recursive resolution on the NVE/PE that installs the RT-5 into one of its IP-VRFs, so the NVE knows which egress NVE/PE to send packets to 💅. Important: recursive resolution of the Overlay Index happens upon installation into an IP-VRF, NOT upon BGP propagation (for instance, on an ASBR) 🙈. Also, because of that resolution, the egress NVE/PE is not necessarily the same NVE that originated the RT-5 ✨.
- The Overlay Index rides along with the RT-5 in the ESI field, the GW IP field, or the EVPN Router's MAC Extended Community, depending on whether the IP prefix next hop is an ESI, an IP address, or a MAC address in tenant space 🎀. The Overlay Index for a given IP prefix is set by local policy at the NVE that originates the RT-5 for that prefix (typically managed by the cloud management system) 🌸.
- So the ingress NVE can do recursive lookup, an NVE that could be an egress NVE for a given Overlay Index MUST originate a route advertising itself as the BGP next hop on the path to the system denoted by that Overlay Index 🧚. Here's how that plays out:
- If an NVE receives an RT-5 that specifies an Overlay Index, it cannot use that RT-5 in its IP-VRF unless (or until) it can recursively resolve the Overlay Index 💀.
- If the RT-5 specifies an ESI as the Overlay Index, resolution only works if the NVE has received and installed an RT-1 (auto-discovery per EVI) route specifying that ESI 🍓.
- If the RT-5 specifies a GW IP address as the Overlay Index, resolution only works if the NVE has received and installed an RT-2 (MAC/IP Advertisement route) specifying that IP address in the IP Address field of its NLRI 🫶.
- If the RT-5 specifies a MAC address as the Overlay Index, resolution only works if the NVE has received and installed an RT-2 (MAC/IP Advertisement route) specifying that MAC address in the MAC Address field of its NLRI 💜.
Note that the RT-1 or RT-2 routes needed for recursive resolution may arrive before OR after the given RT-5 route 🌈.
- Irrespective of recursive resolution, if there is no IGP or BGP route to the BGP next hop of an RT-5, BGP MUST NOT install the RT-5, even if the Overlay Index can be resolved 😭.
- The ESI and GW IP fields may both be zero at the same time. However, they MUST NOT both be non-zero at the same time. A route with a non-zero GW IP and a non-zero ESI (at the same time) SHOULD be treated as withdraw [RFC7606] 🔥.
- If either the ESI or the GW IP is non-zero, then the non-zero one is the Overlay Index, no matter whether the EVPN Router's MAC Extended Community is present or what the label value is 👑. If the GW IP is the Overlay Index (so the ESI is zero), the EVPN Router's MAC Extended Community is ignored if it's present ✨.
- A route where ESI, GW IP, MAC, and Label are all zero at the same time SHOULD be treated as withdraw ⭐.
The indirection provided by the Overlay Index and its recursive lookup resolution is required to achieve fast convergence when the object represented by the Overlay Index fails (see the example described in Section 2.2) 🚀. Main character energy for real.
Table 1 shows the different RT-5 field combinations this specification allows, and which Overlay Index the receiving NVE/PE must use in each case 🍓. Cases with no Overlay Index are marked "None" in Table 1. If there is no Overlay Index, the receiving NVE/PE does NOT perform any recursive resolution, and the actual next hop is just the RT-5's BGP next hop 🌸.
+==========+==========+==========+============+===============+
| ESI | GW IP | MAC* | Label | Overlay Index |
+==========+==========+==========+============+===============+
| Non-Zero | Zero | Zero | Don't Care | ESI |
+----------+----------+----------+------------+---------------+
| Non-Zero | Zero | Non-Zero | Don't Care | ESI |
+----------+----------+----------+------------+---------------+
| Zero | Non-Zero | Zero | Don't Care | GW IP |
+----------+----------+----------+------------+---------------+
| Zero | Zero | Non-Zero | Zero | MAC |
+----------+----------+----------+------------+---------------+
| Zero | Zero | Non-Zero | Non-Zero | MAC or None** |
+----------+----------+----------+------------+---------------+
| Zero | Zero | Zero | Non-Zero | None*** |
+----------+----------+----------+------------+---------------+
Table 1: RT-5 Fields and Indicated Overlay IndexTable notes, bestie 📝:
* MAC with a "Zero" value means no EVPN Router's MAC Extended Community is present along with the RT-5. "Non-Zero" means the extended community is present and carries a valid MAC address. The encoding of a MAC address MUST be the 6-octet MAC address specified by [IEEE-802.1Q]. Examples of invalid MAC addresses are broadcast or multicast MAC addresses. The route MUST be treated as withdraw in case of an invalid MAC address 🙅. The presence of the EVPN Router's MAC Extended Community alone is not enough to indicate use of the MAC address as the Overlay Index, since the extended community can be used for other purposes 💅.
** In this case, the Overlay Index may be the RT-5's MAC address or "None", depending on the local policy of the receiving NVE/PE. Note that the advertising NVE/PE that sets the Overlay Index SHOULD advertise an RT-2 for the MAC Overlay Index if there are receiving NVE/PEs configured to use the MAC as the Overlay Index 🫶. This case in Table 1 is used in the IP-VRF-to-IP-VRF implementations described in Sections 4.4.1 and 4.4.3. Support for a MAC Overlay Index in this model is OPTIONAL 🌈.
*** The Overlay Index is "None". This is a special case used for IP-VRF-to-IP-VRF, where the NVE/PEs are connected by IP NVO tunnels as opposed to Ethernet NVO tunnels 🦋.
If the combination of ESI, GW IP, MAC, and Label in the receiving RT-5 is different from the combinations shown in Table 1, the router will process the route as per the rules described at the beginning of this section (Section 3.2) 🧚.
Table 2 shows the different inter-subnet use cases described in this document and the corresponding coding of the Overlay Index in the route type 5 (RT-5) 🎀.
+=========+=====================+===========================+
| Section | Use Case | Overlay Index in the RT-5 |
+=========+=====================+===========================+
| 4.1 | TS IP address | GW IP |
+---------+---------------------+---------------------------+
| 4.2 | Floating IP address | GW IP |
+---------+---------------------+---------------------------+
| 4.3 | "Bump-in-the-wire" | ESI or MAC |
+---------+---------------------+---------------------------+
| 4.4 | IP-VRF-to-IP-VRF | GW IP, MAC, or None |
+---------+---------------------+---------------------------+
Table 2: Use Cases and Overlay Indexes for Recursive
ResolutionThe above use cases are representative of the different Overlay Indexes supported by the RT-5 (GW IP, ESI, MAC, or None) 💖.
4. Overlay Index Use Cases 🌸
This section walks through some use cases for the Overlay Index types used with the IP Prefix route 🍓. The examples use IPv4 prefixes and subnets, but the RT-5 descriptions are valid for the same cases with IPv6, except that IP Prefixes, IPL, and GW IP get replaced by the corresponding IPv6 values 🦋.
4.1. TS IP Address Overlay Index Use Case ✨
okay so, this one is about inter-subnet forwarding for subnets that sit behind VAs (on TS2 and TS3) 🦋
IP4---+ NVE2 DGW1
| +-----------+ +---------+ +-------------+
SN2---TS2(VA)--| (BD-10) |-| |----| (BD-10) |
| M2/IP2 +-----------+ | | | IRB1\ |
-+---+ | | | (IP-VRF)|---+
| | | +-------------+ _|_
SN1 | VXLAN/ | ( )
| | GENEVE | DGW2 ( WAN )
-+---+ NVE3 | | +-------------+ (___)
| M3/IP3 +-----------+ | |----| (BD-10) | |
SN3---TS3(VA)--| (BD-10) |-| | | IRB2\ | |
| +-----------+ +---------+ | (IP-VRF)|---+
IP5---+ +-------------+
Figure 5: TS IP Address Use CaseSo the example is forwarding between SN1, which uses a 24-bit IP prefix (we'll write it as SN1/24 from here on 💜), and a subnet sitting in the WAN. NVE2, NVE3, DGW1, and DGW2 are all running BGP EVPN 🌸. TS2 and TS3 do not join any dynamic routing protocols. They just have a static route to push traffic toward the WAN. SN1/24 is dual-homed to NVE2 and NVE3, which is very main character energy 👑
In this case, a GW IP is used as the Overlay Index ✨. A different Overlay Index type could have worked here, but this use case assumes the operator already knows the VA's IP addresses, while the VA's MAC address is unknown and the VA's ESI is zero. So the GW IP is the right Overlay Index to pair with the RT-5s 💅. The NVEs know which GW IP to use for a given prefix because of policy.
(1) NVE2 advertises these BGP routes on behalf of TS2:
- Route type 2 (MAC/IP Advertisement route) containing: ML = 48 (MAC address length), M = M2 (MAC address), IPL = 32 (IP prefix length), IP = IP2, and BGP Encapsulation Extended Community [RFC9012] with the matching tunnel type. The MAC and IP addresses may be learned via ARP snooping 🔍
- Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = IP2. The prefix and GW IP are learned by policy 🎀
(2) NVE3 does the same on behalf of TS3:
- Route type 2 (MAC/IP Advertisement route) containing: ML = 48, M = M3, IPL = 32, IP = IP3 (and BGP Encapsulation Extended Community).
- Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = IP3.
(3) DGW1 and DGW2 import both received routes based on the Route Targets:
- Based on the BD-10 Route Target in DGW1 and DGW2, the MAC/IP Advertisement route gets imported, and M2 is added to BD-10 along with its tunnel information 🫶. For instance, if VXLAN is used, the VTEP is derived from the MAC/IP Advertisement route BGP next hop, and the VNI comes from the MPLS Label1 field. M2/IP2 gets added to the ARP table. Then M3 is added to BD-10, and M3/IP3 goes into the ARP table too.
- Based on the BD-10 Route Target in DGW1 and DGW2, the IP Prefix route also gets imported, and SN1/24 is added to the IP-VRF with Overlay Index IP2 pointing at the local BD-10. In this example, the RT-5 from NVE2 is assumed to be preferred over the RT-5 from NVE3. If both routes were equally preferable and ECMP were enabled, SN1/24 would also be added to the routing table with Overlay Index IP3 🌈
(4) When DGW1 gets a packet from the WAN with destination IPx, where IPx belongs to SN1/24:
- A destination IP lookup runs on the DGW1 IP-VRF table, and Overlay Index = IP2 is found. Since IP2 is an Overlay Index, a recursive route resolution is required for IP2 🧚
- IP2 gets resolved to M2 in the ARP table, and M2 gets resolved to the tunnel information from the BD FIB (for example, the remote VTEP and VNI in the VXLAN case).
- The IP packet destined to IPx is encapsulated with:
- Inner source MAC = IRB1 MAC.
- Inner destination MAC = M2.
- Tunnel information provided by the BD (VNI, VTEP IPs, and MACs for the VXLAN case).
(5) When the packet arrives at NVE2:
- Based on the tunnel information (the VNI, in the VXLAN case), the BD-10 context is identified for a MAC lookup.
- The encapsulation is stripped off, and based on a MAC lookup (assuming MAC forwarding on the egress NVE), the packet is forwarded to TS2, where it gets properly routed 💖
(6) Should TS2 move from NVE2 to NVE3, MAC Mobility procedures apply to the MAC route M2/IP2, as defined in [RFC7432]. Route type 5 prefixes are not subject to MAC Mobility procedures, so there are no changes in the DGW IP-VRF table for TS2's mobility. All the prefixes still point at IP2 as the Overlay Index. There's an indirection here: SN1/24 still points at Overlay Index IP2 in the routing table, but IP2 just gets resolved to a different tunnel based on the outcome of the MAC Mobility procedures for the MAC/IP Advertisement route M2/IP2 🦋
Note that in the opposite direction, TS2 sends traffic based on its static-route next-hop information (IRB1 and/or IRB2), and regular EVPN procedures apply 🫶
4.2. Floating IP Overlay Index Use Case 🌸
Sometimes TSs work in active/standby mode, where an upstream floating IP owned by the active TS is used as the Overlay Index to reach some subnets behind the TS. This redundancy mode was already introduced in Sections 2.1 and 2.2, and it's illustrated in Figure 6 💅
NVE2 DGW1
+-----------+ +---------+ +-------------+
+---TS2(VA)--| (BD-10) |-| |----| (BD-10) |
| M2/IP2 +-----------+ | | | IRB1\ |
| <-+ | | | (IP-VRF)|---+
| | | | +-------------+ _|_
SN1 vIP23 (floating) | VXLAN/ | ( )
| | | GENEVE | DGW2 ( WAN )
| <-+ NVE3 | | +-------------+ (___)
| M3/IP3 +-----------+ | |----| (BD-10) | |
+---TS3(VA)--| (BD-10) |-| | | IRB2\ | |
+-----------+ +---------+ | (IP-VRF)|---+
+-------------+
Figure 6: Floating IP Overlay Index for Redundant TSIn this use case, a GW IP is used as the Overlay Index for the same reasons as in Section 4.1. The difference is that this GW IP is a floating IP that belongs to the active TS. Assuming TS2 is the active TS and owns vIP23:
(1) NVE2 advertises these BGP routes for TS2:
- Route type 2 (MAC/IP Advertisement route) containing: ML = 48, M = M2, IPL = 32, and IP = vIP23 (as well as BGP Encapsulation Extended Community). The MAC and IP addresses may be learned via ARP snooping 👀
- Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = vIP23. The prefix and GW IP are learned by policy.
(2) NVE3 advertises this BGP route for TS3 (it does not advertise an RT-2 for M3/vIP23):
- Route type 5 (IP Prefix route) containing: IPL = 24, IP = SN1, ESI = 0, and GW IP address = vIP23. The prefix and GW IP are learned by policy 🎀
(3) DGW1 and DGW2 import both received routes based on the Route Target:
- M2 is added to the BD-10 FIB along with its tunnel information. For the VXLAN use case, the VTEP is derived from the MAC/IP Advertisement route BGP next hop, and the VNI comes from the VNI field. M2/vIP23 gets added to the ARP table ✨
- SN1/24 is added to the IP-VRF in DGW1 and DGW2 with Overlay Index vIP23 pointing at M2 in the local BD-10.
(4) When DGW1 gets a packet from the WAN with destination IPx, where IPx belongs to SN1/24:
- A destination IP lookup runs on the DGW1 IP-VRF table, and Overlay Index = vIP23 is found. Since vIP23 is an Overlay Index, a recursive route resolution for vIP23 is required 🧚
- vIP23 gets resolved to M2 in the ARP table, and M2 gets resolved to the tunnel information from the BD (remote VTEP and VNI for the VXLAN case).
- The IP packet destined to IPx is encapsulated with:
- Inner source MAC = IRB1 MAC.
- Inner destination MAC = M2.
- Tunnel information provided by the BD FIB (VNI, VTEP IPs, and MACs for the VXLAN case).
(5) When the packet arrives at NVE2:
- Based on the tunnel information (the VNI, in the VXLAN case), the BD-10 context is identified for a MAC lookup.
- The encapsulation is stripped off, and based on a MAC lookup (assuming MAC forwarding on the egress NVE), the packet is forwarded to TS2, where it gets properly routed 💜
(6) When the redundancy protocol running between TS2 and TS3 appoints TS3 as the new active TS for SN1, TS3 now owns the floating vIP23 and signals this new ownership using a gratuitous ARP REPLY message (explained in [RFC5227]) or something similar 📣. When NVE3 gets the new owner's notification, it issues a route type 2 for M3/vIP23, and NVE2 withdraws the RT-2 for M2/vIP23. DGW1 and DGW2 update their ARP tables with the new MAC that resolves the floating IP. No changes are made in the IP-VRF table 🌈
4.3. Bump-in-the-Wire Use Case ✨
okay so Figure 7 shows inter-subnet forwarding for an IP Prefix route that carries subnet SN1 🦋. In this use case, TS2 and TS3 are Layer 2 VA devices with no IP addresses, so they can't be included as an Overlay Index in the GW IP field of the IP Prefix route 🙈. Their MAC addresses are M2 and M3, and they're both connected to BD-10. Also, IRB1 and IRB2 (in DGW1 and DGW2) have IP addresses in a subnet that's different from SN1, so keep that in mind 💅.
NVE2 DGW1
M2 +-----------+ +---------+ +-------------+
+---TS2(VA)--| (BD-10) |-| |----| (BD-10) |
| ESI23 +-----------+ | | | IRB1\ |
| + | | | (IP-VRF)|---+
| | | | +-------------+ _|_
SN1 | | VXLAN/ | ( )
| | | GENEVE | DGW2 ( WAN )
| + NVE3 | | +-------------+ (___)
| ESI23 +-----------+ | |----| (BD-10) | |
+---TS3(VA)--| (BD-10) |-| | | IRB2\ | |
M3 +-----------+ +---------+ | (IP-VRF)|---+
+-------------+
Figure 7: Bump-in-the-Wire Use CaseSince TS2 and TS3 can't take part in any dynamic routing protocol, and neither one has an IP address assigned, there are two possible Overlay Index types you can use when advertising SN1 🎀:
- a) an ESI, i.e.,
ESI23, that can be provisioned on the attachment ports of NVE2 and NVE3, as shown in Figure 7, or - b) the VA's MAC address, which can be added to NVE2 and NVE3 by policy.
Using an ESI as the Overlay Index instead of the VA's MAC address is the move, bestie 🫶. The forwarding to the egress NVE can be done purely based on the state of the AC in the Ethernet segment (notified by the Ethernet A-D per EVI route), and all the EVPN multihoming redundancy mechanisms can be reused. For instance, the mass withdrawal mechanism described in [RFC7432] for fast failure detection and propagation can be used 🚀. It's assumed per this section that an ESI Overlay Index is used in this use case, but this use case does not preclude the use of the VA's MAC address as an Overlay Index. If a MAC is used as the Overlay Index, the control plane must follow the procedures described in Section 4.4.3.
The model supports VA redundancy in a similar way to the one described in Section 4.2 for the floating IP Overlay Index use case, except that it uses the EVPN Ethernet A-D per EVI route instead of the MAC advertisement route to advertise the location of the Overlay Index. Here's how it goes 💜:
- Assuming TS2 is the active TS in ESI23, NVE2 advertises the following BGP routes:
- Route type 1 (Ethernet A-D route for BD-10) containing:
ESI = ESI23and the corresponding tunnel information (VNIfield), as well as the BGP Encapsulation Extended Community as per [RFC8365]. - Route type 5 (IP Prefix route) containing:
IPL = 24,IP = SN1,ESI = ESI23, andGW IPaddress= 0. The EVPN Router's MAC Extended Community defined in [RFC9135] is added and carries the MAC address (M2) associated with the TS behind which SN1 sits.M2may be learned by policy; however, the MAC in the Extended Community is preferred if sent with the route.
- Route type 1 (Ethernet A-D route for BD-10) containing:
- NVE3 advertises the following BGP route for TS3 (no AD per EVI route is advertised) 🌸:
- Route type 5 (IP Prefix route) containing:
IPL = 24,IP = SN1,ESI = 23, andGW IPaddress= 0. The EVPN Router's MAC Extended Community is added and carries the MAC address (M3) associated with the TS behind which SN1 sits.M3may be learned by policy; however, the MAC in the Extended Community is preferred if sent with the route.
- Route type 5 (IP Prefix route) containing:
- DGW1 and DGW2 import the received routes based on the Route Target:
- The tunnel information to get to ESI23 is installed in DGW1 and DGW2. For the VXLAN use case, the VTEP will be derived from the Ethernet A-D route BGP next hop and VNI from the VNI/VSID field (see [RFC8365]).
- The RT-5 coming from the NVE that advertised the RT-1 is selected, and
SN1/24is added to the IP-VRF in DGW1 and DGW2 with Overlay IndexESI23andMAC = M2.
- When DGW1 receives a packet from the WAN with destination
IPx, whereIPxbelongs toSN1/24🌈:- A destination IP lookup is performed on the DGW1 IP-VRF table, and Overlay Index
= ESI23is found. Since ESI23 is an Overlay Index, a recursive route resolution is required to find the egress NVE where ESI23 resides. - The IP packet destined to
IPxis encapsulated with:- Inner source MAC
= IRB1 MAC. - Inner destination MAC
= M2(this MAC will be obtained from the EVPN Router's MAC Extended Community received along with the RT-5 for SN1). Note that the EVPN Router's MAC Extended Community is used in this case to carry the TS's MAC address, as opposed to the MAC address of the NVE/PE. - Tunnel information for the NVO tunnel is provided by the Ethernet A-D route per EVI for ESI23 (VNI and VTEP IP for the VXLAN case).
- Inner source MAC
- A destination IP lookup is performed on the DGW1 IP-VRF table, and Overlay Index
- When the packet arrives at NVE2 ⭐:
- Based on the tunnel demultiplexer information (VNI for the VXLAN case), the BD-10 context is identified for a MAC lookup (assuming a MAC-based disposition model [RFC7432]), or the VNI may directly identify the egress interface (for an MPLS-based disposition model, which in this context is a VNI-based disposition model).
- Encapsulation is stripped off and, based on a MAC lookup (assuming MAC forwarding on the egress NVE) or a VNI lookup (in case of VNI forwarding), the packet is forwarded to TS2, where it will be forwarded to SN1.
- If the redundancy protocol running between TS2 and TS3 follows an active/standby model and there is a failure, TS3 is appointed as the new active TS for SN1. TS3 will now own the connectivity to SN1 and will signal this new ownership. Upon receiving the new owner's notification, NVE3's AC will become active and issue a route type 1 for ESI23, whereas NVE2 will withdraw its Ethernet A-D route for ESI23. DGW1 and DGW2 will update their tunnel information to resolve ESI23. The inner destination MAC will be changed to
M3😭.
4.4. IP-VRF-to-IP-VRF Model 💖
This use case is similar to the scenario described in Section 9.1 of [RFC9135]; however, the new requirement here is the advertisement of IP prefixes as opposed to only host routes 🍓.
In the examples described in Sections 4.1, 4.2, and 4.3, the BD instance can connect IRB interfaces and any other Tenant Systems connected to it. EVPN provides connectivity for:
- Traffic destined to the IRB or TS IP interfaces, as well as
- Traffic destined to IP subnets sitting behind the TS, e.g., SN1 or SN2.
To provide connectivity for (1), MAC/IP Advertisement routes (RT-2) are needed so that IRB or TS MACs and IPs can be distributed. Connectivity type (2) is accomplished by the exchange of IP Prefix routes (RT-5) for IPs and subnets sitting behind certain Overlay Indexes, e.g., GW IP, ESI, or TS MAC.
In some cases, IP Prefix routes may be advertised for subnets and IPs sitting behind an IRB. This use case is referred to as the "IP-VRF-to-IP-VRF" model 🧚.
[RFC9135] defines an asymmetric IRB model and a symmetric IRB model based on the required lookups at the ingress and egress NVE. The asymmetric model requires an IP lookup and a MAC lookup at the ingress NVE, whereas only a MAC lookup is needed at the egress NVE; the symmetric model requires IP and MAC lookups at both the ingress and egress NVE. From that perspective, the IP-VRF-to-IP-VRF use case described in this section is a symmetric IRB model 👑.
Note that in an IP-VRF-to-IP-VRF scenario, out of the many subnets that a tenant may have, it may be the case that only a few are attached to a given IP-VRF of the NVE/PE. To provide inter-subnet connectivity among the set of NVE/PEs where the tenant is connected, a new SBD is created on all of them if a recursive resolution is needed. This SBD is instantiated as a regular BD (with no ACs) in each NVE/PE and has an IRB interface that connects the SBD to the IP-VRF. The IRB interface's IP or MAC address is used as the Overlay Index for a recursive resolution.
Depending on the existence and characteristics of the SBD and IRB interfaces for the IP-VRFs, there are three different IP-VRF-to-IP-VRF scenarios identified and described in this document 🌷:
- Interface-less model: no SBD and no Overlay Indexes required.
- Interface-ful with an SBD IRB model: requires SBD as well as GW IP addresses as Overlay Indexes.
- Interface-ful with an unnumbered SBD IRB model: requires SBD as well as MAC addresses as Overlay Indexes.
Inter-subnet IP multicast is outside the scope of this document 🙈.
4.4.1. Interface-less IP-VRF-to-IP-VRF Model 🦋
okay bestie, this one is the "no middleman" version 💅 The IP-VRFs talk to each other straight through tunnels, so there are no IRBs or BDs in between. Figure 8 shows the whole vibe:
NVE1(M1)
+------------+
IP1+----| (BD-1) | DGW1(M3)
| \ | +---------+ +--------+
| (IP-VRF)|----| |-|(IP-VRF)|----+
| / | | | +--------+ |
+---| (BD-2) | | | _+_
| +------------+ | | ( )
SN1| | VXLAN/ | ( WAN )--H1
| NVE2(M2) | GENEVE/| (___)
| +------------+ | MPLS | +
+---| (BD-2) | | | DGW2(M4) |
| \ | | | +--------+ |
| (IP-VRF)|----| |-|(IP-VRF)|----+
| / | +---------+ +--------+
SN2+----| (BD-3) |
+------------+
Figure 8: Interface-less IP-VRF-to-IP-VRF ModelSo here's the deal 👑
- a) The NVEs and DGWs must provide connectivity between hosts in SN1, SN2, and IP1, and hosts sitting at the other end of the WAN, for example H1. It's assumed that the DGWs import/export IP and/or VPN-IP routes to/from the WAN.
- b) The IP-VRF instances in the NVE/DGWs are directly connected through NVO tunnels, and no IRBs and/or BD instances get instantiated to connect the IP-VRFs. 🫶
- c) The solution must provide Layer 3 connectivity among the IP-VRFs for Ethernet NVO tunnels, for instance VXLAN or GENEVE.
- d) The solution may provide Layer 3 connectivity among the IP-VRFs for IP NVO tunnels, for example GENEVE with an IP payload.
To make all that happen, EVPN route type 5 is used to advertise the IP prefixes ✨ The EVPN Router's MAC Extended Community from [RFC9135] comes along for the ride if the advertising NVE/DGW uses Ethernet NVO tunnels. Each NVE/DGW advertises an RT-5 for each of its prefixes, with these fields:
RDas per [RFC7432].Ethernet Tag ID= 0.- IP prefix length and IP address, as explained in the previous sections.
GW IPaddress = 0.ESI= 0.- MPLS label or VNI corresponding to the IP-VRF.
Each RT-5 gets sent with a Route Target that identifies the tenant (IP-VRF). It may also be sent with two BGP extended communities 💜
- The first one is the BGP Encapsulation Extended Community, as per [RFC9012], which identifies the tunnel type.
- The second one is the EVPN Router's MAC Extended Community, as per [RFC9135], holding the MAC address of the NVE that's advertising the route. This MAC address identifies the NVE/DGW and MAY be reused for all the IP-VRFs in the NVE. The EVPN Router's MAC Extended Community must be sent if the route is tied to an Ethernet NVO tunnel, like VXLAN. If the route is tied to an IP NVO tunnel, like GENEVE with an IP payload, the EVPN Router's MAC Extended Community should not be sent. 🙈
Okay, now let's walk through the example of advertising and forwarding packets to SN1/24, which is an IPv4 prefix advertised from NVE1 🧚
- NVE1 advertises the following BGP route:
- Route type 5 (IP Prefix route) containing:
- IPL = 24, IP = SN1, Label = 10.
- GW IP = set to 0.
- BGP Encapsulation Extended Community [RFC9012].
- EVPN Router's MAC Extended Community that contains M1.
- Route Target identifying the tenant (IP-VRF).
- Route type 5 (IP Prefix route) containing:
- DGW1 imports the received routes from NVE1:
- DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
- Since GW IP = ESI = 0, the label is a non-zero value, and the local policy indicates this interface-less model, DGW1 uses the label and next hop of the RT-5. It also uses the MAC address conveyed in the EVPN Router's MAC Extended Community (as the inner destination MAC address) to set up the forwarding state and later encapsulate the routed IP packets. 🎀
- When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:
- A destination IP lookup is performed on the DGW1 IP-VRF table, and the lookup yields SN1/24.
- Since the RT-5 for SN1/24 had GW IP = ESI = 0, a non-zero label, and a next hop, and since the model is interface-less, DGW1 does not need a recursive lookup to resolve the route. 🌸
- The IP packet destined to IPx is encapsulated with: inner source MAC = DGW1 MAC, inner destination MAC = M1, outer source IP (tunnel source IP) = DGW1 IP, and outer destination IP (tunnel destination IP) = NVE1 IP. The source and inner destination MAC addresses are not needed if IP NVO tunnels are used.
- When the packet arrives at NVE1:
- NVE1 identifies the IP-VRF for an IP lookup based on the label. The inner destination MAC is not needed to identify the IP-VRF. 🦋
- An IP lookup is performed in the routing context, where SN1 turns out to be a local subnet associated with BD-2. A subsequent lookup in the ARP table and the BD FIB provides the forwarding information for the packet in BD-2.
This model is called an "interface-less" model because the IP-VRFs are connected directly through tunnels. Those tunnels don't need to be terminated in SBDs, which is what Sections 4.4.2 or 4.4.3 are about. Main character energy, no SBD needed 💅
4.4.2. Interface-ful IP-VRF-to-IP-VRF with SBD IRB 🌈
Now the glow-up version 🌈 Figure 9 shows the Interface-ful IP-VRF-to-IP-VRF with SBD IRB model:
NVE1
+------------+ DGW1
IP10+---+(BD-1) | +---------------+ +------------+
| \ | | | | |
|(IP-VRF)-(SBD)| |(SBD)-(IP-VRF)|-----+
| / IRB(M1/IP1) IRB(M3/IP3) | |
+---+(BD-2) | | | +------------+ _+_
| +------------+ | | ( )
SN1| | VXLAN/ | ( WAN )--H1
| NVE2 | GENEVE/ | (___)
| +------------+ | MPLS | DGW2 +
+---+(BD-2) | | | +------------+ |
| \ | | | | | |
|(IP-VRF)-(SBD)| |(SBD)-(IP-VRF)|-----+
| / IRB(M2/IP2) IRB(M4/IP4) |
SN2+----+(BD-3) | +---------------+ +------------+
+------------+
Figure 9: Interface-ful with SBD IRB ModelIn this model:
- a) Just like in Section 4.4.1, the NVEs and DGWs must provide connectivity between hosts in SN1, SN2, and IP10, and hosts sitting at the other end of the WAN.
- b) But now the NVE/DGWs are connected through Ethernet NVO tunnels that terminate in the SBD instance. The IP-VRFs use IRB interfaces for their connectivity to the SBD. 🫶
- c) Each SBD IRB has an IP and a MAC address, and the IP address must be reachable from other NVEs or DGWs.
- d) The SBD is attached to all the NVE/DGWs in the tenant domain BDs.
- e) The solution must provide Layer 3 connectivity for Ethernet NVO tunnels, for instance VXLAN or GENEVE with an Ethernet payload.
EVPN type 5 routes advertise the IP prefixes, while EVPN RT-2 routes advertise the MAC/IP addresses of each SBD IRB interface 🍓 Each NVE/DGW advertises an RT-5 for each of its prefixes, with these fields:
RDas per [RFC7432].Ethernet Tag ID= 0.- IP prefix length and IP address, as explained in the previous sections.
GW IPaddress = IRB-IP of the SBD. This is the Overlay Index that gets used for the recursive route resolution.ESI= 0.- Label value should be zero, since the RT-5 route requires a recursive lookup resolution to an RT-2 route. It is ignored on reception, and the MPLS label or VNI from the RT-2's MPLS Label1 field is used when forwarding packets.
Each RT-5 gets sent with a Route Target that identifies the tenant (IP-VRF). The EVPN Router's MAC Extended Community should not be sent in this case. 🙅♀️
Time for the example of advertising and forwarding packets to SN1/24, the IPv4 prefix advertised from NVE1 ✨
- NVE1 advertises the following BGP routes:
- Route type 5 (IP Prefix route) containing:
- IPL = 24, IP = SN1, Label = SHOULD be set to 0.
- GW IP = IP1 (SBD IRB's IP).
- Route Target identifying the tenant (IP-VRF).
- Route type 2 (MAC/IP Advertisement route for the SBD IRB) containing:
- ML = 48, M = M1, IPL = 32, IP = IP1, Label = 10.
- A BGP Encapsulation Extended Community [RFC9012].
- Route Target identifying the SBD. This Route Target may be the same as the one used with the RT-5.
- Route type 5 (IP Prefix route) containing:
- DGW1 imports the received routes from NVE1:
- DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
- Since GW IP is different from zero, the GW IP (IP1) is used as the Overlay Index for the recursive route resolution to the RT-2 carrying IP1. 🔥
- DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
- When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:
- A destination IP lookup is performed on the DGW1 IP-VRF table. The lookup yields SN1/24, which is associated with the Overlay Index IP1. The forwarding information is derived from the RT-2 received for IP1.
- The IP packet destined to IPx is encapsulated with: inner source MAC = M3, inner destination MAC = M1, outer source IP (source VTEP) = DGW1 IP, and outer destination IP (destination VTEP) = NVE1 IP.
- When the packet arrives at NVE1:
- NVE1 identifies the IP-VRF for an IP lookup based on the label and the inner MAC DA.
- An IP lookup is performed in the routing context, where SN1 turns out to be a local subnet associated with BD-2. A subsequent lookup in the ARP table and the BD FIB provides the forwarding information for the packet in BD-2. 💜
This model is called an "interface-ful with SBD IRB" model, because the tunnels connecting the DGWs and NVEs need to be terminated into the SBD. The SBD is connected to the IP-VRFs via SBD IRB interfaces, and that's what allows the recursive resolution of RT-5s to GW IP addresses. The whole thing is iconic, periodt 👑✨
4.4.3. Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB 🎀
Okay so Figure 10 shows the Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model 🦋 It's basically the same vibe as Section 4.4.2, except the SBD IRB interfaces have no IP addresses. Main character energy, but make it minimal.
NVE1
+------------+ DGW1
IP1+----+(BD-1) | +---------------+ +------------+
| \ | | | | |
|(IP-VRF)-(SBD)| (SBD)-(IP-VRF) |-----+
| / IRB(M1)| | IRB(M3) | |
+---+(BD-2) | | | +------------+ _+_
| +------------+ | | ( )
SN1| | VXLAN/ | ( WAN )--H1
| NVE2 | GENEVE/ | (___)
| +------------+ | MPLS | DGW2 +
+---+(BD-2) | | | +------------+ |
| \ | | | | | |
|(IP-VRF)-(SBD)| (SBD)-(IP-VRF) |-----+
| / IRB(M2)| | IRB(M4) |
SN2+----+(BD-3) | +---------------+ +------------+
+------------+
Figure 10: Interface-ful with Unnumbered SBD IRB ModelHere's the tea 🫖 in this model:
- a) Just like Sections 4.4.1 and 4.4.2, the NVEs and DGWs must provide connectivity between hosts in SN1, SN2, and IP1 and in hosts sitting at the other end of the WAN. 🌸
- b) Just like Section 4.4.2, the NVE/DGWs are connected through Ethernet NVO tunnels terminated in the SBD instance. The IP-VRFs use IRB interfaces for their connectivity to the SBD. 🔗
- c) BUT the twist is that each SBD IRB has a MAC address only and no IP address (which is why the model is called an "unnumbered" SBD IRB 🙈). There's no need to have IP reachability to the SBD IRB interfaces themselves, and there is a requirement to limit the number of IP addresses used.
- d) Just like Section 4.4.2, the SBD is composed of all the NVE/DGW BDs of the tenant that need inter-subnet forwarding. 💜
- e) Just like Section 4.4.2, the solution must provide Layer 3 connectivity for Ethernet NVO tunnels, for instance VXLAN or GENEVE (with Ethernet payload). ✨
This model also uses the RT-5 recursive resolution 🔁 EVPN type 5 routes advertise the IP prefixes along with the EVPN Router's MAC Extended Community used for the recursive lookup, while EVPN RT-2 routes advertise the MAC addresses of each SBD IRB interface (this time without an IP).
Each NVE/DGW advertises an RT-5 for each of its prefixes, using the same fields as described in Section 4.4.2, except:
- GW IP address = set to 0.
Each RT-5 is sent with a Route Target identifying the tenant (IP-VRF), plus the EVPN Router's MAC Extended Community containing the MAC address associated with the SBD IRB interface. This MAC address may be reused for all the IP-VRFs in the NVE. 🧚
Now the example, which is similar to the one in Section 4.4.2:
- NVE1 advertises the following BGP routes:
- Route type 5 (IP Prefix route) containing the same values as in the example in Section 4.4.2, except:
- GW IP = SHOULD be set to 0.
- EVPN Router's MAC Extended Community containing M1 (this will be used for the recursive lookup to an RT-2).
- Route type 2 (MAC route for the SBD IRB) with the same values as in Section 4.4.2, except:
- ML = 48, M = M1, IPL = 0, Label = 10.
- Route type 5 (IP Prefix route) containing the same values as in the example in Section 4.4.2, except:
- DGW1 imports the received routes from NVE1:
- DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
- The MAC contained in the EVPN Router's MAC Extended Community sent along with the RT-5 (M1) will be used as the Overlay Index for the recursive route resolution to the RT-2 carrying M1. 🎯
- DGW1 installs SN1/24 in the IP-VRF identified by the RT-5 Route Target.
- When DGW1 receives a packet from the WAN with destination IPx, where IPx belongs to SN1/24:
- A destination IP lookup is performed on the DGW1 IP-VRF table. The lookup yields SN1/24, which is associated with the Overlay Index M1. The forwarding information is derived from the RT-2 received for M1.
- The IP packet destined to IPx is encapsulated with: inner source MAC = M3, inner destination MAC = M1, outer source IP (source VTEP) = DGW1 IP, and outer destination IP (destination VTEP) = NVE1 IP. 📦
- When the packet arrives at NVE1:
- NVE1 identifies the IP-VRF for an IP lookup based on the label and the inner MAC DA.
- An IP lookup is performed in the routing context, where SN1 turns out to be a local subnet associated with BD-2. A subsequent lookup in the ARP table and the BD FIB provides the forwarding information for the packet in BD-2. 🎉
The model described above is called an "interface-ful with unnumbered SBD IRB" model (as in Section 4.4.2), but without the SBD IRB having an IP address. Iconic. 👑
5. Security Considerations 🔐
This document provides a set of procedures to achieve inter-subnet forwarding across NVEs or PEs attached to a group of BDs that belong to the same tenant (or VPN). The security considerations discussed in [RFC7432] apply to the intra-subnet forwarding or communication within each of those BDs. In addition, the security considerations in [RFC4364] should also be understood, since this document and [RFC4364] may be used in similar applications. 🚨
Contrary to [RFC4364], this document does not describe PE/CE route distribution techniques. Instead, it considers the CEs as TSs or VAs that do not run dynamic routing protocols. This can be considered a security advantage, since dynamic routing protocols can be blocked on the NVE/PE ACs, which stops the tenant from interacting with the infrastructure's dynamic routing protocols. 🛡️
In this document, the RT-5 may use a regular BGP next hop for its resolution, or an Overlay Index that requires a recursive resolution to a different EVPN route (an RT-2 or an RT-1). In the latter case, it is worth noting that any action that ends up filtering or modifying the RT-2 or RT-1 routes used to convey the Overlay Indexes will modify the resolution of the RT-5, and therefore the forwarding of packets to the remote subnet. 😭 Be careful with those filters, bestie!
6. IANA Considerations 🌈
IANA has registered value 5 in the "EVPN Route Types" registry [EVPNRouteTypes] defined by [RFC7432] as follows:
+=======+=============+===========+
| Value | Description | Reference |
+=======+=============+===========+
| 5 | IP Prefix | RFC 9136 |
+-------+-------------+-----------+
Table 37. References 📚
7.1. Normative References
[EVPNRouteTypes]
IANA, "EVPN Route Types",
<https://www.iana.org/assignments/evpn>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <https://www.rfc-editor.org/info/rfc7432>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.
[RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
"The BGP Tunnel Encapsulation Attribute", RFC 9012,
DOI 10.17487/RFC9012, April 2021,
<https://www.rfc-editor.org/info/rfc9012>.
[RFC9135] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
Rabadan, "Integrated Routing and Bridging in Ethernet VPN
(EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
<https://www.rfc-editor.org/info/rfc9135>.7.2. Informative References
[IEEE-802.1Q]
IEEE, "IEEE Standard for Local and Metropolitan Area
Networks -- Bridges and Bridged Networks",
DOI 10.1109/IEEESTD.2018.8403927, IEEE Std 802.1Q, July
2018,
<https://standards.ieee.org/standard/802_1Q-2018.html>.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, <https://www.rfc-editor.org/info/rfc4364>.
[RFC5227] Cheshire, S., "IPv4 Address Conflict Detection", RFC 5227,
DOI 10.17487/RFC5227, July 2008,
<https://www.rfc-editor.org/info/rfc5227>.
[RFC5798] Nadas, S., Ed., "Virtual Router Redundancy Protocol (VRRP)
Version 3 for IPv4 and IPv6", RFC 5798,
DOI 10.17487/RFC5798, March 2010,
<https://www.rfc-editor.org/info/rfc5798>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.
[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network
Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
2014, <https://www.rfc-editor.org/info/rfc7365>.
[RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
Patel, "Revised Error Handling for BGP UPDATE Messages",
RFC 7606, DOI 10.17487/RFC7606, August 2015,
<https://www.rfc-editor.org/info/rfc7606>.
[RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
"Geneve: Generic Network Virtualization Encapsulation",
RFC 8926, DOI 10.17487/RFC8926, November 2020,
<https://www.rfc-editor.org/info/rfc8926>.Acknowledgments 🫶
The authors would like to thank Mukul Katiyar, Jeffrey Zhang, and Alex Nichol for their valuable feedback and contributions. Tony Przygienda and Thomas Morin also helped improve this document with their feedback. Special thanks to Eric Rosen for his detailed review, which really helped improve the readability and clarify the concepts. We also thank Alvaro Retana for his thorough review. 💖
Contributors 🌸
In addition to the authors listed on the front page, the following coauthors have also contributed to this document:
- Senthil Sathappan
- Florin Balus
- Aldrin Isaac
- Senad Palislamovic
- Samir Thoria
Authors' Addresses 🎀
Okay bestie, this is the squad that wrote it! Jorge is the editor, so he's the one who kept everything organized. 💅
Jorge Rabadan (editor) at Nokia 🌸
Jorge Rabadan (editor)
Nokia
777 E. Middlefield Road
Mountain View, CA 94043
United States of America
Email: jorge.rabadan@nokia.comWim Henderickx also at Nokia 🦋
Wim Henderickx
Nokia
Email: wim.henderickx@nokia.comJohn Drake from Juniper 🧚
John Drake
Juniper
Email: jdrake@juniper.netWen Lin also from Juniper 💜
Wen Lin
Juniper
Email: wlin@juniper.netAli Sajassi from Cisco 👑
Ali Sajassi
Cisco
Email: sajassi@cisco.comThat's the whole document, the full team is here, and every contact is listed so you can reach them. 🫶✨
📦 where this page comes from
- ietf: rfc9136 2021-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms