RFC 9135

Integrated Routing and Bridging in Ethernet VPN (EVPN)

📅 published October 2021, from ietf

RFC 9135: Integrated Routing and Bridging in Ethernet VPN (EVPN) 🦋

Okay bestie, let's get into it! 💖 This is RFC 9135, written by A. Sajassi, S. Salam, and S. Thoria from Cisco Systems, plus J. Drake from Juniper and J. Rabadan from Nokia. It came out in October 2021 and it's a Standards Track document, so it's an official Internet standard, not just a suggestion 👑. It's published under ISSN 2070-1721.

The title block is giving serious main character energy. This RFC doesn't obsolete or update any other RFCs, so it's its own thing, no drama 🎀.

Okay the whole point of this one is making EVPN do inter-subnet routing without hauling everyone's traffic back to a far-off gateway. ✨

Abstract

So EVPN is already super good at letting devices in the same subnet talk to each other across an MPLS or IP network, and it handles multihoming like a pro 💅. But what about devices in different subnets? They still need fast, efficient connectivity, and they need it while keeping all that multihoming goodness. This document is the answer to that, and it's an Integrated Routing and Bridging (IRB) solution built on top of EVPN 🌸.

Status of This Memo

This is an Internet Standards Track document 💜, which means it went through public review and the IESG approved it. It represents the consensus of the IETF community, so it's pretty legit 🫶. If you want the scoop on Internet Standards in general, check out Section 2 of RFC 7841. For the current status, errata, or to send feedback, the place to go is https://www.rfc-editor.org/info/rfc9135 🧚.

Copyright Notice

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication. Please actually read those, bestie, because they explain your rights and restrictions 📜. Any Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty as described in the Simplified BSD License ⚠️.

   1.  Introduction
   2.  Terminology
     2.1.  Requirements Language
   3.  EVPN PE Model for IRB Operation
   4.  Symmetric and Asymmetric IRB
     4.1.  IRB Interface and Its MAC and IP Addresses
     4.2.  Operational Considerations
   5.  Symmetric IRB Procedures
     5.1.  Control Plane - Advertising PE
     5.2.  Control Plane - Receiving PE
     5.3.  Subnet Route Advertisement
     5.4.  Data Plane - Ingress PE
     5.5.  Data Plane - Egress PE
   6.  Asymmetric IRB Procedures
     6.1.  Control Plane - Advertising PE
     6.2.  Control Plane - Receiving PE
     6.3.  Data Plane - Ingress PE
     6.4.  Data Plane - Egress PE
   7.  Mobility Procedure
     7.1.  Initiating a Gratuitous ARP upon a Move
     7.2.  Sending Data Traffic without an ARP Request
     7.3.  Silent Host
   8.  BGP Encoding
     8.1.  EVPN Router's MAC Extended Community
   9.  Operational Models for Symmetric Inter-Subnet Forwarding
     9.1.  IRB Forwarding on NVEs for Tenant Systems
       9.1.1.  Control Plane Operation
       9.1.2.  Data Plane Operation
     9.2.  IRB Forwarding on NVEs for Subnets behind Tenant Systems
       9.2.1.  Control Plane Operation
       9.2.2.  Data Plane Operation
   10. Security Considerations
   11. IANA Considerations
   12. References
     12.1.  Normative References
     12.2.  Informative References
   Acknowledgements
   Authors' Addresses

1. Introduction ✨

Okay so, EVPN [RFC7432] is the thing that gives you an extensible, flexible multihoming VPN over an MPLS/IP network 🌈. It handles intra-subnet connectivity between Tenant Systems (TSs) and end devices, whether those are physical or virtual. An IP subnet shows up as an EVPN instance (EVI) if you're doing a VLAN-based service, or as an (EVI, VLAN) association if you're doing a VLAN-aware bundle service 🎀.

Here's the problem though 😭. There are situations where you need dynamic, efficient inter-subnet connectivity between those same TSs and end devices, and you still want the multihoming features. That's the gap this document fills with an IRB solution based on EVPN 💅.

Inter-subnet communication usually goes through centralized Layer 3 (L3) gateway (GW) devices. Those boxes enforce every inter-subnet policy and do all the inter-subnet forwarding. So imagine two TSs in different subnets, both hanging off the same Provider Edge (PE). They want to chat, but their traffic has to get backhauled all the way from the PE to the centralized gateway, get switched there, and then get sent right back down to the PE 🙃. For today's huge multi-tenant Data Centers (DCs), that is extremely inefficient and sometimes just impractical 💀.

So to fix the centralized L3 GW drama, IRB functionality gets put on the PEs, which are also called EVPN Network Virtualization Edges (NVEs) and sit right next to the TSs 🧚. The goal is to avoid inefficient forwarding, which means no backhauling and no hairpinning 🙅‍♀️. When a PE with IRB capability gets tenant traffic over an Attachment Circuit (AC), it can do two things. It can locally bridge intra-subnet traffic, and it can also locally route inter-subnet traffic on a packet-by-packet basis 🦋. That covers both intra- and inter-subnet forwarding, and it dodges the non-optimal forwarding you get with a centralized L3 GW 👑.

Some TSs also run non-IP protocols alongside their IP traffic, so handling both kinds of traffic optimally really matters 🌸. The idea is to bridge non-IP and intra-subnet traffic, and to route inter-subnet IP traffic. With that in mind, the solution has to meet the following requirements:

R1: The solution must provide each tenant with IP routing of its inter-subnet traffic and Ethernet bridging of its intra-subnet traffic and non-routable traffic, where non-routable traffic refers to both non-IP traffic and IP traffic whose version differs from the IP version configured in IP Virtual Routing and Forwarding (IP-VRF). For example, if an IP-VRF in an NVE is configured for IPv6 and that NVE receives IPv4 traffic on the corresponding VLAN, then the IPv4 traffic is treated as non-routable traffic.

R2: The solution must allow IP routing of inter-subnet traffic to be disabled on a per-VLAN basis on those PEs that are backhauling that traffic to another PE for routing.

2. Terminology 📚

AC: Attachment Circuit

ARP: Address Resolution Protocol

ARP Table: A logical view of a forwarding table on a PE that maintains an IP to a MAC binding entry on an IP interface for both IPv4 and IPv6. These entries are learned through ARP/ND or through EVPN.

BD: Broadcast Domain. As per [RFC7432], an EVI consists of a single BD or multiple BDs. In the case of VLAN-bundle and VLAN-based service models (see [RFC7432]), a BD is equivalent to an EVI. In the case of a VLAN-aware bundle service model, an EVI contains multiple BDs. Also, in this document, "BD" and "subnet" are equivalent terms, and wherever "subnet" is used, it means "IP subnet".

BD Route Target: Refers to the broadcast-domain-assigned Route Target [RFC4364]. In the case of a VLAN-aware bundle service model, all the BD instances in the MAC-VRF share the same Route Target.

BT: Bridge Table. The instantiation of a BD in a MAC-VRF, as per [RFC7432].

CE: Customer Edge

DA: Destination Address

Ethernet NVO Tunnel: Refers to Network Virtualization Overlay tunnels with an Ethernet payload, as specified for VXLAN in [RFC7348] and for NVGRE in [RFC7637].

EVI: EVPN Instance spanning NVE/PE devices that are participating on that EVPN, as per [RFC7432].

EVPN: Ethernet VPN, as per [RFC7432].

IP NVO Tunnel: Refers to Network Virtualization Overlay tunnels with IP payload (no MAC header in the payload) as specified for Generic Protocol Extension (GPE) in [VXLAN-GPE].

IP-VRF: A Virtual Routing and Forwarding table for IP routes on an NVE/PE. The IP routes could be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a Layer 3 VPN in an NVE/PE.

IRB: Integrated Routing and Bridging interface. It connects an IP-VRF to a BD (or subnet).

MAC: Media Access Control

MAC-VRF: A Virtual Routing and Forwarding table for MAC addresses on an NVE/PE, as per [RFC7432]. A MAC-VRF is also an instantiation of an EVI in an NVE/PE.

ND: Neighbor Discovery

NVE: Network Virtualization Edge

NVGRE: Network Virtualization Using Generic Routing Encapsulation, as per [RFC7637].

NVO: Network Virtualization Overlay

PE: Provider Edge

RT-2: EVPN Route Type 2, i.e., MAC/IP Advertisement route, as defined in [RFC7432].

RT-5: EVPN Route Type 5, i.e., IP Prefix route, as defined in Section 3 of [RFC9136].

SA: Source Address

TS: Tenant System

VA: Virtual Appliance

VNI: Virtual Network Identifier. As in [RFC8365], the term is used as a representation of a 24-bit NVO instance identifier, with the understanding that "VNI" will refer to a VXLAN Network Identifier in VXLAN, or a Virtual Subnet Identifier in NVGRE, etc., unless it is stated otherwise.

VTEP: VXLAN Termination End Point, as per [RFC7348].

VXLAN: Virtual eXtensible Local Area Network, as per [RFC7348].

This document also assumes familiarity with the terminology of [RFC7365], [RFC7432], and [RFC8365].

2.1. Requirements Language 🫶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Part 1 of 9. More parts are coming, so stay tuned, bestie! 💜🌸

3. EVPN PE Model for IRB Operation ✨

okay so this doc is basically all about IRB, and IRB is tangled up with a bunch of EVPN pieces: MAC-VRF, IP-VRF, EVI, BD, bridge tables, and IRB interfaces 🎀 so before we go any further we need to know how they all fit together. here's the PE model. it names each piece and shows how they connect 💅

  +-------------------------------------------------------------+
  |                                                             |
  |              +------------------+                    IRB PE |
  | Attachment   | +------------------+                         |
  | Circuit(AC1) | |  +----------+    |                MPLS/NVO tnl
----------------------*Bridge    |    |                    +-----
  |              | |  |Table(BT1)|    |    +-----------+  / \    \
  |              | |  |          *---------*           |<--> |Eth|
  |              | |  |  VLAN x  |    |IRB1|           |  \ /    /
  |              | |  +----------+    |    |           |   +-----
  |              | |     ...          |    |  IP-VRF1  |        |
  |              | |  +----------+    |    |  RD2/RT2  |MPLS/NVO tnl
  |              | |  |Bridge    |    |    |           |   +-----
  |              | |  |Table(BT2)|    |IRB2|           |  / \    \
  |              | |  |          *---------*           |<--> |IP |
----------------------*  VLAN y  |    |    +-----------+  \ /    /
  |  AC2         | |  +----------+    |                    +-----
  |              | |    MAC-VRF1      |                         |
  |              +-+    RD1/RT1       |                         |
  |                +------------------+                         |
  |                                                             |
  |                                                             |
  +-------------------------------------------------------------+

                    Figure 1: EVPN IRB PE Model

so a tenant that wants IRB on a PE needs one IP-VRF table plus one or more MAC-VRF tables 🧚 an IP-VRF is what you get when you instantiate an IP-VPN instance on a PE, as defined in [RFC4364] 🌸 and a MAC-VRF is the instantiation of an EVI on a PE, as defined in [RFC7432] 💜 a MAC-VRF holds one or more bridge tables, and each bridge table lines up with one VLAN (broadcast domain). if EVPN service interfaces are set up in VLAN-based mode (that's Section 6.1 of [RFC7432]), there's just one bridge table per MAC-VRF (per EVI), so one tenant VLAN per EVI. but if they're set up in VLAN-aware bundle mode (that's Section 6.3 of [RFC7432]), you get several bridge tables per MAC-VRF (per EVI), which means several tenant VLANs per EVI 🍓

each bridge table hooks up to an IP-VRF through an L3 interface called an "IRB interface" 🦋 a single tenant subnet is usually (and in this document, always) represented by a VLAN, and that VLAN is supported by a single bridge table. so for one tenant, you've got as many bridge tables as you have subnets, and that means as many IRB interfaces between the tenant IP-VRF and those bridge tables, just like in the PE model above 😭

the IP-VRF is identified by its Route Target and Route Distinguisher, and the MAC-VRF is identified by its own Route Target and Route Distinguisher too 🔥 in VLAN-based mode, a receiving PE that gets an EVPN route with a MAC-VRF Route Target can figure out which bridge table it means. but in VLAN-aware bundle mode, the receiving PE needs both the MAC-VRF Route Target and the VLAN ID to find the right bridge table 👑

4. Symmetric and Asymmetric IRB 🌈

this doc defines two kinds of IRB solutions: symmetric and asymmetric IRB 💅 the data path lookups and tables described here are a logical view of how lookups work. real implementations follow that logical view, but they might not stick to it exactly, because of performance trade-offs. specifically:

references to an ARP table in the context of asymmetric IRB are a logical view of a forwarding table that keeps an IP-to-MAC binding entry on a Layer 3 interface, for both IPv4 and IPv6. these entries aren't subject to ARP or ND protocols. for IP-to-MAC bindings learned via EVPN, an implementation may choose to import them straight into the respective forwarding table (like an adjacency/next-hop table) instead of importing them into ARP or ND protocol tables ✨
references to a host IP lookup followed by a host MAC lookup in the context of asymmetric IRB MAY be collapsed into a single IP lookup in a hardware implementation 🤖

in symmetric IRB, the lookup is symmetric at both the ingress and egress PEs. that means both sides do lookups on both MAC and IP addresses 🎀 the ingress PE does a MAC lookup and then an IP lookup, and the egress PE does an IP lookup and then a MAC lookup, as shown in the figure below.

              Ingress PE                   Egress PE
        +-------------------+        +------------------+
        |                   |        |                  |
        |    +-> IP-VRF ----|---->---|-----> IP-VRF -+  |
        |    |              |        |               |  |
        |   BT1        BT2  |        |  BT3         BT2 |
        |    |              |        |               |  |
        |    ^              |        |               v  |
        |    |              |        |               |  |
        +-------------------+        +------------------+
             ^                                       |
             |                                       |
       TS1->-+                                       +->-TS2

                    Figure 2: Symmetric IRB

in symmetric IRB, like Figure 2 shows, inter-subnet forwarding between two PEs happens between their associated IP-VRFs 🦋 so the tunnel joining those IP-VRFs can be either an IP-only tunnel (for example with MPLS or GPE encapsulation) or an Ethernet NVO tunnel (for example with VXLAN encapsulation). if it's an Ethernet NVO tunnel, TS1's IP packet gets wrapped in an Ethernet header made of the ingress and egress PE MAC addresses 💜 that means the ingress PE doesn't need TS2's MAC address. so in symmetric IRB, the ingress PE doesn't need ARP entries linking TS2's IP and MAC addresses in its ARP table. each PE doing symmetric IRB only keeps ARP entries for locally connected hosts, and only keeps MAC-VRFs/BTs for locally configured subnets 🌸

in asymmetric IRB, the lookup is asymmetric: the ingress PE does three lookups and the egress PE does just one 🍓 the ingress PE does a MAC lookup, then an IP lookup, then another MAC lookup. the egress PE only does a single MAC lookup, as shown in Figure 3 below.

           Ingress PE                       Egress PE
        +-------------------+        +------------------+
        |                   |        |                  |
        |    +-> IP-VRF ->  |        |      IP-VRF      |
        |    |           |  |        |                  |
        |   BT1        BT2  |        |  BT3         BT2 |
        |    |           |  |        |              | | |
        |    |           +--|--->----|--------------+ | |
        |    |              |        |                v |
        +-------------------+        +----------------|-+
             ^                                        |
             |                                        |
       TS1->-+                                        +->-TS2

                    Figure 3: Asymmetric IRB

in asymmetric IRB, like Figure 3 shows, inter-subnet forwarding between two PEs happens between their associated MAC-VRFs/BTs 🧚 so the MPLS or NVO tunnel used for inter-subnet forwarding MUST be of type Ethernet. since the egress PE only does a MAC lookup (no IP lookup), TS1's IP packets have to be encapsulated with TS2's MAC address 😭 for the ingress PE to do that encapsulation, it has to keep TS2's IP and MAC address association in its ARP table. it also has to keep TS2's MAC address in the matching bridge table, even if no TSs on that subnet are attached locally 💀 in other words, each PE doing asymmetric IRB MUST maintain ARP entries for remote hosts (hosts connected to other PEs), and must maintain MAC-VRFs/BTs and IRB interfaces for ALL subnets in an IP-VRF, including subnets that aren't attached locally 🔥 so think carefully about PE scale for the ARP table size, the IRB interfaces, and the number and size of the bridge tables before you use asymmetric IRB 👑

it's worth knowing that whenever a PE does a host IP lookup for a packet that gets routed, the IPv4 Time To Live (TTL) or IPv6 hop limit for that packet goes down by one, and if it hits zero, the packet is discarded 🙈 with symmetric IRB, the TTL / hop limit is decremented by both the ingress and egress PEs (once each). with asymmetric IRB, it's only decremented once, by the ingress PE ✨

the next sections define the control and data plane procedures for symmetric and asymmetric IRB on ingress and egress PEs. the figure below is used to describe those procedures. it shows a single IP-VRF and a bunch of BDs on each PE for one tenant 🌈 basically, an IP-VRF connects one or more EVIs, each EVI contains one MAC-VRF, each MAC-VRF is made of one or more bridge tables (one per BD), and a PE has an IRB interface for each BD 🫶

                PE 1         +---------+
          +-------------+    |         |
  TS1-----|         MACx|    |         |        PE2
(M1/IP1)  |(BT1)        |    |         |   +-------------+
  TS5-----|      \      |    |  MPLS/  |   |MACy  (BT3)  |-----TS3
(M5/IP5)  |IPx/Mx \     |    |  VXLAN/ |   |     /       | (M3/IP3)
          |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
          |       /     |    |         |   |     \       |
  TS2-----|(BT2) /      |    |         |   |      (BT1)  |-----TS4
(M2/IP2)  |             |    |         |   |             |  (M4/IP4)
          +-------------+    |         |   +-------------+
                             |         |
                             +---------+

                    Figure 4: IRB Forwarding

4.1. IRB Interface and Its MAC and IP Addresses ✨

okay so bestie, here's the deal 💅 to support inter-subnet forwarding on a PE, the PE acts as an IP default gateway from the perspective of the attached Tenant Systems. the default gateway MAC and IP addresses get configured on each IRB interface tied to its subnet, and they fall into one of two options:

All the PEs for a given tenant subnet use the same anycast default gateway IP and MAC addresses. On each PE, these default gateway IP and MAC addresses correspond to the IRB interface connecting the bridge table associated with the tenant's VLAN to the corresponding tenant's IP-VRF. 🌸
Each PE for a given tenant subnet uses the same anycast default gateway IP address but its own MAC address. These MAC addresses are aliased to the same anycast default gateway IP address through the use of the Default Gateway extended community as specified in [RFC7432], which is carried in the EVPN MAC/IP Advertisement routes. On each PE, this default gateway IP address, along with its associated MAC addresses, correspond to the IRB interface connecting the bridge table associated with the tenant's VLAN to the corresponding tenant's IP-VRF. 🦋

It is worth noting that if the applications running on the TSs are employing or relying on any form of MAC security, then the first option (i.e., using an anycast MAC address) should be used 🫶 so the applications receive traffic from the same IRB interface MAC address they're sending to. If the second option is used, then the IRB interface MAC address MUST be the one used in the initial ARP reply or ND Neighbor Advertisement (NA) for that TS. 🔥

Although both options work for both symmetric and asymmetric IRB, option 1 is RECOMMENDED 👑 because anycast MAC provisioning is so easy. You set it on the IRB interface for a given subnet across all the PEs for that VLAN, and also on all the IRB interfaces for all the tenant's subnets across all the PEs for all the VLANs. Literally iconic. 🧚 Option 1 also simplifies things since there's no need for the Default Gateway extended community advertisement or its MAC aliasing procedure. And yet another win: after host mobility, the host does NOT need to refresh its default GW ARP/ND entry. 💖

If option 1 is used, an implementation MAY choose to auto-derive the anycast MAC address. If auto-derivation is used, the anycast MAC MUST be auto-derived out of the following ranges (which are defined in [RFC5798]):

Anycast IPv4 IRB case: 00-00-5E-00-01-{VRID}
Anycast IPv6 IRB case: 00-00-5E-00-02-{VRID}

The last octet is generated from a configurable Virtual Router ID (VRID) (range 1-255). If it's not explicitly configured, the default VRID octet is '1'. Auto-derivation of the anycast MAC can ONLY be used if there is certainty that the auto-derived MAC does not collide with any customer MAC address. 🙈

In addition to IP anycast addresses, IRB interfaces can be configured with non-anycast IP addresses for OAM purposes (like sending a traceroute/ping to these interfaces) for both symmetric and asymmetric IRB. These IP addresses need to be distributed as VPN routes when PEs operate in symmetric IRB mode. They DON'T need to be distributed in asymmetric IRB mode, though, because the non-anycast IP addresses are configured along with their individual MACs and get distributed via the EVPN route type 2 advertisement. ✨

For option 1, whether only the anycast MAC address is used or both anycast and non-anycast MAC addresses (the latter for OAM) are used on the same IRB, when a TS sends an ARP request or ND Neighbor Solicitation (NS) to the PE it's attached to, the request is sent for the anycast IP address of the IRB interface associated with the TS's subnet. The reply uses an anycast MAC address (in both the source MAC in the Ethernet header and the sender hardware address in the payload). 🎀

For example, in Figure 4, TS1 is configured with the anycast IPx address as its default gateway IP address. So when it sends an ARP request for IPx (the anycast IP address of the IRB interface for BT1), PE1 sends an ARP reply with MACx, which is the anycast MAC address of that IRB interface. Traffic routed from IP-VRF1 to TS1 uses the anycast MAC address as the source MAC address. 🌈

4.2. Operational Considerations 💅

Symmetric and asymmetric IRB modes may coexist in the same network, and an ingress PE that supports both forwarding modes for a given tenant can interwork with egress PEs that support either IRB mode. The egress PE indicates the desired forwarding mode for a given host based on the presence of the Label2 field and the IP-VRF Route Target in the EVPN MAC/IP Advertisement route. If the Label2 field of the received MAC/IP Advertisement route for host H1 is non-zero, and one of its Route Targets identifies the IP-VRF, the ingress PE will use symmetric IRB mode when forwarding packets destined to H1. If the Label2 field is zero and the MAC/IP Advertisement route for H1 does not carry any Route Target that identifies the IP-VRF, the ingress PE will use asymmetric mode when forwarding traffic to H1. 🫶

As an example that illustrates the previous statement, suppose PE1 and PE2 need to forward packets from TS2 to TS4 in Figure 4. Since both PEs are attached to the bridge table of the destination host, symmetric and asymmetric IRB modes are both possible as long as the ingress PE, PE1, supports both modes. The forwarding mode depends on the mode configured in the egress PE, PE2. That is:

If PE2 is configured for symmetric IRB mode, PE2 advertises TS4 MAC/IP addresses in a MAC/IP Advertisement route with a non-zero Label2 field, e.g., Label2 = Lx, and a Route Target that identifies IP-VRF1 in PE1. IP4 is installed in PE1's IP-VRF1; TS4's ARP and MAC information is also installed in PE1's IRB interface ARP table and BT1, respectively. When a packet from TS2 destined to TS4 is looked up in PE1's IP-VRF route table, a longest prefix match lookup finds IP4 in the IP-VRF, and PE1 forwards using the symmetric IRB mode and Label Lx. 💜
However, if PE2 is configured for asymmetric IRB mode, PE2 advertises TS4 MAC/IP information in a MAC/IP Advertisement route with a zero Label2 field and no Route Target identifying IP-VRF1. In this case, PE2 installs TS4 information in its ARP table and BT1. When a packet from TS2 to TS4 arrives at PE1, a longest prefix match on IP-VRF1's route table yields the local IRB interface to BT1, where a subsequent ARP and bridge table lookup provides the information for an asymmetric forwarding mode to PE2. 🍓

Refer to [EVPN] for more information about interoperability between symmetric and asymmetric forwarding modes. 🌸

The choice between symmetric and asymmetric mode is based on the operator's preference, and it's a trade-off between scale (which is better in symmetric IRB mode) and control plane simplicity (asymmetric IRB mode simplifies the control plane). In cases where a tenant has hosts for every subnet attached to all (or most of) the PEs, the ARP and MAC entries need to be learned by all PEs anyway. So asymmetric IRB mode simplifies the forwarding model and saves space in the IP-VRF route table, since host routes aren't installed in the route table. However, if the tenant does NOT need to stretch subnets (broadcast domains) to multiple PEs and inter-subnet forwarding is needed, symmetric IRB mode will save ARP and bridge table space in all the PEs (compared with asymmetric IRB mode). 🦋

5. Symmetric IRB Procedures 🎀

5.1. Control Plane - Advertising PE

When a PE (e.g., PE1 in Figure 4 above) learns the MAC and IP address of a TS (e.g., via an ARP request or Neighbor Solicitation), it adds the MAC address to the corresponding MAC-VRF/BT of that tenant's subnet and adds the IP address to the IP-VRF for that tenant. It also adds this TS's MAC and IP address association to its ARP table or Neighbor Discovery Protocol (NDP) cache. Then it builds an EVPN MAC/IP Advertisement route (type 2) as follows and advertises it to other PEs participating in that tenant's VPN. 🧚

The Length field of the BGP EVPN Network Layer Reachability Information (NLRI) for an EVPN MAC/IP Advertisement route MUST be either 40 (if the IPv4 address is carried) or 52 (if the IPv6 address is carried).
The Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, IP Address, and MPLS Label1 fields MUST be set per [RFC7432] and [RFC8365].
The MPLS Label2 field is set to either an MPLS label or a VNI corresponding to the tenant's IP-VRF. In the case of an MPLS label, this field is encoded as 3 octets, where the high-order 20 bits contain the label value. 💖

Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, and IP Address fields are part of the route key BGP uses to compare routes. The rest of the fields are NOT part of the route key. ✨

This route is advertised along with the following two extended communities:

Encapsulation Extended Community
EVPN Router's MAC Extended Community

This route is advertised with one or more Encapsulation Extended Communities [RFC9012], one for each encapsulation type the advertising PE supports. If one or more encapsulation types require an Ethernet frame, a single EVPN Router's MAC Extended Community (Section 8.1) is also advertised. This extended community specifies the MAC address to be used as the inner destination MAC address in an Ethernet frame sent to the advertising PE. 🌈

This route MUST be advertised with two Route Targets, one corresponding to the MAC-VRF of the tenant's subnet and another corresponding to the tenant's IP-VRF. 👑

5.2. Control Plane - Receiving PE ✨

Okay so, when a PE (e.g., PE2 in Figure 4 above) gets this EVPN MAC/IP Advertisement route, here's what it does 💅:

The MAC-VRF Route Target and Ethernet Tag, if the latter is non-zero, are used to find the right MAC-VRF and bridge table. If they're found, the MAC address gets imported. The IP-VRF Route Target finds the right IP-VRF, and if that's found too, the IP address gets imported.

Now, if the MPLS Label2 field is non-zero, that means this route is for symmetric IRB 🔥. The MPLS label2 value is the one to use when sending a packet for this IP address to the advertising PE.

If the receiving PE supports asymmetric IRB mode and gets this route with both the MAC-VRF and IP-VRF Route Targets, but the MAC/IP Advertisement route does not include the MPLS Label2 field, then the receiving PE installs the MAC address in the matching MAC-VRF, plus the (IP, MAC) association in the ARP table for that tenant (found through the matching IP-VRF Route Target). Bestie, that's the asymmetric vibe 🦋.

If the receiving PE gets this route with both the MAC-VRF and IP-VRF Route Targets, and it doesn't support asymmetric or symmetric IRB modes but does have the matching MAC-VRF, then it only imports the MAC address. Just the MAC, no cap 🧚.

If the receiving PE gets this route with both the MAC-VRF and IP-VRF Route Targets, and the MAC/IP Advertisement route includes the MPLS Label2 field, but the receiving PE only supports asymmetric IRB mode, then the receiving PE MUST ignore the MPLS Label2 field. It installs the MAC address in the matching MAC-VRF, plus the (IP, MAC) association in the ARP table for that tenant (found through the matching IP-VRF Route Target). Main character energy, but only for the asymmetric side 🙈.

5.3. Subnet Route Advertisement 🌸

Okay, this part is giving big subnet energy. In symmetric IRB, a Layer 3 subnet and IRB interface that correspond to a MAC-VRF/BT are required to be provisioned at a PE only if that PE has locally attached hosts in that subnet. But here's the catch: to get inter-subnet routing working across PEs in a deployment where not every subnet is provisioned at every PE in an EVPN IRB instance, PEs MUST advertise local subnet routes as EVPN RT-5. These subnet routes are required for bootstrapping host (IP, MAC) learning through gleaning procedures that get kicked off by an inter-subnet data packet 💖.

Let me explain it like a story 🫶. If a given host's (IP, MAC) association is unknown, and an ingress PE needs to send a packet to that host, then the ingress PE needs to know which egress PEs are attached to the subnet where the host lives. That way it can send the packet to one of those PEs, which causes the receiving PE to probe for that host.

For example, say subnet A is locally attached to PE1, and subnet B is locally attached to PE2 and PE3. Host A in subnet A, which is attached to PE1, starts a data packet aimed at host B in subnet B, which is attached to PE3. If host B's (IP, MAC) has not been learned yet, either through a gratuitous ARP OR a prior gleaning procedure, then a new gleaning procedure MUST be triggered for host B's (IP, MAC) so it can be learned and advertised across the EVPN network. Since host B's subnet isn't local to PE1, an IP lookup for host B at PE1 won't trigger this gleaning procedure for host B's (IP, MAC) 🙃. So PE1 MUST learn subnet B's prefix route via EVPN RT-5 advertised from PE2 and PE3. That way it can route the packet to one of the PEs that have subnet B locally attached.

Once the packet is received at PE2 OR PE3, and the route lookup gives a glean result, an ARP request is triggered and flooded across the Layer 2 overlay. Host B receives that ARP request and replies to it, so host B's (IP, MAC) gets learned at PE3 and advertised across the EVPN network. After that, packets from host A to host B can be routed directly from PE1 to PE3 ✨. Ta-da!

Advertising the local subnet EVPN RT-5 for an IP-VRF MAY typically be done through provisioning-connected route redistribution to BGP 🧚.

5.4. Data Plane - Ingress PE 💅

Alright, bestie, when an Ethernet frame hits an ingress PE (e.g., PE1 in Figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to figure out the associated MAC-VRF/BT. Then it does a lookup on the destination MAC address. If that MAC address matches its IRB interface MAC address, the ingress PE knows the packet must be inter-subnet routed. So it does an IP lookup in the associated IP-VRF table 🦋. That lookup identifies the BGP next hop of the egress PE, along with the tunnel/encapsulation type and the associated MPLS/VNI values. The ingress PE also drops the TTL / hop limit for that packet by one, and if it hits zero, the ingress PE discards the packet 💀.

If the tunnel type is MPLS or IP-only NVO, then the TS's IP packet goes over the tunnel without any Ethernet header 🌈. But if the tunnel type is Ethernet NVO, then an Ethernet header needs to be added to the TS's IP packet. The source MAC address of this inner Ethernet header is set to the ingress PE's router MAC address. The destination MAC address of this inner Ethernet header is set to the egress PE's router MAC address, which is learned through the EVPN Router's MAC Extended Community attached to the route. The MPLS VPN label is set to the received label2 in the route. For the Ethernet NVO tunnel type, the VNI may be set one of two ways:

downstream mode: The VNI is set to the received label2 in the route, which is downstream assigned.

global mode: The VNI is set to the received label2 in the route, which is assigned domain-wide. This VNI value from the received label2 MUST be the same as the locally configured VNI for the IP-VRF, because all PEs in the NVO MUST be configured with the same IP-VRF VNI for this mode of operation. If the received label2 value doesn't match the locally configured VNI value, the route MUST NOT be used, and an error message SHOULD be logged 🚨.

PEs may be configured to run in one of these two modes, depending on the administrative domain boundaries across the PEs participating in the NVO, and on whether the PE can support downstream VNI mode 🎀.

For NVO tunnel encapsulation, the outer source and destination IP addresses are set to the ingress and egress PE BGP next-hop IP addresses, respectively 🫶.

5.5. Data Plane - Egress PE 🍓

When the egress PE gets the tenant's MPLS or NVO encapsulated packet over an MPLS or NVO tunnel, it removes the NVO tunnel encapsulation. Then it uses the VPN MPLS label (for MPLS encapsulation) or the VNI (for NVO encapsulation) to figure out which IP-VRF to run the IP lookup in 👑. If the VPN MPLS label or VNI identifies a MAC-VRF instead of an IP-VRF, then the procedures in Section 6.4 for asymmetric IRB get executed.

The lookup in the IP-VRF identifies a local adjacency to the IRB interface associated with the egress subnet's MAC-VRF/BT. The egress PE also drops the TTL / hop limit for that packet by one. If it hits zero, the egress PE discards the packet 💀.

The egress PE grabs the destination TS's MAC address for that TS's IP address from its ARP table or NDP cache. It encapsulates the packet with that destination MAC address and a source MAC address that matches that IRB interface, then sends the packet to its destination subnet MAC-VRF/BT 🌸.

The destination MAC address lookup in the MAC-VRF/BT gives you the local adjacency (e.g., local interface) that the Ethernet frame gets sent over. Done and dusted 🔥.

6. Asymmetric IRB Procedures 🧚

6.1. Control Plane - Advertising PE 🦋

Okay so when a PE (e.g., PE1 in Figure 4 above) learns the MAC and IP address of an attached TS (e.g., through an ARP request or ND Neighbor Solicitation), it fills in its MAC-VRF/BT, IP-VRF, and ARP table or NDP cache, just like it would for symmetric IRB. Then it builds an EVPN MAC/IP Advertisement route (type 2) as follows and advertises it to the other PEs in that tenant's VPN 💖.

The Length field of the BGP EVPN NLRI for an EVPN MAC/IP Advertisement route MUST be either 37 (if an IPv4 address is carried) or 49 (if an IPv6 address is carried).
The RD, Ethernet Segment Identifier, Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, IP Address, and MPLS Label1 fields MUST be set per [RFC7432] and [RFC8365].
The MPLS Label2 field MUST NOT be included in this route.

Just like in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length, MAC Address, IP Address Length, and IP Address fields are part of the route key that BGP uses to compare routes. The rest of the fields are not part of the route key 🤓.

This route is advertised along with the following extended community:

Tunnel Type Extended Community

For asymmetric IRB mode, the EVPN Router's MAC Extended Community isn't needed 🙅‍♀️, because forwarding is done using the destination TS's MAC address, which is carried in this EVPN route type 2 advertisement.

This route MUST always be advertised with the MAC-VRF Route Target. It MAY also be advertised with a second Route Target that corresponds to the IP-VRF 🌈.

6.2. Control Plane - Receiving PE ✨

When a PE (e.g., PE2 in Figure 4 above) receives this EVPN MAC/IP Advertisement route, it does the following:

Using the MAC-VRF Route Target, it identifies the matching MAC-VRF and imports the MAC address into it. For asymmetric IRB mode, it's assumed that all PEs in a tenant's VPN are configured with all subnets (i.e., all VLANs) and their corresponding MAC-VRFs/BTs, even if there are no locally attached TSs for some of those subnets. This is because the ingress PE needs to forward based on the destination TS's MAC address and do NVO tunnel encapsulation, as the property of a lookup in the MAC-VRF/BT.
If only the MAC-VRF Route Target is used, then the receiving PE uses the MAC-VRF Route Target to identify the corresponding IP-VRF. In other words, many MAC-VRF Route Targets map to the same IP-VRF for a given tenant. In this case, the receiving PE may use the MAC-VRF to identify the corresponding IP-VRF through the IRB interface associated with the subnet MAC-VRF/BT. The receiving PE may also use the MAC-VRF Route Target to identify the corresponding IP-VRF.
Using the MAC-VRF Route Target, the receiving PE identifies the corresponding ARP table or NDP cache for the tenant, and it adds an entry to that ARP table or NDP cache for the TS's MAC and IP address association. It should be noted that the tenant's ARP table or NDP cache at the receiving PE is identified by all the MAC-VRF Route Targets for that tenant 🧠.
If the IP-VRF Route Target is included, it may be used to import the route into the IP-VRF. If the IP-VRF Route Target is not included, the MAC-VRF is used to derive the corresponding IP-VRF for import, as explained in the prior section. In both cases, an IP-VRF route is installed with the TS MAC binding included in the received route 💜.

If the receiving PE receives the MAC/IP Advertisement route with the MPLS Label2 field, but the receiving PE only supports asymmetric IRB mode, then the receiving PE MUST ignore the MPLS Label2 field. It installs the MAC address in the corresponding MAC-VRF, plus the (IP, MAC) association in the ARP table or NDP cache for that tenant (with the IRB interface identified by the MAC-VRF) 🎀.

6.3. Data Plane - Ingress PE

Okay so bestie, picture this: an Ethernet frame lands at an ingress PE (that's PE1 in Figure 4 up above 🫶). The PE uses the AC ID (like a VLAN ID) to figure out which MAC-VRF/BT it belongs to, and then it does a lookup on the destination MAC address 👀. If that MAC matches its own IRB interface MAC address, the ingress PE knows this packet needs to be inter-subnet routed. Iconic routing moment 💅 So it does an IP lookup in the associated IP-VRF table. That lookup points to a local adjacency for the IRB interface tied to the egress subnet's MAC-VRF/bridge table. The ingress PE also knocks the TTL / hop limit down by one for that packet, and if it hits zero, the ingress PE discards the packet 💀 rip.

Next, the ingress PE grabs the destination TS's MAC address for that TS's IP address from its ARP table or NDP cache 🧚. It wraps the packet up with that destination MAC address plus a source MAC address that matches the IRB interface, and then sends it along to its destination subnet MAC-VRF/BT 🎀.

Now the destination MAC lookup in that MAC-VRF/BT gives us a BGP next-hop address for the egress PE, along with label1 (an L2 VPN MPLS label or a VNI) 🌸. The ingress PE encapsulates the packet with the Ethernet NVO tunnel of your choice (like VXLAN or NVGRE) and sends it to the egress PE 🦋. Since the forwarding happens between the ingress PE's MAC-VRF/BT and the egress PE's MAC-VRF/bridge table, the encapsulation procedures follow [RFC7432] for MPLS and [RFC8365] for VXLAN, no cap 🔥.

6.4. Data Plane - Egress PE

Here's the vibe on the egress side ✨. When a tenant's Ethernet frame comes in over an NVO tunnel, the egress PE strips off the NVO tunnel encapsulation. Then it uses the VPN MPLS label (for MPLS encapsulation) or the VNI (for NVO encapsulation) to find the MAC-VRF/BT where the MAC lookup needs to happen 💜.

That MAC lookup gives us a local adjacency (like a local interface) where the packet needs to get sent out 🍓.

Lowkey the forwarding behavior on the egress PE is the same as the EVPN intra-subnet forwarding described in [RFC7432] for MPLS and [RFC8365] for NVO networks 🙈. In other words, all the inter-subnet forwarding processing is confined to the ingress PE for asymmetric IRB mode. Main character energy for the ingress PE, fr fr 👑.

Also, [RFC7432] gives the EVPN label a different level of granularity 🌈. Besides identifying the bridge domain table, it can also identify the egress interface or a destination MAC address on that interface. If an EVPN label is used for an egress interface or an individual MAC address, then no MAC lookup is needed in the egress PE for MPLS encapsulation, and the packet can go straight to the egress interface based on the EVPN label lookup alone. Iykyk, it's so efficient 🧚✨.

7. Mobility Procedure 🦋

Okay bestie, now we're talking about mobility 🥹. When a TS moves from one NVE (aka the source NVE) to another NVE (aka the target NVE), it is super important that the MAC Mobility procedures run properly and that the MAC-VRF and IP-VRF tables on all participating NVEs get updated 💖. [RFC7432] describes the MAC Mobility procedures for L2-only services, for both single-homed TS and multihomed TS. This section adds the incremental procedures and BGP Extended Communities needed to handle MAC Mobility for IRB 🎀. To keep the spotlight on what's different between L2-only and IRB, the incremental procedure is described for a single-homed TS, and the extra steps for a multihomed TS can be extended per Section 15 of [RFC7432] 💅. This section covers mobility procedures for both symmetric and asymmetric IRB. Even though the language here is about IPv4 ARP, it applies equally to IPv6 ND ✨.

When a TS moves from a source NVE to a target NVE, it can behave in one of the following three ways 👀:

TS initiates an ARP request upon a move to the target NVE.
TS sends a data packet without first initiating an ARP request to the target NVE.
TS is a silent host and neither initiates an ARP request nor sends any packets 🙈.

Depending on the expected TS behavior, an NVE needs to handle at least the first option and should be able to handle the second and third options 🫶. The following subsections describe the procedures for each scenario, and they assume the MAC and IP addresses of a TS have a one-to-one relationship (meaning one IP address per MAC address and vice versa) 🍓. The procedures for host mobility detection in the presence of a many-to-one relationship are outside the scope of this document and are covered in [EXTENDED-MOBILITY] 🌸. The "many-to-one relationship" means many host IP addresses corresponding to a single host MAC address, or many host MAC addresses corresponding to a single IP address 🧚. Also, for IPv6, a link-local IP address doesn't count in a many-to-one relationship, because that address is confined to a single Ethernet segment and isn't used for host mobility (by definition, host mobility is between two different Ethernet segments) 💜. So when an IPv6 host is configured with both a Global Unicast address (or a Unique Local address) and a link-local address, it's considered to have a single IP address for host mobility purposes 🌈.

7.1. Initiating a Gratuitous ARP upon a Move

Scenario time 🎬. When a TS moves from a source NVE to a target NVE, the TS initiates a gratuitous ARP upon the move to the target NVE 🦋.

The target NVE, upon receiving this ARP message, updates its MAC-VRF, IP-VRF, and ARP table with the host MAC, IP, and local adjacency information (like the local interface) ✨.

Since this NVE has previously learned the same MAC and IP addresses from the source NVE, it recognizes that a MAC move happened 👀, and it initiates MAC Mobility procedures per [RFC7432] by advertising an EVPN MAC/IP Advertisement route with both the MAC and IP addresses filled in (per Sections 5.1 and 6.1) along with the MAC Mobility extended community, with the sequence number incremented by one 🔥. The target NVE also exercises the MAC duplication detection procedure in Section 15.1 of [RFC7432] 💅.

The source NVE, upon receiving this MAC/IP Advertisement route, realizes the MAC has moved to the target NVE 😭. It updates its MAC-VRF and IP-VRF tables accordingly with the adjacency information of the target NVE. For asymmetric IRB, the source NVE also updates its ARP table with the received adjacency information, and for symmetric IRB, the source NVE removes the entry associated with the received (IP, MAC) from its local ARP table 🫶. Then it withdraws its EVPN MAC/IP Advertisement route 🍓. Furthermore, it sends an ARP probe locally to make sure the MAC is gone 🙈. If an ARP response is received, the source NVE updates its ARP entry for that (IP, MAC), re-advertises an EVPN MAC/IP Advertisement route for that (IP, MAC) along with the MAC Mobility extended community with the sequence number incremented by one 💖, and it also exercises the MAC duplication detection procedure in Section 15.1 of [RFC7432] 🌸.

All other remote NVE devices, upon receiving the MAC/IP Advertisement route with the MAC Mobility extended community, compare its sequence number with the one they got before 🧐. If the new sequence number is greater than the old one, they update the MAC/IP addresses of the TS in their MAC-VRF and IP-VRF tables so they point to the target NVE 👑. Furthermore, when these remote PEs receive the MAC/IP withdraw for the TS from the source NVE, they perform the cleanups for their BGP tables 🧹✨.

7.2. Sending Data Traffic without an ARP Request 💖

okay so in this scenario, a TS moves from a source NVE to a target NVE, and it just starts sending data traffic 🦋 without initiating an ARP request first. no warm-up, no cap 💅

The target NVE, upon receiving the first data packet, learns the MAC address of the TS in the data plane and updates its MAC-VRF table with the MAC address and the local adjacency information (e.g., local interface) accordingly. the target NVE realizes there has been a MAC move because the same MAC address has been learned remotely from the source NVE. main character energy from the target NVE 👑

If EVPN-IRB NVEs are configured to advertise MAC-only routes in addition to MAC-and-IP EVPN routes, then the following steps are taken:

The target NVE, upon learning this MAC address in the data plane, updates this MAC address entry in the corresponding MAC-VRF with the local adjacency information (e.g., local interface). It also recognizes that this MAC has moved and initiates MAC Mobility procedures per [RFC7432] by advertising an EVPN MAC/IP Advertisement route with only the MAC address filled in along with the MAC Mobility extended community, with the sequence number incremented by one. 🎀
The source NVE, upon receiving this MAC/IP Advertisement route, realizes that the MAC has moved to the new NVE. It updates its MAC-VRF table with the adjacency information for that MAC address to point to the target NVE and withdraws its EVPN MAC/IP Advertisement route that has only the MAC address (if it has advertised such a route previously). Furthermore, it searches for the corresponding MAC-IP entry and sends an ARP probe for this (IP, MAC) pair. The ARP request message is sent both locally to all attached TSs in that subnet as well as to other NVEs participating in that subnet, including the target NVE. Note that the PE needs to maintain a correlation between MAC and MAC-IP route entries in the MAC-VRF to accomplish this. 🧚
The target NVE passes the ARP request to its locally attached TSs, and when it receives the ARP response, it updates its IP-VRF and ARP table with the host (IP, MAC) information. It also sends an EVPN MAC/IP Advertisement route with both the MAC and IP addresses filled in along with the MAC Mobility extended community, with the sequence number set to the same value as the one for the MAC-only Advertisement route it sent previously. ✨
When the source NVE receives the EVPN MAC/IP Advertisement route, it updates its IP-VRF table with the new adjacency information (pointing to the target NVE). In the case of the asymmetric IRB, the source NVE also updates its ARP table with the received adjacency information, and in the case of the symmetric IRB, the source NVE removes the entry associated with the received (IP, MAC) from its local ARP table. Furthermore, it withdraws its previously advertised EVPN MAC/IP route with both the MAC and IP address fields filled in. 🌸
All other remote NVE devices, upon receiving the MAC/IP Advertisement route with the MAC Mobility extended community, compare the sequence number in this advertisement with the one previously received. If the new sequence number is greater than the old one, then they update the MAC/IP addresses of the TS in their corresponding MAC-VRF, IP-VRF, and ARP tables (in the case of asymmetric IRB) to point to the new NVE. Furthermore, upon receiving the MAC/IP withdraw for the TS from the old NVE, these remote PEs perform the cleanups for their BGP tables. 🫶

If an EVPN-IRB NVE is configured not to advertise MAC-only routes, then upon receiving the first data packet, it learns the MAC address of the TS and updates the MAC entry in the corresponding MAC-VRF table with the local adjacency information (e.g., local interface). It also realizes that there has been a MAC move because the same MAC address has been learned remotely from the source NVE. It uses the local MAC route to find the corresponding local MAC-IP route and sends a unicast ARP request to the host. When receiving an ARP response, it follows the procedure outlined in Section 7.1. In the prior case, where MAC-only routes are also advertised, this procedure of triggering a unicast ARP probe at the target PE MAY also be used in addition to the source PE broadcast ARP probing procedure described earlier for better convergence. 🌈

7.3. Silent Host 🙈

okay this one is the sneakiest 😭 in this scenario, a TS moves from a source NVE to a target NVE, but the TS is silent. it neither initiates an ARP request nor sends any data traffic, so neither the target nor the source NVEs are aware of the MAC move. 💀

On the source NVE, an age-out timer (for the silent host that has moved) is used to trigger an ARP probe. This age-out timer can be either an ARP timer or a MAC age-out timer, and this is an implementation choice. The ARP request gets sent both locally to all the attached TSs on that subnet as well as to all the remote NVEs (including the target NVE) participating in that subnet. The source NVE also withdraws the EVPN MAC/IP Advertisement route with only the MAC address (if it has previously advertised such a route). 🍓

The target NVE passes the ARP request to its locally attached TSs, and when it receives the ARP response, it updates its MAC-VRF, IP-VRF, and ARP table with the host (IP, MAC) and local adjacency information (e.g., local interface). It also sends an EVPN MAC/IP Advertisement route with both the MAC and IP address fields filled in along with the MAC Mobility extended community, with the sequence number incremented by one. 🔥

When the source NVE receives the EVPN MAC/IP Advertisement route, it updates its IP-VRF table with the new adjacency information (pointing to the target NVE). In the case of the asymmetric IRB, the source NVE also updates its ARP table with the received adjacency information, and in the case of the symmetric IRB, the source NVE removes the entry associated with the received (IP, MAC) from its local ARP table. Furthermore, it withdraws its previously advertised EVPN MAC/IP route with both the MAC and IP address fields filled in. 💜

All other remote NVE devices, upon receiving the MAC/IP Advertisement route with the MAC Mobility extended community, compare the sequence number in this advertisement with the one previously received. If the new sequence number is greater than the old one, then they update the MAC/IP addresses of the TS in their corresponding MAC-VRF, IP-VRF, and ARP (in the case of asymmetric IRB) tables to point to the new NVE. Furthermore, upon receiving the MAC/IP withdraw for the TS from the old NVE, these remote PEs perform the cleanups for their BGP tables. 🧁

8. BGP Encoding 🎀

This document defines one new BGP Extended Community for EVPN. ✨

8.1. EVPN Router's MAC Extended Community 💅

A new EVPN BGP Extended Community called "EVPN Router's MAC" is introduced here. This new extended community is a transitive extended community with a Type field of 0x06 (EVPN) and a Sub-Type field of 0x03. It MAY be advertised along with the Encapsulation Extended Community defined in Section 4.1 of [RFC9012]. (no wait, it's giving optional energy 🦋)

Wait, correction for accuracy: the sentence above should read that it may be advertised, and the original uses MAY in capitals, so it is written as: It MAY be advertised along with the Encapsulation Extended Community defined in Section 4.1 of [RFC9012].

The EVPN Router's MAC Extended Community is encoded as an 8-octet value as follows:

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Type=0x06     | Sub-Type=0x03 |        EVPN Router's MAC      |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                    EVPN Router's MAC Cont'd                   |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 5: EVPN Router's MAC Extended Community

This extended community is used to carry the PE's MAC address for symmetric IRB scenarios, and it is sent with EVPN RT-2. The advertising PE SHALL only attach a single EVPN Router's MAC Extended Community to a route. In case the receiving PE receives more than one EVPN Router's MAC Extended Community with a route, it SHALL process the first one in the list and not store and propagate the others. 🫧

9. Operational Models for Symmetric Inter-Subnet Forwarding 🌟

The following sections describe two main symmetric IRB forwarding scenarios (within a DC -- i.e., intra-DC) along with the corresponding procedures. In the following scenarios, without loss of generality, it is assumed that a given tenant is represented by a single IP-VPN instance. Therefore, on a given PE, a tenant is represented by a single IP-VRF table and one or more MAC-VRF tables. 🌷

9.1. IRB Forwarding on NVEs for Tenant Systems ✨

okay bestie, this section is about the symmetric IRB setup 💅 where each TS hangs off one or more NVEs, and the NVEs learn its host IP and MAC addresses and share them with every other NVE that wants in on both intra-subnet and inter-subnet chats 🫶

So here's the vibe. We're assuming (no loss of generality, don't worry 🙈) that NVEs run in VLAN-based service interface mode with one bridge table per MAC-VRF. That means each tenant gets one MAC-VRF for every tenant subnet (think: each VLAN) that's set up for extension over VXLAN or NVGRE encapsulation. If you're doing VLAN-aware bundling instead, each MAC-VRF is made of several bridge tables (like one per VLAN). Then the MAC-VRFs on an NVE for a given tenant plug into an IP-VRF for that tenant (or IP-VPN instance) through their IRB interfaces. Main character energy 🎀

Now, VXLAN and NVGRE both need an inner Ethernet header (inner MAC SA/DA), and a TS MAC can't be used for inter-subnet traffic 😭 so the ingress NVE's MAC address gets used as the inner MAC SA instead. That NVE MAC is the device MAC, and it's the same across all MAC-VRFs and IP-VRFs. It gets advertised using the brand-new EVPN Router's MAC Extended Community (Section 8.1) 🦋

Figure 6 shows this off. One tenant (an IP-VPN instance, say) has three subnets, MAC-VRF1, MAC-VRF2, and MAC-VRF3, spread across two NVEs. There are five TSs in the mix:

TS1, TS4, and TS5 all live on the same subnet (same MAC-VRF/VLAN). TS1 and TS5 sit on MAC-VRF1 on NVE1, and TS4 sits on MAC-VRF1 on NVE2.
TS2 is on MAC-VRF2 on NVE1.
TS3 is on MAC-VRF3 on NVE2.
On NVE1, MAC-VRF1 and MAC-VRF2 both hook into IP-VRF1. On NVE2, MAC-VRF1 and MAC-VRF3 both hook into IP-VRF1.

When TS1, TS5, and TS4 chat with each other, only the L2 bridging part of IRB gets used, since they're all on one subnet 💖 But when TS1 wants to talk to TS2 or TS3 (different subnets), both the bridging AND routing parts of IRB kick in ✨ The next subsections break down the control and data plane stuff for this scenario.

                 NVE1         +---------+
           +-------------+    |         |
   TS1-----|         MACx|    |         |        NVE2
 (M1/IP1)  |(MAC-        |    |         |   +-------------+
   TS5-----| VRF1)\      |    |  MPLS/  |   |MACy  (MAC-  |-----TS3
 (M5/IP5)  |       \     |    |  VXLAN/ |   |     / VRF3) | (M3/IP3)
           |    (IP-VRF1)|----|  NVGRE  |---|(IP-VRF1)    |
           |       /     |    |         |   |     \       |
   TS2-----|(MAC- /      |    |         |   |      (MAC-  |-----TS4
 (M2/IP2)  | VRF2)       |    |         |   |       VRF1) | (M4/IP4)
           +-------------+    |         |   +-------------+
                              |         |
                              +---------+

        Figure 6: IRB Forwarding on NVEs for Tenant Systems

9.1.1. Control Plane Operation

okay control plane time 💅 Each NVE advertises a MAC/IP Advertisement route (that's route type 2) for each of its TSs, with these fields set:

RD and Ethernet Segment Identifier (ESI) per [RFC7432]
Ethernet Tag = 0 (assuming VLAN-based service)
MAC Address Length = 48
MAC Address = Mi (where i = 1, 2, 3, 4, or 5) in Figure 6, above
IP Address Length = 32 or 128
IP Address = IPi (where i = 1, 2, 3, 4, or 5) in Figure 6, above
Label1 = MPLS label or VNI corresponding to MAC-VRF
Label2 = MPLS label or VNI corresponding to IP-VRF

Each NVE sends out an EVPN RT-2 route carrying two Route Targets, one for its MAC-VRF and one for its IP-VRF 🎀 The EVPN RT-2 also comes with two BGP Extended Communities. The first one tells you the tunnel type, and it's the "Encapsulation Extended Community" defined in [RFC9012]. The second one carries the NVE's MAC address (MACx for NVE1, MACy for NVE2) as defined in Section 8.1. The EVPN Router's MAC Extended Community MUST be added when the Ethernet NVO tunnel is used. If the IP NVO tunnel type is used, you don't need to send that second Extended Community at all 🙈 Also, it should be noted that the IP NVO tunnel type only applies to symmetric IRB procedures.

When the receiving NVE gets this advertisement, here's what it does 👑

It uses the Route Targets for its MAC-VRF and IP-VRF to find those tables, then imports the MAC and IP addresses into them, respectively.
It imports the MAC address from the MAC/IP Advertisement route into the MAC-VRF. The BGP next-hop address becomes the underlay tunnel destination address (for example, the VTEP DA for VXLAN encapsulation), and label1 becomes the VNI for VXLAN encapsulation or an EVPN label for MPLS encapsulation.
If the route carries the new EVPN Router's MAC Extended Community, and the receiving NVE uses an Ethernet NVO tunnel, then it imports the IP address into the IP-VRF. The inner MAC DA is the NVE's MAC address (taken from the new EVPN Router's MAC Extended Community), the underlay tunnel destination is the BGP next-hop address, the VTEP DA is used for VXLAN encapsulation, and label2 becomes the IP-VPN VNI for VXLAN encapsulation.
If the receiving NVE uses MPLS encapsulation, it imports the IP address into the IP-VRF with the BGP next-hop address as the underlay tunnel destination and label2 as the IP-VPN label for MPLS encapsulation.

⚠️ Heads up, bestie: if the receiving NVE gets an EVPN RT-2 with only label1 and only a single Route Target for the IP-VRF; or an EVPN RT-2 with only a single Route Target for the MAC-VRF but with both label1 and label2; or an EVPN RT-2 with a MAC address length of zero, then it MUST use the treat-as-withdraw approach [RFC7606] and SHOULD log an error message 😭

9.1.2. Data Plane Operation

Now the data plane 🦋 Just so you know, this describes the logical functions only, and the real implementation might look different. Let's follow TS1 in subnet-1 (MAC-VRF1) on NVE1 as it sends traffic to TS3 in subnet-3 (MAC-VRF3) on NVE2 🧚

NVE1 gets a packet whose MAC DA matches the MAC-VRF1 IRB interface on NVE1 (the interface between MAC-VRF1 and IP-VRF1) and whose VLAN tag matches MAC-VRF1.
NVE1 uses that VLAN tag to identify MAC-VRF1. Then it looks up the MAC DA and forwards the frame to its IRB interface.
The Ethernet header gets stripped, and the packet goes into the IP-VRF, where NVE1 does an IP lookup on the destination IP address. NVE1 also drops the TTL / hop limit by one, and if it hits zero, NVE1 throws the packet away 🙈 That lookup gives the outgoing NVO tunnel and the encapsulation it needs. If the encapsulation is for the Ethernet NVO tunnel, it includes the egress NVE's MAC address as the inner MAC DA, the egress NVE's IP address (e.g., the BGP next-hop address) as the VTEP DA, and the VPN-ID as the VNI. The inner MAC SA and VTEP SA are set to the NVE's own MAC and IP addresses, respectively. If it's MPLS encapsulation, the matching EVPN and LSP labels get added. Then the packet heads to the egress NVE 💖
When the egress NVE gets a packet over the Ethernet NVO tunnel (say, VXLAN encapsulated), it strips the Ethernet header. Since the inner MAC DA is the egress NVE's MAC address, the egress NVE knows it needs to do an IP lookup. It uses the VNI to find the IP-VRF table. If the packet is MPLS encapsulated, the EVPN label lookup finds the IP-VRF table instead. Next, an IP lookup for the destination TS (TS3) gives an access-facing IRB interface to send the packet out of. Before the packet goes out that interface, the ARP table gets checked to find the destination TS's MAC address. NVE2 also drops the TTL / hop limit by one, and if it reaches zero, NVE2 discards the packet ✨
The IP packet gets wrapped in an Ethernet header. The MAC SA is set to the IRB interface MAC address (the IRB interface between MAC-VRF3 and IP-VRF1 on NVE2), and the MAC DA is set to TS3's MAC address. The packet goes to MAC-VRF3, where a MAC DA lookup sends it to TS3 over the right interface 🎀

In this symmetric IRB setup, inter-subnet traffic between NVEs always uses the IP-VRF VNI or MPLS label 🌈 For example, traffic from TS2 to TS4 gets encapsulated by NVE1 with NVE2's IP-VRF VNI/MPLS label, as long as TS4's host IP is present in NVE1's IP-VRF 👑

9.2. IRB Forwarding on NVEs for Subnets behind Tenant Systems ✨

okay bestie, this section is about the symmetric IRB setup for when some TSs are actually running one or more subnets 🌸 and those TSs hang off one or more NVEs. So on top of advertising the MAC/IP addresses for each TS (which can be multihomed with All-Active redundancy mode 💅), the NVE also has to advertise the subnets that are statically configured on each TS. Main character energy, fr 👑

The big difference from the last approach is that extra advertisement for each subnet. Those subnet advertisements use the EVPN IP Prefix route defined in [RFC9136] 🦋 and each prefix gets advertised with the IP address of its TS (which lives in the overlay address space) as its next hop. The receiving NVEs then do recursive route resolution to match that subnet prefix to the NVE that advertised it, so they know exactly where to send packets meant for that subnet 🎀

Here's why this is so iconic: when a TS moves from one NVE to another, you do NOT have to re-advertise any of that TS's subnet prefixes 😭 You only advertise the IP/MAC addresses for the TS itself and run the MAC Mobility procedures for it. The recursive route resolution handles the subnet prefix updates automatically. No cap, that's so efficient 🧚

Check out Figure 7 🦋 It shows one tenant (think an IP-VPN service) with three subnets, represented by MAC-VRF1, MAC-VRF2, and MAC-VRF3, spread across two NVEs. There are four TSs tied to these three MAC-VRFs: TS1 is on MAC-VRF1 on NVE1, TS2 is on MAC-VRF2 on NVE1, TS3 is on MAC-VRF3 on NVE2, and TS4 is on MAC-VRF1 on NVE2. TS1 has two subnet prefixes (SN1 and SN2), and TS3 has one (SN3) 🌈 The MAC-VRFs on each NVE hook up to their matching IP-VRF through their IRB interfaces.

Now the fun part 💖 When TS4 and TS1 chat within the same subnet, only the L2 forwarding (bridging) side of IRB is used, so the traffic just goes through their MAC-VRFs. But when TS3 wants to reach SN1 or SN2 behind TS1 (that's inter-subnet traffic), BOTH the bridging and routing parts of IRB kick in, so the traffic goes through the MAC-VRFs and the IP-VRFs 🔥 If TS4 wants to reach SN1, it just uses its default route and sends the packet to the MAC address of the IRB interface on NVE2. Then NVE2 does an IP lookup in its IP-VRF and finds the entry for SN1 ✨

The next subsections walk through the control and data plane operations for this IRB scenario in full detail 🫶

                                NVE1      +----------+
        SN1--+          +-------------+   |          |
             |--TS1-----|(MAC- \      |   |          |
        SN2--+ M1/IP1   | VRF1) \     |   |          |
                        |     (IP-VRF)|---|          |
                        |       /     |   |          |
                TS2-----|(MAC- /      |   |  MPLS/   |
               M2/IP2   | VRF2)       |   |  VXLAN/  |
                        +-------------+   |  NVGRE   |
                        +-------------+   |          |
        SN3--+--TS3-----|(MAC-\       |   |          |
               M3/IP3   | VRF3)\      |   |          |
                        |     (IP-VRF)|---|          |
                        |       /     |   |          |
                TS4-----|(MAC- /      |   |          |
               M4/IP4   | VRF1)       |   |          |
                        +-------------+   +----------+
                               NVE2

Figure 7: IRB Forwarding on NVEs for Subnets behind TSs

Quick note on Figure 7 👀 SN1 and SN2 are configured on NVE1, which advertises each of them in an IP Prefix route. SN3 is configured on NVE2, which advertises it in an IP Prefix route too 💅

9.2.1. Control Plane Operation 🧠

Each NVE advertises a route type 5 (EVPN RT-5, the IP Prefix route defined in [RFC9136]) for each of its subnet prefixes. The IP address of its TS goes in as the next hop (the Gateway Address field), and the route looks like this:

RD associated with the IP-VRF
ESI = 0
Ethernet Tag = 0
IP Prefix Length = 0 to 32 or 0 to 128
IP Prefix = SNi
Gateway Address = IPi (IP address of TS)
MPLS Label = 0

Each EVPN RT-5 gets advertised with one or more Route Targets tied to the IP-VRF it came from 🎀

Each NVE also advertises an EVPN RT-2 (MAC/IP Advertisement route) for each of its TSs, along with the associated Route Targets and Extended Communities, exactly as described in Section 9.1.1 🌸

When a receiving NVE gets an EVPN RT-5 advertisement, it does the following:

It uses the Route Target to figure out which IP-VRF it belongs to 🦋
It imports the IP prefix into the matching IP-VRF, which is configured with an import RT that's one of the RTs carried by the EVPN RT-5 route. The IP address of the associated TS becomes its next hop ✨

When a receiving NVE gets an EVPN RT-2 advertisement, it imports the MAC/IP addresses of the TS into the matching MAC-VRF and IP-VRF per Section 9.1.1 💖 Once both routes exist, recursive route resolution is performed to resolve the IP prefix (from the EVPN RT-5) to the IP address of the corresponding NVE (e.g., its BGP next hop). That BGP next hop is then used as the underlay tunnel destination address (e.g., the VTEP DA for VXLAN encapsulation), and the EVPN Router's MAC is used as the inner MAC for VXLAN encapsulation 🧚

9.2.2. Data Plane Operation 🚀

Quick disclaimer first: this describes just the logical functions, and the actual implementation may differ 🙈 Let's walk through what happens when a host in SN1 behind TS1 wants to send traffic to a host in SN3 behind TS3.

TS1 sends a packet with a MAC DA that matches the MAC-VRF1 IRB interface of NVE1, and a VLAN tag that matches MAC-VRF1 🌈
The ingress NVE1 uses the VLAN tag to identify MAC-VRF1. Then it looks up the MAC DA and forwards the frame to its IRB interface, as in Section 9.1.1 ✨
The Ethernet header of the packet is stripped, and the packet goes to the IP-VRF, where an IP lookup happens on the destination address. That lookup gives you the fields needed for VXLAN encapsulation: NVE2's MAC address as the inner MAC DA, NVE2's IP address as the VTEP DA, and the VNI. The MAC SA is set to NVE1's MAC address, and the VTEP SA is set to NVE1's IP address. NVE1 also decrements the TTL / hop limit for that packet by one, and if it hits zero, NVE1 discards the packet 💀
The packet then gets encapsulated with the proper header based on all that info and forwarded to the egress NVE (NVE2) 🦋
On the egress NVE (NVE2), assuming the packet is VXLAN encapsulated, the VXLAN and inner Ethernet headers get removed, and the resulting IP packet goes to the IP-VRF associated with that VNI 🫶
Next, a lookup happens on the IP DA (which is in SN3) in NVE2's associated IP-VRF. That IP lookup gives you the access-facing IRB interface the packet should go out on. Before sending it out that interface, the ARP table is checked to get the destination TS (TS3) MAC address. NVE2 also decrements the TTL / hop limit by one, and if it reaches zero, NVE2 discards the packet 🔥
The IP packet gets an Ethernet header with the MAC SA set to that of the access-facing IRB interface of the egress NVE (NVE2), and the MAC DA set to the destination TS (TS3) MAC address. The packet goes to MAC-VRF3 and, after a lookup on the MAC DA, is forwarded to TS3 over the corresponding interface 💖

10. Security Considerations 🔐

Okay, this part is SERIOUS bestie 😭 The security considerations for Layer 2 forwarding in this document follow those of [RFC7432] for MPLS encapsulation and those of [RFC8365] for VXLAN or NVGRE encapsulations. This section adds some extra considerations on top of that.

This document describes a set of procedures for inter-subnet forwarding of tenant traffic across PEs (or NVEs). These procedures cover both Layer 2 forwarding and Layer 3 routing, on a packet-by-packet basis. The security consideration for Layer 3 routing in this document follows that of [RFC4365], with one exception: the application of routing protocols between CEs and PEs. Unlike [RFC4364], this document does not describe route distribution techniques between CEs and PEs. Instead, it treats the CEs as TSs or VAs that do not run dynamic routing protocols. That can be seen as a security advantage, since dynamic routing protocols can be blocked on the NVE/PE ACs, which stops the tenant from interacting with the infrastructure's dynamic routing protocols 🛡️✨

The VPN scheme described in this document does not provide the quartet of security properties mentioned in [RFC4365] (confidentiality protection, source authentication, integrity protection, and replay protection) 😱 If you want those, they must be provided by mechanisms that are outside the scope of the VPN mechanisms, so go get them somewhere else, periodt 💅

Also, in this document, the EVPN RT-5 is used for certain scenarios. This route uses an Overlay Index that requires a recursive resolution to a different EVPN route (an EVPN RT-2). Because of this, it's worth knowing that any action that ends up filtering or modifying the EVPN RT-2 route used to convey the Overlay Indexes will modify the resolution of the EVPN RT-5, and therefore the forwarding of packets to the remote subnet. Watch that RT-2, bestie 👀💀

11. IANA Considerations 📋

IANA has allocated Sub-Type value 0x03 in the "EVPN Extended Community Sub-Types" registry, like this 🎀

+================+======================================+===========+
| Sub-Type Value | Name                                 | Reference |
+================+======================================+===========+
| 0x03           | EVPN Router's MAC                    | RFC 9135  |
|                | Extended Community                   |           |
+----------------+--------------------------------------+-----------+

                                  Table 1

This document has also been listed as an additional reference for the MAC/IP Advertisement route in the "EVPN Route Types" registry 🌸

12. References 📚

12.1. Normative References

Okay, bestie, these are the must-reads 📚 Without them the rest of the doc doesn't really click ✨

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, https://www.rfc-editor.org/info/rfc2119.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006, https://www.rfc-editor.org/info/rfc4364.
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015, https://www.rfc-editor.org/info/rfc7432.
[RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K. Patel, "Revised Error Handling for BGP UPDATE Messages", RFC 7606, DOI 10.17487/RFC7606, August 2015, https://www.rfc-editor.org/info/rfc7606.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, https://www.rfc-editor.org/info/rfc8174.
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., Uttaro, J., and W. Henderickx, "A Network Virtualization Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI 10.17487/RFC8365, March 2018, https://www.rfc-editor.org/info/rfc8365.
[RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder, "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI 10.17487/RFC9012, April 2021, https://www.rfc-editor.org/info/rfc9012.
[RFC9136] Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and A. Sajassi, "IP Prefix Advertisement in Ethernet VPN (EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021, https://www.rfc-editor.org/info/rfc9136.

12.2. Informative References

These ones are the extra reading 🦋 Not required, but super helpful for going deeper 💜

[EVPN] Krattiger, L., Ed., Sajassi, A., Ed., Thoria, S., Rabadan, J., and J. Drake, "EVPN Interoperability Modes", Work in Progress, Internet-Draft, draft-ietf-bess-evpn-modes-interop-00, 26 May 2021, https://datatracker.ietf.org/doc/html/draft-ietf-bess-evpn-modes-interop-00.
[EXTENDED-MOBILITY] Malhotra, N., Ed., Sajassi, A., Pattekar, A., Rabadan, J., Lingala, A., and J. Drake, "Extended Mobility Procedures for EVPN-IRB", Work in Progress, Internet-Draft, draft-ietf-bess-evpn-irb-extended-mobility-07, 2 October 2021, https://datatracker.ietf.org/doc/html/draft-ietf-bess-evpn-irb-extended-mobility-07.
[RFC4365] Rosen, E., "Applicability Statement for BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4365, DOI 10.17487/RFC4365, February 2006, https://www.rfc-editor.org/info/rfc4365.
[RFC5798] Nadas, S., Ed., "Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6", RFC 5798, DOI 10.17487/RFC5798, March 2010, https://www.rfc-editor.org/info/rfc5798.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, https://www.rfc-editor.org/info/rfc7348.
[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. Rekhter, "Framework for Data Center (DC) Network Virtualization", RFC 7365, DOI 10.17487/RFC7365, October 2014, https://www.rfc-editor.org/info/rfc7365.
[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network Virtualization Using Generic Routing Encapsulation", RFC 7637, DOI 10.17487/RFC7637, September 2015, https://www.rfc-editor.org/info/rfc7637.
[VXLAN-GPE] Maino, F., Ed., Kreeger, L., Ed., and U. Elzur, Ed., "Generic Protocol Extension for VXLAN (VXLAN-GPE)", Work in Progress, Internet-Draft, draft-ietf-nvo3-vxlan-gpe-12, 22 September 2021, https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-vxlan-gpe-12.

Acknowledgements

Huge shoutout to the squad 🫶 The authors want to thank Sami Boutros, Jeffrey Zhang, Krzysztof Szarkowicz, Lukas Krattiger, and Neeraj Malhotra for their really valuable comments. They also want to thank Linda Dunbar, Florin Balus, Yakov Rekhter, Wim Henderickx, Lucy Yong, and Dennis Cai for their feedback and contributions 💖

Authors' Addresses

Ali Sajassi
Cisco Systems

Email: sajassi@cisco.com


Samer Salam
Cisco Systems

Email: ssalam@cisco.com


Samir Thoria
Cisco Systems

Email: sthoria@cisco.com


John E Drake
Juniper

Email: jdrake@juniper.net


Jorge Rabadan
Nokia

Email: jorge.rabadan@nokia.com

📦 where this page comes from

ietf: rfc9135 2021-10
💖 girlified by claude-haiku-5-5 on 2026-10-18: a language model's rewrite of the original, under its terms