RFC 9574
Optimized Ingress Replication Solution for Ethernet VPNs (EVPNs)
RFC 9574: Optimized Ingress Replication Solution for Ethernet VPNs (EVPNs) ✨
okay bestie, let's start with the title block 🎀 This one is an IETF doc, numbered RFC 9574, and it's on the Standards Track 🔥 It's an official Internet Standard, so people are actually supposed to implement it, no cap.
The ISSN is 2070-1721, and it was published in May 2024 🌸
The authors are:
- J. Rabadan, Ed. (Nokia, the editor 👑)
- S. Sathappan (Nokia)
- W. Lin (Juniper Networks)
- M. Katiyar (Versa Networks)
- A. Sajassi (Cisco Systems)
There's no Obsoletes or Updates line in this title block, so nothing in it is replacing or patching another RFC 🦋
Abstract 💜
okay so the big idea is about Network Virtualization Overlay (NVO) networks that use Ethernet VPNs (EVPNs) as their control plane 🧚 These networks can move overlay Broadcast, Unknown Unicast, or Multicast (BUM) traffic using either ingress replication trees or Protocol Independent Multicast (PIM) trees.
PIM is super efficient because it stops the same packet from being sent multiple times over the same physical link ✨ The catch is that PIM might not be deployed in the NVO network core at all 😭
That's where ingress replication comes in. It doesn't need PIM in the core, which is a major vibe. But some NVO networks running really demanding multicast apps need something more efficient, and they still don't want PIM in the core 💅
So this document describes a way to optimize how efficient ingress replication trees are 🫶
Status of This Memo 🌈
This is an Internet Standards Track document.
It's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community 💖 It went through public review and got the green light from the Internet Engineering Steering Group (IESG) 🎀 If you want more on how Internet Standards work, check out Section 2 of RFC 7841 🌸
You can find info about the current status of this document, any errata, and how to send feedback at https://www.rfc-editor.org/info/rfc9574 🦋
Copyright Notice 💅
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which are at https://trustee.ietf.org/license-info and apply as of the date this document was published 📜 Please read them carefully, because they spell out your rights and restrictions for this document, and it's not giving skipping-the-fine-print energy 🙈
Any Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're also provided without warranty, as described in the Revised BSD License ✨
Table of Contents 🧚
1. Introduction
2. Terminology and Conventions
3. Solution Requirements
4. EVPN BGP Attributes for Optimized Ingress Replication
5. Non-selective Assisted Replication (AR) Solution Description
5.1. Non-selective AR-REPLICATOR Procedures
5.2. Non-selective AR-LEAF Procedures
5.3. RNVE Procedures
6. Selective Assisted Replication (AR) Solution Description
6.1. Selective AR-REPLICATOR Procedures
6.2. Selective AR-LEAF Procedures
7. Pruned Flooding Lists (PFLs)
7.1. Example of a Pruned Flooding List
8. AR Procedures for Single-IP AR-REPLICATORS
9. AR Procedures and EVPN All-Active Multihoming Split-Horizon
9.1. Ethernet Segments on AR-LEAF Nodes
9.2. Ethernet Segments on AR-REPLICATOR Nodes
10. Security Considerations
11. IANA Considerations
12. References
12.1. Normative References
12.2. Informative References
Acknowledgements
Contributors
Authors' Addresses1. Introduction ✨
Okay so bestie, Ethernet Virtual Private Networks (EVPNs) can be used as the control plane for a Network Virtualization Overlay (NVO) network [RFC8365] 🦋. Network Virtualization Edge (NVE) and Provider Edge (PE) devices that are part of the same EVPN Broadcast Domain (BD) use Ingress Replication (IR) or PIM-based trees to move the tenant's Broadcast, Unknown Unicast, or Multicast (BUM) traffic 💖.
With ingress replication, the ingress NVE gets a BUM frame from the Tenant System (TS) and makes one copy for every remote NVE/PE attached to the BD 🍓. Each copy gets wrapped in an IP packet, and the outer IP Destination Address (IP DA) is the loopback of the egress NVE/PE 🧚. The IP fabric core nodes (aka spines) just route those IP-encapsulated BUM frames based on that outer IP DA 🌸.
With PIM-based trees, the NVEs/PEs in the same BD join a PIM-based tree 🌈. The ingress NVE sends a single copy of the frame, wrapped in an IP packet where the outer IP DA is the multicast address representing that tree. The fabric core nodes are part of the PIM tree and keep multicast state for the group, so the IP-encapsulated BUM frames can reach every NVE/PE that joined 🔥.
The two approaches are shown in Figure 1 🎀. On the left, NVE1 uses ingress replication to send a BUM frame (from Tenant System TS1) to the remote nodes attached to the BD, which are NVE2, NVE3, and PE1. On the right, it's the same example but with a PIM-based tree, i.e., (S1,G1), instead of ingress replication. Only one copy of the tunneled BUM frame is generated that way, but every router in the fabric has to keep multicast state. For example, the spine keeps a PIM routing entry for (S1,G1) with an Incoming Interface (IIF) and three Outgoing Interfaces (OIFs) 👑.
To WAN To WAN
^ ^
| |
+-----+ +-----+
+----------| PE1 |-----------+ +----------| PE1 |-----------+
| +--^--+ | | +--^--+ |
| | IP Fabric | | | IP Fabric |
| PE | | (S1,G1) |OIF to G1 |
| +----PE->+-----+ No State | | IIF +-----+ OIF to G1 |
| | +---2->|Spine|------+ | | +------>Spine|------+ |
| | | +-3->+-----+ | | | | +-----+ | |
| | | | 2 3 | | |PIM |OIF to G1| |
| | | |IR | | | | |tree | | |
|+-----+ +--v--+ +--v--+ | |+-----+ +--v--+ +--v--+ |
+| NVE1|---| NVE2|---| NVE3|-+ +| NVE1|---| NVE2|---| NVE3|-+
+--^--+ +-----+ +-----+ +--^--+ +-----+ +-----+
| | | | | |
| v v | v v
TS1 TS2 TS3 TS1 TS2 TS3
Figure 1: Ingress Replication vs. PIM-Based Trees in NVO NetworksIn NVO networks where PIM-based trees cannot be used, ingress replication is the only option 😭. That happens when the core nodes don't support PIM, or when the network operator doesn't want to run PIM in the core.
In some use cases, the amount of replication for BUM traffic stays under control on the NVEs, thanks to these pretty common assumptions 💅:
a. Broadcast traffic is greatly reduced thanks to the proxy Address Resolution Protocol (ARP) and proxy Neighbor Discovery (ND) capabilities that EVPNs support [RFC9161] on the NVEs. Some NVEs can even provide Dynamic Host Configuration Protocol (DHCP) server functions for the attached TSs, which cuts broadcast traffic down even more 🧁.
b. Unknown unicast traffic is greatly reduced in NVO networks where all the Media Access Control (MAC) and IP addresses from the TSs are learned in the control plane 🫶.
c. Multicast applications are not used 🙈.
If those assumptions hold for a given NVO network, then ingress replication is a simple solution for multi-destination traffic ✨. However, statement c. above isn't always true, and multicast applications are required in lots of use cases 🥹.
When the multicast sources are attached to NVEs living in hypervisors or on low-performance-replication Top-of-Rack (ToR) switches, replicating a large amount of multicast traffic to a big pile of remote NVEs/PEs can seriously hurt the NVE's performance and impact the application 😩.
This document describes a solution that uses two ingress replication optimizations 💜:
- Assisted Replication (AR)
- Pruned Flooding Lists (PFLs)
Assisted Replication is a set of procedures that lets the ingress NVE/PE send a single copy of a broadcast or multicast frame received from a TS to the BD, without needing PIM in the underlay 🦋. It defines the roles of AR-REPLICATOR and AR-LEAF routers. The AR-LEAF is the ingress NVE/PE attached to the TS. It sends a single copy of a broadcast or multicast packet to a selected AR-REPLICATOR, which replicates the packet multiple times toward remote AR-LEAF or AR-REPLICATOR routers. That makes the AR-REPLICATOR "assisting" the ingress AR-LEAF in getting the broadcast or multicast traffic to the remote NVEs/PEs attached to the same BD 🎀. Assisted Replication can use a single AR-REPLICATOR, or two AR-REPLICATOR routers in the path between the ingress AR-LEAF and the remote destination NVEs/PEs. The procedures that use a single AR-REPLICATOR (the non-selective Assisted Replication solution) are specified in Section 5. Section 6 describes how multi-stage replication, meaning two AR-REPLICATOR routers in the path between the ingress AR-LEAF and the destination NVEs/PEs, gets done (the selective Assisted Replication solution) 🌸. The Assisted Replication procedures don't touch unknown unicast traffic, which follows the same forwarding procedures as known unicast traffic so that packet reordering doesn't happen 🧠.
PFLs give the ingress NVE/PE a way to prune, or remove, certain destination NVEs/PEs from a flooding list, depending on whether those NVEs/PEs are interested in receiving BUM traffic ✂️. As specified in [RFC8365], an NVE/PE builds a flooding list for BUM traffic from the next hops of the received EVPN Inclusive Multicast Ethernet Tag routes for the BD. [RFC8365] says the flooding list is used for all BUM traffic, but this document lets you prune certain next hops from that list 💅. Here's an example. Say an ingress NVE creates a flooding list with next hops PE1, PE2, and PE3. If PE2 and PE3 didn't signal any interest in receiving unknown unicast traffic in their Inclusive Multicast Ethernet Tag routes, then when the ingress NVE gets an unknown unicast frame from a TS, it only replicates it to PE1. PE2 and PE3 are "pruned" from the NVE's flooding list for unknown unicast traffic 🫧. PFLs can be used with ingress replication or Assisted Replication, and they're described in Section 7 🌟.
Both optimizations, Assisted Replication and PFLs, can be used together or on their own, so the performance and efficiency of transporting multicast across the network can improve 🚀. Both solutions need some extensions to the BGP attributes used in [RFC7432], so check Section 4 for the details 📝.
The Assisted Replication solution in this document is focused on NVO networks (that's why it uses IP tunnels). MPLS transport networks are out of scope for this document 🙅♀️. The PFLs solution MAY be used in NVO and MPLS transport networks 🦋.
Section 3 lists the requirements of the combined optimized ingress replication solution. Sections 5 and 6 describe the Assisted Replication solution for the non-selective and selective procedures, respectively. Section 7 provides the PFLs solution 💖.
2. Terminology and Conventions ✨
okay bestie, so the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" 💅 only count as the official BCP 14 [RFC2119] [RFC8174] key words when, and only when, they show up in all capitals, just like here 🎀
here's the glossary we're using all throughout this doc 🦋
- AR-IP (Assisted Replication - IP): an IP address owned by the AR-REPLICATOR. It's how the AR-REPLICATOR spots the incoming traffic that must follow the AR procedures 🔥 It also shows up in the Tunnel Identifier and Next Hop fields of the Replicator-AR route.
- AR-LEAF (Assisted Replication - LEAF): an NVE/PE that sends all of its BM traffic to an AR-REPLICATOR, which can replicate that traffic further on its behalf 🧚 An AR-LEAF is typically an NVE/PE with poor replication performance capabilities 💀
- AR-REPLICATOR (Assisted Replication - REPLICATOR): an NVE/PE that can replicate broadcast or multicast traffic received on overlay tunnels out to other overlay tunnels and local Attachment Circuits (ACs) 👑 This document defines the control and data plane procedures that an AR-REPLICATOR needs to follow.
- AR-VNI (Assisted Replication - VNI): a Virtual eXtensible Local Area Network (VXLAN) Network Identifier (VNI) that the AR-REPLICATOR advertises along with the Replicator-AR route 🌸 It's used to identify the incoming packets that must follow the AR procedures ONLY in the single-IP AR-REPLICATOR case (see Section 8).
- Assisted Replication forwarding mode: for an AR-LEAF, this means sending an AC Broadcast and Multicast (BM) packet to a single AR-REPLICATOR with a tunnel destination address AR-IP ✨ For an AR-REPLICATOR, it means sending a BM packet to a selected number of, or all of, the overlay tunnels when the packet was previously received from an overlay tunnel.
- BD: Broadcast Domain, as defined in [RFC7432].
- BD label: the MPLS label that identifies the BD. It's advertised in Regular-IR or Replicator-AR routes when the encapsulation is MPLS over GRE (MPLSoGRE) or MPLS over UDP (MPLSoUDP) 🍓
- BM traffic: broadcast and multicast frames (excluding unknown unicast frames) 🦋
- DF and NDF: Designated Forwarder and Non-Designated Forwarder. These are roles defined in NVEs/PEs attached to multihomed TSs, as per [RFC7432] and [RFC8365] 🫶
- ES and ESI: Ethernet Segment and Ethernet Segment Identifier. EVPN multihoming concepts as specified in [RFC7432].
- EVI: EVPN Instance. A group of Provider Edge (PE) devices participating in the same EVPN service, as specified in [RFC7432] 🌈
- GRE: Generic Routing Encapsulation [RFC4023].
- Ingress Replication forwarding mode: refers to the ingress replication behaviour explained in [RFC7432] 🔥 In this mode, an AC BM packet copy is sent to each remote PE/NVE in the BD, and an overlay BM packet is sent only to the ACs and not to other overlay tunnels.
- IR-IP (Ingress Replication - IP): the local IP address of an NVE/PE that's used for the ingress replication signaling and procedures provided in [RFC7432] 💖 Encapsulated incoming traffic with an outer destination IP address matching the IR-IP will follow the procedures for ingress replication and not the procedures for Assisted Replication. The IR-IP is also used in the Tunnel Identifier and Next Hop fields of the Regular-IR route.
- IR-VNI (Ingress Replication - VNI): a VNI advertised along with the Inclusive Multicast Ethernet Tag route for the ingress replication tunnel type ✨
- MPLS: Multi-Protocol Label Switching.
- NVE: Network Virtualization Edge [RFC8365].
- NVGRE: Network virtualization using Generic Routing Encapsulation [RFC7637].
- PE: Provider Edge.
- PMSI (P-Multicast Service Interface): a conceptual interface that lets a PE send customer multicast traffic to all or some PEs in the same VPN [RFC6513] 🧚
- RD: Route Distinguisher.
- Regular-IR route: an EVPN Inclusive Multicast Ethernet Tag route [RFC7432] that uses the ingress replication tunnel type 🌸
- Replicator-AR route: an EVPN Inclusive Multicast Ethernet Tag route advertised by an AR-REPLICATOR to signal its capabilities, as described in Section 4 👑
- RNVE (Regular NVE): an NVE that supports the procedures provided in [RFC8365] and does not support the procedures provided in this document 🙈 However, this document defines procedures to interoperate with RNVEs, so nobody gets left out 🫶
- ToR switch: Top-of-Rack switch.
- TS and VM: Tenant System and Virtual Machine. In this document, TSs and VMs are the devices connected to the ACs of the PEs and NVEs 💅
- VNI: VXLAN Network Identifier. Used in VXLAN tunnels.
- VSID: Virtual Segment Identifier. Used in NVGRE tunnels.
- VXLAN: Virtual eXtensible Local Area Network [RFC7348] 🦋
3. Solution Requirements 💅
okay so the ingress replication optimization solution in this doc has to hit these requirements, no cap 🔥
- a. The solution provides an ingress replication optimization for BM traffic without the need for PIM while preserving the packet order for unicast applications, i.e., unknown unicast traffic should follow the same path as known unicast traffic. This optimization is required in low-performance NVEs 🥺
- b. The solution reduces the flooded traffic in NVO networks where some NVEs do not need broadcast/multicast and/or unknown unicast traffic ✨
- c. The solution is compatible with [RFC7432] and [RFC8365] and has no impact on the Customer Edge (CE) procedures for BM traffic 🎀 In particular, the solution supports the following EVPN functions:
- All-active multihoming, including the split-horizon and DF functions.
- Single-active multihoming, including the DF function.
- Handling of multi-destination traffic and processing of BM traffic as per [RFC7432].
- d. The solution is backward compatible with existing NVEs using a non-optimized version of ingress replication 🫶 A given BD can have NVEs/PEs supporting regular ingress replication and optimized ingress replication, so everyone can hang 🌈
- e. The solution is independent of the NVO-specific data plane encapsulation and the virtual identifiers being used, e.g., VXLAN VNIs, NVGRE VSIDs, or MPLS labels, as long as the tunnel is IP based 🧚
4. EVPN BGP Attributes for Optimized Ingress Replication ✨
Okay so this section is where the magic happens 🔥 The ingress replication optimization in this doc extends the Inclusive Multicast Ethernet Tag routes and attributes from [RFC7432] so that an NVE/PE can signal its optimized ingress replication capabilities. It's giving "upgrade your group chat" energy 💅
The NLRI of the Inclusive Multicast Ethernet Tag route [RFC7432] is shown in Figure 2, and this doc uses it with zero changes to its format 🙌 The PMSI Tunnel Attribute's general format from [RFC7432] (which borrows it from [RFC6514]) also gets reused. The only new things are a new tunnel type and some new flags, as shown in Figure 3 🦋
+------------------------------------+
| RD (8 octets) |
+------------------------------------+
| Ethernet Tag ID (4 octets) |
+------------------------------------+
| IP Address Length (1 octet) |
+------------------------------------+
| Originating Router's IP Address |
| (4 or 16 octets) |
+------------------------------------+
Figure 2: EVPN Inclusive Multicast Ethernet Tag Route's NLRI 0 1 2 3 4 5 6 7
+---------------------------------+ +--+--+--+--+--+--+--+--+
| Flags (1 octet) | -> |x |E |x | T |BM|U |L |
+---------------------------------+ +--+--+--+--+--+--+--+--+
| Tunnel Type (1 octet) | T = Assisted Replication Type
+---------------------------------+ BM = Broadcast and Multicast
| MPLS Label (3 octets) | U = Unknown (unknown unicast)
+---------------------------------+ x = unassigned
| Tunnel Identifier (variable) |
+---------------------------------+
Figure 3: PMSI Tunnel AttributeThe Flags field in Figure 3 is 8 bits long, per [RFC7902] 🌸 The Extension (E) flag was allocated by [RFC7902], and the Leaf Information Required (L) flag was allocated by [RFC6514]. This doc defines 4 bits of that Flags field:
- Bits 3 and 4, which together form the Assisted Replication Type (T) field
- Bit 5, called the Broadcast and Multicast (BM) flag
- Bit 6, called the Unknown (U) flag
Bits 5 and 6 are collectively known as the Pruned Flooding Lists (PFLs) flags 🍓
Here's how the T field and the PFLs flags work:
- T is the Assisted Replication Type field (2 bits). It tells you the AR role of the advertising router:
00(decimal 0) = RNVE (non-AR support)01(decimal 1) = AR-REPLICATOR10(decimal 2) = AR-LEAF11(decimal 3) = RESERVED
- The PFLs flags set the behavior the advertising router wants for each type of traffic:
- Broadcast and Multicast (BM) flag.
BM = 1means "prune me from the BM flooding list".BM = 0means regular behavior. - Unknown (U) flag.
U = 1means "prune me from the Unknown flooding list".U = 0means regular behavior.
- Broadcast and Multicast (BM) flag.
- The L flag (bit 7) is defined in [RFC6514] and only gets used in the selective AR solution 🧚
Please refer to Section 11 for the IANA considerations related to the PMSI Tunnel Attribute flags 📚
Now, the Inclusive Multicast Ethernet Tag route (Figure 2) and the PMSI Tunnel Attribute (Figure 3) can be used in two different modes for the same BD. Let's get into it 👑
Regular-IR route: In this route, the Originating Router's IP Address, Tunnel Type (0x06), MPLS Label, and Tunnel Identifier MUST be used as described in [RFC7432] when ingress replication is in use. The NVE/PE that advertises the route sets the Next Hop to an IP address this document calls IR-IP. When an AR-LEAF node advertises the Regular-IR route, it MUST set the T field to 10 (AR-LEAF) 💖
Replicator-AR route: This route is how the AR-REPLICATOR advertises its AR capabilities, and its fields get set like this:
- The Originating Router's IP Address MUST be set to an IP address of the advertising router that's common to all the EVIs on the PE (usually a loopback address of the PE) 🫶
- The Tunnel Identifier and Next Hop fields SHOULD be set to the same IP address as the Originating Router's IP Address field when the NVE/PE originates the route. That means when the NVE/PE is not an ASBR, see Section 10.2 of [RFC8365]. No matter what the Tunnel Identifier and Originating Router's IP Address fields say, the ingress NVE/PE will process the received Replicator-AR route and use the IP address in the Next Hop field to build IP tunnels to the AR-REPLICATOR. Iconic 🦋
- The Next Hop address is called the AR-IP, and it MUST be different from the IR-IP for a given PE/NVE, unless the procedures in Section 8 are followed 🙈
- The Tunnel Type MUST be set to Assisted Replication Tunnel. Section 11 has the allocated type value ✨
- The T (Assisted Replication type) field MUST be set to
01(AR-REPLICATOR) 🌈 - The L (Leaf Information Required) flag MUST be set to 0 for non-selective AR and MUST be set to 1 for selective AR 💜
An NVE/PE configured as an AR-REPLICATOR for a BD MUST advertise a Replicator-AR route for that BD, and it MAY advertise a Regular-IR route too. The Replicator-AR route tells the AR-LEAFs which outer IP DA (aka which AR-IP) to use for IP-encapsulated BM frames that use the Assisted Replication forwarding mode. The AR-REPLICATOR forwards an IP-encapsulated BM frame in Assisted Replication forwarding mode if the outer IP DA matches its AR-IP. If the outer IP DA matches its IR-IP instead, it forwards in Ingress Replication forwarding mode 🎀
Okay, and this doc also uses the Leaf Auto-Discovery (Leaf A-D) route from [RFC9572] when selective AR mode is in play. An AR-LEAF MAY send a Leaf A-D route in response to receiving a Replicator-AR route with its L flag set. The Leaf A-D route is only used for selective AR, and its fields get set like this 🍭
- The Originating Router's IP Address is set to the advertising router's IP address, which is the same IP address the AR-LEAF uses in its Regular-IR routes. The Next Hop address is set to the IR-IP, which SHOULD be the same IP address as the advertising router's IP address when the NVE/PE originates the route (so when it is not an ASBR, see Section 10.2 of [RFC8365]) 🌷
- Route Key [RFC9572] is the "Route Type Specific" NLRI of the Replicator-AR route that this Leaf A-D route was generated for 🔑
- The AR-LEAF builds an IP-address-specific Route Target, the same way [RFC9572] does it. It puts the IP address carried in the Next Hop field of the received Replicator-AR route into the Global Administrator field of the extended community, sets the Local Administrator field of that community to 0, and sets the Extended Communities attribute of the Leaf A-D route to that community. The AR-REPLICATOR that sent the Replicator-AR route auto-configures that same IP-address-specific import Route Target, which it uses to control which Leaf A-D routes it accepts 🦋
- The Leaf A-D route MUST include the PMSI Tunnel Attribute with the Tunnel Type set to Assisted Replication Tunnel (Section 11), the T (Assisted Replication type) set to AR-LEAF, and the Tunnel Identifier set to the IP address of the advertising AR-LEAF. The PMSI Tunnel Attribute MUST also carry a downstream-assigned MPLS label or VNI, which the AR-REPLICATOR uses to send traffic to the AR-LEAF 💅
Every AR-enabled node understands and processes the T (Assisted Replication type) field in the PMSI Tunnel Attribute (Flags field) of these routes. It MUST signal the matching type (AR-REPLICATOR or AR-LEAF) based on its administrative choice. An NVE/PE following this spec isn't expected to set the Assisted Replication Type field to decimal 3, since that's a RESERVED value 🙅♀️ If an AR-REPLICATOR or AR-LEAF receives a route with the Assisted Replication Type field set to decimal 3, it processes that route as a Regular-IR route advertised by an RNVE. Not me treating a reserved value like a real one 💀
Each node attached to the BD may understand and process the BM/U flags (PFLs flags). Note that these BM/U flags can be used to optimize delivery of multi-destination traffic. Their use SHOULD be an administrative choice and independent of the AR role 🌸 When the PFL capability is enabled, the BM/U flags can be used with the Regular-IR, Replicator-AR, and Leaf A-D routes ✨
Last but not least, non-optimized ingress replication NVEs/PEs won't know about the new PMSI Tunnel Attribute flag definition or the new tunnel type (AR). So non-upgraded NVEs/PEs ignore the information in the Flags field, or an unknown tunnel type (type AR in this case), for any Inclusive Multicast Ethernet Tag route 🧸
5. Non-selective Assisted Replication (AR) Solution Description ✨
Okay bestie, check out Figure 4 🦋 It shows an NVO network where non-selective AR is turned on. Every BD gets one of three roles: AR-REPLICATOR, AR-LEAF, and RNVE. It's called "non-selective" because the chosen AR-REPLICATOR for a given flow MUST replicate the BM traffic to all the NVEs/PEs in the BD except for the source NVE/PE 💅 NVO tunnels (aka IP tunnels) exist among all the PEs and NVEs in the diagram, and the PEs and NVEs have TSs or VMs plugged into their ACs. Iconic setup 🌸
( )
(_ WAN _)
+---(_ _)----+
| (_ _) |
PE1 | PE2 |
+------+----+ +----+------+
TS1--+ (BD-1) | | (BD-1) +--TS2
|REPLICATOR | |REPLICATOR |
+--------+--+ +--+--------+
| |
+--+----------------+--+
| |
| |
+----+ VXLAN/NVGRE/MPLSoGRE +----+
| | IP Fabric | |
| | | |
NVE1 | +-----------+----------+ | NVE3
Hypervisor| ToR | NVE2 |Hypervisor
+---------+-+ +-----+-----+ +-+---------+
| (BD-1) | | (BD-1) | | (BD-1) |
| LEAF | | RNVE | | LEAF |
+--+-----+--+ +--+-----+--+ +--+-----+--+
| | | | | |
VM11 VM12 TS3 TS4 VM31 VM32
Figure 4: Non-selective AR ScenarioSo in AR BDs like BD-1 in Figure 4, BM traffic between two NVEs can take a different path than unicast traffic 🧚 This solution recommends pushing BM traffic through the AR-REPLICATOR node, while unknown/known unicast traffic gets delivered straight from the source node to the destination node, with no intermediate node replicating it. It's giving efficient 💖
Quick note: known unicast forwarding is NOT impacted by this solution. Like, at all. Unknown unicast traffic SHALL follow the same path as known unicast traffic 🙈
5.1. Non-selective AR-REPLICATOR Procedures
An AR-REPLICATOR is an NVE/PE that can replicate incoming BM traffic (received on an overlay tunnel) out to other overlay tunnels and local ACs. It signals its role in the control plane and knows where the other roles live: AR-LEAF nodes, RNVEs, and other AR-REPLICATORs 👑 An AR-enabled BD service may have zero, one, or more AR-REPLICATORs. In our Figure 4 example, PE1 and PE2 are the AR-REPLICATORs. These are the rules for the AR-REPLICATOR role, bestie:
a. The AR-REPLICATOR role SHOULD be an administrative choice in any NVE/PE that is part of an AR-enabled BD. This admin option to turn on AR-REPLICATOR capabilities MAY be implemented as a system-level option instead of a per-BD option ✨
b. An AR-REPLICATOR MUST advertise a Replicator-AR route and MAY advertise a Regular-IR route. The AR-REPLICATOR MUST NOT generate a Regular-IR route if it does not have local ACs. If the Regular-IR route is advertised, the Assisted Replication Type field of the Regular-IR route MUST be set to 0 😭
c. The Replicator-AR and Regular-IR routes are generated according to Section 4. The AR-IP and IR-IP are different IP addresses owned by the AR-REPLICATOR 🎀
d. When a node defined as an AR-REPLICATOR receives a BM packet on an overlay tunnel, it does a tunnel destination IP address lookup and applies these procedures:
- If the destination IP address is the AR-REPLICATOR IR-IP address, the node will process the packet normally, as discussed in [RFC7432]. Standard vibes 🌈
- If the destination IP address is the AR-REPLICATOR AR-IP address, the node MUST replicate the packet to local ACs and overlay tunnels (excluding the overlay tunnel to the source of the packet). When replicating to remote AR-REPLICATORs, the tunnel destination IP address will be an IR-IP. This tells the remote AR-REPLICATOR that it MUST NOT replicate to overlay tunnels. The tunnel source IP address used by the AR-REPLICATOR MUST be its IR-IP when replicating to AR-REPLICATOR or AR-LEAF nodes 🍓
An AR-REPLICATOR MUST follow a data path implementation compatible with the following rules, no cap 👑
- The AR-REPLICATORs will build a flooding list made of ACs and overlay tunnels to remote nodes in the BD. Some of those overlay tunnels MAY be flagged as non-BM receivers, based on the BM flag received from the remote nodes in the BD 💜
- When an AR-REPLICATOR receives a BM packet on an AC, it will forward the BM packet to its flooding list (including local ACs and remote NVEs/PEs), skipping the non-BM overlay tunnels.
- When an AR-REPLICATOR receives a BM packet on an overlay tunnel, it will check the destination IP address of the underlay IP header and:
- If the destination IP address matches its IR-IP, the AR-REPLICATOR will skip all the overlay tunnels from the flooding list, i.e., it will only replicate to local ACs. This is the regular ingress replication behavior described in [RFC7432]. Basic, but make it work 💅
- If the destination IP address matches its AR-IP, the AR-REPLICATOR MUST forward the BM packet to its flooding list (ACs and overlay tunnels), excluding the non-BM overlay tunnels. The AR-REPLICATOR will make sure the traffic is not sent back to the originating AR-LEAF 🦋
- If the encapsulation is MPLSoGRE or MPLSoUDP and the received BD label that the AR-REPLICATOR advertised in the Replicator-AR route is not at the bottom of the stack, the AR-REPLICATOR MUST copy all the labels below the BD label and propagate them when forwarding the packet to the egress overlay tunnels. Label stack drama, but handled 🧚
- The AR-REPLICATOR/LEAF nodes will build an unknown unicast flooding list made of ACs and overlay tunnels to the IR-IP addresses of the remote nodes in the BD. Some of those overlay tunnels MAY be flagged as non-U (unknown unicast) receivers, based on the U flag received from the remote nodes in the BD 🌸
- When an AR-REPLICATOR/LEAF receives an unknown unicast packet on an AC, it will forward the unknown unicast packet to its flooding list, skipping the non-U overlay tunnels.
- When an AR-REPLICATOR/LEAF receives an unknown unicast packet on an overlay tunnel, it will forward the unknown unicast packet to its local ACs and never to an overlay tunnel. This is the regular ingress replication behavior described in [RFC7432]. Never to a tunnel, periodt 🫶
5.2. Non-selective AR-LEAF Procedures 🦋
okay so an AR-LEAF is an NVE/PE that's not great at replicating traffic itself 🥺, so it sends ALL the BM traffic to an AR-REPLICATOR that can copy it further on its behalf. It MAY signal its AR-LEAF capability in the control plane, and it knows where the other roles live (AR-REPLICATORs and RNVEs). A given service can have zero, one, or more AR-LEAF nodes. In Figure 4, NVE1 and NVE3 (both chilling in hypervisors) are the AR-LEAF nodes. Here's what applies to the AR-LEAF role 💅:
a. The AR-LEAF role SHOULD be an administrative choice in any NVE/PE that's part of an AR-enabled BD. This administrative option to turn on AR-LEAF capabilities MAY be implemented as a system-level option instead of a per-BD option. 🌸
b. In this non-selective AR solution, the AR-LEAF MUST advertise a single Regular-IR Inclusive Multicast Ethernet Tag route, as described in [RFC7432]. The AR-LEAF SHOULD set the Assisted Replication Type field to AR-LEAF. Note that even though this field doesn't affect the remote nodes when they create an EVPN destination to the AR-LEAF, it's super useful for ease of operation and troubleshooting of the BD 🔍✨.
c. In a BD where there are no AR-REPLICATORs (because the AR-REPLICATORs are down or reconfigured), the AR-LEAF MUST use regular ingress replication based on the remote Regular-IR Inclusive Multicast Ethernet Tag routes, as described in [RFC7432]. This may happen in the following cases:
- The AR-LEAF has a list of AR-REPLICATORs for the BD, but it detects that all of them are down (via next-hop tracking in the IGP or some other detection mechanism). 😬
- The AR-LEAF receives updates from all the former AR-REPLICATORs containing a non-REPLICATOR AR type in the Inclusive Multicast Ethernet Tag routes. 🙈
- The AR-LEAF never discovered an AR-REPLICATOR for the BD. 💀
d. In a service where there are one or more AR-REPLICATORs (based on the received Replicator-AR routes for the BD), the AR-LEAF can locally pick which AR-REPLICATOR it sends the BM traffic to 🎯:
- A single AR-REPLICATOR MAY be selected for all the BM packets received on the AR-LEAF ACs for a given BD. This pick is a local decision and doesn't have to match what other AR-LEAFs picked within the same BD. 🫶
- An AR-LEAF MAY select more than one AR-REPLICATOR and do either per-flow or per-BD load balancing. ⚖️
- If the selected AR-REPLICATOR fails, another AR-REPLICATOR SHOULD be selected by the AR-LEAF. 🔄
- When an AR-REPLICATOR is selected for a given flow or BD, the AR-LEAF MUST send all the BM packets targeted to that AR-REPLICATOR using the forwarding information given by the Replicator-AR route for the chosen AR-REPLICATOR, with Tunnel Type =
0x0A(AR tunnel). The underlay destination IP address MUST be the AR-IP advertised by the AR-REPLICATOR in the Replicator-AR route. 🚀 - An AR-LEAF MAY change its AR-REPLICATOR selection(s) dynamically due to an administrative or policy configuration change. 🔀
- AR-LEAF nodes SHALL send service-level BM control plane packets, following the procedures for regular ingress replication. Examples are IGMP, Multicast Listener Discovery (MLD), or PIM packets, and, in general, any packets using link-local scope multicast IPv4 or IPv6 packets. The AR-REPLICATORs MUST NOT replicate these control plane packets to other overlay tunnels, since they'll use the IR-IP address. 🙅♀️
e. Using an AR-REPLICATOR-activation-timer (in seconds, with a default value of 3) on the AR-LEAF nodes is RECOMMENDED. ⏲️ When an AR-LEAF gets a new Replicator-AR route where the AR-REPLICATOR is selected, it runs a timer before programming the new AR-REPLICATOR. If there's a newly added AR-REPLICATOR, or an AR-REPLICATOR reboots, this timer gives it some time to program the AR-LEAF nodes before the AR-LEAF starts sending BM traffic. The AR-REPLICATOR-activation-timer SHOULD be configurable in seconds, and its value needs to account for how long it takes the AR-LEAF Regular-IR Inclusive Multicast Ethernet Tag route to reach the AR-REPLICATOR and get programmed. While the AR-REPLICATOR-activation-timer is running, the AR-LEAF node uses regular ingress replication. ⏳
f. If the AR-LEAF has already selected an AR-REPLICATOR, whether to switch to a new preferred AR-REPLICATOR for the existing BM traffic flows is a matter of local policy. 🧠
An AR-LEAF MUST follow a data path implementation compatible with the following rules 📋:
- The AR-LEAF nodes will build two flooding lists:
- Flooding list #1: Composed of ACs and an AR-REPLICATOR-set of overlay tunnels. The AR-REPLICATOR-set is defined as one or more overlay tunnels to the AR-IP addresses of the remote AR-REPLICATOR(s) in the BD. Picking more than one AR-REPLICATOR is described in item d. above and is a local AR-LEAF decision.
- Flooding list #2: Composed of ACs and overlay tunnels to the remote IR-IP addresses.
- When an AR-LEAF receives a BM packet on an AC, it checks the AR-REPLICATOR-set:
- If the AR-REPLICATOR-set is empty, the AR-LEAF MUST send the packet to flooding list #2.
- If the AR-REPLICATOR-set is NOT empty, the AR-LEAF MUST send the packet to flooding list #1, where only one of the overlay tunnels of the AR-REPLICATOR-set is used.
- When an AR-LEAF receives a BM packet on an overlay tunnel, it forwards the BM packet to its local ACs and never to an overlay tunnel. This is the regular ingress replication behavior described in [RFC7432].
- AR-LEAF nodes process unknown unicast traffic in the same way AR-REPLICATORS do, as described in Section 5.1.
5.3. RNVE Procedures 🌈
An RNVE is an NVE/PE without AR-REPLICATOR or AR-LEAF capabilities that does ingress replication as described in [RFC7432]. The RNVE doesn't signal any AR role and is totally unaware of the AR-REPLICATOR/LEAF roles in the BD. It ignores the flags in the Regular-IR routes, and it ignores the Replicator-AR routes (because of an unknown tunnel type in the PMSI Tunnel Attribute) and the Leaf A-D routes (because of the IP-address-specific Route Target). 🙃
This role gives EVPNs the backward compatibility needed in optimized ingress replication BDs. In Figure 4, NVE2 is the RNVE. 🫶
6. Selective Assisted Replication (AR) Solution Description ✨
Figure 5 is used to describe the selective AR solution. 🎀
( )
(_ WAN _)
+---(_ _)----+
| (_ _) |
PE1 | PE2 |
+------+----+ +----+------+
TS1--+ (BD-1) | | (BD-1) +--TS2
|REPLICATOR | |REPLICATOR |
+--------+--+ +--+--------+
| |
+--+----------------+--+
| |
| |
+----+ VXLAN/NVGRE/MPLSoGRE +----+
| | IP Fabric | |
| | | |
NVE1 | +-----------+----------+ | NVE3
Hypervisor| ToR | NVE2 |Hypervisor
+---------+-+ +-----+-----+ +-+---------+
| (BD-1) | | (BD-1) | | (BD-1) |
|LEAF-set-1 | |LEAF-set-1 | |LEAF-set-2 |
+--+-----+--+ +--+-----+--+ +--+-----+--+
| | | | | |
VM11 VM12 TS3 TS4 VM31 VM32
Figure 5: Selective AR ScenarioThe solution is called "selective" because a given AR-REPLICATOR MUST replicate the BM traffic to only the AR-LEAFs that requested the replication (as opposed to all the AR-LEAF nodes), and it MUST replicate the BM traffic to the RNVEs (if there are any). The same AR roles defined in Sections 4 and 5 are used here, but the procedures are different. 🔥
The selective AR procedures create multiple AR-LEAF-sets in the EVPN BD and build single-hop trees among AR-LEAFs of the same set (AR-LEAF->AR-REPLICATOR->AR-LEAF) and two-hop trees among AR-LEAFs of different sets (AR-LEAF->AR-REPLICATOR->AR-REPLICATOR->AR-LEAF). Compared to the selective solution, the non-selective AR method assumes all the AR-LEAFs of the BD are in the same set and always creates single-hop trees among AR-LEAFs. The selective solution is more efficient than the non-selective one in multi-stage IP fabrics, but the trade-off is additional signaling and an additional outer source IP address lookup. 🧩
The following subsections describe the differences in the procedures for AR-REPLICATORs/LEAFs compared to the non-selective AR solution. There are no changes applicable to RNVEs. 💖
6.1. Selective AR-REPLICATOR Procedures 🦋
Okay so in our example in Figure 5, PE1 and PE2 are the selective AR-REPLICATORs 💅 Here's what that role looks like, bestie:
a. The selective AR-REPLICATOR role SHOULD be an administrative choice in any NVE/PE that is part of an AR-enabled BD. This administrative option MAY be implemented as a system-level option as opposed to a per-BD option. 🌸
b. Each AR-REPLICATOR will build a list of AR-REPLICATOR, AR-LEAF, and RNVE nodes. Even with the "selective" administrative option turned on, an AR-REPLICATOR MUST NOT behave as a selective AR-REPLICATOR if at least one of the AR-REPLICATORs has the L flag NOT set. If at least one AR-REPLICATOR sends a Replicator-AR route with L = 0 (in the BD context), the rest of the AR-REPLICATORs will fall back to non-selective AR mode. It's giving "one person says no and everybody reverts" energy 🙈
c. The selective AR-REPLICATOR MUST follow the procedures described in Section 5.1, with a few differences that are a whole mood ✨
- The AR-REPLICATOR MUST have the L flag set to 1 when advertising the Replicator-AR route. This flag is how AR-REPLICATORs advertise their "selective" AR-REPLICATOR capabilities 🎀 On top of that, the AR-REPLICATOR auto-configures its IP-address-specific import Route Target as described in the third bullet of the procedures for Leaf A-D routes in Section 4.
- The AR-REPLICATOR will build a "selective" AR-LEAF-set with the list of nodes that requested replication to its own AR-IP. For example, say NVE1 and NVE2 advertise a Leaf A-D route with PE1's IP-address-specific Route Target, and NVE3 advertises a Leaf A-D route with PE2's IP-address-specific Route Target. Then PE1 only adds NVE1/NVE2 to its selective AR-LEAF-set for BD-1 and excludes NVE3. Likewise, PE2 only adds NVE3 to its selective AR-LEAF-set for BD-1 and excludes NVE1/NVE2. Sorted, no drama 🧚
- When a node defined and operating as a selective AR-REPLICATOR receives a packet on an overlay tunnel, it does a tunnel destination IP lookup. If the destination IP address is the AR-REPLICATOR AR-IP address, the node MUST replicate the packet to:
- Local ACs.
- Overlay tunnels in the selective AR-LEAF-set, excluding the overlay tunnel to the source AR-LEAF.
- Overlay tunnels to the RNVEs if the tunnel source IP address is the IR-IP of an AR-LEAF. In any other case, the AR-REPLICATOR MUST NOT replicate the BM traffic to remote RNVEs. In other words, only the first-hop selective AR-REPLICATOR will replicate to all the RNVEs 👑
- Overlay tunnels to the remote selective AR-REPLICATORs if the tunnel source IP address (of the encapsulated packet that arrived on the overlay tunnel) is an IR-IP of its own AR-LEAF-set. In any other case, the AR-REPLICATOR MUST NOT replicate the BM traffic to remote AR-REPLICATORs. When doing this replication, the tunnel destination IP address is the AR-IP of the remote selective AR-REPLICATOR. That tunnel destination address AR-IP tells the remote selective AR-REPLICATOR that the packet needs further replication to its AR-LEAFs 🫶
A selective AR-REPLICATOR data path implementation MUST be compatible with the following rules, no cap 💀
- The selective AR-REPLICATORs will build two flooding lists:
- Flooding list #1: Composed of ACs and overlay tunnels to the remote nodes in the BD, always using the IR-IPs in the tunnel destination IP addresses.
- Flooding list #2: Composed of ACs, a selective AR-LEAF-set, and a selective AR-REPLICATOR-set, where:
- The selective AR-LEAF-set is composed of the overlay tunnels to the AR-LEAFs that advertise a Leaf A-D route for the local AR-REPLICATOR. This set is updated with every Leaf A-D route received or withdrawn from a new AR-LEAF.
- The selective AR-REPLICATOR-set is composed of the overlay tunnels to all the AR-REPLICATORs that send a Replicator-AR route with L = 1. The AR-IP addresses are used as tunnel destination IP addresses.
- Some of the overlay tunnels in the flooding lists MAY be flagged as non-BM receivers based on the BM flag received from the remote nodes in the routes.
- When a selective AR-REPLICATOR receives a BM packet on an AC, it MUST forward the BM packet to its flooding list #1, skipping the non-BM overlay tunnels.
- When a selective AR-REPLICATOR receives a BM packet on an overlay tunnel, it checks the destination and source IPs of the underlay IP header, and then:
- If the destination IP address matches its AR-IP and the source IP address matches an IP of its own selective AR-LEAF-set, the AR-REPLICATOR MUST forward the BM packet to its flooding list #2, unless some AR-REPLICATOR within the BD has advertised L = 0. In the latter case, the node reverts to Non-selective mode, and flooding list #1 MUST be used. Non-BM overlay tunnels are skipped when sending BM packets.
- If the destination IP address matches its AR-IP and the source IP address does not match any IP address of its selective AR-LEAF-set, the AR-REPLICATOR MUST forward the BM packet to flooding list #2, skipping the AR-REPLICATOR-set. Non-BM overlay tunnels are skipped when sending BM packets.
- If the destination IP address matches its IR-IP, the AR-REPLICATOR MUST use flooding list #1 but MUST skip all the overlay tunnels from the flooding list, i.e., it only replicates to local ACs. This is the regular ingress replication behavior described in [RFC7432]. Non-BM overlay tunnels are skipped when sending BM packets.
- In any case, the AR-REPLICATOR ensures that the traffic is not sent back to the originating source. If the encapsulation is MPLSoGRE or MPLSoUDP and the received BD label (the label that the AR-REPLICATOR advertised in the Replicator-AR route) is not at the bottom of the stack, the AR-REPLICATOR MUST copy the rest of the labels when forwarding them to the egress overlay tunnels. 🔥
6.2. Selective AR-LEAF Procedures 🌈
A selective AR-LEAF is picky in the best way. It chooses a single selective AR-REPLICATOR per BD, and then:
- It sends all the BD's BM traffic to that AR-REPLICATOR, and
- It expects to receive all the BM traffic for a given BD from the same AR-REPLICATOR (except for the BM traffic from the RNVEs, which comes directly from the RNVEs) 🍓
In the example in Figure 5, we consider NVE1/NVE2/NVE3 as selective AR-LEAFs. NVE1 selects PE1 as its selective AR-REPLICATOR. If that's the case, NVE1 sends all its BM traffic for BD-1 to PE1. If other AR-LEAFs or REPLICATORs send BM traffic, NVE1 receives that traffic from PE1. A selective AR-LEAF and a non-selective AR-LEAF behave differently, and here's how 🦋
a. The selective AR-LEAF role SHOULD be an administrative choice in any NVE/PE that is part of an AR-enabled BD. This administrative option to enable AR-LEAF capabilities MAY be implemented as a system-level option as opposed to a per-BD option. 💖
b. The AR-LEAF MAY advertise a Regular-IR route if there are RNVEs in the BD. The selective AR-LEAF MUST advertise a Leaf A-D route after receiving a Replicator-AR route with L = 1. It is RECOMMENDED that the selective AR-LEAF wait for a period specified by an AR-LEAF-join-wait-timer (in seconds, with a default value of 3) before sending the Leaf A-D route, so the AR-LEAF can collect all the Replicator-AR routes for the BD before advertising the Leaf A-D route. If the Replicator-AR route with L = 1 is withdrawn, the corresponding Leaf A-D route is withdrawn too. ⭐
c. In a service where there is more than one selective AR-REPLICATOR, the selective AR-LEAF MUST locally select a single selective AR-REPLICATOR for the BD. Once selected:
- The selective AR-LEAF MUST send a Leaf A-D route, including the route key and IP-address-specific Route Target of the selected AR-REPLICATOR.
- The selective AR-LEAF MUST send all the BM packets received on the ACs for a given BD to that AR-REPLICATOR.
- In the case of failure of the selected AR-REPLICATOR (detected when the Replicator-AR route becomes infeasible as a result of any of the underlying BGP mechanisms), another AR-REPLICATOR will be selected and a new Leaf A-D update will be issued for the new AR-REPLICATOR. This new route updates the selective list in the new selective AR-REPLICATOR. In the case of failure of the active selective AR-REPLICATOR, it is RECOMMENDED that the selective AR-LEAF revert to ingress replication behavior for an AR-REPLICATOR-activation-timer (in seconds, with a default value of 3) to mitigate the traffic impact. When the timer expires, the selective AR-LEAF resumes its AR mode with the new selective AR-REPLICATOR. The AR-REPLICATOR-activation-timer MAY be the same configurable parameter as the parameter discussed in Section 5.2. 🧚
- A selective AR-LEAF MAY change the selection of AR-REPLICATOR(s) dynamically due to an administrative or policy configuration change.
All the AR-LEAFs in a BD are expected to be configured as either selective or non-selective. A mix of selective and non-selective AR-LEAFs SHOULD NOT coexist in the same BD. If a non-selective AR-LEAF is present, its BM traffic sent to a selective AR-REPLICATOR will not be replicated to other AR-LEAFs that are not in its selective AR-LEAF-set. Mixing them is a recipe for chaos, bestie 😭
A selective AR-LEAF MUST follow a data path implementation compatible with the following rules:
- The selective AR-LEAF nodes will build two flooding lists:
- Flooding list #1: Composed of ACs and the overlay tunnel to the selected AR-REPLICATOR (using the AR-IP as the tunnel destination IP address).
- Flooding list #2: Composed of ACs and overlay tunnels to the remote IR-IP addresses.
- Some of the overlay tunnels in the flooding lists MAY be flagged as non-BM receivers based on the BM flag received from the remote nodes in the routes.
- When an AR-LEAF receives a BM packet on an AC, it checks to see if an AR-REPLICATOR was selected. If one is found, flooding list #1 MUST be used. Otherwise, flooding list #2 MUST be used. Non-BM overlay tunnels are skipped when sending BM packets.
- When an AR-LEAF receives a BM packet on an overlay tunnel, it MUST forward the BM packet to its local ACs and never to an overlay tunnel. This is the regular ingress replication behavior described in [RFC7432]. 🎀
7. Pruned Flooding Lists (PFLs) 🦋
Okay bestie, so besides AR, the second optimization in this ingress replication solution is that every BD node can signal PFLs ✨. Like we covered in Section 4, an EVPN node can set a value for the BM and U PFLs flags in its Regular-IR, Replicator-AR, or Leaf A-D routes 💅. Here's the deal:
BMis the Broadcast and Multicast flag.BM= 1 means "prune me from the BM flooding list".BM= 0 is regular behavior.Uis the Unknown flag.U= 1 means "prune me from the Unknown flooding list".U= 0 is regular behavior.
Signaling and processing these PFLs flags SHOULD be an administrative choice 👑. If a node is set up to process them, then whenever it gets a non-zero PFLs flag for a route, the NVE/PE adds the matching flag to the overlay tunnel it created in the flooding list 🌸. When it replicates a BM packet from a flooding list, it skips any overlay tunnels marked BM = 1, because the NVEs/PEs at the other end of those tunnels aren't expecting BM packets 🙈. Same vibe for unknown unicast: the NVE/PE skips overlay tunnels marked U = 1 🫶.
An NVE/PE that isn't following this document, or isn't configured for this optimization, just ignores any PFLs flags it receives 💀. An AR-LEAF or RNVE that gets BUM traffic on an overlay tunnel MUST replicate that traffic to its local ACs, no matter what BM/U flags the overlay tunnels have 🔥. Not up for debate, periodt 💅.
This optimization MAY be used along with the Assisted Replication solution ✨.
7.1. Example of a Pruned Flooding List 🧚
Let's walk through how PFLs actually feels in practice 🎀. We'll assume that BD-1 in Figure 4 has optimized ingress replication enabled, and:
- PE1 and PE2 are administratively configured as AR-REPLICATORs because they have high-performance replication powers 💪. PE1 and PE2 will send a Replicator-AR route with
BM/Uflags =00💜. - NVE1 and NVE3 are administratively configured as AR-LEAF nodes because their replication is software-based and kinda low-performance 🥺. They will advertise a Regular-IR route with type AR-LEAF. Assume both NVEs advertise all of their attached VMs' MAC and IP addresses in EVPNs as soon as they come up, and these NVEs don't have any VMs into multicast apps. Then they'll be configured to signal
BM/Uflags =11for BD-1 🙅♀️. That means neither NVE1 nor NVE3 wants BM or unknown unicast traffic, because:- Their attached VMs (VM11, VM12, VM31, VM32) don't support multicast applications 🙈.
- Their attached VMs won't receive ARP Requests. Proxy ARP [RFC9161] on the remote NVEs/PEs will answer ARP Requests locally, and no other broadcast traffic is expected 🌈.
- Their attached VMs won't receive unknown unicast traffic, since the VMs' MAC and IP addresses are always advertised by EVPNs as long as the VMs are active 🦋.
- NVE2 isn't optimized ingress replication aware, so it takes on the RNVE role in BD-1 🌸.
Based on those assumptions, here's the forwarding behavior that happens ✨:
- Any BM packets sent from VM11 go to VM12 and PE1. PE1 then forwards the BM packets on to TS1, the WAN link, PE2, and NVE2, but not to NVE3 🚫. PE2 and NVE2 replicate the BM packets to their local ACs, but NVE3 is spared from replicating those BM packets to VM31 and VM32 for no reason 😭.
- Any BM packets received on PE2 from the WAN go to PE1 and NVE2, but not to NVE1 and NVE3, so those two hypervisors don't have to replicate to their local VMs for nothing 💅. PE1 and NVE2 replicate to their local ACs only.
- Any unknown unicast packet sent from VM31 is forwarded by NVE3 to NVE2, PE1, and PE2, but not to NVE1 🧠. The solution stops unnecessary replication to NVE1, since the destination of that unknown traffic can't be NVE1 👑.
- Any unknown unicast packet sent from TS1 is forwarded by PE1 to the WAN link, PE2, and NVE2, but not to NVE1 and NVE3, since the target of that unknown traffic can't be NVE1 or NVE3 🎀.
8. AR Procedures for Single-IP AR-REPLICATORS 🧩
Sections 5 and 6 assume the AR-REPLICATOR can use two local routable IP addresses to terminate and originate NVO tunnels, i.e., IR-IP and AR-IP addresses 💖. That's usually the case for PE-based AR-REPLICATOR nodes 🌷.
Sometimes, though, the AR-REPLICATOR doesn't support more than one IP address for terminating and originating NVO tunnels, meaning the IR-IP and AR-IP are the same IP address 🥹. That can happen with some software-based or low-end AR-REPLICATOR nodes 💻. If that's your situation, the procedures in Sections 5 and 6 MUST be modified in the following way 🔥:
- The Replicator-AR routes generated by the AR-REPLICATOR use an AR-IP that matches its IR-IP ✨. To tell apart the data plane packets that need ingress replication from the packets that must use Assisted Replication forwarding mode, the Replicator-AR route MUST advertise a different VNI/VSID than the one used by the Regular-IR route 🦋. For instance, the AR-REPLICATOR advertises an AR-VNI along with the Replicator-AR route, and an IR-VNI along with the Regular-IR route 💜. Since both routes have the same key, each route needs a different Route Distinguisher 🌈.
- An AR-REPLICATOR performs Ingress Replication forwarding mode or Assisted Replication forwarding mode for incoming overlay packets based on an ingress VNI lookup, not the tunnel IP DA lookup 🧚. When replicating to remote AR-REPLICATOR nodes, the IR-VNI or AR-VNI advertised by the egress node decides whether Ingress Replication forwarding mode or Assisted Replication forwarding mode gets used at the subsequent AR-REPLICATOR 🫶.
The rest of the procedures follow those described in Sections 5 and 6 🎀.
9. AR Procedures and EVPN All-Active Multihoming Split-Horizon 🌸
This section extends the procedures for cases where two or more AR-LEAF nodes are attached to the same ES, and two or more AR-REPLICATOR nodes are attached to the same ES in the BD 👑. The mixed case, where an AR-LEAF node and an AR-REPLICATOR node are attached to the same ES, would need extended procedures that are out of scope for this document 🙃.
9.1. Ethernet Segments on AR-LEAF Nodes 🦋
If VXLAN or NVGRE is used, and the split-horizon is based on the tunnel source IP address and "local bias" as described in [RFC8365], then the split-horizon check won't work when an ES is shared between two AR-LEAF nodes and the AR-REPLICATOR replaces the tunnel source IP address of packets with its own AR-IP 😭.
To stay compatible with the source IP address split-horizon check, the AR-REPLICATOR MAY keep the original received tunnel source IP address when replicating packets to a remote AR-LEAF or RNVE ✨. That lets AR-LEAF nodes apply split-horizon check procedures for BM packets before sending them to the local ES 💅. Even if the AR-LEAF's source IP address is preserved when replicating to AR-LEAFs or RNVEs, the AR-REPLICATOR MUST always use its IR-IP as the source IP address when replicating to other AR-REPLICATORs 🔥.
When EVPNs are used for MPLSoGRE or MPLSoUDP, the ESI-label-based split-horizon procedure provided in [RFC7432] won't work for multihomed ESs defined on AR-LEAF nodes 🥺. Local bias is recommended in this case, just like with VXLAN or NVGRE explained above 🌈. The local-bias and tunnel source IP address preservation mechanisms together provide the required split-horizon behavior in non-selective or selective AR 🧠.
If the AR-REPLICATOR implementation keeps the received tunnel source IP address, then unicast Reverse Path Forwarding (uRPF) checks in the IP fabric based on the tunnel source IP address MUST be disabled ⚠️. Don't skip this one, bestie 💀.
9.2. Ethernet Segments on AR-REPLICATOR Nodes 💜
AR-REPLICATOR nodes attached to the same all-active ES follow local-bias procedures [RFC8365] as follows 🫶:
a. For BUM traffic received on a local AR-REPLICATOR's AC, local-bias procedures as provided in [RFC8365] MUST be followed 💅.
b. For BUM traffic received on an AR-REPLICATOR overlay tunnel with AR-IP as the IP DA, local bias MUST also be followed 🔥. That means traffic received with AR-IP as the IP DA gets treated as though it came in on a local AC that's part of the ES, and it gets forwarded to all local ESs, regardless of their DF or NDF state 👑.
c. BUM traffic received on an AR-REPLICATOR overlay tunnel with IR-IP as the IP DA follows regular local-bias rules [RFC8365] and won't be forwarded to local ESs that are shared with the AR-LEAF or AR-REPLICATOR originating the traffic 🎀.
d. When the AR-REPLICATOR supports a single IP address, the IR-IP and the AR-IP are the same IP address, as discussed in Section 8 🧩. The received BUM traffic gets treated as in item b above if the received VNI is the AR-VNI, and as in item c if the VNI is the IR-VNI 🌸.
10. Security Considerations ✨
okay bestie, this section is SERIOUS but also kind of iconic, so let's go 💅 The security considerations in [RFC7432] and [RFC8365] apply to this document. The security considerations related to the Leaf A-D route in [RFC9572] apply too. 🔐
Lowkey the Assisted Replication method introduced by this document may bring some new risks that could mess with the successful delivery of BM traffic 😭. Unicast traffic is NOT affected by Assisted Replication (although unknown unicast traffic is affected by the procedures for PFLs) 🦋. The forwarding of BM traffic is modified, so BM traffic from the AR-LEAF nodes gets drawn toward AR-REPLICATORs in the BD 🧲. An AR-LEAF forwards BM traffic to its selected AR-REPLICATOR, which means an attack on the AR-REPLICATOR could impact delivery of the BM traffic using that node 🚨. Also, an attack on the AR-REPLICATOR and any change to the advertised AR type will modify the selections made by the AR-LEAF nodes. If no other AR-REPLICATOR is selected, the AR-LEAF nodes are forced to use Ingress Replication forwarding mode, which hurts their performance, since AR-LEAF nodes are usually NVEs/PEs with poor replication performance 💀.
This document also introduces the ability for the AR-REPLICATOR to forward traffic received on an overlay tunnel to another overlay tunnel. You might think this brings the risk of BM loops 🌀, that is, an AR-LEAF receiving a BM-encapsulated packet that the AR-LEAF originated in the first place, because one or two AR-REPLICATORs "looped" the BM traffic back to the AR-LEAF. Following the procedures in this document prevents these BM loops, since the AR-REPLICATOR will always forward the BM traffic using the correct tunnel IP DA (or the correct VNI in the case of single-IP AR-REPLICATORs), which tells the remote nodes how to forward the traffic 🙌. This is true for both the Non-selective and Selective modes defined in this document. However, incorrect implementation of the procedures in this document may still lead to those unexpected BM loops 😬.
The Selective mode provides a multi-stage replication solution, where proper configuration of all the AR-REPLICATORs will prevent any issues 🎀. A mix of mistakenly configured selective and non-selective AR-REPLICATORs in the same BD could theoretically create packet duplication in some AR-LEAFs 🙈. That said, this document specifies a fallback solution, which is falling back to Non-selective mode in cases where the AR-REPLICATORs advertised an inconsistent AR mode 🫶.
This document allows the AR-REPLICATOR to preserve the tunnel source IP address of the AR-LEAF (as an option) when forwarding BM packets from one overlay tunnel to another overlay tunnel 🌸. Preserving the AR-LEAF source IP address makes the local-bias filtering procedures possible for AR-LEAF nodes attached to the same ES. If the AR-REPLICATOR does not preserve the AR-LEAF source IP address, AR-LEAF nodes attached to all-active ESs will cause packet duplication on the multihomed CE 😱.
The AR-REPLICATOR nodes are, by design, using more bandwidth than PEs [RFC7432] or NVEs [RFC8365] would use. Certain network events or unexpected low performance may exceed the AR-REPLICATOR's local bandwidth and cause service disruption 📉.
Finally, PFLs (Section 7) should be used with care. Intentional or unintentional misconfiguration of the BDs on a given leaf node may result in the leaf not receiving the required BM or unknown unicast traffic. Handle with care, bestie 💜.
11. IANA Considerations 🍓
IANA has allocated the following Border Gateway Protocol (BGP) parameters:
- Allocation in the "P-Multicast Service Interface Tunnel (PMSI Tunnel) Tunnel Types" registry:
+=======+=============================+===========+
| Value | Meaning | Reference |
+=======+=============================+===========+
| 0x0A | Assisted Replication Tunnel | RFC 9574 |
+-------+-----------------------------+-----------+
Table 1- Allocations in the "P-Multicast Service Interface (PMSI) Tunnel Attribute Flags" registry:
+=======+===============================+===========+
| Value | Name | Reference |
+=======+===============================+===========+
| 3-4 | Assisted Replication Type (T) | RFC 9574 |
+-------+-------------------------------+-----------+
| 5 | Broadcast and Multicast (BM) | RFC 9574 |
+-------+-------------------------------+-----------+
| 6 | Unknown (U) | RFC 9574 |
+-------+-------------------------------+-----------+
Table 212. References 📚
12.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC6513] Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/
BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February
2012, <https://www.rfc-editor.org/info/rfc6513>.
[RFC6514] Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
Encodings and Procedures for Multicast in MPLS/BGP IP
VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,
<https://www.rfc-editor.org/info/rfc6514>.
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <https://www.rfc-editor.org/info/rfc7432>.
[RFC7902] Rosen, E. and T. Morin, "Registry and Extensions for
P-Multicast Service Interface Tunnel Attribute Flags",
RFC 7902, DOI 10.17487/RFC7902, June 2016,
<https://www.rfc-editor.org/info/rfc7902>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.
[RFC9572] Zhang, Z., Lin, W., Rabadan, J., Patel, K., and A.
Sajassi, "Updates to EVPN Broadcast, Unknown Unicast, or
Multicast (BUM) Procedures", RFC 9572,
DOI 10.17487/RFC9572, May 2024,
<https://www.rfc-editor.org/info/rfc9572>.12.2. Informative References
[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed.,
"Encapsulating MPLS in IP or Generic Routing Encapsulation
(GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005,
<https://www.rfc-editor.org/info/rfc4023>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.
[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
Virtualization Using Generic Routing Encapsulation",
RFC 7637, DOI 10.17487/RFC7637, September 2015,
<https://www.rfc-editor.org/info/rfc7637>.
[RFC9161] Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Hankins, G.,
and T. King, "Operational Aspects of Proxy ARP/ND in
Ethernet Virtual Private Networks", RFC 9161,
DOI 10.17487/RFC9161, January 2022,
<https://www.rfc-editor.org/info/rfc9161>.Acknowledgements 💖
The authors would like to thank Neil Hart, David Motz, Dai Truong, Thomas Morin, Jeffrey Zhang, Shankar Murthy, and Krzysztof Szarkowicz for their valuable feedback and contributions. Also, thanks to John Scudder for his thorough review, which improved the quality of the document significantly 🥹.
Contributors 🌈
In addition to the authors listed on the front page, the following people also contributed to this document and should be considered coauthors:
- Wim Henderickx, Nokia
- Kiran Nagaraj, Nokia
- Ravi Shekhar, Juniper Networks
- Nischal Sheth, Juniper Networks
- Aldrin Isaac, Juniper
- Mudassir Tufail, Citibank
Authors' Addresses 🦋
- Jorge Rabadan (editor), Nokia, 777 Middlefield Road, Mountain View, CA 94043, United States of America. Email: jorge.rabadan@nokia.com
- Senthil Sathappan, Nokia. Email: senthil.sathappan@nokia.com
- Wen Lin, Juniper Networks. Email: wlin@juniper.net
- Mukul Katiyar, Versa Networks. Email: mukul@versa-networks.com
- Ali Sajassi, Cisco Systems. Email: sajassi@cisco.com
📦 where this page comes from
- ietf: rfc9574 2024-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms