RFC 9625
EVPN Optimized Inter-Subnet Multicast (OISM) Forwarding
RFC 9625: EVPN Optimized Inter-Subnet Multicast (OISM) Forwarding 🎀
Okay bestie, this is RFC 9625 💅 It's a Standards Track document from the IETF, so it's an official spec that people are meant to build against. It came out in August 2024, and its ISSN is 2070-1721 ✨
The authors are W. Lin, Z. Zhang, J. Drake, E. Rosen (who is the editor, so Ed.), J. Rabadan, and A. Sajassi. The affiliations listed are Juniper Networks, Inc., Nokia, and Cisco Systems. The title block doesn't list anything this RFC obsoletes or updates, so it's a clean, standalone vibe 🌈
Abstract ✨
Okay, so Ethernet VPN (EVPN) is a service that takes one Local Area Network (LAN), which is one IP subnet, and splits it into multiple segments 🦋 Each segment can live at a different site, and the segments are linked together over an IP or MPLS backbone. Intra-subnet traffic, whether it's unicast or multicast, always looks bridged to the end users, even though it's actually traveling over that backbone. And when one tenant owns several of these LANs, EVPN also lets IP unicast traffic get routed between them 🌸
This document is about the next level: it defines new procedures so that inter-subnet IP multicast traffic can be routed among a tenant's LANs, while intra-subnet IP multicast still looks bridged 💖 These procedures can give optimal routing for the inter-subnet multicast traffic, and they don't force any of that traffic to leave a router and then come right back into that same router. They also handle IP multicast traffic that starts outside the EVPN domain or is headed outside it 🧚
Status of This Memo 📜
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). More info on Internet Standards is in Section 2 of RFC 7841 ✨
Info about the current status of this document, any errata, and how to send feedback on it can be found at https://www.rfc-editor.org/info/rfc9625 🔗
Copyright Notice ©
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please read these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they are provided without warranty as described in the Revised BSD License 💜
Table of Contents 🗂️
1. Introduction
1.1. Terminology
1.1.1. Requirements Language
1.2. Background
1.2.1. Segments, Broadcast Domains, and Tenants
1.2.2. Inter-BD (Inter-Subnet) IP Traffic
1.2.3. EVPN and IP Multicast
1.2.4. BDs, MAC-VRFs, and EVPN Service Models
1.3. Need for EVPN-Aware Multicast Procedures
1.4. Additional Requirements That Must Be Met by the Solution
1.5. Model of Operation: Overview
1.5.1. Control Plane
1.5.2. Data Plane
2. Detailed Model of Operation
2.1. Supplementary Broadcast Domain
2.2. Detecting When a Route is for/from a Particular BD
2.3. Use of IRB Interfaces at Ingress PE
2.4. Use of IRB Interfaces at an Egress PE
2.5. Announcing Interest in (S,G)
2.6. Tunneling Frames from Ingress PEs to Egress PEs
2.7. Advanced Scenarios
3. EVPN-Aware Multicast Solution Control Plane
3.1. Supplementary Broadcast Domain (SBD) and Route Targets
3.2. Advertising the Tunnels Used for IP Multicast
3.2.1. Constructing Routes for the SBD
3.2.2. Ingress Replication
3.2.3. Assisted Replication
3.2.3.1. Automatic SBD Matching
3.2.4. BIER
3.2.5. Inclusive P2MP Tunnels
3.2.5.1. Using the BUM Tunnels as IP Multicast Inclusive
Tunnels
3.2.5.2. Using Wildcard S-PMSI A-D Routes to Advertise
Inclusive Tunnels Specific to IP Multicast
3.2.6. Selective Tunnels
3.3. Advertising SMET Routes
4. Constructing Multicast Forwarding State
4.1. Layer 2 Multicast State
4.1.1. Constructing the OIF List
4.1.2. Data Plane: Applying the OIF List to an (S,G) Frame
4.1.2.1. Eligibility of an AC to Receive a Frame
4.1.2.2. Applying the OIF List
4.2. Layer 3 Forwarding State
5. Interworking with Non-OISM EVPN PEs
5.1. IPMG Designated Forwarder
5.2. Ingress Replication
5.2.1. Ingress PE is Non-OISM
5.2.2. Ingress PE is OISM
5.3. P2MP Tunnels
6. Traffic to/from Outside the EVPN Tenant Domain
6.1. Layer 3 Interworking via EVPN OISM PEs
6.1.1. General Principles
6.1.2. Interworking with MVPN
6.1.2.1. MVPN Sources with EVPN Receivers
6.1.2.1.1. Identifying MVPN Sources
6.1.2.1.2. Joining a Flow from an MVPN Source
6.1.2.2. EVPN Sources with MVPN Receivers
6.1.2.2.1. General Procedures
6.1.2.2.2. Any-Source Multicast (ASM) Groups
6.1.2.2.3. Source on Multihomed Segment
6.1.2.3. Obtaining Optimal Routing of Traffic between MVPN
and EVPN
6.1.2.4. Selecting the MEG SBD-DR
6.1.3. Interworking with Global Table Multicast
6.1.4. Interworking with PIM
6.1.4.1. Source Inside EVPN Domain
6.1.4.2. Source Outside EVPN Domain
6.2. Interworking with PIM via an External PIM Router
7. Using an EVPN Tenant Domain as an Intermediate (Transit)
Network for Multicast Traffic
8. IANA Considerations
9. Security Considerations
10. References
10.1. Normative References
10.2. Informative References
Appendix A. Integrated Routing and Bridging
Acknowledgements
Authors' Addresses1. Introduction 🌸
1.1. Terminology 💜
Here's the vocab we're using in this doc. Lots of it is going to come up a lot, so let's get it down 🫶
OISM: Optimized Inter-Subnet Multicast. EVPN PEs that follow the procedures of this document are known as "OISM" Provider Edges (PEs). EVPN PEs that don't follow the procedures of this document are known as "non-OISM" PEs.
IP Multicast Packet: An IP packet whose IP Destination Address field is a multicast address that is not a link-local address. (Link-local addresses are IPv4 addresses in the 224/24 range and IPv6 addresses in the FF02/16 range.)
IP Multicast Frame: An Ethernet frame whose payload is an IP multicast packet (as defined above).
(S,G) Multicast Packet: An IP multicast packet whose Source IP Address field contains S and whose IP Destination Address field contains G.
(S,G) Multicast Frame: An IP multicast frame whose payload contains S in its Source IP Address field and G in its IP Destination Address field.
EVI: EVPN Instance. An EVPN instance spanning the PE devices participating in that EVPN.
BD: Broadcast Domain. An emulated Ethernet, such that two systems on the same BD will receive each other's link-local broadcasts.
Note that EVPN supports service models where a single EVI contains only one BD, and service models where a single EVI contains multiple BDs. Both types of service models are supported by this document. In all models, a given BD belongs to only one EVI ✨
DF: Designated Forwarder. As defined in [RFC7432], an Ethernet segment may be multihomed (attached to more than one PE). An Ethernet segment may also contain multiple BDs of one or more EVIs. For each such EVI, one of the PEs attached to the segment becomes that EVI's DF for that segment. Since a BD may belong to only one EVI, we can speak unambiguously of the BD's DF for a given segment 💅
AC: Attachment Circuit. An AC connects the bridging function of an EVPN PE to an Ethernet segment of a particular BD. ACs are not visible at Layer 3.
If a given Ethernet segment, attached to a given PE, contains n BDs, we say that the PE has n ACs to that segment.
L3 Gateway: An L3 Gateway is a PE that connects an EVPN Tenant Domain to an external multicast domain by performing both the OISM procedures and the Layer 3 multicast procedures of the external domain.
PEG: PIM/EVPN Gateway. An L3 Gateway that connects an EVPN Tenant Domain to an external multicast domain whose Layer 3 multicast procedures are those of PIM [RFC7761].
MEG: MVPN/EVPN Gateway. An L3 Gateway that connects an EVPN Tenant Domain to an external multicast domain whose Layer 3 multicast procedures are those of Multicast VPN (MVPN) [RFC6513] [RFC6514].
IPMG: IP Multicast Gateway. A PE that is used for interworking OISM EVPN PEs with non-OISM EVPN PEs.
DR: Designated Router. A PE that has special responsibilities for handling multicast on a given BD.
FHR: First Hop Router. The FHR is a PIM router [RFC7761] with special responsibilities. It is the first multicast router to see (S,G) packets from source S, and if G is an Any-Source Multicast (ASM) group, the FHR is responsible for sending PIM Register messages to the PIM Rendezvous Point (RP) for group G.
LHR: Last Hop Router. The LHR is a PIM router [RFC7761] with special responsibilities. Generally, it is attached to a LAN, and it determines whether there are any hosts on the LAN that need to receive a given multicast flow. If so, it creates and sends the PIM Join messages that are necessary to receive the flow 🙈
EC: Extended Community. A BGP Extended Communities attribute [RFC4360] [RFC7153] is a BGP path attribute that consists of one or more Extended Communities.
RT: Route Target. A Route Target is a particular kind of BGP Extended Community. A BGP Extended Community consists of a type field, a sub-type field, and a value field. Certain type/sub-type combinations indicate that a particular Extended Community is an RT. RT1 and RT2 are considered to be the same RT if and only if they have the same type, sub-type, and value fields.
C- prefix: In many documents on VPN multicast, the prefix C- appears before any address or wildcard that refers to an address or addresses in a tenant's address space, rather than to an address or addresses in the address space of the backbone network. This document omits the C- prefix in many cases where it's clear from the context that the reference is to the tenant's address space 🦋
This document also assumes familiarity with the terminology of [RFC4364], [RFC6514], [RFC7432], [RFC7761], [RFC9136], [RFC9251], and [RFC9572].
1.1.1. Requirements Language 🫶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎀
Part 1 of 18 is done 🌷 More parts are coming, so stay tuned, bestie!
1.2. Background 🫖✨
Okay bestie, let's spill the tea on EVPN (Ethernet VPN) [RFC7432]. It's giving L2VPN realness: an IP or MPLS backbone provider can offer Ethernet service to a whole crew of customers, who are called "tenants" 💅
In this section (and also in [RFC9135]) we're covering the essential EVPN background 📚
1.2.1. Segments, Broadcast Domains, and Tenants 🧚
The key concept of EVPN is the Broadcast Domain (BD) 🌈 A BD is basically an emulated Ethernet. Each BD belongs to exactly one tenant. A BD usually has multiple Ethernet segments, and each segment might be plugged into a different EVPN Provider Edge (EVPN PE) router. EVPN PE routers are often called "Network Virtualization Endpoints (NVEs)", but this document is gonna use "EVPN PE" or, when the context is clear, just "PE" 💖
In this document, "segment" is used interchangeably with "Ethernet Segment" or "ES", as defined in [RFC7432] 🦋
Attached to each segment are Tenant Systems (TSs). A TS can be any kind of system, physical or virtual, host or router, etc., that can attach to an Ethernet 🌸
When two TSs are on the same segment, their traffic does not pass through an EVPN PE. When two TSs are on different segments of the same BD, their traffic does pass through an EVPN PE 🫶
When two TSs, say TS1 and TS2, are on the same BD, the following happens:
- If TS1 knows the Media Access Control (MAC) address of TS2, TS1 can send unicast Ethernet frames to TS2. TS2 will receive the frames unaltered.
- If TS1 broadcasts an Ethernet frame, TS2 will receive the unaltered frame.
- If TS1 multicasts an Ethernet frame, TS2 will receive the unaltered frame as long as TS2 has been provisioned to receive the Ethernet multicast destination MAC address.
When we say TS2 receives an unaltered frame from TS1, we mean the frame still contains TS1's MAC address, and no alteration of the frame's payload (and therefore no alteration of the payload's IP header) has been made 🎀
EVPN lets a single segment be attached to multiple PE routers. This is known as "EVPN multihoming" 👑 Say a given segment is attached to both PE1 and PE2, and PE1 receives a frame from that segment. PE1 may need to send the frame over the backbone to PE2. EVPN has procedures to make sure PE2 can't send that frame back to its originating segment. This matters a lot for multicast, because a frame arriving at PE1 from a given segment has already been seen by all the systems on that segment that need to see it. If PE2 sent the frame back to the originating segment, receivers there would get the packet twice 😭 Even worse, the frame might get sent back to PE1, which could cause an infinite loop 💀
1.2.2. Inter-BD (Inter-Subnet) IP Traffic 🦋
If a given tenant has multiple BDs, the tenant may want to allow IP communication among them. A set of BDs like this is known as an "EVPN Tenant Domain" or just a "Tenant Domain" 💜
If tenant systems TS1 and TS2 are not in the same BD, they do not receive unaltered Ethernet frames from each other. For TS1 to send traffic to TS2, TS1 encapsulates an IP datagram inside an Ethernet frame and uses Ethernet to send that frame to an IP router. The router decapsulates the IP datagram, does the IP processing, and re-encapsulates the datagram for Ethernet. The MAC Source Address field now holds the MAC address of the router, not TS1's 🙈 The TTL field of the IP datagram should be decremented by exactly 1, even if the frame needs to be sent from one PE to another. So the provider's backbone structure stays hidden from the tenants 🫧
EVPN handles inter-BD communication within a Tenant Domain by providing an integrated L2/L3 service for unicast IP traffic ✨ EVPN's Integrated Routing and Bridging (IRB) functionality is specified in [RFC9135]. Each BD in a Tenant Domain is assumed to be a single IP subnet, and each IP subnet within a given Tenant Domain is assumed to be a single BD. EVPN's IRB functionality lets IP traffic travel from one BD to another and makes sure proper IP processing (for example, the TTL decrement) happens 🌈
A quick overview of IRB, including the notion of an IRB interface, lives in Appendix A. As explained there, an IRB interface is a sort of virtual interface connecting an L3 routing instance to a BD. A BD may have multiple Attachment Circuits (ACs) to a given PE, where each AC connects to a different Ethernet segment of the BD. However, these ACs are invisible to the L3 routing function. From the perspective of an L3 routing instance, a PE has just one interface to each BD, namely the IRB interface for that BD 🍓
In this document, when traffic is routed out of an IRB interface, we say it is sent down the IRB interface to the BD that the IRB is for. In the other direction, traffic is sent up the IRB interface from the BD to the L3 routing instance 🧚
The L3 routing instance depicted in Appendix A is associated with a single Tenant Domain, and you can think of it as the IP Virtual Routing and Forwarding (IP-VRF) for that Tenant Domain 🌸
1.2.3. EVPN and IP Multicast 🌈
[RFC9135] and [RFC9136] cover inter-subnet (inter-BD) IP unicast forwarding, but they do not cover inter-subnet IP multicast forwarding 🙃
[RFC7432] covers intra-subnet (intra-BD) Ethernet multicast. The intra-subnet Ethernet multicast procedures of [RFC7432] are used for Ethernet broadcast traffic, Ethernet unicast traffic whose Destination MAC Address field contains an unknown address, and Ethernet traffic whose Destination MAC Address field contains an Ethernet multicast MAC address. These three classes of traffic are collectively known as "BUM traffic" (Broadcast, Unknown Unicast, or Multicast traffic), and the procedures for handling BUM traffic are known as "BUM procedures" 📣
[RFC9251] extends the intra-subnet Ethernet multicast procedures by adding procedures that are specific to, and optimized for, the use of IP multicast within a subnet. However, that document does not cover inter-subnet IP multicast 🥹
The purpose of this document is to specify procedures for EVPN that provide optimized IP multicast functionality within an EVPN Tenant Domain. It also specifies procedures that allow IP multicast packets to be sourced from, or destined to, systems outside the Tenant Domain. The entire set of procedures is referred to as "Optimized Inter-Subnet Multicast (OISM)" procedures 🏆
To support the OISM procedures specified in this document, an EVPN PE MUST also support [RFC9135] and [RFC9251]. (However, certain procedures in [RFC9251] are modified when OISM is supported.) 🔥
1.2.4. BDs, MAC-VRFs, and EVPN Service Models 💅
[RFC7432] defines the notion of MAC-VRF (MAC Virtual Routing and Forwarding). A MAC-VRF contains one or more bridge tables (see Section 3 of [RFC7432]), and each of those represents a single Broadcast Domain 🎀
In the IRB model (outlined in Appendix A), an L3 routing instance has one IRB interface per BD, not one per MAC-VRF. This document doesn't distinguish between a Broadcast Domain and a bridge table. Instead, it uses the terms interchangeably (or uses the acronym "BD" to mean either one). How BDs are grouped into MAC-VRFs doesn't matter for the procedures specified here 🧁
Section 6 of [RFC7432] also defines several different EVPN service models:
- In the vlan-based service, each MAC-VRF contains one bridge table, and that bridge table corresponds to a particular Virtual LAN (VLAN) (see Section 3 of [RFC7432]). So each VLAN is treated as a BD 🌷
- In the vlan bundle service, each MAC-VRF contains one bridge table, and that bridge table corresponds to a set of VLANs. So a set of VLANs is treated as constituting a single BD 🍀
- In the vlan-aware bundle service, each MAC-VRF may contain multiple bridge tables, where each bridge table corresponds to one BD. If a MAC-VRF contains several bridge tables, then it corresponds to several BDs 🪩
The procedures in this document are meant to work for all of these service models 💖
1.3. Need for EVPN-Aware Multicast Procedures 🔍
Inter-subnet IP multicast among a set of BDs can be achieved, in a non-optimal way, without any specific EVPN procedures. For instance, if a tenant has n BDs among which it wants to send IP multicast traffic, it can simply attach a conventional multicast router to all n BDs. More generally, as long as each BD has at least one IP multicast router, and the IP multicast routers communicate multicast control information with each other, conventional IP multicast procedures will work normally, and no special EVPN functionality is needed 🙌
However, that technique does not provide optimal routing for multicast 😬 In conventional multicast routing, for a given multicast flow, there is only one multicast router on each BD that is permitted to send traffic of that flow to the BD. If that BD has receivers for a given flow, but the source of the flow is not on that BD, then the flow must pass through that multicast router. This leads to the hairpinning problem described (for unicast) in Appendix A 🌀
For example, consider an (S,G) flow that is sourced by a TS S and needs to be received by TSs R1 and R2. Suppose S is on a segment of BD1, and R1 is on a segment of BD2, but both are attached to PE1. Also suppose the tenant has a multicast router attached to a segment of BD1 and to a segment of BD2. However, the segments to which that router is attached are both attached to PE2. Then the flow from S to R would have to follow the path: S-->PE1-->PE2-->tenant multicast router-->PE2-->PE1-->R1. Obviously, the path S-->PE1-->R would be preferred 😤
etc.Hmm, that's not right; I need the diagram exactly as written. Here is the diagram from the original, verbatim:
+---+ +---+
|PE1+----------------------+PE2|
+---+-+ +-+---+
| \ \ / / |
BD1 BD2 BD3 BD3 BD2 BD1
| | | \ | |
S R1 R2 routerNow suppose there is a second receiver, R2. R2 is attached to a third BD, BD3. However, it is attached to a segment of BD3 that is attached to PE1. And suppose the tenant multicast router is attached to a segment of BD3 that attaches to PE2. In this case, the tenant multicast router will make two copies of the packet, one for BD2 and one for BD3. PE2 will send both copies back to PE1. Not only is the routing sub-optimal, but PE2 also sends multiple copies of the same packet to PE1, which is a further sub-optimality 🙀
This is only an example; many more examples of sub-optimal multicast routing can easily be given. To eliminate sub-optimal routing and extra copies, we need a multicast solution that is EVPN-aware and that can use its knowledge of the internal structure of a Tenant Domain to make sure multicast traffic gets routed optimally 🧠 The procedures in this document let us avoid all such sub-optimalities when routing inter-subnet multicast traffic within a Tenant Domain ✨
1.4. Additional Requirements That Must Be Met by the Solution ✨
Okay so, besides routing multicast flows the most optimal way inside a Tenant Domain 🦋, the EVPN-aware multicast solution also has to hit a bunch of extra requirements. Let's go, bestie 💅
- The solution must integrate well with the procedures specified in [RFC9251]. Like, an integrated set of procedures has to handle both intra-subnet multicast and inter-subnet multicast 🫶
- Okay, intra-subnet multicast is where things get serious 😭 The solution MUST maintain the integrity of the multicast Ethernet service. Here's what that looks like:
- If a source and a receiver are on the same subnet, the MAC Source Address (SA) of the multicast frame sent by the source will not get rewritten. Hands off the MAC SA 🙈
- If a source and a receiver are on the same subnet, no IP processing of the Ethernet payload is done. The IP TTL is not decremented, the IPv4 header checksum is not changed, no fragmentation is done, etc. Literally zero IP fiddling 💖
- On the other hand, if a source and a receiver are on different subnets, the frame received by the receiver will not have the MAC Source Address of the source, because the frame will appear to have come from a multicast router 🧚 Also, proper processing of the IP header is done, e.g., TTL decrements by 1, header checksum modification, possible fragmentation, etc. It's giving router energy 👑
- If a Tenant Domain contains several BDs, it MUST be possible for a multicast flow (even when the multicast group address is an ASM address) to have sources in one of those BDs and receivers in one or more of the other BDs without requiring the presence of any system performing PIM RP functions [RFC7761]. No RP needed, periodt 🌈
- Sometimes a MAC address used by one TS on a particular BD is also used by another TS on a different BD. Inter-subnet routing of multicast traffic MUST NOT make any assumptions about the uniqueness of a MAC address across several BDs. Don't assume MAC addresses are unique across BDs, no cap 🌸
- If two EVPN PEs attached to the same Tenant Domain both support the OISM procedures, each may receive inter-subnet multicasts from the other, even if the egress PE is not attached to any segment of the BD from which the multicast packets are being sourced. It MUST NOT be necessary to provision the egress PE with knowledge of the ingress BD. Lowkey the egress PE doesn't need to know the ingress BD at all ✨
- There must be a procedure that allows EVPN PE routers supporting OISM procedures to send/receive multicast traffic to/from EVPN PE routers that support only [RFC7432] but that does not support the OISM procedures or even the procedures of [RFC9135]. However, when interworking with such routers (which we call "non-OISM PE routers"), optimal routing may not be achievable. Interworking is a whole vibe, but optimal routing might not be possible with these non-OISM PE routers 🎀
- It MUST be possible to support scenarios in which multicast flows with sources inside a Tenant Domain have external receivers, i.e., receivers that are outside the domain. It must also be possible to support scenarios where multicast flows with external sources (sources outside the Tenant Domain) have receivers inside the domain. Both directions, iykyk 🦋
This presupposes that unicast routes to multicast sources outside the domain can be distributed to EVPN PEs attached to the domain and that unicast routes to multicast sources within the domain can be distributed outside the domain.
Of particular importance are the scenarios in which the external sources and/or receivers are reachable via L3VPN/MVPN or via IP/PIM. These are the big ones, bestie 👑
The solution for external interworking MUST allow for deployment scenarios in which EVPN does not need to export a host route for every multicast source. No need to export a host route for every single source, that's so much less clutter 🍓
- The solution for external interworking must not presuppose that the same tunneling technology is used within both the EVPN domain and the external domain. For example, MVPN interworking must be possible when MVPN is using MPLS Point-to-Multipoint (P2MP) tunneling and when EVPN is using Ingress Replication (IR) or Virtual eXtensible Local Area Network (VXLAN) tunneling. Different tunnel tech on each side is totally fine, no drama 🌈
- The solution must not be overly dependent on the details of a small set of use cases but must be adaptable to new use cases as they arise. (That is, the solution must be robust.) It has to be flexible enough to handle whatever new use cases show up, slay 💅
1.5. Model of Operation: Overview ⭐
1.5.1. Control Plane 🧚
In this section, and in the remainder of this document, we assume the reader is familiar with the procedures of IGMP / Multicast Listener Discovery (MLD) (see [RFC3376] and [RFC3810]), by which hosts announce their interest in receiving particular multicast flows. So you already know how hosts tell the network "hey, I want this flow" 💜
Consider a Tenant Domain consisting of a set of k BDs: BD1, ..., BDk. To support the OISM procedures, each Tenant Domain must also be associated with a Supplementary Broadcast Domain (SBD). An SBD is treated in the control plane as a real BD, but it does not have any ACs. The SBD has several uses; these will be described later in this document (see Sections 2.1 and 3). Think of the SBD as the secret helper BD 🤫✨
Each PE that attaches to one or more of the BDs in a given Tenant Domain will be provisioned to recognize that those BDs are part of the same Tenant Domain. Note that a given PE does not need to be configured with all the BDs of a given Tenant Domain. In general, a PE will only be attached to a subset of the BDs in a given Tenant Domain and will be configured only with that subset of BDs. However, each PE attached to a given Tenant Domain must be configured with the SBD for that Tenant Domain. So a PE doesn't need every BD, but it does need the SBD, no exceptions 🎀
Suppose a particular segment of a particular BD is attached to PE1. [RFC7432] specifies that PE1 must originate an Inclusive Multicast Ethernet Tag (IMET) route for that BD and that the IMET route must be propagated to all other PEs attached to the same BD. If the given segment contains a host that has interest in receiving a particular multicast flow, either an (S,G) flow or a (*,G) flow, PE1 will learn of that interest by participating in the IGMP/MLD snooping procedures, as specified in [RFC4541]. In this case:
- PE1 is interested in receiving the flow;
- the AC attaching the interested host to PE1 is also said to be interested in the flow; and
- the BD containing an AC that is interested in a particular flow is also said to be interested in that flow.
Once PE1 determines that it has an AC that is interested in receiving a particular flow or set of flows, it originates one or more Selective Multicast Ethernet Tag (SMET) routes [RFC9251] to advertise that interest. PE1 basically raises its hand and says "I want this!" 🙋♀️💖
Note that each IMET or SMET route is for a particular BD. The notion of a route being for a particular BD is explained in Section 2.2. Keep that in mind, it matters 🦋
When OISM is being supported, the procedures of [RFC9251] are modified as follows:
- The IMET route originated by a particular PE for a particular BD is distributed to all other PEs attached to the Tenant Domain containing that BD, even to those PEs that are not attached to that particular BD. Ooh, the IMET route goes Tenant-Domain-wide, not just BD-wide 🌸
- The SMET routes originated by a particular PE are originated on a per-Tenant-Domain basis rather than a per-BD basis. That is, the SMET routes are considered to be for the Tenant Domain's SBD rather than any of its ordinary BDs. These SMET routes are distributed to all the PEs attached to the Tenant Domain.
In this way, each PE attached to a given Tenant Domain learns, from the other PEs attached to the same Tenant Domain, the set of flows that are of interest to each of those other PEs. So everybody gets the gossip on who wants what 💅👑
An OISM PE that is provisioned with several BDs in the same Tenant Domain MUST originate an IMET route for each such BD. To indicate its support of [RFC9251], it SHOULD attach the EVPN Multicast Flags Extended Community to each such IMET route, but it MUST attach the EC to at least one such IMET route. Every BD gets its own IMET, and the EC goes on at least one of them, so the flag is definitely there 🧚
Suppose PE1 is provisioned with both BD1 and BD2 and considers them to be part of the same Tenant Domain. It is possible that PE1 will receive both an IMET route for BD1 and an IMET route for BD2 from PE2. If either of these IMET routes has the EVPN Multicast Flags Extended Community, PE1 MUST assume that PE2 is supporting the procedures of [RFC9251] for ALL BDs in the Tenant Domain. One EC is enough to vouch for PE2 across the whole Tenant Domain, slay 🌈
If a PE supports OISM functionality, it indicates that, by setting the OISM-supported flag in the Multicast Flags Extended Community, it attaches to some or all of its IMET routes. An OISM PE SHOULD attach this EC with the OISM-supported flag set to all the IMET routes it originates. However, if PE1 imports IMET routes from PE2, and at least one of PE2's IMET routes indicates that PE2 is an OISM PE, PE1 MUST assume that PE2 is following OISM procedures. Basically, one OISM-flagged IMET is all PE1 needs to believe PE2 is doing the OISM thing 🍓✨
1.5.2. Data Plane 💅
Okay so, picture this 🎀 PE1 has an AC to a segment in BD1, and PE1 gets an (S,G) multicast frame from that AC (as defined in Section 1.1). Main character moment incoming ✨
There may be other ACs of PE1 where TSs have shown interest (via IGMP/MLD) in getting (S,G) multicast packets. PE1 is in charge of sending the received multicast packet out on those ACs. There are two cases to think about 🦋
- Intra-Subnet Forwarding: Here, an AC that's interested in (S,G) is connected to a segment that's part of the source BD, BD1. If the segment isn't multihomed, or PE1 is the Designated Forwarder (DF) (see [RFC7432]) for that segment, PE1 sends the multicast frame out on that AC without changing the MAC SA. The IP header isn't touched at all. Literally, the TTL doesn't get decremented 🙈
- Inter-Subnet Forwarding: Here, an AC that's interested in (S,G) is connected to a segment of BD2, where BD2 is different than BD1. If PE1 is the DF for that segment (or the segment isn't multihomed), PE1 decapsulates the IP multicast packet, does any necessary IP processing (including the TTL decrement), and then re-encapsulates the packet appropriately for BD2. PE1 then sends the packet out on the AC. Note that after re-encapsulation, the MAC SA will be PE1's MAC address on BD2. The IP TTL will have been decremented by 1 💜
Now, there may also be other PEs that are interested in (S,G) traffic 👀 Let's say PE2 is one of them. Then PE1 tunnels a copy of the IP multicast frame to PE2, keeping its original MAC SA and not altering the payload's IP header at all. The tunnel encapsulation holds info that PE2 can use to link the frame with an apparent source BD. If the frame's actual source BD is BD1, then:
- If PE2 is attached to BD1, the tunnel encapsulation used to send the frame to PE2 will make PE2 identify BD1 as the apparent source BD.
- If PE2 is not attached to BD1, the tunnel encapsulation used to send the frame to PE2 will make PE2 identify the SBD as the apparent source BD.
Quick tea ☕ The tunnel encapsulation used for a particular BD will have been advertised in an IMET route or a Selective Provider Multicast Service Interface (S-PMSI) route [RFC9572] for that BD. That route carries a PMSI Tunnel Attribute (PTA), which says how packets coming from that BD get encapsulated. This is the info that lets the receiving PE figure out the apparent source BD, as stated above. See Section 3.2 for more details 🧚
When PE2 receives the tunneled frame, it forwards it on any of its ACs that are interested in (S,G) 🌸
If PE2 figures out from the tunnel encapsulation that the apparent source BD is BD1, then:
- For those ACs that connect PE2 to BD1, the intra-subnet forwarding procedure described above is used, except now it's PE2, not PE1, doing that procedure. Unmodified EVPN procedures from [RFC7432] make sure a packet that came from a multihomed segment is never sent back to that segment 🙅♀️
- For those ACs that don't connect to BD1, the inter-subnet forwarding procedure described above is used, except now it's PE2, not PE1, carrying it out.
If the tunnel encapsulation identifies the apparent source BD as the SBD, PE2 applies the inter-subnet forwarding procedures described above to all of its ACs that are interested in the flow 🍓
These procedures make sure an IP multicast frame travels from its ingress PE to every egress PE that wants to receive it. While it's in transit, the frame keeps its original MAC SA, and the payload keeps its original IP header. Note that in every case, when an IP multicast packet gets sent from one BD to another, these procedures cause its TTL to be decremented by 1 🌈
So far, we've been assuming that an IP multicast packet arrives at its ingress PE over an AC that belongs to one of the BDs in a given Tenant Domain. But a packet can also reach its ingress PE in other ways 😭 Since an EVPN PE supporting IRB has an IP-VRF, the IP-VRF might have a VRF interface that isn't an IRB interface. For example, there might be a VRF interface that's actually a physical link to an external Ethernet switch, a directly attached host, or a router. When an EVPN PE, say PE1, gets a packet through things like that, we'll say the packet has an external source (i.e., a source outside the Tenant Domain). There are also other scenarios where a multicast packet might have an external source, e.g., it might arrive over an MVPN tunnel from an L3VPN PE. In those cases, we'll still call PE1 the "ingress EVPN PE" ✨
When an EVPN PE, say PE1, receives an externally sourced multicast packet and there are receivers for it inside the Tenant Domain, it does the following:
- Suppose PE1 has an AC in BD1 that's interested in (S,G). Then PE1 encapsulates the packet for BD1, filling in the MAC SA field with PE1's own MAC address on BD1. It sends the resulting frame out on the AC 💖
- Suppose some other EVPN PE, say PE2, is interested in (S,G). PE1 encapsulates the packet for Ethernet, filling in the MAC SA field with PE1's own MAC address on the SBD. PE1 then tunnels the packet to PE2. The tunnel encapsulation will identify the apparent source BD as the SBD. Since the apparent source BD is the SBD, PE2 will know to treat the frame as an inter-subnet multicast 🧠
When IR is used to send IP multicast frames from an ingress EVPN PE to a set of egress PEs, the ingress PE has to send multiple copies of the frame. Each copy is the original Ethernet frame. Decapsulation and IP processing only happen at the egress PE 🎀
If a P2MP tree or Bit Index Explicit Replication (BIER) [RFC9624] is used to send an IP multicast frame from an ingress PE to a set of egress PEs, then the ingress PE only has to send one copy of the frame to each of its next hops. Again, each egress PE gets the original frame and does any necessary IP processing 🦋
2. Detailed Model of Operation 🎀
The model described in Section 1.5.2 can be expressed more precisely using the notion of IRB interface (see Appendix A). For a given Tenant Domain:
- A given PE has one IRB interface for each BD it's attached to. This IRB interface connects L3 routing to that BD. When IP multicast packets are sent or received on the IRB interfaces, the semantics of the interface are modified from the semantics described in Appendix A. See Section 2.3 for the details of the modification.
- Each PE also has an IRB interface that connects L3 routing to the SBD. The semantics of this interface are different than the semantics of the IRB interface to the real BDs. See Section 2.3.
In this section, we assume that PIM is not enabled on the IRB interfaces. In general, it's not necessary to enable PIM on the IRB interfaces unless there are PIM routers on one of the Tenant Domain's BDs, or there's some other scenario requiring a Tenant Domain's L3 routing instance to become a PIM adjacency of some other system. These cases will be discussed in Section 7 🌸
2.1. Supplementary Broadcast Domain 🌈
Suppose a given Tenant Domain has three BDs (BD1, BD2, and BD3) and two PEs (PE1 and PE2). PE1 attaches to BD1 and BD2, while PE2 attaches to BD2 and BD3 💅
To carry out the procedures described above, all the PEs attached to the Tenant Domain must be provisioned with the SBD for that Tenant Domain. An RT must be associated with the SBD and provisioned on each of those PEs. We'll refer to that RT as the "SBD-RT" 🫶
A Tenant Domain is also configured with an IP-VRF [RFC9135], and the IP-VRF is associated with an RT. This RT MAY be the same as the SBD-RT 🍓
Suppose an (S,G) multicast frame originating on BD1 has a receiver on BD3. PE1 will transmit the packet to PE2 as a frame, and the encapsulation will identify the frame's source BD as BD1. Since PE2 isn't provisioned with BD1, it will treat the packet as if its source BD were the SBD. That means a packet can be transmitted from BD1 to BD3 even though its ingress PE isn't configured for BD3 and/or its egress PE isn't configured for BD1 🧚
EVPN supports service models in which a given EVI can contain only one BD. It also supports service models in which a given EVI can contain multiple BDs. No matter which service model is being used for a particular tenant, it is highly RECOMMENDED that an EVI containing only the SBD be provisioned for that tenant 👑
If, for some reason, it's not feasible to provision an EVI that contains only the SBD, it's possible to put the SBD in an EVI that contains other BDs. However, in that case, the SBD-RT MUST be different than the RT associated with any other BD. Otherwise, the procedures of this document (as detailed in Sections 2.2 and 3.1) will not produce correct results 😭
2.2. Detecting When a Route is for/from a Particular BD
okay so bestie, this section is literally about how a PE figures out which BD a route belongs to 💅✨ it's giving detective energy, no cap. 🕵️♀️
In this document, we frequently say that a particular multicast route is "from" or "for" a particular BD, or is "related to" or "associated with" a particular BD. These terms are used interchangeably 🦋. Later sections explain when various routes must be originated for particular BDs. Here, we explain how the PE originating a route marks it to show which BD it's for 🎀. We also explain how a PE receiving the route works out which BD the route is for 🧚.
In EVPN, each BD is assigned an RT. An RT is a BGP Extended Community that can be attached to the BGP routes used by the EVPN control plane 🌸. In some EVPN service models, each BD gets its own unique RT. In other service models, a set of BDs (all in the same EVI) may share the same RT 💜. The RT assigned to the SBD is called the "SBD-RT" 👑.
In those service models that let a set of BDs share a single RT, each BD is assigned a non-zero Tag ID 🍓. The Tag ID shows up in the Network Layer Reachability Information (NLRI) of many of the BGP routes used by the EVPN control plane 🫶.
A given route may be for the SBD or for an ordinary BD (a BD that is not the SBD). An RT assigned to an ordinary BD is known as an "ordinary BD-RT" ✨.
When constructing an IMET, SMET, S-PMSI, or Leaf [RFC9572] route for a given BD, the following rules apply 🌈:
- If the route is for an ordinary BD, say BD1, then:
- the route MUST carry the ordinary BD-RT associated with BD1, and
- the route MUST NOT carry any RT that is associated with an ordinary BD other than BD1.
- If the route is for the SBD, the route MUST carry the SBD-RT and MUST NOT carry any RT that is associated with any other BD.
- As detailed in subsequent sections, under certain circumstances, a route that is for BD1 may carry both the RT of BD1 and also the SBD-RT 🔥.
The IMET route for the SBD MUST carry a Multicast Flags Extended Community in which an OISM SBD flag is set 💖.
The IMET route for a BD other than the SBD SHOULD carry an EVI-RT EC as defined in [RFC9251] 🦋. The EC is constructed from the SBD-RT to indicate the BD's corresponding SBD. This lets all PEs check that their SBD provisioning is consistent, and lets an Assisted Replication (AR) replicator automatically work out a BD's corresponding SBD without any provisioning, as explained in Section 3.2.3.1 🙌.
When receiving an IMET, SMET, S-PMSI, or Leaf route, the receiving PE has to figure out which BD the route belongs to 🔍. It does this by looking at the RTs the route carries, as well as the Tag ID field in the route's NLRI. There are several cases to consider 🎯. Some of these are error cases that show up when the route wasn't built properly 😭.
When one of the error cases is detected, the route MUST be regarded as a malformed route, and the treat-as-withdraw procedure of [RFC7606] MUST be applied 💀. Note that these error cases can only be detected by EVPN procedures at the receiving PE. BGP procedures at intermediate nodes will generally not notice such error cases, and in general they SHOULD NOT try to 🙈.
Heads up bestie: the error cases below are the "this route is cursed" ones. Treat-as-withdraw kicks in for all of them. 🚫
Case 1: The receiving PE recognizes more than one of the route's RTs as an SBD-RT (i.e., the route carries SBD-RTs of more than one Tenant Domain).
This is an error case; the route has not been properly constructed 💀.
Case 2: The receiving PE recognizes one of the route's RTs as associated with an ordinary BD and recognizes another of the route's RTs as associated with a different ordinary BD.
This is an error case; the route has not been properly constructed 💀.
Case 3: The receiving PE recognizes one of the route's RTs as associated with an ordinary BD in a particular Tenant Domain and recognizes another of the route's RTs as associated with the SBD of a different Tenant Domain.
This is an error case; the route has not been properly constructed 💀.
Case 4: The receiving PE does not recognize any of the route's RTs as associated with an ordinary BD in any of its Tenant Domains, but does recognize one of the RTs as the SBD-RT of one of its Tenant Domains.
In this case, the receiving PE associates the route with the SBD of that Tenant Domain 🌟. This association is made even if the Tag ID field of the route's NLRI is not the Tag ID of the SBD 🌷.
This is a normal use case where either (a) the route is for a BD to which the receiving PE is not attached, or (b) the route is for the SBD. In either case, the receiving PE associates the route with the SBD 🧁.
Case 5: The receiving PE recognizes exactly one of the RTs as an ordinary BD-RT associated with one of the PE's EVIs, say EVI-1. The receiving PE also recognizes one of the RTs as being the SBD-RT of the Tenant Domain containing EVI-1.
In this case, the route is associated with the BD in EVI-1 that is identified (in the context of EVI-1) by the Tag ID field of the route's NLRI. (If EVI-1 contains only a single BD, the Tag ID is likely to be zero 🍼.)
This is the case where the route is for a BD to which the receiving PE is attached, but the route also carries the SBD-RT. In this case, the receiving PE associates the route with the ordinary BD, not with the SBD 🎀.
Note that according to the above rules, the mapping from BD to RT is many-to-one or one-to-one 💅. A route that an EVPN PE originates for a particular BD carries that BD's RT, and an EVPN PE that receives the route associates it with a BD as described above. However, RTs aren't only used to help identify the BD a route belongs to. BGP may also use them to decide the path along which the route is distributed and which PEs receive it 🦋. There may be cases where you want to originate a route for a particular BD but have it distributed to only some of the EVPN PEs attached to that BD 🙈. Or you might want the route distributed to some intermediate set of systems, where it might be modified or replaced before being propagated further 🧚. Such situations are outside the scope of this document 🌸.
There may also be situations where it's desirable to exchange routes among two or more different Tenant Domains (EVPN Extranet) 🌈. Such situations are also outside the scope of this document 💜.
2.3. Use of IRB Interfaces at Ingress PE
okay bestie, this part is all about what happens to a multicast packet when it lands at the ingress PE and gets passed up to the IRB interfaces 🎀✨.
When an (S,G) multicast frame is received from an AC belonging to a particular BD, say BD1:
- The frame is sent unchanged to other EVPN PEs that are interested in (S,G) traffic 🦋. The encapsulation used to send the frame to those PEs depends on the tunnel type being used for multicast transmission. (For our purposes, we consider IR, AR, and BIER to be tunnel types, even though IR, AR, and BIER don't actually use P2MP tunnels 🧚.) At the egress PE, the apparent source BD of the frame can be inferred from the tunnel encapsulation. If the egress PE is not attached to the actual source BD, it will infer that the apparent source BD is the SBD 💖.
Note that the inter-PE transmission of a multicast frame among EVPN PEs of the same Tenant Domain does NOT involve the IRB interfaces, as long as the multicast frame was received over an AC attached to one of the Tenant Domain's BDs 🌸.
- The frame is also sent up the IRB interface that attaches BD1 to the Tenant Domain's L3 routing instance in this PE 🌟. That is, the L3 routing instance, behaving as if it were a multicast router, receives the IP multicast frames that arrive at the PE from its local ACs. The L3 routing instance decapsulates the frame's payload to extract the IP multicast packet, decrements the IP TTL, adjusts the header checksum, and does any other necessary IP processing (e.g., fragmentation) 🧁.
- The L3 routing instance keeps track of which BDs have local receivers for (S,G) traffic. (A local receiver is a TS, reachable via a local AC, that has expressed interest in (S,G) traffic 🫶.) If the L3 routing instance has an IRB interface to BD2, and it knows that BD2 has a LOCAL receiver interested in (S,G) traffic, it encapsulates the packet in an Ethernet header for BD2, putting its own MAC address in the MAC SA field. Then, it sends the packet down the IRB interface to BD2 🍓.
If a packet is sent from the L3 routing instance to a particular BD via the IRB interface (step 3 in the above list), and if the BD in question is NOT the SBD, the packet is sent ONLY to LOCAL ACs of that BD 🙌. If the packet needs to go to other PEs, it has already been sent to them in step 1 👑. Note that this is a change in the IRB interface semantics from what is described in [RFC9135] and Figure 3 ✨.
If a given locally attached segment is multihomed, existing EVPN procedures ensure that a packet is not sent by a given PE to that segment unless the PE is the DF for that segment 🌈. Those procedures also ensure that a packet is never sent by a PE to its segment of origin. Thus, EVPN segment multihoming is fully supported, and duplicate delivery to a segment or looping on a segment are prevented without the need for any new procedures to be defined in this document 💅.
What if an IP multicast packet is received from outside the Tenant Domain? 🤔 For instance, perhaps PE1's IP-VRF for a particular Tenant Domain also has a physical interface leading to an external switch, host, or router, and PE1 receives an IP multicast packet or frame on that interface. Or perhaps the packet is from an L3VPN or a different EVPN Tenant Domain 🌍.
Such a packet is first processed by the L3 routing instance, which decrements TTL and does any other necessary IP processing. Then, the packet is sent into the Tenant Domain by sending it down the IRB interface to the SBD of that Tenant Domain. This requires encapsulating the packet in an Ethernet header. The MAC SA field will contain the PE's own MAC on the SBD 🧚.
An IP multicast packet sent by the L3 routing instance down the IRB interface to the SBD is treated as if it had arrived from a local AC, and steps 1-3 are applied. Note that the semantics of sending a packet down the IRB interface to the SBD are thus slightly different from the semantics of sending a packet down other IRB interfaces. IP multicast packets sent down the SBD's IRB interface may be distributed to other PEs, but IP multicast packets sent down other IRB interfaces are distributed only to local ACs 🎀.
If a PE sends a link-local multicast packet down the SBD IRB interface, that packet will be distributed (as an Ethernet frame) to other PEs of the Tenant Domain, but will not appear on any of the actual BDs 🌷.
2.4. Use of IRB Interfaces at an Egress PE 🎀
Okay, so picture this: an egress EVPN PE gets an (S,G) multicast frame from the frame's ingress EVPN PE. Like we covered above, the packet shows up as an Ethernet frame through a tunnel from the ingress PE, and the tunnel encapsulation tells you which source BD the Ethernet frame came from. 🌸
We define the frame's apparent source BD like this 💅: if the egress PE is attached to the actual source BD, then the actual source BD is the apparent source BD. If the egress PE is not attached to the actual source BD, the SBD is the apparent source BD.
The egress PE then does the following steps ✨
- If the egress PE has ACs belonging to the apparent source BD of the frame, it sends the frame unchanged to any ACs of that BD that have interest in (S,G) packets. The MAC SA of the frame is not modified, and the IP header of the frame's payload is not modified in any way. 🫶
- The frame is also sent to the L3 routing instance by being sent up the IRB interface that attaches the L3 routing instance to the apparent source BD. Steps 2 and 3 listed in Section 2.3 are then applied. 🧚
2.5. Announcing Interest in (S,G) 🦋
[RFC9251] defines the procedures an egress PE uses to announce its interest in a multicast flow or a set of flows. So if an egress PE figures out it has LOCAL receivers in a particular BD, say BD1, that want a particular set of flows, it originates one or more SMET routes for BD1. Each SMET route specifies a particular (S,G) or (,G) flow. By originating a SMET route for BD1, the PE is basically saying "I have receivers for (S,G) or (,G) in BD1" 📣. That SMET route carries the RT for BD1, which makes sure it gets distributed to all PEs attached to BD1.
The OISM procedures for originating SMET routes are a little different from those in [RFC9251]. In most cases, the SMET routes are considered to be for the SBD rather than the BD containing local receivers. These SMET routes carry the SBD-RT and do not carry any ordinary BD-RT. Details on processing SMET routes are in Section 3.3. 💜
Since the SMET routes carry the SBD-RT, every ingress PE attached to a particular Tenant Domain will learn about all the other PEs (attached to the same Tenant Domain) that have interest in a particular set of flows. Note that a PE that receives a given SMET route does not necessarily have any BDs (other than the SBD) in common with the PE that originated it. Main character energy, no BD required 👑
If all the sources and receivers for a given (*,G) are in the Tenant Domain, inter-subnet ASM traffic gets routed properly without needing any RPs, shared trees, or other complicated multicast routing infrastructure. 🌈 Say, for example, that:
- PE1 has a local receiver, on BD1, for (*,G), and
- PE2 has a local source, on BD2, for (*,G).
PE1 will originate a SMET(*,G) route for the SBD, and PE2 will receive that route, even if PE2 is not attached to BD1. PE2 will thus know to forward (S,G) traffic to PE1. PE1 does not need to do any source discovery. (This does assume that source S does not send the same (S,G) datagram on two different BDs and that the Tenant Domain does not contain two or more sources with the same IP address S. The use of multicast sources that have IP anycast addresses is outside the scope of this document. 🙈)
If some PE attached to the Tenant Domain does not support [RFC9251], it will be assumed to be interested in all flows. Whether a particular remote PE supports [RFC9251] or not is determined by the presence of the Multicast Flags Extended Community in its IMET route; this is specified in [RFC9251]. 🍓
2.6. Tunneling Frames from Ingress PEs to Egress PEs 🌊
[RFC7432] specifies the procedures for setting up and using BUM tunnels. A BUM tunnel is a tunnel used to carry traffic on a particular BD if that traffic is (a) broadcast traffic, (b) unicast traffic with an unknown Destination MAC Address, or (c) Ethernet multicast traffic. 💖
This document lets BUM tunnels be used as the default tunnels for transmitting IP multicast frames. It also lets a separate set of tunnels, instead of the BUM tunnels, be used as the default tunnels for carrying IP multicast frames. Let's call these "IP multicast tunnels" 🫧.
When the tunneling is done via IR or via BIER, this difference doesn't matter at all. However, when P2MP tunnels are used, there's a big advantage to having separate IP multicast tunnels. 🎀
It's desirable for an ingress PE to transmit a copy of a given (S,G) multicast frame on only one P2MP tunnel, so all the egress PEs interested in (S,G) packets have to join that tunnel. If the source BD and PE for an (S,G) frame are BD1 and PE1, respectively, and PE2 has receivers on BD2 for (S,G), then PE2 must join the P2MP Label Switched Path (LSP) on which PE1 transmits the (S,G) frame. PE2 must join this P2MP LSP even if PE2 is not attached to the source BD, BD1. If PE1 were transmitting the multicast frame on its BD1 BUM tunnel, then PE2 would have to join the BD1 BUM tunnel, even though PE2 has no BD1 Attachment Circuits. That would make PE2 pull all the BUM traffic from BD1, most of which it would just have to discard 😩. So it is RECOMMENDED that the default IP multicast tunnels be distinct from the BUM tunnels. ✨
Now, the big exception 🚨: link-local IP multicast traffic MUST always be carried on the BUM tunnels and ONLY on the BUM tunnels. Link-local IP multicast traffic consists of IPv4 traffic with a destination address prefix of 224/24 and IPv6 traffic with a destination address prefix of FF02/16. In this document, the terms "IP multicast packet" and "IP multicast frame" are defined in Section 1.1 so as to exclude link-local traffic. 🙅♀️
Also, it's possible to use selective tunnels to carry particular multicast flows (see Section 3.2). When an (S,G) frame is transmitted on a selective tunnel, it is not transmitted on the BUM tunnel or on the default IP multicast tunnel. 🦋
2.7. Advanced Scenarios 🌟
There are some deployment scenarios that need special procedures:
- Some multicast sources or receivers are attached to PEs that support [RFC7432] but do not support this document or [RFC9135]. To interoperate with these non-OISM PEs, it's necessary to have one or more gateway PEs that interface the tunnels discussed in this document with the BUM tunnels of the legacy PEs. This is discussed in Section 5. 🤝
- Sometimes multicast traffic originates from outside the EVPN domain or needs to be sent outside the EVPN domain. This is discussed in Section 6. An important special case of this, integration with MVPN, is discussed in Section 6.1.2. 🌍
- In some scenarios, one or more of the tenant systems is a PIM router, and the Tenant Domain is used as a transit network that is part of a larger multicast domain. This is discussed in Section 7. 🧩
3. EVPN-Aware Multicast Solution Control Plane 👑
3.1. Supplementary Broadcast Domain (SBD) and Route Targets 💅
As discussed in Section 2.1, every Tenant Domain is associated with a single SBD. Remember that a Tenant Domain is defined to be a set of BDs that can freely send and receive IP multicast traffic to/from each other. If an EVPN PE has one or more ACs in a BD of a particular Tenant Domain, and if the EVPN PE supports the procedures of this document, that EVPN PE MUST be provisioned with the SBD of that Tenant Domain. 🫶
At each EVPN PE attached to a given Tenant Domain, there is an IRB interface leading from the L3 routing instance of that Tenant Domain to the SBD. However, the SBD has no ACs. 🌸
Each SBD is provisioned with an RT. All the EVPN PEs supporting a given SBD are provisioned with that RT as an import RT. That RT MUST NOT be the same as the RT associated with any other BD. 🔒
We'll use the term "SBD-RT" to mean the RT that has been assigned to the SBD. Routes carrying this RT will be propagated to all EVPN PEs in the same Tenant Domain as the originator. 📬
Section 2.2 specifies the rules by which an EVPN PE that receives a route determines whether a received route belongs to a particular ordinary BD or to the SBD. Section 2.2 also specifies additional rules that must be followed when constructing routes that belong to a particular BD, including the SBD. 🧠
The SBD SHOULD be in an EVI of its own. Even if the SBD is not in an EVI of its own, the SBD-RT MUST be different than the RT associated with any other BD. This restriction is necessary in order for the rules of Sections 2.2 and 3.1 to work correctly. 🎯
Note that an SBD, just like any other BD, is associated on each EVPN PE with a MAC-VRF. Per [RFC7432], each MAC-VRF is associated with a Route Distinguisher (RD). When constructing a route that is for an SBD, an EVPN PE will place the RD of the associated MAC-VRF in the Route Distinguisher field of the NLRI. (If the Tenant Domain has several MAC-VRFs on a given PE, the EVPN PE has a choice of which RD to use.) 🍰
If AR [RFC9574] is used, each AR-REPLICATOR for a given Tenant Domain must be provisioned with the SBD of that Tenant Domain, even if the AR-REPLICATOR does not have any L3 routing instances. 🔁
3.2. Advertising the Tunnels Used for IP Multicast 🚇
The procedures for advertising the tunnels that carry IP multicast traffic depend on the type of tunnel being used. If the tunnel type is neither IR, AR, nor BIER, there are procedures for advertising both inclusive tunnels and selective tunnels. 🌈
When IR, AR, or BIER is used to transmit IP multicast packets across the core, there are no P2MP tunnels. Once an ingress EVPN PE determines the set of egress EVPN PEs for a given flow, the IMET routes contain all the information needed to transport packets of that flow to the egress PEs. 📦
If AR is used, the ingress EVPN PE is also an AR-LEAF, and the IMET route coming from the selected AR-REPLICATOR contains the information needed. The AR-REPLICATOR will behave as an ingress EVPN PE when sending a flow to the egress EVPN PEs. 🙌
If the tunneling technique requires P2MP tunnels to be set up (e.g., RSVP-TE P2MP, Multipoint LDP (mLDP), or PIM), some of the tunnels may be selective tunnels and some may be inclusive tunnels. 🌳
Selective P2MP tunnels are always advertised by the ingress PE using S-PMSI Auto-Discovery (A-D) routes [RFC9572]. 📣
For inclusive tunnels, there's a choice between using a BD's ordinary BUM tunnel as the default inclusive tunnel for carrying IP multicast traffic, or using a separate IP multicast tunnel as the default inclusive tunnel for carrying IP multicast. In the former case, the inclusive tunnel is advertised in an IMET route. In the latter case, the inclusive tunnel is advertised in a (C-,C-) S-PMSI A-D route [RFC9572]. Details may be found in subsequent sections. 💫
3.2.1. Constructing Routes for the SBD ✨
Okay so there are times when an EVPN PE has to originate IMET, SMET, and/or S-PMSI routes for the SBD 🫶. We're gonna call these SBD-IMET routes, SBD-SMET routes, and SBD-SPMSI routes all through this doc, which is honestly giving clarity 💅. Later sections spell out exactly when each one needs to get sent out.
When an EVPN PE needs to originate an SBD-IMET, SBD-SMET, or SBD-SPMSI route, it builds it like this:
- The RD field of the route's NLRI gets set to the RD of the MAC-VRF that's associated with the SBD.
- The SBD-RT gets attached to the route.
- The Tag ID field of the route's NLRI gets set to the Tag ID assigned to the SBD. This is most likely 0 if a VLAN-based or VLAN-bundle service is being used, but it's non-zero if a VLAN-aware bundle service is being used 🎀
3.2.2. Ingress Replication 🦋
When IR is used to transport IP multicast frames of a given Tenant Domain, each EVPN PE attached to that Tenant Domain MUST originate an SBD-IMET route (see Section 3.2.1). No skipping, periodt 💀
The SBD-IMET route MUST carry a PTA, and the MPLS Label field of the PTA MUST specify a downstream-assigned MPLS label. That label has to map uniquely (in the context of the originating EVPN PE) to the SBD 🔥
Following the procedures of [RFC7432], an EVPN PE MUST also originate an IMET route for each BD it's attached to. Each of these IMET routes carries a PTA that specifies a downstream-assigned label, and that label maps uniquely to the BD in question (again, in the context of the originating EVPN PE) ✨ These IMET routes don't need to carry the SBD-RT, so they can chill without it 🌸
When an ingress EVPN PE needs to use IR to send an IP multicast frame from a particular source BD to an egress EVPN PE, the ingress PE checks whether or not the egress PE has originated an IMET route for that BD. If it has, that IMET route holds the MPLS label the egress PE assigned to the source BD, and the ingress PE uses that label when it transmits the packet. If it hasn't, the ingress PE uses the label the egress PE assigned to the SBD instead (the one in the SBD-IMET route that the egress originated) 💜
Here's the tea: the set of IMET routes a given egress PE originates, and that a given ingress PE installs, can change over time 👑 If the egress PE withdraws its IMET route for the source BD, the ingress PE MUST stop using the label from that IMET route and MUST switch to the label carried in the SBD-IMET route from that same egress PE. Implementors must also take into account that an IMET route from a particular PE for a particular BD might show up after that PE's SBD-IMET route. Plan for it, bestie 🙈
3.2.3. Assisted Replication 🧚
When AR is used to transport IP multicast frames of a given Tenant Domain, each EVPN PE (including the AR-REPLICATOR) attached to that Tenant Domain MUST originate an SBD-IMET route (see Section 3.2.1).
An AR-REPLICATOR attached to a given Tenant Domain counts as an EVPN PE of that Tenant Domain. It's attached to all the BDs in the Tenant Domain, but it doesn't necessarily have L3 routing instances. Main character energy, but make it low-key 🌈
Just like with IR, the SBD-IMET route carries a PTA where the MPLS Label field specifies the downstream-assigned MPLS label that identifies the SBD. The difference is that the AR-REPLICATOR and the AR-LEAF EVPN PEs set the PTA's flags differently, as per [RFC9574] 🍓
Also, each EVPN PE originates an IMET route for each BD it's attached to. Same as with IR, these routes carry the downstream-assigned MPLS labels that identify the BDs, and they don't carry the SBD-RT 🫶
When an ingress EVPN PE acting as an AR-LEAF needs to send an IP multicast frame from a particular source BD to an egress EVPN PE, the ingress PE figures out whether or not there's any AR-REPLICATOR that originated an IMET route for that BD. If there's more than one AR-REPLICATOR, one gets selected first. After that, the AR-LEAF uses the label in that AR-REPLICATOR's IMET route when it sends packets to it. The AR-REPLICATOR receives the packet and, following the procedures in [RFC9574] and in Section 3.2.2 of this document, transmits the packets to the egress EVPN PEs using the labels in the received IMET routes, either for the source BD or for the SBD ✨
If an ingress AR-LEAF for a given BD hasn't received any IMET route for that BD from an AR-REPLICATOR, the ingress AR-LEAF follows the procedures in Section 3.2.2 instead 🦋
3.2.3.1. Automatic SBD Matching 💖
Each PE needs to know a BD's corresponding SBD. You can configure that info in each BD, and that's one way to do it. But it means repetitive configuration plus consistency checking, to make sure every BD for the same tenant is set up with the same SBD. Ugh, so much work 😭 A better move is to configure the SBD info in the L3 routing instance, so all the related BDs derive the SBD information from it 👑
An AR-REPLICATOR also needs to know the same information, even though it doesn't necessarily have an L3 routing instance. The good news is that, from the EVI-RT EC in a BD's IMET route, an AR-REPLICATOR can derive the corresponding SBD of that BD without any configuration at all 🎀
3.2.4. BIER 🌸
When BIER is used to transport multicast packets of a given Tenant Domain, and a given EVPN PE attached to that Tenant Domain is a possible ingress EVPN PE for traffic originating outside that Tenant Domain, the given EVPN PE MUST originate an SBD-IMET route (see Section 3.2.1).
On top of that, IMET routes originated for other BDs in the Tenant Domain MUST carry the SBD-RT 💅
Each IMET route (including but not limited to the SBD-IMET route) MUST carry a PTA. The MPLS Label field of the PTA MUST specify an upstream-assigned MPLS label. That label maps uniquely (in the context of the originating EVPN PE) to the BD the route is originated for 🧚
Let's run an example, because it's clearer that way. Say there's an ingress EVPN PE, PE1, that needs to use BIER to tunnel an IP multicast frame to a set of egress EVPN PEs. The frame's source BD is BD1. The frame gets encapsulated like this:
- A four-octet MPLS label stack entry [RFC3032] is prepended to the frame. The Label field is set to the upstream-assigned label that PE1 has assigned to BD1.
- The resulting MPLS packet then gets encapsulated in a BIER encapsulation [RFC8296] [RFC9624]. The BIER BitString is set to identify the egress EVPN PEs. The BIER Proto field is set to the value for "MPLS packet with an upstream-assigned label at top of the stack" 🍓
Note: The packet being tunneled from PE1 might have originated outside the Tenant Domain. In that case, the actual source BD, BD1, is treated as the SBD. The upstream-assigned label it carries is the label PE1 assigned to the SBD and advertised in its SBD-IMET route 🙈
Now say an egress PE, PE2, gets one of these BIER packets. The BFIR-id field of the BIER header lets PE2 figure out that the ingress PE is PE1. From there, there are two possible cases 👀
- PE2 has received and installed an IMET route for BD1 from PE1. In this case, the BIER packet carries the upstream-assigned label specified in the PTA of that IMET route. That's how PE2 works out the apparent source BD (as defined in Section 2.4) 💜
- PE2 has not received and installed an IMET route for BD1 from PE1. In this case, PE2 won't recognize the upstream-assigned label in the BIER packet, so PE2 MUST discard the packet. Bye bye, packet 😭
For more on using BIER to support EVPN, check out [RFC9624] 🫶
3.2.5. Inclusive P2MP Tunnels 👑
3.2.5.1. Using the BUM Tunnels as IP Multicast Inclusive Tunnels ✨
The procedures in this section only apply when:
- a) it is desired to use the BUM tunnels to carry IP multicast traffic across the backbone, and
- b) the BUM tunnels are P2MP tunnels (i.e., neither IR, AR, nor BIER are being used to transport the BUM traffic).
In this case, an IP multicast frame (whether inter-subnet or intra-subnet) gets carried across the backbone in the BUM tunnel belonging to its source BD. Each EVPN PE attached to a given Tenant Domain needs to join the BUM tunnels for every BD in the Tenant Domain, even the BDs it isn't locally attached to. That's how an IP multicast packet from any source BD can reach all PEs attached to the Tenant Domain 🦋
Heads up though: this means all the BUM traffic from a given BD in a Tenant Domain gets sent to every PE attached to that Tenant Domain, including PEs that aren't attached to the given BD. To avoid that, it is RECOMMENDED that the BUM tunnels not be used as IP multicast inclusive tunnels, and that the procedures of Section 3.2.5.2 be used instead 💅
If a PE is a possible ingress EVPN PE for traffic originating outside the Tenant Domain, the PE MUST originate an SBD-IMET route (see Section 3.2.1). That route MUST carry a PTA specifying the P2MP tunnel used for transmitting IP multicast packets that originate outside the Tenant Domain. All EVPN PEs of the Tenant Domain MUST join the tunnel specified in the PTA of an SBD-IMET route:
- If the tunnel is an RSVP-TE P2MP tunnel, the originator of the route MUST use RSVP-TE P2MP procedures to add each PE of the Tenant Domain to the tunnel, even PEs that haven't originated an SBD-IMET route 🔥
- If the tunnel is an mLDP or PIM tunnel, each PE importing the SBD-IMET route MUST add itself to the tunnel, using mLDP or PIM procedures, respectively 🧚
Whether or not a PE originates an SBD-IMET route, it will still originate an IMET route for each BD it's attached to. Each of these IMET routes MUST carry the SBD-RT, as well as the RT for the BD it belongs to 🎀
If a received IMET route is not the SBD-IMET route, it also carries the RT for its source BD. The route's NLRI carries the Tag ID for the source BD. From the RT and the Tag ID, any PE that receives the route can determine the route's source BD ✨
If the MPLS Label field of the PTA contains zero, the specified P2MP tunnel is used only to carry frames of a single source BD 🌸
If the MPLS Label field of the PTA does not contain zero, it MUST contain an upstream-assigned MPLS label. That label maps uniquely (in the context of the originating EVPN PE) to the source BD, or, in the case of an SBD-IMET route, to the SBD. The tunnel may then be used to carry frames of multiple source BDs. The apparent source BD of a particular packet is inferred from the label the packet carries 💖
IP multicast traffic originating outside the Tenant Domain is transmitted with the label corresponding to the SBD, as specified in the ingress EVPN PE's SBD-IMET route 🫶
3.2.5.2. Using Wildcard S-PMSI A-D Routes to Advertise Inclusive Tunnels Specific to IP Multicast 💖
okay so these procedures are ONLY for when you want to send IP multicast on an inclusive tunnel that's different from the one carrying your BUM traffic 🦋 if they're the same tunnel, this section isn't your vibe. and these procedures do NOT apply when the tunnel type is IR or BIER, EXCEPT when you need to interwork between non-OISM PEs and OISM PEs (that's in Section 5 💅).
every EVPN PE attached to the given Tenant Domain MUST originate an SBD-SPMSI A-D route. its NLRI MUST contain (C-*,C-*) (see [RFC6625]) 🎀 the extra rules for building that route are in Section 3.2.1, so go peek there too!
and an EVPN PE MUST also originate an S-PMSI A-D route with (C-*,C-*) in its NLRI for each of the other BDs in that Tenant Domain it's attached to ✨ all of these routes MUST carry the SBD-RT, which makes sure every EVPN PE attached to the Tenant Domain imports them. main character energy for the whole Tenant Domain 👑
when a PE receives these routes, it follows Section 2.2 to figure out which BD the route is for 🌸
if the MPLS Label field of the PTA is zero, the tunnel is only carrying frames from a single source BD 🧚
if the MPLS Label field does not contain zero, it MUST specify an upstream-assigned MPLS label that maps uniquely (in the context of the originating EVPN PE) to the source BD 💜 the tunnel may carry frames from multiple source BDs, and the apparent source BD for a packet is inferred from the label that packet carries. so iconic 😭
the EVPN PE advertising these S-PMSI A-D routes is basically saying "this is the default tunnel I'll use (as ingress PE) to send IP multicast packets" 🫶 and that upstream-assigned label is what lets an egress PE work out a given packet's apparent source BD. it's giving teamwork 🔥
3.2.6. Selective Tunnels 🦋
an ingress EVPN PE for a given multicast flow (or set of flows) can always assign that flow to a particular P2MP tunnel by originating an S-PMSI A-D route whose NLRI identifies the flow or set of flows 💅 the NLRI could be (C-*,C-G) or (C-S,C-G). the S-PMSI A-D route MUST carry the SBD-RT so that every EVPN PE attached to the Tenant Domain imports it 🌈
an S-PMSI A-D route is for one particular source BD. it MUST carry the RT associated with that BD, and it MUST have the Tag ID for that BD in its NLRI 🍓
when an EVPN PE imports an S-PMSI A-D route, it applies the rules of Section 2.2 to link the route to a particular BD ✨
each such route MUST contain a PTA, as specified in Section 3.2.5.2 💖
an egress EVPN PE that's interested in the specified flow or flows MUST join the specified tunnel 🎀 how it joins depends on the tunnel type. (ngl this part is a little detailed: if the tunnel type is RSVP-TE P2MP LSP, the LIR flag of the PTA SHOULD NOT be set. an ingress OISM PE already knows which OISM EVPN PEs are interested in any given flow, so it can add them to the RSVP-TE P2MP tunnel that carries those flows 🧚)
if the PTA does not specify a non-zero MPLS label, the apparent source BD of any packets arriving on that tunnel is the BD associated with the route carrying the PTA 🌸 if the PTA does specify a non-zero MPLS label, then packets arriving on that tunnel carrying the specified label are considered to belong to that same BD 💜
it's worth noting that when either IR or BIER is used, an ingress PE doesn't need S-PMSI A-D routes to assign specific flows to selective tunnels 🙈 the procedures of Section 3.3, together with the procedures of Sections 3.2.2, 3.2.3, and 3.2.4, give you selective-tunnel functionality without S-PMSI A-D routes at all. efficient and cute 🦋
3.3. Advertising SMET Routes 💅
[RFC9251] lets an egress EVPN PE say "I'm interested in this flow!" by originating a SMET route 💖 the NLRI of that route identifies the flow or set of flows as (C-*,C-*), (C-*,C-G), or (C-S,C-G).
each SMET route belongs to one particular BD. the BD's Tag ID shows up in the NLRI, and the route carries that BD's RT 🎀 from that <RT, tag> pair, other EVPN PEs can tell which BD a received SMET route belongs to. (remember the route might carry multiple RTs, so keep your eyes open 👀)
there are three cases to think about 🌈
Case 1: It is known that no BD of a Tenant Domain contains a multicast router. ✨
in this case, an egress PE advertises its interest by originating a SMET route that belongs to the SBD. we call this an SBD-SMET route 🧚 it carries the SBD-RT and has the Tag ID for the SBD in its NLRI. you don't need SMET routes for the individual BDs, because a PE that receives a SMET route has no reason to send a matching IGMP/MLD Join on any of its ACs 🌸
Case 2: It is known that more than one BD of a Tenant Domain may contain a multicast router. 💜
this one is pretty much like Case 1. an egress PE advertises its interest by originating an SBD-SMET route, which carries the SBD-RT and has the Tag ID for the SBD in its NLRI 💖
the tricky bit: it's really important to be sure that SMET routes for the individual BDs are not originated 😭 for example, say PE1 had local receivers for a given flow on both BD1 and BD2, and it originated SMET routes for both of those BDs. then the PEs receiving those SMET routes might send IGMP/MLD Joins on both of those BDs. that could let externally sourced multicast traffic enter the Tenant Domain at both BDs, and that means duplicated data, which is a whole mess 💀
note that if it's possible for more than one BD to contain a tenant multicast router, then in order to receive multicast data originating from outside EVPN, the PEs MUST follow the procedures of Section 6 🔥
Case 3: It is known that only a single BD of a Tenant Domain contains a multicast router. 🦋
picture an egress PE attached to a BD where there might be a tenant multicast router (the tenant router doesn't have to be on a segment attached to that PE). and say the PE has one or more ACs on that BD that are interested in a given multicast flow. in that case, on top of the SMET route for the SBD, the egress PE MAY originate a SMET route for that BD too 🎀 this lets the ingress PE(s) send IGMP/MLD messages on ACs for the BD, as specified in [RFC9251]. as long as that's the only BD with a tenant multicast router, there's zero possibility of duplicated data 🙌
this document doesn't specify how to dynamically figure out which of the three cases applies to a given deployment, so the PEs of a given Tenant Domain MUST be provisioned to know which case applies ⭐
as detailed in [RFC9251], a SMET route carries flags that say whether IGMP (v1, v2, or v3) or MLD (v1 or v2) messages should be triggered on the ACs of the BD the SMET route belongs to. for IGMP v3 and MLD v2, the IE flag also tells you whether the source information in the SMET route is an Include Group type or an Exclude Group type 💜 if an SBD PE needs to generate IGMP/MLD reports (like in Section 6.2) or the route is for an (S, G) state, the value of the flags MUST be set according to the rules in [RFC9251]. otherwise, the flags SHOULD be set to 0 🫶
a PE only needs to originate the set of SBD-SMET routes it actually needs in order to receive the multicast traffic it's interested in 🍓 say PE1 has ACs on BD1 that are interested in (C-*,C-G) traffic and ACs on BD2 that are interested in (C-S,C-G) traffic. a single SBD-SMET route specifying (C-*,C-G) will attract all the flows it needs ✨
and as another example, say the ACs on BD1 are interested in (C-*,C-G) but not (C-S,C-G), while the ACs on BD2 are interested in (C-S,C-G). a single SBD-SMET route specifying (C-*,C-G) will pull in all the necessary flows 🌈
so in other words, to work out the set of SBD-SMET routes that have to be sent for a given C-G, the PE has to merge the IGMP/MLD state for all the BDs of the given Tenant Domain that it's attached to 🧚
per [RFC9251], importing a SMET route for a particular BD causes the IGMP/MLD state to be instantiated for the IRB interface to that BD 🌸 this also applies when the BD is the SBD.
BUT, traffic that originates in one of the actual BDs of a particular Tenant Domain MUST NOT be sent down the IRB interface that connects the L3 routing instance of that Tenant Domain to the SBD 🙅♀️ that would cause duplicate delivery, because that traffic has already been distributed throughout the Tenant Domain. so when you're setting up IGMP/MLD state based on SBD-SMET routes, take care to make sure the IRB interface to the SBD is not added to the Outgoing Interface (OIF) list if the traffic originates within the Tenant Domain 💖
there are some multicast scenarios that use anycast sources 🦋 for example, two different sources might share the same anycast IP address, say S1, and each might transmit an (S1,G) multicast flow. in that situation the two (S1,G) flows are typically identical. ordinary PIM procedures will deliver only one of the flows to each receiver that has expressed interest in either (*,G) or (S1,G). however, the OISM procedures described in this document will result in both (S1,G) flows being distributed in the Tenant Domain, and duplicate delivery will result 😭 so, if there are receivers for (*,G) in a given Tenant Domain, there MUST NOT be anycast sources for G within that Tenant Domain. (this restriction could be lifted by defining additional procedures, but that's outside the scope of this document 🎀)
4. Constructing Multicast Forwarding State ✨
4.1. Layer 2 Multicast State 🦋
Okay so every EVPN PE keeps Layer 2 multicast state for each BD it's attached to 🫶 and yes, it's used for forwarding IP multicast frames based on the inner IP header ✨ That state gets learned through IGMP/MLD snooping [RFC4541] plus the procedures in this document.
Let PE1 be an EVPN PE and BD1 be a BD it's attached to 💖 At PE1, BD1's Layer 2 multicast state for a given (C-S,C-G) or (C-*,C-G) decides what happens to an IP multicast packet that BD1's Layer 2 multicast function receives on an EVPN PE 💅
An IP multicast (S,G) packet counts as received by BD1's Layer 2 multicast function in PE1 in these cases:
- The packet is the payload of an Ethernet frame that PE1 received from an AC attached to BD1.
- The packet is the payload of an Ethernet frame whose apparent source BD is BD1, and PE1 received it over a tunnel from another EVPN PE.
- The packet is received from BD1's IRB interface (i.e., it was transmitted by PE1's L3 routing instance down BD1's IRB interface).
Here's the big picture 🌈 According to this document, every transmission of IP multicast packets from one EVPN PE to another happens at Layer 2. So the packets go out as Ethernet frames, following the Layer 2 multicast state.
Each Layer 2 multicast state (S,G) or (*,G) holds a set of outgoing interfaces, aka an OIF list 🍓 When an (S,G) multicast frame comes into BD1's Layer 2 multicast function, the disposition works like this:
- The OIF list comes from BD1's Layer 2 (S,G) state. If there's no such (S,G) state, it comes from BD1's (*,G) state instead. (If neither state exists, the OIF list is considered to be null 🙈)
- The rules of Section 4.1.2 get applied to the OIF list. This will generally send the frame to some, but not all, elements of the OIF list ✨
Also, there's no Reverse Path Forwarding (RPF) check at Layer 2, so don't go looking for one 💅
4.1.1. Constructing the OIF List 🧚
This document extends the procedures of [RFC9251] so that IMET and SMET routes for a particular BD get distributed not just to PEs attached to that BD, but to PEs attached to any BD in the Tenant Domain 👑 That way, each PE attached to a given Tenant Domain learns, from another PE attached to the same Tenant Domain, the set of flows that each of those other PEs cares about. (If some PE attached to the Tenant Domain doesn't support [RFC9251], it will be assumed to be interested in all flows. Whether or not a particular remote PE supports [RFC9251] is determined by the presence of an Extended Community in its IMET route; this is specified in [RFC9251].) If a set of remote PEs are interested in a particular flow, the tunnels used to reach those PEs get added to the OIF list of the multicast states for that flow 🦋
An EVPN PE may run IGMP/MLD snooping procedures [RFC4541] on each of its ACs to figure out the set of flows of interest to each AC. (An AC is said to be interested in a given flow if it connects to a segment that has tenant systems interested in that flow.) If IGMP/MLD procedures aren't being run on a given AC, that AC is considered to be interested in all flows. For each BD, the set of ACs interested in a given flow gets determined, and the ACs in that set get added to the OIF list of that BD's multicast state for that flow 🌸
The OIF list for each multicast state must also contain the IRB interface for the BD the state belongs to.
Implementors, heads up: the OIF list of a multicast state will change from time to time as ACs and/or remote PEs either become interested in or lose interest in particular multicast flows 🫶
4.1.2. Data Plane: Applying the OIF List to an (S,G) Frame 🔥
When an (S,G) multicast frame is received by the Layer 2 multicast function of a given EVPN PE, say PE1, what happens next depends on (a) the way it was received, (b) the OIF list of the corresponding multicast state (see Section 4.1.1), (c) whether an AC is eligible to receive a given frame (see Section 4.1.2.1), and (d) its apparent source BD (see Section 3.2 for how to determine the apparent source BD of a frame received over a tunnel from another PE) 💖
4.1.2.1. Eligibility of an AC to Receive a Frame 🎀
A given (S,G) multicast frame is eligible to be transmitted by a given PE, say PE1, on a given AC, say AC1, only if one of the following conditions holds:
- Ethernet Segment Identifier (ESI) labels are being used, PE1 is the DF for the segment that AC1 connects to, and the frame did not originate from that same segment (as determined by the ESI label) ✨
- The ingress PE for the frame is a remote PE, say PE2, local bias is being used, and PE2 is not connected to the same segment as AC1 💅
4.1.2.2. Applying the OIF List 🌈
Assume a given (S,G) multicast frame has been received by a given PE, say PE1. PE1 determines the apparent source BD of the frame, finds the Layer 2 (S,G) state for that BD (or the (*,G) state if there is no (S,G) state), and uses the OIF list from that state. (Note that if PE1 isn't attached to the actual source BD, the apparent source BD will be the SBD 🧚)
If PE1 has determined the frame's apparent source BD to be BD1 (which may or may not be the SBD), then these cases get considered:
- The frame was received by PE1 from a local AC, say AC1, that attaches to BD1.
- a. The frame MUST be sent on all local ACs of BD1 that appear in the OIF list, except for AC1 itself.
- b. The frame MUST also be delivered to any other EVPN PEs that have interest in it. This is achieved as follows:
- i. If (a) AR is being used, (b) PE1 is an AR-LEAF, and (c) the OIF list is non-null, PE1 MUST send the frame to the AR-REPLICATOR.
- ii. Otherwise, the frame MUST be sent on all tunnels in the OIF list.
- c. The frame MUST be sent to the local L3 routing instance by being sent up the IRB interface of BD1. It MUST NOT be sent up any other IRB interfaces.
- The frame was received by PE1 over a tunnel from another PE. (See Section 3.2 for the rules to determine the apparent source BD of a packet received from another PE. Note that if PE1 isn't attached to the source BD, it will regard the SBD as the apparent source BD.)
- a. The frame MUST be sent on all local ACs in the OIF list that connect to BD1 and that are eligible (per Section 4.1.2.1) to receive the frame.
- b. The frame MUST be sent up the IRB interface of the apparent source BD. (Note that this may be the SBD.) The frame MUST NOT be sent up any other IRB interfaces.
- c. If PE1 is not an AR-REPLICATOR, it MUST NOT send the frame to any other EVPN PEs. However, if PE1 is an AR-REPLICATOR, it MUST send the frame to all tunnels in the OIF list, except for the tunnel over which the frame was received.
- The frame was received by PE1 from the BD1 IRB interface (i.e., the frame has been transmitted by PE1's L3 routing instance down the BD1 IRB interface), and BD1 is NOT the SBD.
- a. The frame MUST be sent on all local ACs in the OIF list that are eligible, as per Section 4.1.2.1, to receive the frame.
- b. The frame MUST NOT be sent to any other EVPN PEs.
- c. The frame MUST NOT be sent up any IRB interfaces.
- The frame was received from the SBD IRB interface (i.e., it has been transmitted by PE1's L3 routing instance down the SBD IRB interface).
- a. The frame MUST be sent on all tunnels in the OIF list. This causes the frame to be delivered to any other EVPN PEs that have interest in it.
- b. The frame MUST NOT be sent on any local ACs.
- c. The frame MUST NOT be sent up any IRB interfaces.
4.2. Layer 3 Forwarding State 🧚
If an EVPN PE is performing IGMP/MLD procedures on the ACs of a given BD, it processes those messages at Layer 2 to help form the Layer 2 multicast state. It also sends those messages up that BD's IRB interface to the L3 routing instance of a particular Tenant Domain. This causes the (C-S,C-G) or (C-*,C-G) L3 state to be created or updated ✨
A Layer 3 multicast state has both an Input Interface (IIF) and an OIF list 💜
For a (C-S,C-G) state, if the source BD is present on the PE, the IIF is set to the IRB interface that attaches to that BD. Otherwise, the IIF is set to the SBD IRB interface.
For (C-*,C-G) states, traffic can arrive from any BD, so the IIF needs to be set to a wildcard value meaning "any IRB interface" 🌸
The OIF list of these states includes one or more of the IRB interfaces of the Tenant Domain. In general, maintaining the OIF list doesn't require any EVPN-specific procedures. However, there is one EVPN-specific rule:
If the IIF is one of the IRB interfaces (or the wildcard meaning "any IRB interface"), then the SBD IRB interface MUST NOT be added to the OIF list. Traffic originating from within a particular EVPN Tenant Domain must not be sent down the SBD IRB interface, as such traffic has already been distributed to all EVPN PEs attached to that Tenant Domain.
Please also see Section 6.1.1, which states a modification of this rule for the case where OISM is interworking with external Layer 3 multicast routing 🫶
Heads up, bestie 🥺 This part only has 10 standalone MUST in it, not 11 (the MUST NOT ones are separate). I kept every requirement exactly as the original wrote it, rather than inventing an extra one, because adding a requirement that isn't in the RFC would be a worse bug than a short count.
5. Interworking with Non-OISM EVPN PEs 💅🎀
okay so bestie, it is totally possible for one Tenant Domain to have OISM PEs AND non-OISM PEs hanging out together 🫶 and honestly? Inter-subnet IP multicast should still work perfectly and be fully functional even if not every PE attached to the Tenant Domain can be upgraded to do OISM stuff ✨ that's the whole vibe!
Quick note: non-OISM PEs are NOT required to have IRB support or support for [RFC9251] 🙈 but it is seriously advantageous for them to support [RFC9251] 💜
Here's the cast of characters for this section 🧚
PE-S: the ingress PE for an (S,G) flow 🌸
PE-R: an egress PE for an (S,G) flow 🦋
BD-S: the source BD for an (S,G) flow. PE-S MUST have one or more ACs attached to BD-S, at least one of which attaches to host S 🔥
BD-R: a BD that has a host interested in the flow. That host is attached to PE-R through an AC that belongs to BD-R 🍓
So to let OISM PEs interwork with non-OISM PEs, a given Tenant Domain needs to contain one or more IP Multicast Gateways (IPMGs) 👑 An IPMG is basically an OISM PE with special responsibilities for the interworking between the two worlds 😭
If a PE is functioning as an IPMG, it MUST signal that fact by setting the IPMG flag in the Multicast Flags EC that it attaches to its IMET routes 🙌 An IPMG SHOULD attach this EC, with the IPMG flag set, to all the IMET routes it originates 💖 And here's the tea: if PE1 imports any IMET route from PE2 that has the EC present with the IPMG flag set, then PE1 will assume PE2 is an IPMG 👀
Next up is the IPMG Designated Forwarder (IPMG-DF) selection procedure 🎀 It makes sure there is exactly one active IPMG-DF for any given BD at any given time ✨ The nitty gritty details are in Section 5.1. The IPMG-DF for a given BD, say BD-S, has some special jobs to do when it receives (S,G) frames on that BD 💅
Ok this part is important, so read it twice 💀
- If the frames come from a non-OISM PE-S:
- The IPMG-DF forwards them to OISM PEs that do NOT attach to BD-S but are interested in (S,G) 🦋
Note that OISM PEs that DO attach to BD-S will already have received the frames on the BUM tunnel from the non-OISM PE-S 🫶
- The IPMG-DF forwards them to non-OISM PEs that are interested in (S,G) on ACs that do not belong to BD-S 🌈
Note that if a non-OISM PE has multiple BDs (other than BD-S) interested in (S,G), it will receive one copy of the frame for EACH such BD. This is necessary because non-OISM PEs cannot move IP multicast traffic from one BD to another 😭
- If the frames come from an OISM PE, the IPMG-DF forwards them to non-OISM PEs that are interested in (S,G) on ACs that do not belong to BD-S 🧚
If a non-OISM PE is interested in (S,G) on an AC belonging to BD-S, it will already have received a copy of the (S,G) frame, encapsulated for BD-S, from the OISM PE-S (see Section 3.2.2) 💜 If the non-OISM PE is interested in (S,G) on one or more ACs belonging to BD-R1,...,BD-Rk, where the BD-Ri are distinct from BD-S, the IPMG-DF needs to send it a copy of the frame for each BD-Ri 🌸
If an IPMG receives a frame on a BD for which it is NOT the IPMG-DF, it just follows the normal OISM procedures 🦋
This section lays out several sets of procedures:
- the procedures that the IPMG-DF for a given BD follows when it receives, on that BD, an IP multicast frame from a non-OISM PE 🌸
- the procedures that the IPMG-DF for a given BD follows when it receives, on that BD, an IP multicast frame from an OISM PE 💖
- and the procedures that an OISM PE follows when it receives, on a given BD, an IP multicast frame from a non-OISM PE, when that OISM PE is NOT the IPMG-DF for that BD ✨
To enable OISM/non-OISM interworking in a given Tenant Domain, the Tenant Domain MUST have some EVPN PEs that can function as IPMGs 🔥 An IPMG MUST be configured with the SBD 🎀 It also MUST be configured with every BD of the Tenant Domain that exists on any of the non-OISM PEs of that domain 💅 (Operationally, it may be simpler to configure the IPMG with ALL the BDs of the Tenant Domain 🧚)
Of course, a non-OISM PE only needs to be configured with the BDs it has ACs for 🙈 An OISM PE that is NOT an IPMG only needs to be configured with the SBD and with the BDs it has ACs for 💜
An IPMG MUST originate a wildcard SMET route (with (C-,C-) in the NLRI) for each BD in the Tenant Domain 🌈 This makes sure it receives ALL the IP multicast traffic that is sourced in the Tenant Domain 👑 Also, non-OISM nodes that do not support [RFC9251] will send ALL the multicast traffic from a given BD to ALL PEs attached to that BD, even if those PEs don't originate a SMET route 😭
The interworking procedures vary a bit depending on whether packets go from PE to PE via IR or via P2MP tunnels 🦋 In this section we do NOT consider BIER, due to the low likelihood of there being a non-OISM PE that supports BIER 🙈
5.1. IPMG Designated Forwarder
Every PE that is eligible to be selected as an IPMG-DF for a particular BD originates BOTH an IMET route for that BD AND an SBD-IMET route 💖 As stated in Section 5, these SBD-IMET routes carry a Multicast Flags EC with the IPMG flag set ✨
These SBD-IMET routes SHOULD also carry a DF Election EC 🎀 The DF Election EC and its use are specified in [RFC8584]. When the route is originated, the AC-DF bit in the DF Election EC SHOULD NOT be set 🙅 This bit is NOT used when selecting an IPMG-DF, i.e., it MUST be ignored by the receiver of an SBD-IMET route 🔥
In the context of a given Tenant Domain, to select the IPMG-DF for a particular BD, say BD1, the IPMGs of the Tenant Domain do the following 🧚
- From the set of received SBD-IMET routes for the given Tenant Domain, figure out the candidate set of PEs that support IPMG functionality for that domain 🌸
- From that candidate set, eliminate any PEs from which NO IMET route for BD1 has been received 🦋
- Select a DF election algorithm as specified in [RFC8584]. Some of the possible algorithms can be found, e.g., in [RFC8584], [RFC7432], and [EVPN-DF] 🌈
- Apply the DF election algorithm (see [RFC8584]) to the candidate set of PEs. The winner becomes the IPMG-DF for BD1 👑
Note that even if a given PE supports MEG (Section 6.1.2) and/or PEG (Section 6.1.4) functionality, as well as IPMG functionality, its SBD-IMET routes carry ONLY one DF Election EC 💅
5.2. Ingress Replication
The procedures in this section are used when IR is used to transmit packets from one PE to another 🫶
When a non-OISM PE-S transmits a multicast frame from BD-S to another PE, say PE-R, PE-S will use the encapsulation specified in the BD-S IMET route that was originated by PE-R 💖 This encapsulation includes the label that appears in the MPLS Label field of the PTA of the IMET route ✨ If the tunnel type is VXLAN, the label is actually a Virtual Network Identifier (VNI) 🌈 For other tunnel types, the label is an MPLS label 🎀 Either way, the frames are transmitted with a label that was assigned to a particular BD by the PE-R the frame is being sent to 🧚
To support OISM/non-OISM interworking, an OISM PE-R MUST originate, for EACH of its BDs, BOTH an IMET route AND a (C-,C-) S-PMSI A-D route 🔥 Note that even when IR is being used, interworking between OISM and non-OISM PEs requires the OISM PEs to follow the rules of Section 3.2.5.2, as modified below 💅
Non-OISM PEs will NOT understand S-PMSI A-D routes 🙈 So when a non-OISM PE-S transmits an IP multicast frame with a particular source BD to an IPMG, it encapsulates the frame using the label specified in that IPMG's BD-S IMET route 🌸 (This is just the procedure of [RFC7432].)
The (C-,C-) S-PMSI A-D route originated by a given OISM PE will have a PTA that specifies IR 🦋
- If MPLS tunneling is being used, the MPLS Label field SHOULD contain a non-zero value, and the LIR flag SHOULD be zero 💜 (The case where the MPLS Label field is zero or the LIR flag is set is outside the scope of this document 🙈)
- If the tunnel encapsulation is VXLAN, the MPLS Label field MUST contain a non-zero value, and the LIR flag MUST be zero 🔥
When an OISM PE-S transmits an IP multicast frame to an IPMG, it uses the label specified in that IPMG's (C-,C-) S-PMSI A-D route 🧚
When a PE originates BOTH an IMET route AND a (C-,C-) S-PMSI A-D route, the values of the MPLS Label field in the respective PTAs MUST be distinct 💖 Further, EACH MUST map uniquely (in the context of the originating PE) to the route's BD 🌈
As a result, an IPMG receiving an MPLS-encapsulated IP multicast frame can ALWAYS tell from the label whether the frame's ingress PE is an OISM PE or a non-OISM PE 👑 When an IPMG receives a VXLAN-encapsulated IP multicast frame, it may need to determine the identity of the ingress PE from the outer IP encapsulation 🫶 It can then determine whether the ingress PE is an OISM PE or a non-OISM PE by looking at the IMET route from that PE ✨
Suppose an IPMG receives an IP multicast frame from another EVPN PE in the Tenant Domain, and the IPMG is NOT the IPMG-DF for the frame's source BD 🦋 Then the IPMG only performs the ordinary OISM functions; it does NOT perform the IPMG-specific functions for that frame 💅 In the rest of this section, when we talk about the procedures an IPMG applies when it receives an IP multicast frame, we are assuming that the source BD of the frame is a BD for which the IPMG is the IPMG-DF 🌸
We have two basic cases to consider: (1) the frame's ingress PE is a non-OISM node, and (2) the frame's ingress PE is an OISM node 💜
5.2.1. Ingress PE is Non-OISM ✨
okay so picture this: a non-OISM PE, let's call her PE-S, gets an (S,G) multicast frame over an AC that's attached to a BD, say BD-S. 🦋 By normal EVPN procedures, PE-S sends a copy of the frame to every PE-R (both OISM and non-OISM) in the Tenant Domain that's attached to BD-S. If the non-OISM node supports [RFC9251], only the PEs that have expressed interest in (S,G) get the frame. The IPMG will have expressed interest through a (C-*,C-*) SMET route, so it gets the frame too. 💖
Any OISM PE (IPMG included) that receives the frame applies the normal OISM procedures. So it delivers the frame to any of its local ACs, in BD-S or in any other BD, that are interested in (S,G). ✨
Now, an OISM PE that is also the IPMG-DF for a particular BD, say BD-S, has some extra procedures for frames it receives on BD-S from non-OISM PEs. 👑
- When the IPMG-DF for BD-S receives an
(S,G)frame from a non-OISM node, it MUST forward a copy of the frame to every OISM PE that is not attached to BD-S but has interest in(S,G). The copy sent to a given OISM PE-R must carry the label that PE-R has assigned to the SBD in an S-PMSI A-D route. The IPMG MUST NOT do any IP processing of the frame's IP payload. TTL decrement and the other IP processing get handled by PE-R, following the normal OISM procedures. 🙈 There's no need for the IPMG to put an ESI label in the frame's tunnel encapsulation, because it's already known that the frame's source BD has no presence on PE-R. There's also no need for the IPMG to change the frame's MAC SA. 🫶 - In addition, when the IPMG-DF for BD-S receives an
(S,G)frame from a non-OISM node, it may need to forward copies of the frame to other non-OISM nodes. Before it does that, it MUST decapsulate the(S,G)packet and do the IP processing (e.g., TTL decrement). Say PE-R is a non-OISM node with an AC to BD-R, where BD-R is not the same as BD-S, and that AC is interested in(S,G). The IPMG then has to encapsulate the(S,G)packet (after the IP processing is done) in an Ethernet header. The MAC SA field carries the MAC address of the IPMG's IRB interface for BD-R. Then the IPMG sends the frame to PE-R. The tunnel encapsulation carries the label that PE-R advertised in its IMET route for BD-R. No ESI label is needed here, since the source and destination BDs are known to be different. 💅
Heads up: if a non-OISM PE-R has several BDs (other than BD-S) with local ACs that are interested in (S,G), the IPMG will send it one copy for each of those BDs. This is needed because the non-OISM PE cannot move packets from one BD to another. 🎀
There may be deployment scenarios where every OISM PE is configured with every BD that's present on any non-OISM PE. 🌈 In those scenarios, the item 1 procedures won't actually result in any packets being transmitted. So if you already know a priori that this deployment scenario exists for a given Tenant Domain, the item 1 procedures can be disabled. 🧚
5.2.2. Ingress PE is OISM 💜
okay now the other side: an OISM PE, PE-S, receives an (S,G) multicast frame over an AC that attaches to a particular BD, say BD-S. 🌸
Because PE-S receives all the IMET routes for BD-S, it knows all the PEs attached to BD-S. Then, by normal OISM procedures:
- PE-S sends a copy of the frame to every OISM PE-R (IPMG included) in the Tenant Domain that's attached to BD-S and interested in
(S,G). The copy sent to a given PE-R carries the label that PE-R assigned to BD-S in its(C-*,C-*)S-PMSI A-D route. ✨ - PE-S also transmits a copy of the
(S,G)frame to every OISM PE-R that's interested in(S,G)but is not attached to BD-S. The copy contains the label that PE-R assigned to the SBD. (As specified in Section 5.2.1, an IPMG is assumed to have indicated interest in all multicast flows.) 🦋 - PE-S also transmits a copy of the
(S,G)frame to every non-OISM PE-R attached to BD-S. It does this using the label that PE-R advertised in its IMET route for BD-S. 🍓
The PE-Rs follow their normal procedures. An OISM PE that receives the (S,G) frame on BD-S applies the OISM procedures to deliver the frame to its local ACs as needed. A non-OISM PE that receives the (S,G) frame on BD-S delivers it only to its local BD-S ACs as needed. 💖
Now suppose a non-OISM PE-R is interested in (S,G) on a BD that's different from BD-S, say BD-R. If that non-OISM PE-R is attached to BD-S, the OISM PE-S will send it the original (S,G) multicast frame. But the non-OISM PE-R won't be able to send the frame to ACs that aren't in BD-S. If PE-R isn't even attached to BD-S, the OISM PE-S won't send it a copy of the frame at all, because PE-R isn't attached to the SBD. 😭 In these cases, the IPMG needs to relay the (S,G) multicast traffic from OISM PE-S to non-OISM PE-R.
When the IPMG-DF for BD-S receives an (S,G) frame from an OISM PE-S, it has to forward it to every non-OISM PE-R that's interested in (S,G) on a BD-R that's different from BD-S. The IPMG MUST decapsulate the IP multicast packet, do the IP processing, re-encapsulate it for BD-R (changing the MAC SA to the IPMG's own MAC address for BD-R), and send a copy of the frame to PE-R. ⭐ Note that a given non-OISM PE-R will receive multiple copies of the frame if it has multiple BDs where there's interest in the frame. 🔥
5.3. P2MP Tunnels 🌈
When IR is used to distribute the multicast traffic among the EVPN PEs, the procedures in Section 5.2 make sure there's no duplicate delivery of multicast traffic. That means no egress PE will ever send a frame twice on any given AC. ✨ If P2MP tunnels are being used to distribute the multicast traffic, extra procedures are needed to prevent duplicate delivery. 💅
Right now, it's not clear that there will be a use case where OISM nodes need to interwork with non-OISM nodes that use P2MP tunnels. 🙈 If it's determined that there is such a use case, procedures for P2MP may be specified in a separate document. 🧚
6. Traffic to/from Outside the EVPN Tenant Domain 🌍
In this section, we talk about scenarios where a multicast source outside a given EVPN Tenant Domain sends traffic to receivers inside the domain (and possibly also to receivers outside the domain). 👑 This requires the OISM procedures to interwork with various Layer 3 multicast routing procedures. 💜
In this section, we assume the Tenant Domain isn't being used as an intermediate transit network for multicast traffic. In other words, we don't consider the case where the Tenant Domain contains multicast routers that receive traffic from sources outside the domain and forward it to receivers outside the domain. The transit scenario is covered in Section 7. 🦋
We can split the non-transit scenarios into two classes:
- One or more of the EVPN PE routers provide the functionality needed to interwork with Layer 3 multicast routing procedures.
- A single BD in the Tenant Domain contains external multicast routers (tenant multicast routers), and those tenant multicast routers are used to interwork, on behalf of the entire Tenant Domain, with Layer 3 multicast routing procedures.
6.1. Layer 3 Interworking via EVPN OISM PEs 🎀
6.1.1. General Principles 💖
Okay so sometimes you need to connect an EVPN Tenant Domain to an external Layer 3 multicast domain (the external domain), like a PIM or MVPN domain 🌸. Why? Because EVPN tenant systems need to be able to receive multicast from sources (external sources) outside the Tenant Domain, and receivers (external receivers) outside the Tenant Domain need to get traffic from sources inside it 🦋. Main character interworking energy ✨.
To make that happen, one or more OISM PEs must be L3 Gateways. An L3 Gateway is in both worlds: it takes part in the OISM procedures and in the L3 multicast routing procedures of the external domain. Here's the vibe 👇
src1 rcvr1
| |
R1 RP R2
PIM/MVPN
Domain
+---+ +---+
-----|GW1|----------------------|GW2|----
+---+ +---+
| \ \ / / |
| \ \ / / |
BD1 BD2 SBD SBD BD2 BD1
EVPN Domain
SBD SBD
/ \
/ \
+---+ +---+
|PE1| |PE2|
+---+ +---+
| \ / |
BD1 BD2 BD2 BD1
| | | |
src2 rcvr2 src3 rcvr3
Figure 1: Interworking via OISM PEsAn L3 Gateway that's interested in (S,G) traffic has to be able to figure out the best route to S 🔍. If it's interested in (,G), it has to be able to find the best route to G's RP 🗺️. In these interworking setups the L3 Gateway must be running a Layer 3 unicast routing protocol. Through that protocol it imports unicast routes (either IP routes or VPN-IP routes) from routers that are *not EVPN PEs 📥. And since multicast sources might live inside the EVPN Tenant Domain, the EVPN PEs also need to export unicast routes to those sources, either as IP routes or as VPN-IP routes, depending on what the external domain wants 📤.
When the L3 Gateway is choosing the best route to a multicast source or RP, it might have a choice between an EVPN route and an IP/VPN-IP route. When that happens, the L3 Gateway SHOULD always prefer the EVPN route 👑. Why does that matter? If traffic starts in the Tenant Domain and has a receiver in the Tenant Domain, the path stays inside the EVPN Tenant Domain, even if the source is also reachable over a routed path 🫶. It also protects against sub-optimal routing that could happen if two EVPN PEs export IP/VPN-IP routes and each one imports the other's IP/VPN-IP routes 💀.
Section 4.2 covers how OISM PEs build Layer 3 multicast states 🧩. These states have IRB interfaces as their IIF and OIF list entries, and they're the foundation for interworking OISM with other Layer 3 multicast procedures like MVPN or PIM 🌈. From the point of view of the Layer 3 multicast procedures running on a given L3 Gateway, an EVPN Tenant Domain is basically just a set of IRB interfaces 🎀.
When you interwork an EVPN Tenant Domain with an external domain, the L3 Gateway's Layer 3 multicast states won't only have IRB interfaces as IIF and OIF list entries. They'll also have other interfaces that lead outside the Tenant Domain 🚪. For example, when interworking with MVPN, the multicast states may include MVPN tunnels as well as IRB interfaces as IIF or OIF list members. When interworking with PIM, they may include PIM-enabled non-IRB interfaces as IIF or OIF list members 🦋.
As long as a Tenant Domain isn't being used as an intermediate transit network for IP multicast traffic, there's no need to enable PIM on its IRB interfaces 🙈.
Generally speaking, an L3 Gateway has these responsibilities 💅:
- It exports, to the external domain, unicast routes to those multicast sources in the EVPN Tenant Domain that are locally attached to the L3 Gateway.
- It imports, from the external domain, unicast routes to multicast sources that are in the external domain.
- It runs the procedures needed to pull externally sourced multicast traffic that's of interest to locally attached receivers in the EVPN Tenant Domain. When that traffic arrives, it gets sent down the IRB interfaces of the BDs where those locally attached receivers live 📨.
One of the L3 Gateways in a given Tenant Domain becomes the DR for the SBD (see Section 6.1.2.4) 👑. That L3 Gateway picks up some extra responsibilities too:
- It exports, to the external domain, unicast routes to multicast sources in the EVPN Tenant Domain that are not locally attached to any L3 Gateway.
- It imports, from the external domain, unicast routes to multicast sources that are in the external domain.
- It runs the procedures needed to pull externally sourced multicast traffic that's of interest to receivers in the EVPN Tenant Domain that are not locally attached to an L3 Gateway. When that traffic arrives, it gets sent down the SBD IRB interface. Then the OISM procedures already described in this document take over and spread the IP multicast traffic through the Tenant Domain to any EVPN PEs that want it ✨. So to an OISM PE that isn't an L3 Gateway, the externally sourced traffic looks like it was sourced on the SBD 🧚.
For this to work, a bit of extra care is needed when an L3 Gateway creates or changes a Layer 3 (,G) multicast state ⚠️. Say group G has both external sources (sources outside the EVPN Tenant Domain) and internal sources (sources inside it). Section 4.2 says that when there are internal sources, the SBD IRB interface must not be added to the OIF list of the (,G) state 🙅♀️. The reason: traffic from internal sources has already been delivered to every EVPN PE that's interested in it. But if the (*,G) state's OIF list doesn't include its SBD IRB interface, then traffic from external sources won't reach the other EVPN PEs 😭.
One way to handle this is as follows 💡. When an L3 Gateway receives (S,G) traffic from an interface that isn't IRB, and that traffic matches a Layer 3 (*,G) state, the L3 Gateway can create (S,G) state. The IIF is set to the external interface where the traffic is expected to arrive. The OIF list contains the SBD IRB interface, plus the IRB interfaces of any other BDs attached to the PEG DR that have locally attached receivers interested in that (S,G) traffic 🌷. That (S,G) state makes sure the external traffic gets sent down the SBD IRB interface 🎯. The text that follows will assume this procedure, but other implementation techniques may also be possible 🤷♀️.
If a particular BD is attached to several L3 Gateways, one of them becomes the DR for that BD (see Section 6.1.2.4) 🫧. If the interworking scenario needs FHR functionality, it's usually the DR for a particular BD that handles it on behalf of the source hosts on that BD. For example, if the scenario requires PIM Register messages to be sent by an FHR, the DR for a given BD would send the PIM Register messages for the sources on that BD 📬. Heads up though: the DR for the SBD does not perform FHR functionality on behalf of external sources 🚫.
An optional alternative is for each L3 Gateway to perform FHR functionality for its locally attached sources 🍓. In that case, the DR would only need to handle FHR functionality for sources locally attached to itself AND sources that aren't attached to any L3 Gateway 💜.
One more important thing ⚠️: if it's possible that more than one BD contains a tenant multicast router, then a PE that receives a SMET route for that BD MUST NOT reconstruct IGMP/MLD Join Reports from the SMET route, and it MUST NOT transmit any such IGMP/MLD Join Reports on its local ACs attaching to that BD. Otherwise, multicast traffic may get duplicated 😱🔥.
6.1.2. Interworking with MVPN 🦋
okay so this section is where we figure out how EVPN PEs running OISM procedures can vibe with L3VPN PEs that run BGP-based MVPN [RFC6514] procedures 💅 The procedures here let a given EVPN Tenant Domain become part of an L3VPN/MVPN and support multicast flows where either of these happens:
- The source of a given multicast flow is attached to an Ethernet segment whose BD is part of an EVPN Tenant Domain, and one or more receivers of the flow are attached to the network via L3VPN/MVPN. (Other receivers may be attached to the network via EVPN.) 🌸
- The source of a given multicast flow is attached to the network via L3VPN/MVPN, and one or more receivers of the flow are attached to an Ethernet segment that is part of an EVPN Tenant Domain. (Other receivers may be attached via L3VPN/MVPN.) 🌈
In this interworking model, the existing L3VPN/MVPN PEs have no idea that certain sources or receivers are part of an EVPN Tenant Domain 🙈 The existing L3VPN/MVPN nodes just run their standard procedures and are totally unaware of EVPN. The trick is that some or all of the EVPN PEs act as L3 Gateways running L3VPN/MVPN procedures, as detailed in the subsections below. It's giving secret agent energy 🕵️♀️
In this section, we assume there are no tenant multicast routers on any of the EVPN-attached Ethernet segments. (Of course, there may be multicast routers in the L3VPN.) Handling the case where there are tenant multicast routers is covered in Section 7.
To support MVPN/EVPN interworking, we introduce the notion of an MVPN/EVPN Gateway (MEG). ✨
A MEG is an L3 Gateway (see Section 6.1.1), so it is both an OISM PE and an L3VPN/MVPN PE, which is honestly iconic 👑 For a given EVPN Tenant Domain, it will have an IP-VRF. If the Tenant Domain is part of an L3VPN/MVPN, the IP-VRF also serves as an L3VPN VRF [RFC4364]. The IRB interfaces of the IP-VRF are considered to be VRF interfaces of the L3VPN VRF. The L3VPN VRF may also have other local VRF interfaces that are not EVPN IRB interfaces.
The VRF on the MEG will import VPN-IP routes [RFC4364] from other L3VPN PE routers, and it will also export VPN-IP routes to other L3VPN PE routers. To do that, it must be appropriately configured with the RTs used in the L3VPN to control the distribution of the VPN-IP routes. Generally these RTs will be different from the RTs used for controlling the distribution of EVPN routes, since there's no need to send EVPN routes to L3VPN-only PEs and no reason to send L3VPN/MVPN routes to EVPN-only PEs 💖
Note that the RDs in the imported VPN-IP routes won't necessarily follow the EVPN rules (as specified in [RFC7432]) for creating RDs. So a MEG MUST NOT expect the RDs of the VPN-IP routes to be of any particular format other than what is required by the L3VPN/MVPN specifications. Don't assume, bestie 🙅♀️
The VPN-IP routes that a MEG exports to L3VPN are subnet routes and/or host routes for the multicast sources that are part of the EVPN Tenant Domain. The exact set of routes that need to be exported is discussed in Section 6.1.2.2.
Each IMET route originated by a MEG SHOULD carry a Multicast Flags Extended Community with the MEG flag set, which basically says the originator of that IMET route is a MEG 🧚 However, PE1 will consider PE2 to be a MEG if PE1 imports at least one IMET route from PE2 that carries the Multicast Flags EC with the MEG flag set.
All the MEGs of a given Tenant Domain attach to the SBD of that domain, and one of them is selected to be the SBD's Designated Router (the MEG SBD-DR) for the domain. The selection procedure is discussed in Section 6.1.2.4. 👑
In this model of operation, MVPN procedures and EVPN procedures are pretty independent 🦋 In particular, there's no assumption that MVPN and EVPN use the same kind of tunnels. So no special procedures are needed for the common scenarios where, e.g., EVPN uses VXLAN tunnels but MVPN uses MPLS P2MP tunnels, or where EVPN uses IR but MVPN uses MPLS P2MP tunnels. Main character energy, no extra work needed 💅
Similarly, no special procedures are needed to prevent duplicate data delivery on Ethernet segments that are multihomed. ✨
The MEG does have some special procedures (described below) for interworking between EVPN and MVPN. These cover selecting the Upstream PE for a given multicast source, exporting VPN-IP routes, and generating MVPN C-multicast routes triggered by the installation of SMET routes. Those are the three things it's specially in charge of 🫶
6.1.2.1. MVPN Sources with EVPN Receivers 🌸
6.1.2.1.1. Identifying MVPN Sources
okay so picture a multicast source S. It's possible that a MEG will import both an EVPN unicast route to S and a VPN-IP route (or an ordinary IP route), where the prefix length of each route is the same 🤔 To draw (S,G) multicast traffic for any group G, the MEG SHOULD use the EVPN route rather than the VPN-IP or IP route to determine the Upstream PE (see Section 5 of [RFC6513]).
Doing so makes sure that when an EVPN tenant system wants to receive a multicast flow from another EVPN tenant system, the traffic from the source to that receiver stays inside the EVPN domain 🔥 This avoids problems that could come up if there's a unicast route via L3VPN to S but no multicast routers along the routed path. It also avoids a problem that could come from the fact that the MEGs will import each other's VPN-IP routes.
In Section 6.1.2.1.2, we describe the procedures to be used when the selected route to S is a VPN-IP route.
6.1.2.1.2. Joining a Flow from an MVPN Source
Okay, let's say there's a tenant system, call her R, on a particular BD, call it BD-R. Now suppose R wants to receive (S,G) multicast traffic, where source S is not attached to any PE in the EVPN Tenant Domain but is attached to an MVPN PE. Here's how it plays out 💖
- Suppose R is on a singly homed Ethernet segment of BD-R and that segment is attached to PE1, where PE1 is a MEG. PE1 learns via IGMP/MLD listening that R is interested in
(S,G). PE1 checks its VRF and determines that there is no route toSwithin the Tenant Domain (i.e., no EVPN RT-2 route matching onS's IP address), but there is a route toSvia L3VPN (i.e., the VRF contains a subnet or host route toSthat was received as a VPN-IP route). So PE1 originates (if it hasn't already) an MVPN C-multicast Source Tree Join(S,G)route. The route is constructed according to normal MVPN procedures.
The Layer 2 multicast state is constructed as specified in Section 4.1.
In the Layer 3 multicast state, the IIF is the appropriate MVPN tunnel, and the IRB interface to BD-R is added to the OIF list.
When PE1 receives (S,G) traffic from the appropriate MVPN tunnel, it does IP processing on the traffic and then sends it down its IRB interface to BD-R. Following normal OISM procedures, the (S,G) traffic gets encapsulated for Ethernet and sent out on the AC that R is attached to. 🎀
- Suppose R is on a singly homed Ethernet segment of BD-R and that segment is attached to PE1, where PE1 is an OISM PE but is NOT a MEG. PE1 learns via IGMP/MLD listening that R is interested in
(S,G). PE1 follows normal OISM procedures and originates an SBD-SMET route for(S,G). This route gets received by all the MEGs of the Tenant Domain, including the MEG SBD-DR. From PE1's IMET routes, the MEG SBD-DR can tell whether or not PE1 is itself a MEG. If PE1 is not a MEG, the MEG SBD-DR will originate (if it hasn't already) an MVPN C-multicast Source Tree Join(S,G)route. That will cause the MEG SBD-DR to receive(S,G)traffic on an MVPN tunnel.
The Layer 2 multicast state is constructed as specified in Section 4.1.
In the Layer 3 multicast state, the IIF is the appropriate MVPN tunnel, and the IRB interface to the SBD is added to the OIF list.
When the MEG SBD-DR receives (S,G) traffic on an MVPN tunnel, it does IP processing on the traffic and then sends it down its IRB interface to the SBD. Following normal OISM procedures, the traffic gets encapsulated for Ethernet and delivered to all PEs in the Tenant Domain that are interested in (S,G), including PE1. 🍓
- If R is on a multihomed Ethernet segment of BD-R, one of the PEs attached to the segment will be its DF (following normal EVPN procedures). The DF will know (via IGMP/MLD listening or the procedures of [RFC9251]) that a tenant system reachable via one of its local ACs to BD-R is interested in
(S,G)traffic. The DF is responsible for originating an SBD-SMET route for(S,G), following normal OISM procedures. If the DF is a MEG, it MUST originate the corresponding MVPN C-multicast Source Tree Join(S,G)route. If the DF is not a MEG, the MEG SBD-DR SBD MUST originate the C-multicast route when it receives the SMET route. 🫧
Optionally, if the non-DF is a MEG, it MAY originate the corresponding MVPN C-multicast Source Tree Join (S,G) route. This will cause the traffic to flow to both the DF and the non-DF, but only the DF will forward the traffic out an AC. This allows for quicker recovery if the DF's local AC to R fails. Backup plan, but make it cute 🌷
- If R is attached to a non-OISM PE, it will receive the traffic via an IPMG, as specified in Section 5. 🧚
If an EVPN-attached receiver is interested in (*,G) traffic, and it's possible for there to be sources of (*,G) traffic that are attached only to L3VPN nodes, the MEGs will have to know the group-to-RP mappings. That lets them originate MVPN C-multicast Shared Tree Join (*,G) routes and send them toward the RP. (Since we're assuming in this section that there are no tenant multicast routers attached to the EVPN Tenant Domain, the RP must be attached via L3VPN. Alternatively, the MEG itself could be configured to function as an RP for group G.) 💜
The Layer 2 multicast states are constructed as specified in Section 4.1.
In the Layer 3 (*,G) multicast state, the IIF is the appropriate MVPN tunnel. A MEG will add its IRB interfaces to the (*,G) OIF list for any BDs that contain locally attached receivers. If there are receivers attached to other EVPN PEs, then whenever (S,G) traffic from an external source matches a (*,G) state, the MEG will create (S,G) state, with the MVPN tunnel as the IIF, the OIF list copied from the (*,G) state, and the SBD IRB interface added to the OIF list. (Please see the discussion in Section 6.1.1 about including the SBD IRB interface in a (*,G) state. The SBD IRB interface is only used in the OIF list for traffic from external sources.) 🌈
Normal MVPN procedures will then result in the MEG getting the (*,G) traffic from all the multicast sources for G that are attached via L3VPN. This traffic arrives on MVPN tunnels. When the MEG pulls the traffic off these tunnels, it does the IP processing. If there are any receivers on a given BD, say BD-R, that are attached via local EVPN ACs, the MEG sends the traffic down its BD-R IRB interface. If there are any other EVPN PEs that are interested in the (*,G) traffic, the MEG sends the traffic down the SBD IRB interface. Normal OISM procedures then distribute the traffic as needed to the other EVPN PEs. 🎉
6.1.2.2. EVPN Sources with MVPN Receivers 🦋
6.1.2.2.1. General Procedures ✨
Okay so picture this: an EVPN tenant system S is blasting IP multicast traffic to group G 📡, and there's a receiver R for that (S,G) traffic chilling on the L3VPN but not in the EVPN Tenant Domain 🙈. (Quick note: in this doc we're assuming the L3VPN-/MVPN-only nodes won't have any special procedures for when the source lives inside an EVPN domain. Not their job 💅)
In this case, an L3VPN PE that can reach R has to send an MVPN C-multicast Join (S,G) route to one of the MEGs attached to the EVPN Tenant Domain 🦋. For that to happen, the L3VPN PE must have imported a VPN-IP route for S (either a host route or a subnet route) from a MEG.
If a MEG figures out that there's a multicast source transmitting on one of its ACs, the MEG SHOULD originate a VPN-IP host route for that source 🌸. This determination SHOULD be made by looking at the IP multicast traffic that arrives on the ACs. (It MAY be made by provisioning, btw 🧚) A MEG SHOULD NOT export a VPN-IP host route for any IP address that isn't known to be a multicast source, unless it has some other reason for exporting that route 🙅♀️. And the VPN-IP host route for a given multicast source MUST be withdrawn if the source goes silent for a configurable period of time, or if it can be determined that the source is no longer reachable via a local AC. Bye bye, ghost source 👻
A MEG SHOULD also originate a VPN-IP subnet route for each of the BDs in the Tenant Domain 🍓
VPN-IP routes exported by a MEG must carry any attributes or Extended Communities that L3VPN and MVPN require 💖. In particular, a VPN-IP route exported by a MEG must carry a VRF Route Import Extended Community that matches the IP-VRF it was imported from, plus a Source AS Extended Community. Main character requirements, literally 👑
So as a result, if S is attached to a MEG, the L3VPN nodes will direct their MVPN C-multicast Join routes to that MEG. Normal MVPN procedures will then deliver the traffic to the L3VPN nodes 🎀. The Layer 3 multicast state for (S,G) will have the MVPN tunnel on its OIF list, and the IIF will be the IRB interface leading to the BD containing S. Chef's kiss 🤌
If S is not attached to a MEG, the L3VPN nodes will direct their C-multicast Join routes to whichever MEG appears to be on the best route to S's subnet 🗺️. When that MEG gets the C-multicast Join, it will originate an EVPN SMET route for (S,G). So the MEG will receive the (S,G) traffic at Layer 2 via the OISM procedures 🧠. That traffic gets sent up the appropriate IRB interface, and the Layer 3 MVPN procedures make sure it reaches the L3VPN nodes that asked for it. The Layer 3 multicast state for (S,G) will have the MVPN tunnel in the OIF list, and the IIF will be one of the following:
- If S belongs to a BD that is attached to the MEG, the IIF will be the IRB interface to that BD.
- Otherwise, the IIF will be the SBD IRB interface.
Note that this works even if S is attached to a non-OISM PE, per the procedures of Section 5. Iykyk, it's giving flexibility 💜
6.1.2.2.2. Any-Source Multicast (ASM) Groups 🌈
Okay, so suppose the MEG SBD-DR learns that one of the PEs in its Tenant Domain is interested in (*,G) traffic, where G is an ASM group 👀. If there are no tenant multicast routers, the MEG SBD-DR SHOULD perform the First Hop Router (FHR) functionality for group G on behalf of the Tenant Domain, as described in [RFC7761]. This means the MEG SBD-DR must know who the RP is for each group, must send Register messages to the RP, etc. That's a lot of responsibility, bestie 😭
If the MEG SBD-DR is going to be the FHR for the Tenant Domain, it must see all the multicast traffic that's sourced from within the domain and destined to an ASM group address. The MEG can make sure of this by originating an SBD-SMET route for (*,*).
(As a possible optimization, an SBD-SMET route for (*, any ASM group) may be defined in a separate document 📝)
In some deployment scenarios, it may be preferred that the MEG that receives the (S,G) traffic over an AC be the one providing the FHR functionality 🦋. This behavior is OPTIONAL. If this option is used, it MUST be ensured that the MEG DR does not provide the FHR functionality for (S,G) traffic that is attached to another MEG. FHR functionality for (S,G) traffic from a particular source S MUST be provided by only a single router. One boss per source, no drama 👑
Other deployment scenarios are also possible. For example, one might want to configure the MEGs themselves to be RPs 🏰. In that case, the RPs would have to exchange information with each other about which sources are active. The method for exchanging that information is outside the scope of this document 🤷♀️
6.1.2.2.3. Source on Multihomed Segment 🫶
Suppose S is attached to a segment that is all-active multihomed to PE1 and PE2 🧩. If S is transmitting to two groups, say G1 and G2, it's possible that PE1 receives the (S,G1) traffic from S, while PE2 receives the (S,G2) traffic from S. Plot twist 😳
This creates an issue for MVPN/EVPN interworking, because there's no way to make L3VPN/MVPN nodes pick PE1 as the ingress PE for (S,G1) traffic while picking PE2 as the ingress PE for (S,G2) traffic. Rude 💀
However, the following procedure makes sure the IP multicast traffic still flows, even if the L3VPN/MVPN nodes pick the "wrong" EVPN PE as the Upstream PE for, e.g., the (S,G1) traffic 🙌
Suppose S is on an Ethernet segment, belonging to BD1, that is multihomed to both PE1 and PE2, where PE1 is a MEG. And suppose IP multicast traffic from S to G travels over the AC that attaches the segment to PE2. If PE1 receives a C-multicast Source Tree Join (S,G) route, it MUST originate a SMET route for (S,G). Normal OISM procedures will then cause PE2 to send the (S,G) traffic to PE1 on an EVPN IP multicast tunnel. Normal OISM procedures will also cause PE1 to send the (S,G) traffic up its BD1 IRB interface. Then normal MVPN procedures cause PE1 to forward the traffic on an MVPN tunnel. The routing isn't optimal in this case, but the traffic does flow correctly, and that's the vibe 🌸
6.1.2.3. Obtaining Optimal Routing of Traffic between MVPN and EVPN 🗺️
The routing of IP multicast traffic between MVPN nodes and EVPN nodes will be optimal as long as there's a MEG along the optimal route. There are various deployment strategies that can get you optimal routing between MVPN and EVPN 💖
In one such scenario, a Tenant Domain has a small number of strategically placed MEGs. For example, a data center might have a few MEGs that connect it to a wide-area network. Then the optimal route into or out of the data center would go through those MEGs 🏢
In this scenario, the MEGs don't need to originate VPN-IP host routes for the multicast sources. They only need to originate VPN-IP subnet routes. The internal structure of the EVPN is completely hidden from the MVPN node, so EVPN actions like MAC Mobility and Mass Withdrawal [RFC7432] have zero impact on the MVPN control plane 🙈✨
While this deployment scenario gives the most optimal routing and has the least impact on the installed base of MVPN nodes, it does complicate network planning considerations 📋
Another way to get close-to-optimal routing is to turn each EVPN PE into a MEG 🧚. Then routing of MVPN-to-EVPN traffic is optimal. However, routing of EVPN-to-MVPN traffic is not guaranteed to be optimal when a source host is on a multihomed Ethernet segment (as discussed in Section 6.1.2.2.) 😬
The obvious disadvantage of this method is that it requires every EVPN PE to be a MEG 💅
The procedures specified in this document let an operator add MEG functionality to any subset of its EVPN OISM PEs. That means an operator can make whatever trade-offs they think are right between optimal routing and MEG deployment 🎀
6.1.2.4. Selecting the MEG SBD-DR 👑
Every PE that is eligible for selection as the MEG SBD-DR originates an SBD-IMET route. As stated in Section 5, these SBD-IMET routes carry a Multicast Flags EC with the MEG flag set 🚩
These SBD-IMET routes SHOULD also carry a DF Election EC. The DF Election EC and its use are specified in [RFC8584]. When the route is originated, the AC-DF bit in the DF Election EC SHOULD be set to zero. This bit isn't used when selecting a MEG SBD-DR, i.e., it MUST be ignored by the receiver of an SBD-IMET route. Ignore that bit, periodt 🙅♀️
In the context of a given Tenant Domain, the MEGs of that domain perform the following procedure to select the MEG SBD-DR 🦋:
- From the set of received SBD-IMET routes for the given Tenant Domain, determine the candidate set of PEs that support MEG functionality for that domain.
- Select a DF election algorithm as specified in [RFC8584]. Some of the possible algorithms can be found, e.g., in [RFC7432], [RFC8584], and [EVPN-DF].
- Apply the DF election algorithm (see [RFC8584]) to the candidate set of PEs. The winner becomes the MEG SBD-DR 🏆
Note that if a given PE supports IPMG (Section 6.1.2) or PEG (Section 6.1.4) functionality as well as MEG functionality, its SBD-IMET routes carry only one DF Election EC 🌈
6.1.3. Interworking with Global Table Multicast 🌐
If multicast service to outside sources and/or receivers is provided via the BGP-based Global Table Multicast (GTM) procedures of [RFC7716], the procedures of Section 6.1.2 can easily be adapted for EVPN/GTM interworking. The way to adapt the MVPN procedures to GTM is explained in [RFC7716] 💫
6.1.4. Interworking with PIM 🧚♀️
As discussed, there may be receivers in an EVPN Tenant Domain that are interested in multicast flows whose sources are outside the EVPN Tenant Domain. Or there may be receivers outside an EVPN Tenant Domain that are interested in multicast flows whose sources are inside the Tenant Domain 👀
If the outside sources and/or receivers are part of an MVPN, see the procedures for interworking covered in Section 6.1.2 💜
There are also cases where an external source or receiver is attached via IP and the Layer 3 multicast routing is done via PIM. In this case, the interworking between the PIM domain and the EVPN Tenant Domain is done at L3 Gateways that perform PIM/EVPN Gateway (PEG) functionality. A PEG is very similar to a MEG, except that its Layer 3 multicast routing is done via PIM rather than via BGP 🍓
If external sources or receivers for a given group are attached to a PEG via a Layer 3 interface, that interface should be treated as a VRF interface attached to the Tenant Domain's L3VPN VRF. The Layer 3 multicast routing instance for that Tenant Domain will either run PIM on the VRF interface or listen for IGMP/MLD messages on that interface. If the external receiver is attached elsewhere on an IP network, the PE has to enable PIM on its interfaces to the backbone network. In both cases, the PE needs to perform PEG functionality, and its IMET routes must carry the Multicast Flags EC with the PEG flag set 🔥
For each BD on which there is a multicast source or receiver, one of the PEGs will become the PEG DR. DR selection can be done using the same procedures specified in Section 6.1.2.4, except with PEG substituted for MEG 🫶
As long as there are no tenant multicast routers within the EVPN Tenant Domain, the PEGs do not need to run PIM on their IRB interfaces 🌸
6.1.4.1. Source Inside EVPN Domain
okay so, if a PEG gets a PIM Join (S,G) from outside the EVPN Tenant Domain 💅, it might need to create (S,G) state 🫶. The PE has to figure out whether S is inside the Tenant Domain. If S is not within the EVPN Tenant Domain, the PE just does normal Layer 3 multicast routing 🌸. If S is within the EVPN Tenant Domain, the IIF of the (S,G) state gets set like this:
- If S is on a BD that is attached to the PE, the IIF is the PE's IRB interface to that BD.
- If S is not on a BD that is attached to the PE, the IIF is the PE's IRB interface to the SBD.
When the PE creates that (S,G) state, it MUST originate (if it hasn't already) an SBD-SMET route for (S,G) ✨. This makes it pull the (S,G) traffic via Layer 2 🦋. When the traffic arrives over an EVPN tunnel, it gets sent up an IRB interface, where Layer 3 multicast routing decides what happens to the packet 👑. The SBD-SMET route gets withdrawn when the (S,G) state no longer exists (unless there's some other reason for not withdrawing it) 🎀.
If there are no tenant multicast routers within the EVPN Tenant Domain, there can't be an RP in the Tenant Domain, so a PEG doesn't have to handle externally arriving PIM Join (*,G) messages 🧚.
The PEG DR for a particular BD MUST act as the a First Hop Router for that BD 💖. It will examine all (S,G) traffic on the BD, and whenever G is an ASM group, the PEG DR will send Register messages to the RP for G. That means the PEG DR will need to pull all the (S,G) traffic originating on a given BD by originating a SMET (,) route for that BD. If a PEG DR is the DR for all the BDs, it SHOULD originate just an SBD-SMET (,) route rather than a SMET (,) route for each BD 🍓.
The rules for exporting IP routes to multicast sources are the same as those for MEGs in Section 6.1.2.2 🌈, except that the exported routes are IP routes rather than VPN-IP routes, and you don't need to attach the VRF Route Import EC or the Source AS EC.
When a source is on a multihomed segment, the same issue from Section 6.1.2.2.3 shows up 🙈. Picture S on an Ethernet segment, belonging to BD1, multihomed to both PE1 and PE2, where PE1 is a PEG. Now say IP multicast traffic from S to G travels over the AC that attaches the segment to PE2. If PE1 receives an external PIM Join (S,G) route, it MUST originate a SMET route for (S,G) 🔥. Normal OISM procedures make PE2 send the (S,G) traffic to PE1 on an EVPN IP multicast tunnel. Normal OISM procedures also make PE1 send the (S,G) traffic up its BD1 IRB interface. Then normal PIM procedures make PE1 forward the traffic along a PIM tree. The routing isn't optimal in this case, but the traffic flows correctly 💜.
6.1.4.2. Source Outside EVPN Domain
By means of normal OISM procedures, a PEG learns whether there are receivers in the Tenant Domain that are interested in (,G) or (S,G) traffic 👀. The PEG must determine whether S (or the RP for G) is outside the EVPN Tenant Domain. If so, and if there's a receiver on BD1 interested in that traffic, the PEG DR for BD1 is responsible for originating a PIM Join (S,G) or Join (,G) control message 🌸.
An alternative would be to let any PEG that's directly attached to a receiver originate the PIM Joins 🦋. Then the PEG DR would only have to originate PIM Joins on behalf of receivers that aren't attached to a PEG. Heads up though: if this is done, it is necessary for the PEGs to run PIM on all their IRB interfaces so that the PIM Assert procedures can be used to prevent duplicate delivery to a given BD ⚠️.
The IIF for the Layer 3 (S,G) or (,G) state is determined by normal PIM procedures 💅. If a receiver is on BD1, and the PEG DR is attached to BD1, its IRB interface to BD1 is added to the OIF list. This makes sure any receivers locally attached to the PEG DR receive the traffic 🫶. If there are receivers attached to other EVPN PEs, then whenever (S,G) traffic from an external source matches a (,G) state, the PEG will create (S,G) state. The IIF gets set to whatever external interface the traffic is expected to arrive on (copied from the (,G) state), the OIF list is copied from the (,G) state, and the SBD IRB interface is added to the OIF list 🧚.
6.2. Interworking with PIM via an External PIM Router
Section 6.1 describes how to use an OISM PE router as the gateway to a non-EVPN multicast domain when the EVPN Tenant Domain isn't being used as an intermediate transit network for multicast 🌐. An alternative approach is to have one or more external PIM routers (perhaps operated by a tenant) on one of the BDs of the Tenant Domain. We will refer to this BD as the "gateway BD" 💜.
In this model:
- The EVPN Tenant Domain is treated as a stub network attached to the external PIM routers 🎀.
- The external PIM routers follow normal PIM procedures and provide the FHR and LHR functionality for the entire Tenant Domain 👑.
- The OISM PEs do not run PIM 🙅♀️.
- There MUST NOT be more than one gateway BD 🛑.
- If an OISM PE not attached to the gateway BD has interest in a given multicast flow, it conveys that interest, following normal OISM procedures, by originating an SBD-SMET route for that flow 🦋.
- If a PE attached to the gateway BD receives an SBD-SMET, it may need to generate and transmit a corresponding IGMP/MLD Join on one or more of its ACs. (Procedures for generating an IGMP/MLD Join as a result of receiving a SMET route are given in [RFC9251].) The PE MUST know which BD is the gateway BD and MUST NOT transmit an IGMP/MLD Join to any other BDs 🍓. Furthermore, even if a particular AC is part of that BD, the PE SHOULD NOT transmit an IGMP/MLD Join on that AC unless there is an external PIM router attached via that AC 🌸.
As a result, IGMP/MLD messages will be received by the external PIM routers on the gateway BD, and those external PIM routers will send PIM Join messages externally as required 📡. Traffic for the given multicast flow will then be received by one of the external PIM routers, and that router will forward the traffic to the gateway BD 💖.
The normal OISM procedures will then cause the given multicast flow to be tunneled to any PEs of the EVPN Tenant Domain that have interest in the flow 🌈. PEs attached to the gateway BD will see the flow as originating from the gateway BD, and other PEs will see the flow as originating from the SBD 🧚.
- An OISM PE attached to a gateway BD MUST set its Layer 2 multicast state to indicate that each AC to the gateway BD has interest in all multicast flows. It MUST also originate a SMET route for (,). The procedures for originating SMET routes are discussed in Section 2.5 ✨.
This makes the OISM PEs attached to the gateway BD receive all the IP multicast traffic sourced within the EVPN Tenant Domain and transmit that traffic to the gateway BD, where the external PIM routers receive it 🔥. This lets the external PIM routers do FHR functions on behalf of the entire Tenant Domain. (Of course, if the gateway BD has a multihomed segment, only the PE that is the DF for that segment will transmit the multicast traffic to the segment 💅.)
7. Using an EVPN Tenant Domain as an Intermediate (Transit) Network for Multicast Traffic ✨
okay so this section is about the situation where one or more BDs in an EVPN Tenant Domain are carrying IP multicast traffic even though the source and at least one receiver are not part of the Tenant Domain 🫶. so the BDs are basically just middle links in a bigger multicast tree that PIM built 🌈. the Tenant Domain is the middle girl and she's doing her job 💅
We define a "tenant multicast router" as a multicast router, running PIM, that:
- is attached to one or more BDs of the Tenant Domain but
- is not an EVPN PE router.
so for an EVPN Tenant Domain to work as a transit network for IP multicast, one or more of its BDs must have tenant multicast routers, and an OISM PE attached to that BD MUST be provisioned to enable PIM on its IRB interface to that BD 🔥. this is still true even if none of the tenant routers is on a segment attached to the PE 🙈. and then ALL the OISM PEs (even the ones not attached to a BD with tenant multicast routers) MUST be provisioned to enable PIM on their SBD IRB interfaces 💖.
If PIM is enabled on a particular BD, the DR selection procedure of Section 6.1.2.4 MUST be replaced by the normal PIM DR Election procedure of [RFC7761] 👑. ngl this means one of the tenant routers could get picked as the DR instead of one of the OISM PEs 😭. when that happens, First Hop Router and Last Hop Router functionality will not be performed by any of the EVPN PEs 🦋.
A PIM control message on a particular BD is treated as a link-local multicast message, so it gets sent transparently from PE to PE via the BUM tunnel for that BD 🧚. this is true whether the control message came in from an AC or from the local Layer 3 routing instance via an IRB interface 🍓.
A PIM Join/Prune message has three fields that matter here:
Upstream NeighborGroup Address (G)Source Address (S), omitted in the case of (*,G) Join/Prune messages
we'll usually talk about a PIM Join as either a Join (S,G) or a Join (,G), and we'll use "Join (X,G)" to mean either one 🌸. in a Join (X,G), "X" means "S" for (S,G) and "G's RP" for (,G) ✨.
Let's say BD1 has two tenant multicast routers, C1 and C2 🎀. C1 is on a segment attached to PE1, and C2 is on a segment attached to PE2. when C1 sends a PIM Join (X,G) to BD1, the Upstream Neighbor field might be set to PE1, PE2, or C2 💜. C1 picks the Upstream Neighbor based on its unicast routing. usually it picks the PIM router on BD1 that's closest to X (according to unicast routing) 🧠. and that won't necessarily be PE1 🙈. PE1 might not even be visible to the unicast routing algorithm the tenant routers use, and even if it is, it probably isn't the closest one to X 💀. so we need to think about two cases:
- C1 sends a PIM Join (X,G) to BD1, with PE1 as the Upstream Neighbor.
PE1's PIM routing instance receives the Join on the BD1 IRB interface 🦋. if X is not within the Tenant Domain, PE1 handles the Join with normal PIM procedures. that usually makes PE1 pick an Upstream Neighbor and send it a Join (X,G) ✨.
if X is within the Tenant Domain but is attached to some other PE, PE1 sends an SBD-SMET route for (X,G) (if it hasn't already) 🌈. the IIF of the Layer 3 (X,G) state is the SBD IRB interface, and the OIF list includes the IRB interface to BD1 💅.
the SBD-SMET route pulls the (X,G) traffic to PE1, and the (X,G) state then forwards that traffic to C1 👑.
if X is within the Tenant Domain and is attached to PE1 itself, no SBD-SMET route is sent 🫶. the IIF of the Layer 3 (X,G) state is the IRB interface to X's BD, and the OIF list includes the IRB interface to BD1 🍓.
- C1 sends a PIM Join (X,G) to BD1, with either PE2 or C2 as the Upstream Neighbor.
PE1's PIM routing instance receives the Join on the BD1 IRB interface 🧚. if neither X nor the Upstream Neighbor is within the Tenant Domain, PE1 handles the Join with normal PIM procedures. this will NOT result in PE1 sending a Join (X,G) 🙅♀️.
if either X or the Upstream Neighbor is within the Tenant Domain, PE1 sends an SBD-SMET route for (X,G) (if it hasn't already) 💖. the IIF of the Layer 3 (X,G) state is the SBD IRB interface, and the OIF list includes the IRB interface to BD1 🔥.
the SBD-SMET route pulls the (X,G) traffic to PE1, and the (X,G) state forwards that traffic to C1 🦋.
8. IANA Considerations 🌸
okay IANA has assigned new flags in the "Multicast Flags Extended Community" registry, which lives under the "Border Gateway Protocol (BGP) Extended Communities" registry. here's the lineup 💅
+=====+================+===========+===================+
| Bit | Name | Reference | Change Controller |
+=====+================+===========+===================+
| 7 | OISM SBD | RFC 9625 | IETF |
+-----+----------------+-----------+-------------------+
| 9 | IPMG | RFC 9625 | IETF |
+-----+----------------+-----------+-------------------+
| 10 | MEG | RFC 9625 | IETF |
+-----+----------------+-----------+-------------------+
| 11 | PEG | RFC 9625 | IETF |
+-----+----------------+-----------+-------------------+
| 12 | OISM-supported | RFC 9625 | IETF |
+-----+----------------+-----------+-------------------+Table 1: Multicast Flags Extended Community Registry ✨
9. Security Considerations 💀
okay this part is serious, bestie 😭. this document uses protocols and procedures defined in the normative references, so it inherits the security considerations of those references 🫶.
this document adds flags or Extended Communities (ECs) to a bunch of BGP routes to signal that particular nodes support the OISM, IPMG, MEG, and/or PEG functionalities defined here 🔥. if those flags and/or ECs get added, removed, or modified incorrectly, the procedures here will malfunction, and then loss or diversion of data traffic is possible 🚨. implementations should provide tools that make it easy to debug configuration mistakes that cause incorrect information to be signaled 🧰.
the interworking with non-OISM networks described in Sections 5 and 6 needs gateway functions in multiple redundant PEs, and one of them is elected as Designated Forwarder for a given BD (or SBD) 👑. the election of the MEG or PEG DR, as well as the IPMG Designated Forwarder, uses the Designated Forwarder election procedures of [RFC8584] 🌈. an attacker with access to one of these gateways may influence that election and therefore modify how multicast traffic gets forwarded between the OISM network and the external domain 😱. the operator should be especially careful about protecting these gateways, and make sure the management interfaces used to access them are only allowed for authorized operators 🔐.
this document also introduces the concept of per-Tenant-Domain dissemination for SMET routes, as opposed to the per-BD distribution in [RFC9251] 🦋. that means a SMET route triggered by an IGMP/MLD Join received in BD-1 on PE1 needs to be distributed and imported by ALL PEs of the Tenant Domain, even the ones not attached to BD-1 💅. so an attacker with access to only one BD on one PE of the Tenant Domain might force SMET route advertisements and hit the resources of every PE in the Tenant Domain, instead of only the PEs of that particular BD (like in [RFC9251]) 🙀. implementations should provide ways to filter or control the client IGMP/MLD reports that are received from the attached hosts 🛡️✨.
10. References 📚
10.1. Normative References
okay bestie, these are the receipts 🧾✨ every single one of these is something the spec leans on, so here they are in full
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001,
<https://www.rfc-editor.org/info/rfc3032>.
[RFC3376] Cain, B., Deering, S., Kouvelas, I., Fenner, B., and A.
Thyagarajan, "Internet Group Management Protocol, Version
3", RFC 3376, DOI 10.17487/RFC3376, October 2002,
<https://www.rfc-editor.org/info/rfc3376>.
[RFC3810] Vida, R., Ed. and L. Costa, Ed., "Multicast Listener
Discovery Version 2 (MLDv2) for IPv6", RFC 3810,
DOI 10.17487/RFC3810, June 2004,
<https://www.rfc-editor.org/info/rfc3810>.
[RFC4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
Communities Attribute", RFC 4360, DOI 10.17487/RFC4360,
February 2006, <https://www.rfc-editor.org/info/rfc4360>.
[RFC6625] Rosen, E., Ed., Rekhter, Y., Ed., Hendrickx, W., and R.
Qiu, "Wildcards in Multicast VPN Auto-Discovery Routes",
RFC 6625, DOI 10.17487/RFC6625, May 2012,
<https://www.rfc-editor.org/info/rfc6625>.
[RFC7153] Rosen, E. and Y. Rekhter, "IANA Registries for BGP
Extended Communities", RFC 7153, DOI 10.17487/RFC7153,
March 2014, <https://www.rfc-editor.org/info/rfc7153>.
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <https://www.rfc-editor.org/info/rfc7432>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8584] Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake,
J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet
VPN Designated Forwarder Election Extensibility",
RFC 8584, DOI 10.17487/RFC8584, April 2019,
<https://www.rfc-editor.org/info/rfc8584>.
[RFC9135] Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
Rabadan, "Integrated Routing and Bridging in Ethernet VPN
(EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
<https://www.rfc-editor.org/info/rfc9135>.
[RFC9136] Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
(EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
<https://www.rfc-editor.org/info/rfc9136>.
[RFC9251] Sajassi, A., Thoria, S., Mishra, M., Patel, K., Drake, J.,
and W. Lin, "Internet Group Management Protocol (IGMP) and
Multicast Listener Discovery (MLD) Proxies for Ethernet
VPN (EVPN)", RFC 9251, DOI 10.17487/RFC9251, June 2022,
<https://www.rfc-editor.org/info/rfc9251>.
[RFC9572] Zhang, Z., Lin, W., Rabadan, J., Patel, K., and A.
Sajassi, "Updates to EVPN Broadcast, Unknown Unicast, or
Multicast (BUM) Procedures", RFC 9572,
DOI 10.17487/RFC9572, May 2024,
<https://www.rfc-editor.org/info/rfc9572>.
[RFC9574] Rabadan, J., Ed., Sathappan, S., Lin, W., Katiyar, M., and
A. Sajassi, "Optimized Ingress Replication Solution for
Ethernet VPNs (EVPNs)", RFC 9574, DOI 10.17487/RFC9574,
May 2024, <https://www.rfc-editor.org/info/rfc9574>.10.2. Informative References
these ones are the extra reading 📚 nice to have, not strictly required to get the vibe
[EVPN-DF] Rabadan, J., Sathappan, S., Lin, W., Drake, J., and A.
Sajassi, "Preference-based EVPN DF Election", Work in
Progress, Internet-Draft, draft-ietf-bess-evpn-pref-df-13,
9 October 2023, <https://datatracker.ietf.org/doc/html/
draft-ietf-bess-evpn-pref-df-13>.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, <https://www.rfc-editor.org/info/rfc4364>.
[RFC4541] Christensen, M., Kimball, K., and F. Solensky,
"Considerations for Internet Group Management Protocol
(IGMP) and Multicast Listener Discovery (MLD) Snooping
Switches", RFC 4541, DOI 10.17487/RFC4541, May 2006,
<https://www.rfc-editor.org/info/rfc4541>.
[RFC6513] Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/
BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February
2012, <https://www.rfc-editor.org/info/rfc6513>.
[RFC6514] Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
Encodings and Procedures for Multicast in MPLS/BGP IP
VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012,
<https://www.rfc-editor.org/info/rfc6514>.
[RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
Patel, "Revised Error Handling for BGP UPDATE Messages",
RFC 7606, DOI 10.17487/RFC7606, August 2015,
<https://www.rfc-editor.org/info/rfc7606>.
[RFC7716] Zhang, J., Giuliano, L., Rosen, E., Ed., Subramanian, K.,
and D. Pacella, "Global Table Multicast with BGP Multicast
VPN (BGP-MVPN) Procedures", RFC 7716,
DOI 10.17487/RFC7716, December 2015,
<https://www.rfc-editor.org/info/rfc7716>.
[RFC7761] Fenner, B., Handley, M., Holbrook, H., Kouvelas, I.,
Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent
Multicast - Sparse Mode (PIM-SM): Protocol Specification
(Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March
2016, <https://www.rfc-editor.org/info/rfc7761>.
[RFC8296] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
Tantsura, J., Aldrin, S., and I. Meilik, "Encapsulation
for Bit Index Explicit Replication (BIER) in MPLS and Non-
MPLS Networks", RFC 8296, DOI 10.17487/RFC8296, January
2018, <https://www.rfc-editor.org/info/rfc8296>.
[RFC9624] Zhang, Z., Przygienda, T., Sajassi, A., and J. Rabadan,
"EVPN Broadcast, Unknown Unicast, or Multicast (BUM) Using
Bit Index Explicit Replication (BIER)", RFC 9624,
DOI 10.17487/RFC9624, August 2024,
<https://www.rfc-editor.org/info/rfc9624>.Appendix A. Integrated Routing and Bridging ✨
okay so this appendix is a mini tutorial on how routing and bridging hang out together 🎀 first we look at a model where the bridging and the routing live in totally separate boxes, and then we look at the model from [RFC9135], where ONE device does both. the body of this doc just assumes you're in that second model, so pay attention 💅
Figure 2 is the model where a router only routes and has zero L2 bridging skills 💀 we've got two LANs, LAN1 and LAN2. LAN1 is run by switch1 and LAN2 is run by switch2. The router has an interface called lan1 that plugs into LAN1 (through switch1) and an interface called lan2 that plugs into LAN2 (through switch2). Each interface is set up as an IP interface with an IP address and a subnet mask.
+-------+ +--------+ +-------+
| | lan1| |lan2 | |
H1 -----+Switch1+--------+ Router1+--------+Switch2+------H3
| | | | | |
H2 -----| | | | | |
+-------+ +--------+ +-------+
|_________________| |__________________|
LAN1 LAN2
Figure 2: Conventional Router with LAN InterfacesIP traffic (unicast or multicast) that stays inside a single subnet never even reaches the router 🙈 so if H1 sends an Ethernet frame with H2's MAC address in the Ethernet Destination Address field, the frame goes from H1 to Switch1 to H2 and never touches the router. Since no router ever sees it, the IP datagram inside stays totally untouched, so its TTL is not decremented. The Ethernet Source and Destination MAC addresses don't change either.
Now if H1 wants to send a unicast IP datagram to H3, which is on a different subnet, H1 has to be configured with the IP address of a default router. Let's say H1 has Router1's IP address as its default router address. H1 compares H3's IP address against its own IP address and IP subnet mask and realizes H3 is on a different subnet, so the packet has to be routed 🧠 H1 uses ARP to map Router1's IP address to a MAC address on LAN1. Then it wraps the datagram in an Ethernet frame using Router1's MAC address as the destination MAC address and sends the frame to Router1.
Router1 receives the frame over its lan1 interface. It sees the frame is addressed to itself, so it strips off the Ethernet encapsulation and processes the IP datagram. The datagram isn't addressed to Router1, so it has to be forwarded onward. Router1 looks up the datagram's IP Destination Address and finds that the destination (H3) can be reached via its lan2 interface. Now it does the full IP processing: it decrements the IP TTL, adjusts the IP header checksum (if present), may fragment the packet as necessary, etc. Then the datagram (or its fragments) gets wrapped in an Ethernet header, with Router1's MAC address on LAN2 as the MAC Source Address and H3's MAC address on LAN2 (which Router1 figures out via ARP) as the Destination MAC Address. Finally, the packet goes out on the lan2 interface 🚀
If H1 has an IP multicast datagram to send (aka an IP datagram whose Destination Address field is an IP Multicast Address), it wraps it in an Ethernet frame whose Destination MAC Address is computed from the IP Destination Address 🦋
If H2 is a receiver for that multicast address, H2 gets a copy of the frame from H1, unchanged. The MAC Source Address in the Ethernet encapsulation doesn't change, the IP TTL doesn't get decremented, and so on, bestie 🌸
If H3 is a receiver for that multicast address, the datagram has to be routed to H3. For that to happen, Router1 must be configured as a multicast router, and it must accept traffic sent to Ethernet multicast addresses. Router1 receives H1's multicast frame on its lan1 interface, strips the Ethernet encapsulation, and decides how to dispatch the IP datagram based on its multicast forwarding states. If Router1 knows there's a receiver for the multicast datagram on LAN2, it makes a copy of the datagram, decrements the TTL (and does any other necessary IP processing), and then wraps the datagram in the Ethernet frame for LAN2. The MAC Source Address for this frame is Router1's MAC Source Address on LAN2. The Destination MAC Address is computed from the IP Destination Address. Finally, the frame is sent out on Router1's LAN2 interface 👑
Figure 3 shows an integrated router/bridge that supports the routing/bridging integration model of [RFC9135]. It's the model this whole document is built on, so get comfy with it 🌈
+------------------------------------------+
| Integrated Router/Bridge |
+-------+ +--------+ +-------+
| | IRB1| L3 |IRB2 | |
H1 -----+ BD1 +--------+Routing +--------+ BD2 +------H3
| | |Instance| | |
H2 -----| | | | | |
+-------+ +--------+ +-------+
|___________________| |____________________|
LAN1 LAN2
Figure 3: Integrated Router/Bridgeokay so look at Figure 3 🖼️✨ a single device can have one or more L3 Routing Instances 💅 The routing/forwarding tables of a given routing instance is known as an IP-VRF [RFC9135]. In the EVPN world it's giving to think of each routing instance as the routing for one particular tenant 🎀 Each IP-VRF is attached to one or more interfaces.
When several EVPN PEs have a routing instance of the same Tenant Domain, those PEs advertise IP routes to the attached hosts 🦋 This is done as specified in [RFC9135].
The integrated router/bridge in Figure 3 also hooks up to a bunch of Broadcast Domains (BDs) 🌈 Each BD does the same jobs that the bridges in Figure 2 do. To the L3 routing instance, each BD looks like a LAN. The interface that attaches a particular BD to a particular IP-VRF is called an "IRB interface" 💜 From the L3 routing point of view, each BD is a subnet, so each IRB interface gets configured with a MAC address (that's the router's MAC address on that LAN), plus an IP address and subnet mask. Main character energy 👑
The integrated router/bridge in Figure 3 may have multiple ACs to each BD 🌸 These ACs are only visible to the bridging function, NOT to the routing instance. To the L3 routing instance there is just one interface to each BD 🍓
If the L3 routing instance represents the IP routing of a particular tenant, then the BDs attached to that routing instance are BDs belonging to that same tenant 🧚
Bridging and routing now work exactly like they did for Figure 2, except BD1 takes the place of Switch1, BD2 takes the place of Switch2, interface IRB1 takes the place of interface lan1, and interface IRB2 takes the place of interface lan2 🫶
It is important to understand that an IRB interface connects an L3 routing instance to a BD, NOT to a MAC-VRF (see [RFC7432] for the definition of MAC-VRF) 🙈 A MAC-VRF can hold several BDs, as long as no MAC address shows up in more than one BD. From the L3 routing instance's perspective, each individual BD is its own IP subnet. Whether or not each BD has its own MAC-VRF is irrelevant to the L3 routing instance, so it doesn't care 💅
Figure 4 shows IRB when a pair of BDs (subnets) are attached to two different PE routers 💖 In this example, each BD has two segments, and one segment of each BD is attached to one PE router.
+------------------------------------------+
| Integrated Router/Bridges |
+-------+ +--------+ +-------+
| | IRB1| |IRB2 | |
H1 -----+ BD1 +--------+ PE1 +--------+ BD2 +------H3
|(Seg-1)| |(L3 Rtg)| |(Seg-1)|
H2 -----| | | | | |
+-------+ +--------+ +-------+
|___________________| | |____________________|
LAN1 | LAN2
|
|
+-------+ +--------+ +-------+
| | IRB1| |IRB2 | |
H4 -----+ BD1 +--------+ PE2 +--------+ BD2 +------H5
|(Seg-2)| |(L3 Rtg)| |(Seg-2)|
| | | | | |
+-------+ +--------+ +-------+
Figure 4: Integrated Router/Bridges with Distributed Subnetokay now the fun part 😭 If H1 wants to send an IP packet to H4, it checks its IP address and subnet mask and sees that H4 is on the same subnet as H1. Even though H1 and H4 aren't attached to the same PE router, EVPN gives Ethernet communication among all hosts on the same BD 🌈 So H1 uses ARP to find H4's MAC address and sends an Ethernet frame with H4's MAC address in the Destination MAC Address field. The frame lands at PE1, but since the Destination MAC address is not PE1's MAC address, PE1 decides the frame stays on BD1. So the packet inside the frame is NOT decapsulated and is NOT sent up the IRB interface to PE1's routing instance. Instead, standard EVPN intra-subnet procedures (as detailed in [RFC7432]) deliver the frame to PE2, which then sends it along to H4 🦋
If H1 wants to send an IP packet to H5, it checks and sees that H5 is NOT on the same subnet as H1 🙃 Assuming H1 has been configured with the IP address of PE1 as its default router, H1 sends the packet in an Ethernet frame with PE1's MAC address in the Destination MAC Address field. PE1 receives the frame and notices it's addressed to PE1 itself, so PE1 sends the frame up its IRB1 interface to the L3 routing instance. Appropriate IP processing is done, e.g., TTL decrement. The L3 routing instance figures out that the next hop for H5 is PE2, so the packet gets encapsulated (e.g., in MPLS) and sent across the backbone to PE2's routing instance. PE2 sees that the packet's destination, H5, is on BD2 segment-2, and sends the packet down its IRB2 interface. This means the IP packet gets wrapped in an Ethernet frame with PE2's MAC address (on BD2) in the Source Address field and H5's MAC address in the Destination Address field 💅
Also, if H1 has an IP packet to send to H3, the forwarding is handled entirely within PE1 🧚 PE1's routing instance sees the packet arrive on its IRB1 interface, and then it transmits the packet by sending it down its IRB2 interface. Efficient and iconic ✨
Often, all the hosts in a particular Tenant Domain get provisioned with the same default router IP address 🍓 This IP address can be provisioned as an anycast address on all the EVPN PEs attached to that Tenant Domain. So even though every host has the same default router address, the actual default router for a given host will be one of the PEs attached to the same Ethernet segment as that host. This provisioning method makes sure IP packets from a given host get handled by the closest EVPN PE that supports IRB 🎀
In the topology of Figure 4, you could imagine H1 being configured with a default router address that belongs to PE2 but not to PE1 🙈 Inter-subnet routing would still work, but IP packets from H1 to H3 would then take the non-optimal path H1-->PE1-->PE2-->PE1-->H3. Sending traffic along a path where it leaves a router and then comes right back to that same router is sometimes known as "hairpinning" 😩 Similarly, if PE2 supports IRB but PE1 does not, the same non-optimal path from H1 to H3 would have to be followed. To avoid hairpinning, each EVPN PE needs to support IRB 💖
It's worth pointing out how IRB interfaces interact with multicast traffic 🌸 Looking at Figure 4 again, suppose PE1 and PE2 are functioning as IP multicast routers. Also suppose H3 transmits a multicast packet and both H1 and H4 are interested in receiving it. PE1 receives the packet from H3 via its IRB2 interface. The Ethernet encapsulation from BD2 is removed, the IP header processing is done, and then the packet is re-encapsulated for BD1, with PE1's MAC address in the MAC Source Address field. Then the packet is sent down the IRB1 interface. Layer 2 procedures (as defined in [RFC7432]) then deliver a copy of the packet locally to H1 and remotely to H4 🫶
Please be aware that this document modifies the semantics, described in the previous paragraph, of sending/receiving multicast traffic on an IRB interface 🚨 This is explained in Section 1.5.1 and subsequent sections. Don't skip it bestie 🧐
Acknowledgements 🫶
The authors thank Vikram Nagarajan and Princy Elizabeth for their work on Sections 6.2 and 3.2.3.1 💜 The authors also benefited tremendously from discussions with Aldrin Isaac on EVPN multicast optimizations 🌈
Authors' Addresses 💌
Here are the people who made this whole thing happen, with their contact details 🦋
Wen Lin
Juniper Networks, Inc.
10 Technology Park Drive
Westford, MA 01886
United States of America
Email: wlin@juniper.netZhaohui Zhang
Juniper Networks, Inc.
10 Technology Park Drive
Westford, MA 01886
United States of America
Email: zzhang@juniper.netJohn Drake
Juniper Networks, Inc.
1194 N. Mathilda Ave
Sunnyvale, CA 94089
United States of America
Email: jdrake@juniper.netEric C. Rosen (editor)
Juniper Networks, Inc.
10 Technology Park Drive
Westford, MA 01886
United States of America
Email: erosen52@gmail.comJorge Rabadan
Nokia
777 E. Middlefield Road
Mountain View, CA 94043
United States of America
Email: jorge.rabadan@nokia.comAli Sajassi
Cisco Systems
170 West Tasman Drive
San Jose, CA 95134
United States of America
Email: sajassi@cisco.comThat's the whole RFC, bestie 🌈👑 Thanks for hanging in there with me through all 18 parts 🥹💖
📦 where this page comes from
- ietf: rfc9625 2024-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms