RFC 8388

Usage and Applicability of BGP MPLS-Based Ethernet VPN

📅 published May 2018, from ietf

Okay bestie, let's get into it! ✨ This is part 1 of 7, so don't panic, more is coming. 💖

Here's the vibe check on RFC 8388: it's Informational, not a Standards Track spec, and it's published by the IETF with the IESG's approval. Authored by J. Rabadan (Ed.), S. Palislamovic, W. Henderickx (all Nokia), A. Sajassi (Cisco), and J. Uttaro (AT&T). It's dated May 2018, sits in ISSN 2070-1721, and it neither obsoletes nor updates any other RFC. 🎀 Its whole job is to be a friendly guide to deploying BGP MPLS-based Ethernet VPN (EVPN) and to explain the trade-offs. 🦋

Status of This Memo 🌸

Quick reality check: this doc is not an Internet Standards Track specification. It's published for informational purposes only. 💅

It is still a product of the IETF and represents the consensus of the IETF community. 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved document is a candidate for an Internet Standard, though. See Section 2 of RFC 7841 for that. 🤓

For the latest status, errata, and how to give feedback, go to https://www.rfc-editor.org/info/rfc8388. 🔥

Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stood on the date of publication. Please actually read these, bestie, because they spell out your rights and restrictions. 📜 Any Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in that license. 🙈

Abstract ✨

This document looks at how BGP MPLS-based Ethernet VPN (EVPN) gets used and where it fits in a simple, pretty common deployment scenario. 🧚 It walks through the different EVPN procedures using that example, and it covers the benefits and trade-offs of each option. Basically, it's a simplified guide for deploying EVPN networks. 🍓

1. Introduction 🌸

So this doc is the sidekick to [RFC7432]. 👑 It shows how the tech applies in a simple, fairly common deployment scenario, which we meet in Section 3.

After we describe the topology and requirements, Section 4 covers the provisioning model. 🎀

Once we've analyzed the provisioning model, Sections 5, 6, and 7 dig into the control-plane and data-plane procedures for the example scenario. There are two possible disposition/forwarding models: MAC-based and MPLS-based. 🦋 Both can interoperate in the same network, but each one has different trade-offs, and Section 8 breaks those down. 💅

Last but not least, EVPN has some traffic flow optimization tools too. Section 9 covers those in the context of our example scenario. 🔥

2. Terminology 💖

Here's the glossary, so we're all speaking the same language. 🌈

  • VID: VLAN Identifier 🏷️
  • CE: Customer Edge (device) 🏠
  • EVI: EVPN Instance ✨
  • MAC-VRF: A Virtual Routing and Forwarding (VRF) table for Media Access Control (MAC) addresses on a Provider Edge (PE) router. 🧚
  • ES: An Ethernet Segment is a set of links through which a CE is connected to one or more PEs. Each ES is identified by an Ethernet Segment Identifier (ESI) in the control plane. 🔗
  • CE-VIDs: The VLAN Identifier tags being used at CE1, CE2, and CE3 to tag customer traffic sent to the service provider EVPN network. 🎀
  • CE1-MAC, CE2-MAC, and CE3-MAC: The source MAC addresses "behind" each CE, respectively. These MAC addresses can belong to the CEs themselves or to devices connected to the CEs. 💜
  • CE1-IP, CE2-IP, and CE3-IP: The IP addresses associated with the above MAC addresses. 🌸
  • LACP: Link Aggregation Control Protocol 🦋
  • RD: Route Distinguisher 🍓
  • RT: Route Target 🎯
  • PE: Provider Edge (router) 👑
  • AS: Autonomous System 🌐
  • PE-IP: The IP address of a given PE. 💅

3. Use Case Scenario Description and Requirements ✨

Figure 1 is the scenario we'll keep coming back to for the rest of the doc. 🫶

                            +--------------+
                            |              |
          +----+     +----+ |              | +----+   +----+
          | CE1|-----|    | |              | |    |---| CE3|
          +----+    /| PE1| |   IP/MPLS    | | PE3|   +----+
                   / +----+ |   Network    | +----+
                  /         |              |
                 /   +----+ |              |
          +----+/    |    | |              |
          | CE2|-----| PE2| |              |
          +----+     +----+ |              |
                            +--------------+

                     Figure 1: EVPN Use Case Scenario

So here's the cast: three PEs (PE1, PE2, and PE3) and three CEs (CE1, CE2, and CE3). 🎀 The big requirement is that broadcast domains must be extended among all three CEs. 💖

Part 1 of 7 is done, bestie! 🦋 Say "next" when you're ready for part 2. 💅

3.1. Service Requirements

okay so, for this scenario we're assuming the following service requirements 💅

  • Redundancy requirements:
    • CE2 needs multihoming connectivity to PE1 and PE2, and not just for backup 🫶 It also wants more upstream/downstream bandwidth to and from the network, so two links are better than one.
    • Fast convergence. For example, if the link between CE2 and PE1 goes down, a fast convergence mechanism must be supported so that PE3 can immediately send the traffic to PE2, no matter how many services and MAC addresses are affected 🏃‍♀️💨
  • Service interface requirements:
    • The service definition must be flexible when it comes to mapping CE-VIDs to broadcast domains in the core 🎀
    • Three EVI services are required in this example ✨
      • EVI100 uses VLAN-based service interfaces on the three CEs, with a 1:1 VLAN-to-EVI mapping. The CE-VIDs at the three CEs can be the same (for example, VID 100) or different at each CE (for instance, VID 101 in CE1, VID 102 in CE2, and VID 103 in CE3). Either way, one single broadcast domain has to be created for EVI100, so CE-VIDs will need translation at the egress PEs if they aren't consistent across the three CEs. When the same CE-VID is used across all three CEs for EVI100, [RFC7432] calls this the "Unique VLAN" EVPN case. We'll use that term throughout this document too 🌸
      • EVI200 uses VLAN bundle service interfaces on CE1, CE2, and CE3, based on an N:1 VLAN-to-EVI mapping. The operator needs to preconfigure a range of CE-VIDs and their mapping to the EVI, and this mapping should be consistent across all the PEs (no translation is supported). A single broadcast domain is created for the customer, and the customer is responsible for keeping users in different CE-VIDs separated 🧚
      • EVI300 uses VLAN-aware bundling service interfaces on CE1, CE2, and CE3. Just like EVI200, an N:1 VLAN-to-EVI mapping is created at the ingress PEs, but here a separate broadcast domain is required per CE-VID. The CE-VIDs can be different, so CE-VID translation is required 🦋

Note that in Section 4.2.1, only EVI100 is used as an example of VLAN-based service provisioning. In Sections 6.2 and 7.2, 4k VLAN-based EVIs (EVI1 to EVI4k) are used so that the impact of the MAC versus MPLS disposition models in the control plane can be evaluated 🔍 In the same way, EVI200 and EVI300 will be described with a 4k:1 mapping (CE-VIDs-to-EVI mapping) in Sections 6.3, 6.4, 7.3, and 7.4 🌈

  • Broadcast, Unknown Unicast, Multicast (BUM) optimization requirements:
    • The solution must support ingress replication or P2MP MPLS LSPs on a per-EVI service basis. For example, we can use ingress replication for EVI100 and EVI200, assuming those EVIs won't carry much BUM traffic. On the flip side, if EVI300 is presumably carrying a significant amount of multicast traffic, P2MP MPLS LSPs can be used for that service 💜
    • The benefit of ingress replication over P2MP LSPs is that the core routers won't need to maintain any multicast state 👑

3.2. Why EVPN Is Chosen to Address This Use Case

okay bestie, so Virtual Private LAN Service (VPLS) solutions based on [RFC4761], [RFC4762], and [RFC6074] cannot meet the requirements in Section 3, but EVPN can 😭✨

For example:

  • If CE2 has a single CE-VID (or just a few CE-VIDs), the current VPLS multihoming solutions (based on load-balancing per CE-VID or per service) don't give us the optimized link utilization this example needs 💅 EVPN brings the flow-based, load-balancing multihoming solution this scenario calls for, which optimizes the upstream/downstream link utilization between CE2 and PE1-PE2 🔥
  • EVPN has a fast convergence solution that doesn't depend on the CE-VIDs in the multihomed PEs. When the link between CE2 and PE1 fails, PE3 can immediately send traffic to PE2 based on a single notification message sent by PE1. VPLS can't do this 🙈
  • When it comes to service interfaces and mapping to broadcast domains, VPLS might meet the requirements for EVI100 and EVI200, but the VLAN-aware bundling service interfaces required by EVI300 aren't supported by the current VPLS tools 💀

The rest of the document describes how EVPN can meet the service requirements from Section 3, and even optimize the network further by:

  • giving the user an option to reduce (and even suppress) ARP (Address Resolution Protocol) flooding; and
  • supporting ARP termination and inter-subnet forwarding 🦋

4. Provisioning Model

One of the requirements stated in [RFC7209] is ease of provisioning, and it's a whole vibe 🌸 BGP parameters and service context parameters should be auto-provisioned, so that adding a new MAC-VRF to the EVI takes a minimum number of single-sided provisioning touches. Honestly though, this is only possible in a limited number of cases 😅 This section walks through the provisioning tasks for the services from Section 3, i.e., EVI100 (VLAN-based service interfaces), EVI200 (VLAN bundle service interfaces), and EVI300 (VLAN-aware bundling service interfaces) 🎀

4.1. Common Provisioning Tasks

Regardless of the service interface type (VLAN-based, VLAN bundle, or VLAN-aware), the subsections below describe the parameters that need to be provisioned in the three PEs 💖

4.1.1. Non-Service-Specific Parameters

The multihoming function in EVPN requires provisioning some parameters that aren't service specific and that are shared by all the MAC-VRFs on the node using multihoming capabilities. In our use case, these parameters are only provisioned or auto-derived in PE1 and PE2, and they're listed below 🧚

  • Ethernet Segment Identifier (ESI): Only the ESI associated with CE2 needs to be considered in our example. Single-homed CEs like CE1 and CE3 don't need an ESI to be provisioned (the ESI will be coded as zero in the BGP Network Layer Reachability Information (NLRI)) 🍓 In our example, a Link Aggregation Group (LAG) is used between CE2 and PE1-PE2 (since all-active multihoming is a requirement), so the ESI can be auto-derived from the LACP information as described in [RFC7432]. Note that the ESI must be unique across all the PEs in the network, so auto-provisioning the ESI is recommended only when the CEs are managed by the operator. Otherwise, the ESI should be manually provisioned (Type 0, as in [RFC7432]) to avoid potential conflicts ⚠️
  • ES-Import Route Target (ES-Import RT): This is the RT that PE1 and PE2 send along with the ES route. No matter how the ESI is provisioned in PE1 and PE2, the ES-Import RT must always be auto-derived from the 6-byte MAC address portion of the ESI value 💜
  • Ethernet Segment Route Distinguisher (ES RD): This is the RD encoded in the ES route, and it's the Ethernet Auto-Discovery (A-D) route that PE1 and PE2 send for the CE2 ESI. This RD should always be auto-derived from the PE-IP address, as described in [RFC7432] 🌈
  • Multihoming type: The user must be able to provision the multihoming type used in the network. In our use case, the multihoming type is set to all-active for the CE2 ESI. This info is encoded in the ESI Label extended community flags and sent by PE1 and PE2 along with the Ethernet A-D route for the CE2 ESI 👑

On top of that, the same LACP parameters are configured on PE1 and PE2 for the ES, so CE2 can send frames to PE1 and PE2 as though they were one single system 🫶

4.1.2. Service-Specific Parameters

The following parameters must be provisioned in PE1, PE2, and PE3 per EVI service 🎀

  • EVI Identifier: This is the global identifier per EVI that's shared by all the PEs that are part of the EVI. So PE1, PE2, and PE3 get provisioned with EVI100, 200, and 300. The EVI identifier can be associated with (or be the same value as) the EVI default Ethernet Tag (the 4-byte default broadcast domain identifier for the EVI). The Ethernet Tag is only different from zero in the EVPN BGP routes if the service interface type (of the source PE) is a VLAN-aware bundle 🦋
  • EVI Route Distinguisher (EVI RD): This RD is a unique value across all the MAC-VRFs in a PE. Auto-derivation of this RD might be possible depending on the service interface type used in the EVI. The next section gets into the specifics of each service interface type 💅
  • EVI Route Target(s) (EVI RT): One or more RTs can be provisioned per MAC-VRF. The imported and exported RT(s) can be equal or different, just like the RT(s) in IP-VPNs. Auto-derivation of these RT(s) might be possible depending on the service interface type used in the EVI. The next section gets into the specifics of each service interface type 🌸
  • CE-VID and port/LAG binding to EVI identifier or Ethernet Tag: For more information, please see Section 4.2. 🧚

4.2. Service-Interface-Dependent Provisioning Tasks

Depending on the service interface type used in the EVI, a given CE-VID binding provision must be specified 💖

4.2.1. VLAN-Based Service Interface EVI

In our use case, EVI100 is a VLAN-based service interface EVI ✨

EVI100 can be a "unique-VLAN" service if the CE-VID used for this service on CE1, CE2, and CE3 is identical (for example, VID 100). In that case, the VID 100 binding must be provisioned on PE1, PE2, and PE3 for EVI100 and the associated port or LAG. The MAC-VRF RD and RT can be auto-derived from the CE-VID 🎀

  • The auto-derived MAC-VRF RD will be a Type 1 RD, as recommended in [RFC7432], and it's comprised of [PE-IP]:[zero-padded-VID], where [PE-IP] is the IP address of the PE (a loopback address) and [zero-padded-VID] is a 2-byte value where the low-order 12 bits are the VID (VID 100 in our example) and the high-order 4 bits are zero 🌈
  • The auto-derived MAC-VRF RT will be composed of [AS]:[zero-padded-VID], where [AS] is the Autonomous System that the PE belongs to, and [zero-padded-VID] is a 2- or 4-byte value where the low-order 12 bits are the VID (VID 100 in our example) and the high-order bits are zero. Note that auto-deriving the RT implies supporting a basic any-to-any topology in the EVI and using the same import and export RT in the EVI 💜

If EVI100 is not a "unique-VLAN" instance, each individual CE-VID must be configured on each PE, and MAC-VRF RDs and RTs cannot be auto-derived, so they must be provisioned by the user 😤

4.2.2. VLAN Bundle Service Interface EVI

Assuming EVI200 is a VLAN bundle service interface EVI and VIDs 200-250 are assigned to EVI200, the CE-VID bundle 200-250 must be provisioned on PE1, PE2, and PE3. Note that this model does not allow CE-VID translation, so the CEs must use the same CE-VIDs for EVI200 🍓 No auto-derived EVI RDs or EVI RTs are possible here 🙈

4.2.3. VLAN-Aware Bundling Service Interface EVI ✨

okay so if EVI300 is a VLAN-aware bundling service interface EVI, the CE-VID binding to EVI300 does not have to match across all three PEs 💅. Only PE1 and PE2 need to agree, since they're part of the same ES 🫶. For example, PE1 and PE2 could bind CE-VIDs to EVI300 in the range 300-310, while PE3 uses 321-330 🦋. Each individual CE-VID gets its own broadcast domain, and that domain shows up as an Ethernet Tag in the control plane 🌸.

So besides the CE-VID bundle range bound to EVI300 on each PE, the user has to manually provision the association between each individual CE-VID and its matching EVPN Ethernet Tag 😭. No auto-derived EVI RDs/RTs are possible 💀.

5. BGP EVPN NLRI Usage 🌈

[RFC7432] defines four different route types and four different extended communities 🎀. Not every PE in an EVPN network has to generate and process all of them, though 🙈. Table 1 lays out which routes must be exported and imported in the use case covered by this document. "Export" here means the PE is capable of generating and exporting a given route, assuming there are no BGP policies blocking it. "Import" works the same way, meaning the PE can import and process a given route, assuming the right RTs and policies are in place. "N/A" means neither import nor export is required ✨.

            +-----------------+---------------+---------------+
            | BGP EVPN Routes | PE1-PE2       | PE3           |
            +-----------------+---------------+---------------+
            | ES              | Export/Import | N/A           |
            | A-D per ESI     | Export/Import | Import        |
            | A-D per EVI     | Export/Import | Import        |
            | MAC             | Export/Import | Export/Import |
            | Inclusive Mcast | Export/Import | Export/Import |
            +-----------------+---------------+---------------+

Table 1: Base EVPN Routes and Export/Import Actions

PE3 is required to export only MAC and Inclusive Multicast (Mcast) routes 💅. It also has to be able to import and process A-D routes, along with MAC and Inclusive Multicast routes 🔥. If PE3 couldn't support importing and processing A-D routes per ESI and per EVI, then fast convergence and aliasing functions (respectively) wouldn't be possible in this use case 😭.

6. MAC-Based Forwarding Model Use Case 🧚

This section walks through how the PEs export and import BGP EVPN routes in our use case, and how traffic gets forwarded when PE1, PE2, and PE3 all support a MAC-based forwarding model 👑. To compare the control- and data-plane impact across the two forwarding models (MAC-based and MPLS-based) and different service types, we'll assume CE1, CE2, and CE3 need to exchange traffic for up to 4k CE-VIDs 💖.

6.1. EVPN Network Startup Procedures

Before any EVI is provisioned in the network, the following procedures are required:

  • Infrastructure setup: The proper MPLS infrastructure has to be set up among PE1, PE2, and PE3, so the EVPN services can use Point-to-Point (P2P) and P2MP LSPs 🌈. On top of the MPLS transport, PE1 and PE2 must be configured with the same LACP configuration toward CE2 ✨. Details are provided in [RFC7432]. Once the LAG is set up, PE1 and PE2 can auto-generate the ESI for the CE2 Ethernet Segment (for example, ESI12) from the LACP information exchanged with CE2 (ESI Type 1), as discussed in Section 4.1 🦋. The ESI can also be manually provisioned on PE1 and PE2 (ESI Type 0) 🎀. PE1 and PE2 will auto-configure a BGP policy that imports any ES route matching the auto-derived ES-Import RT for ESI12 💅.
  • Ethernet Segment route exchange and Designated Forwarder (DF) election: PE1 and PE2 advertise a BGP Ethernet Segment route for ESI12, with the ESI RD and ES-Import RT auto-generated as discussed in Section 4.1.1 🫶. They import each other's ES routes and run the DF election algorithm for any EVIs that already exist at this point 🌸. PE3 simply discards the route 🙈. The DF election algorithm can support service carving, so downstream BUM traffic from the network to CE2 can be load-balanced across PE1 and PE2 on a per-service basis 🔥.

At the end of this process, the network infrastructure is ready to start deploying EVPN services 🎉. PE1 and PE2 know about the shared Ethernet Segment, i.e., ESI12 👑.

6.2. VLAN-Based Service Procedures

Assuming the EVPN network has to carry traffic among CE1, CE2, and CE3 for up to 4k CE-VIDs, the service provider can choose to implement VLAN-based service interface EVIs to make that happen 💖. Here, each CE-VID gets individually mapped to its own EVI 🧚. That does mean a total of 4k MAC-VRFs are required per PE 😭, but the upside is auto-provisioning of most service parameters when no VLAN translation is needed (see Section 4.2.1) plus great control over each individual customer broadcast domain ✨. For this section, we assume the range of EVIs from 1 to 4k is provisioned in the network 🌈.

6.2.1. Service Startup Procedures

As soon as the EVIs are created on PE1, PE2, and PE3, the following control-plane actions happen 💅:

  • Flooding tree setup per EVI (4k routes): Each PE sends one Inclusive Multicast Ethernet Tag route per EVI (up to 4k routes per PE) so the flooding tree per EVI can be set up 🌸. Ingress replication or P2MP LSPs can optionally be signaled in the Provider Multicast Service Interface (PMSI) Tunnel attribute, and the corresponding tree can then be created 🦋.
  • Ethernet A-D routes per ESI (a set of routes for ESI12): PE1 and PE2 issue a set of A-D routes for ESI12, carrying a total list of 4k RTs (one per EVI). It has to be a set of routes so the total number of RTs can be conveyed 🫶. As per [RFC7432], each Ethernet A-D route per ESI is told apart from the others in the set by a different Route Distinguisher (ES RD) 🎀. The set also includes ESI Label extended communities with the active-standby flag set to zero (all-active multihoming type) and an ESI Label that is different from zero (used for split-horizon functions) 🔥. All three PEs import these routes, since the RTs match the locally configured EVI RTs 👑. The A-D routes per ESI are used for fast convergence and split-horizon functions, as discussed in [RFC7432] 💖.
  • Ethernet A-D routes per EVI (4k routes): PE1 and PE2 send an A-D route per EVI for ESI12 🧚. Each individual route carries the corresponding EVI RT and an MPLS Label that PE3 uses for the aliasing function ✨. All three PEs import these routes 🌈.

6.2.2. Packet Walk-Through ✨

Okay bestie, the services are all set up, so the traffic can finally start flowing 🦋. For this walk-through, we're assuming there are no MAC addresses learned yet, and that MAC learning at the access happens in the data plane in our use case. Here's what goes down when frames show up from each CE (for example, EVI1) 💖.

BUM frame example from CE1:

a. An ARP request with CE-VID=1 is issued from source MAC CE1-MAC (the MAC address coming from CE1 or from a device connected to CE1) to find the MAC address of CE3-IP. 💌

b. Based on the CE-VID, the frame gets identified to be forwarded in the MAC-VRF-1 (EVI1) context. A source MAC lookup is done in the MAC FIB, and the sender's CE1-IP is looked up in the proxy ARP table within the MAC-VRF-1 (EVI1) context. If CE1-MAC/CE1-IP are unknown in both tables, three actions are carried out (assuming the source MAC is accepted by PE1) 🫶:

  1. the forwarding state is added for the CE1-MAC associated with the corresponding port and CE-VID;
  2. the ARP request is snooped and the tuple CE1-MAC/CE1-IP is added to the proxy ARP table; and
  3. a BGP MAC Advertisement route is triggered from PE1 containing the EVI1 RD and RT, ESI=0, Ethernet-Tag=0, and CE1-MAC/CE1-IP, along with an MPLS Label assigned to MAC-VRF-1 from the PE1 Label space. Note that depending on the implementation, the MAC FIB and proxy ARP learning processes can independently send two BGP MAC advertisements instead of one (one containing only the CE1-MAC and another one containing CE1-MAC/CE1-IP). 🤓

Since we're assuming a MAC forwarding model, a label per MAC-VRF is normally allocated and signaled by the three PEs for MAC Advertisement routes. Based on the RT, the route gets imported by PE2 and PE3, and the forwarding state plus the ARP entry are added to their MAC-VRF-1 context. From this moment on, any ARP request from CE2 or CE3 destined to CE1-IP can be directly replied to by PE1, PE2, or PE3, and ARP flooding for CE1-IP is not needed in the core. Iconic, right? 👑

c. Since the ARP frame is a broadcast frame, it's forwarded by PE1 using the Inclusive Multicast Tree for EVI1 (the CE-VID=1 tag should be kept if translation is required). Depending on the type of tree, the label stack may vary. For example, assuming ingress replication, the packet is replicated to PE2 and PE3 with the downstream allocated labels and the P2P LSP transport labels. No other labels are added to the stack. 🧚

d. Assuming PE1 is the DF for EVI1 on ESI12, the frame is locally replicated to CE2. 🎀

e. The MPLS-encapsulated frame gets to PE2 and PE3. Since PE2 is non-DF for EVI1 on ESI12, and there's no other CE connected to PE2, the frame is discarded. At PE3, the frame is de-encapsulated, the CE-VID is translated if needed, and it's forwarded to CE3. 🌸

Any other type of BUM frame from CE1 follows the same procedures. BUM frames from CE3 follow the same procedures too. ✨

BUM frame example from CE2:

a. An ARP request with CE-VID=1 is issued from source MAC CE2-MAC to find the MAC address of CE3-IP. 💌

b. CE2 will hash the frame and forward it to, for example, PE2. Based on the CE-VID, the frame gets identified to be forwarded in the EVI1 context. A source MAC lookup is done in the MAC FIB, and the sender's CE2-IP is looked up in the proxy ARP table within the MAC-VRF-1 context. If both are unknown, three actions are carried out (assuming the source MAC is accepted by PE2) 🫶:

  1. the forwarding state is added for the CE2-MAC associated with the corresponding LAG/ESI and CE-VID;
  2. the ARP request is snooped and the tuple CE2-MAC/CE2-IP is added to the proxy ARP table; and
  3. a BGP MAC Advertisement route is triggered from PE2 containing the EVI1 RD and RT, ESI=12, Ethernet-Tag=0, and CE2-MAC/CE2-IP, along with an MPLS Label assigned from the PE2 Label space (one label per MAC-VRF). Again, depending on the implementation, the MAC FIB and proxy ARP learning processes can independently send two BGP MAC advertisements instead of one. 🤓

Note that since PE3 is not part of ESI12, it will install the forwarding state for CE2-MAC as long as the A-D routes for ESI12 are also active on PE3. On the contrary, PE1 is part of ESI12, so PE1 won't modify the forwarding state for CE2-MAC if it has previously learned CE2-MAC locally attached to ESI12. Otherwise, it adds the forwarding state for CE2-MAC associated with the local ESI12 port. 👀

c. Assuming PE2 doesn't have the ARP information for CE3-IP yet, and since the ARP is a broadcast frame and PE2 is the non-DF for EVI1 on ESI12, the frame is forwarded by PE2 in the Inclusive Multicast Tree for EVI1, thus adding the ESI Label for ESI12 at the bottom of the stack. The ESI Label has been previously allocated and signaled by the A-D routes for ESI12. Note that, as per [RFC7432], if the result of the CE2 hashing is different and the frame is sent to PE1, PE1 should add the ESI Label too (PE1 is the DF for EVI1 on ESI12). 🦋

d. The MPLS-encapsulated frame gets to PE1 and PE3. PE1 de-encapsulates the Inclusive Multicast Tree Label(s) and, based on the ESI Label at the bottom of the stack, it decides not to forward the frame to the ESI12. It pops the ESI Label and replicates the frame to CE1, since CE1 is not part of the ESI identified by the ESI Label. At PE3, the Inclusive Multicast Tree Label is popped and the frame is forwarded to CE3. If a P2MP LSP is used as the Inclusive Multicast Tree for EVI1, PE3 will find an ESI Label after popping the P2MP LSP Label. The ESI Label simply gets popped, since CE3 is not part of ESI12. 💅

Unicast frame example from CE3 to CE1:

a. A unicast frame with CE-VID=1 is issued from source MAC CE3-MAC and destination MAC CE1-MAC (we assume PE3 has previously resolved an ARP request from CE3 to find the MAC of CE1-IP and has added CE3-MAC/CE3-IP to its proxy ARP table). 💌

b. Based on the CE-VID, the frame gets identified to be forwarded in the EVI1 context. A source MAC lookup is done in the MAC FIB within the MAC-VRF-1 context, and this time, since we assume CE3-MAC is known, no further actions are carried out as a result of the source lookup. A destination MAC lookup is performed next, and the label stack associated with the MAC CE1-MAC is found (including the label associated with MAC-VRF-1 in PE1 and the P2P LSP Label to get to PE1). The unicast frame is then encapsulated and forwarded to PE1. 🚀

c. At PE1, the packet is identified to be part of EVI1, and a destination MAC lookup is performed in the MAC-VRF-1 context. The labels are popped and the frame is forwarded to CE1 with CE-VID=1. 🎀

Unicast frames from CE1 to CE3 or from CE2 to CE3 follow the same procedures described above. ✨

Unicast frame example from CE3 to CE2:

a. A unicast frame with CE-VID=1 is issued from source MAC CE3-MAC and destination MAC CE2-MAC (we assume PE3 has previously resolved an ARP request from CE3 to find the MAC of CE2-IP). 💌

b. Based on the CE-VID, the frame gets identified to be forwarded in the MAC-VRF-1 context. We assume CE3-MAC is known. A destination MAC lookup is performed next, and PE3 finds CE2-MAC associated with PE2 on ESI12. ESI12 is an Ethernet Segment for which PE3 has two active A-D routes per ESI (from PE1 and PE2) and two active A-D routes for EVI1 (from PE1 and PE2). Based on a hashing function for the frame, PE3 may decide to forward the frame using the label stack associated with PE2 (the label received from the MAC Advertisement route) or the label stack associated with PE1 (the label received from the A-D route per EVI for EVI1). Either way, the frame is encapsulated and sent to the remote PE. 🌈

c. At PE2 (or PE1), the packet is identified to be part of EVI1 based on the bottom label, and a destination MAC lookup is performed. At either PE (PE2 or PE1), the FIB lookup yields a local ESI12 port to which the frame is sent. 🫧

Unicast frames from CE1 to CE2 follow the same procedures. 💜

6.3. VLAN Bundle Service Procedures 🎀

Instead of using VLAN-based interfaces, the operator can choose to implement VLAN bundle interfaces to carry the traffic for the 4k CE-VIDs among CE1, CE2, and CE3. If that's the case, the 4k CE-VIDs can be mapped to the same EVI (for example, EVI200) at each PE. The main advantage of this approach is the low control-plane overhead (reduced number of routes and labels) and easiness of provisioning. The tradeoff? There's no control over the customer broadcast domains, i.e., a single Inclusive Multicast Tree for all the CE-VIDs and no CE-VID translation in the provider network. Tradeoffs are literally everything in networking, bestie 🙈

6.3.1. Service Startup Procedures 🚀

As soon as EVI200 is created in PE1, PE2, and PE3, the following control-plane actions are carried out:

  • Flooding tree setup per EVI (one route): Each PE will send one Inclusive Multicast Ethernet Tag route per EVI (hence, only one route per PE) so that the flooding tree per EVI can be set up. Note that ingress replication or P2MP LSPs can optionally be signaled in the PMSI Tunnel attribute, and the corresponding tree can be created. 🌊
  • Ethernet A-D routes per ESI (one route for ESI12): A single A-D route for ESI12 will be issued from PE1 and PE2. This route includes a single RT (RT for EVI200), an ESI Label extended community with the active-standby flag set to zero (all-active multihoming type), and an ESI Label different from zero (used by the non-DF for split-horizon functions). This route will be imported by the three PEs, since the RT matches the locally configured EVI200 RT. The A-D routes per ESI will be used for fast convergence and split-horizon functions, as described in [RFC7432]. ⚡
  • Ethernet A-D routes per EVI (one route): An A-D route (EVI200) will be sent by PE1 and PE2 for ESI12. This route includes the EVI200 RT and an MPLS Label to be used by PE3 for the aliasing function. This route will be imported by the three PEs. 🍓

6.3.2. Packet Walk-Through 🦋

The packet walk-through for the VLAN bundle case is similar to the one described for EVI1 in the VLAN-based case, except for the way the CE-VID is handled by the ingress PE and the egress PE:

  • No VLAN translation is allowed, and the CE-VIDs are kept untouched from CE to CE, i.e., the ingress CE-VID must be kept at the imposition PE and at the disposition PE. 🔒
  • The frame is identified to be forwarded in the MAC-VRF-200 context as long as its CE-VID belongs to the VLAN bundle defined in the PE1/PE2/PE3 port to CE1/CE2/CE3. Our example is a special VLAN bundle case, since the entire CE-VID range is defined in the ports; therefore, any CE-VID would be part of EVI200. 🌈

Please refer to Section 6.2.2 for more information about the control-plane and forwarding-plane interaction for BUM and unicast traffic from the different CEs. 💖

6.4. VLAN-Aware Bundling Service Procedures 💖

okay so the last potential service type we're looking at is VLAN-aware bundling 💅 When this type of service interface carries the 4k CE-VIDs among CE1, CE2, and CE3, every CE-VID gets mapped to the same EVI (like EVI300). The difference from the VLAN bundle type in the last section is that each incoming CE-VID also gets mapped to its own "normalized" Ethernet Tag, on top of EVI300 🎀 If no translation is needed, the Ethernet Tag matches the CE-VID. If translation is needed, a translation between CE-VID and Ethernet Tag is needed at the imposition PE and at the disposition PE too. The big win here is being able to control customer broadcast domains while still giving the customer a single EVI 🫶

6.4.1. Service Startup Procedures ✨

okay so as soon as EVI300 is created on PE1, PE2, and PE3, these control-plane actions happen 🧚

  • Flooding tree setup per EVI per Ethernet Tag (4k routes): Each PE sends one Inclusive Multicast Ethernet Tag route per EVI and per Ethernet Tag (hence, 4k routes per PE) so the flooding tree per customer broadcast domain can be set up 🌸 Ingress replication or P2MP LSPs can optionally be signaled in the PMSI Tunnel attribute, and the matching tree gets created. In this use case, since all the CE-VIDs and Ethernet Tags are defined on the three PEs, multicast tree aggregation might make sense to save forwarding states 💜
  • Ethernet A-D routes per ESI (one route for ESI12): A single A-D route for ESI12 is issued from PE1 and PE2 🍓 It includes a single RT (RT for EVI300), an ESI Label extended community with the active-standby flag set to zero (all-active multihoming type), and an ESI Label that is different than zero (used by the non-DF for split-horizon functions). All three PEs import it, since the RT matches the locally configured EVI300 RT. These A-D routes per ESI are used for fast convergence and split-horizon functions, as described in [RFC7432] ⚡
  • Ethernet A-D routes per EVI: A single A-D route (EVI300) may be sent by PE1 and PE2 for ESI12 in case no CE-VID translation is required 🦋 It includes the EVI300 RT and an MPLS Label that PE3 uses for the aliasing function. All three PEs import this route. If CE-VID translation is required, you need an A-D per EVI route per Ethernet Tag (4k) 💀

6.4.2. Packet Walk-Through 🌈

the packet walk-through for the VLAN-aware case is pretty similar to the one we did before 🙈 But compared to the other two cases, VLAN-aware services allow for CE-VID translation and for an N:1 CE-VID to EVI mapping. Those two things can't be done at once in either of the other two service interfaces 👑 Here are the main differences from the packet walk-through in Section 6.2.2:

  • At the ingress PE, frames get identified to be forwarded in the EVI300 context as long as their CE-VID belongs to the range defined in the PE port to the CE 🔥 On top of that, CE-VID=x is mapped to a "normalized" Ethernet-Tag=y at the MAC-VRF-300 (where x and y might be equal if no translation is needed). Qualified learning is now required, since a different bridge table is allocated within MAC-VRF-300 for each Ethernet Tag. The same MAC could be learned in two different Ethernet Tag bridge tables of the same MAC-VRF 😭
  • Any new locally learned MAC on the MAC-VRF-300/Ethernet-Tag=y interface gets advertised by the ingress PE in a MAC Advertisement route, using the now Ethernet Tag field (Ethernet-Tag=y), so the remote PE learns the MAC associated with the MAC-VRF-300/Ethernet-Tag=y FIB 🧠 The Ethernet Tag field is not used in advertisements of MACs learned on VLAN-based or VLAN-bundle service interfaces, just FYI 🫶
  • At the ingress PE, BUM frames get sent to the corresponding flooding tree for the particular Ethernet Tag they're mapped to 🌊 Each individual Ethernet Tag can have a different flooding tree within the same EVI300. For instance, Ethernet-Tag=y can use ingress replication to reach the remote PEs, whereas Ethernet-Tag=z can use a P2MP LSP 🍭
  • At the egress PE, Ethernet-Tag=y (for a given broadcast domain within MAC-VRF-300) can be translated to egress CE-VID=x. That's not possible for VLAN bundle interfaces. It is possible for VLAN-based interfaces, but it requires a separate MAC-VRF per CE-VID 🙅‍♀️

7. MPLS-Based Forwarding Model Use Case 🧚‍♀️

okay so EVPN supports an alternative forwarding model, usually called the MPLS-based forwarding or disposition model 🌷 It's the opposite of the MAC-based forwarding or disposition model we covered in Section 6. Switching to the MPLS-based model instead of the MAC-based one might have an impact on the following:

  • the number of forwarding states required; and
  • the FIB where the forwarding states are handled (MAC FIB or MPLS Label FIB (LFIB)).

The MPLS-based forwarding model skips the destination MAC lookup at the egress PE MAC FIB, at the expense of increasing the number of next-hop forwarding states at the egress MPLS LFIB 💅 This also affects the control plane and the label allocation model, since an MPLS-based disposition PE has to send as many routes and labels as the required next-hops in the egress MAC-VRF 🫠 This concept is equivalent to the forwarding models supported in IP-VPNs at the egress PE, where an IP lookup in the IP-VPN FIB may or may not be necessary, depending on the available next-hop forwarding states in the LFIB ✨

The following subsections highlight the impact on the control- and data-plane procedures described in Section 6 when an MPLS-based forwarding model is used 🎀

Note that both forwarding models are compatible and interoperable in the same network 🌈 Which model each PE implements is a local decision for that PE node, so it's totally up to each one 🍓

7.1. Impact of MPLS-Based Forwarding on the EVPN Network Startup 🌸

the MPLS-based forwarding model has no impact on the procedures explained in Section 6.1 🙌

7.2. Impact of MPLS-Based Forwarding on the VLAN-Based Service Procedures 🦋

Compared to the MAC-based forwarding model, the MPLS-based forwarding model has no impact in terms of the number of routes when all the service interfaces are VLAN-based 💖 The differences for the use case in this document are summarized in this list:

  • Flooding tree setup per EVI (4k routes per PE): There's no impact compared to the MAC-based model 🫶
  • Ethernet A-D routes per ESI (one set of routes for ESI12 per PE): There's no impact compared to the MAC-based model 💜
  • Ethernet A-D routes per EVI (4k routes per PE/ESI): There's no impact compared to the MAC-based model 🧠
  • MAC Advertisement routes: Instead of allocating and advertising the same MPLS Label for all the new MACs locally learned on the same MAC-VRF, a different label has to be advertised per CE next-hop or MAC, so no MAC FIB lookup is needed at the egress PE 😭 In general, this means a different label (at least per CE) must be advertised, although the PE can decide to implement a label per MAC if more granularity (hence, less scalability) is required in terms of forwarding states. For example, if CE2 sends traffic from two different MACs to PE1, CE2-MAC1 and CE2-MAC2, the same MPLS Label=x can be re-used for both MAC advertisements, since they both share the same source ESI12 🔥 It's up to the PE1 implementation to use a different label per individual MAC within the same ES (even if only one label per ESI is enough) 🍭
  • PE1, PE2, and PE3 won't add forwarding states to the MAC FIB when they learn new local CE MAC addresses on the data plane. They'll add forwarding states to the MPLS LFIB instead 👑

7.3. Impact of MPLS-Based Forwarding on the VLAN Bundle Service Procedures 🌊

Compared to the MAC-based forwarding model, the MPLS-based forwarding model has no impact in terms of number of routes when all the service interfaces are VLAN bundle type 🌷 The differences for the use case in this document are summarized in this list:

  • Flooding tree setup per EVI (one route): There's no impact compared to the MAC-based model 🎀
  • Ethernet A-D routes per ESI (one route for ESI12 per PE): There's no impact compared to the MAC-based model 🍓
  • Ethernet A-D routes per EVI (one route per PE/ESI): There's no impact compared to the MAC-based model, since no VLAN translation is required 🫶
  • MAC Advertisement routes: Instead of allocating and advertising the same MPLS Label for all the new MACs locally learned on the same MAC-VRF, a different label has to be advertised per CE next-hop or MAC, so no MAC FIB lookup is needed at the egress PE 😭 In general, this means a different label (at least per CE) must be advertised, although the PE can decide to implement a label per MAC if more granularity (hence, less scalability) is required in terms of forwarding states. It's up to the PE1 implementation to use a different label per individual MAC within the same ES (even if only one label per ESI is enough) 🧠
  • PE1, PE2, and PE3 won't add forwarding states to the MAC FIB when they learn new local CE MAC addresses on the data plane, but will add forwarding states to the MPLS LFIB instead 💅

7.4. Impact of MPLS-Based Forwarding on the VLAN-Aware Service Procedures 💅

Okay so, compared to the MAC-based forwarding model, the MPLS-based one has zero impact on the number of A-D routes when all the service interfaces are VLAN-aware bundle type 🎀. The differences for this use case are summed up in this list, bestie:

  • Flooding tree setup per EVI (4k routes per PE): No change compared to the MAC-based model. Same energy ✨
  • Ethernet A-D routes per ESI (one route for ESI12 per PE): No change compared to the MAC-based model 🦋
  • Ethernet A-D routes per EVI (1 route per ESI or 4k routes per PE/ESI): PE1 and PE2 may send one route per ESI if no CE-VID translation is needed. However, 4k routes are normally sent for EVI300, one per <ESI, Ethernet Tag ID> tuple. This lets the egress PE find all the forwarding info in the MPLS LFIB and even support Ethernet Tag to CE-VID translation at the egress 🧚
  • MAC Advertisement routes: Instead of allocating and advertising the same MPLS Label for all the new MACs locally learned on the same MAC-VRF, a different label must be advertised per CE next-hop or MAC, so no MAC FIB lookup is needed at the egress PE. Generally, this means a different label (at least per CE) must be advertised, although the PE can decide to implement a label per MAC if more granularity (hence, less scalability) is required in terms of forwarding states. It's up to the PE1 implementation to use a different label per individual MAC within the same ES. Also, the Ethernet Tag will be set to a non-zero value for the MAC Advertisement routes. The same MAC address can be announced with a different Ethernet Tag value, which makes the advertising PE install two different forwarding states in the MPLS LFIB. Literally so main character 👑
  • PE1, PE2, and PE3 will not add forwarding states to the MAC FIB upon learning new local CE MAC addresses on the data plane, but will rather add forwarding states to the MPLS LFIB 🌸

8. Comparison between MAC-Based and MPLS-Based Egress Forwarding Models 🍓

Both forwarding models are possible in a network deployment, and each one has its own trade-offs. Iykyk 🙈

Both forwarding models can save A-D routes per EVI when VLAN-aware bundling services are deployed and no CE-VID translation is required. It's giving efficiency, but customers normally require CE-VID translation, so we assume an A-D per EVI route per <ESI, Ethernet Tag> is needed 💜.

The MAC-based model saves a significant amount of MPLS Labels compared to the MPLS-based forwarding model. All the MACs and A-D routes for the same EVI can signal the same MPLS Label, saving labels from the local PE space. A MAC FIB lookup at the egress PE is required in order to do so 🔥.

The MPLS-based forwarding model can save forwarding states at the egress PEs if labels per next-hop CE (as opposed to per MAC) are implemented. No egress MAC lookup is required. Also, a different label per next-hop CE per MAC-VRF is consumed, as opposed to a single label per MAC-VRF ⭐.

Table 2 summarizes the resource implementation details of both models 🫶:

+-----------------------------+-----------------+------------------+
| Resources                   | MAC-Based Model | MPLS-Based Model |
+-----------------------------+-----------------+------------------+
| MPLS Labels Consumed        | 1 per MAC-VRF   | 1 per CE/EVI     |
| Egress PE Forwarding States | 1 per MAC       | 1 per Next-Hop   |
| Egress PE Lookups           | 2 (MPLS+MAC)    | 1 (MPLS)         |
+-----------------------------+-----------------+------------------+

Table 2: Resource Comparison between MAC-Based and MPLS-Based Models

The egress forwarding model is an implementation local to the egress PE and is independent of the model supported on the rest of the PEs. In our use case, PE1, PE2, and PE3 could have either egress forwarding model without any dependencies. No drama, no cap 💖.

9. Traffic Flow Optimization ✨

In addition to the procedures described across Sections 3 through 8, EVPN [RFC7432] procedures allow for optimized traffic handling so we can minimize unnecessary flooding across the entire infrastructure 🌈. The optimization comes from specific ARP termination and the ability to block unknown unicast flooding. EVPN procedures also allow for intelligent, close-to-the-source inter-subnet forwarding, and they solve the commonly known suboptimal routing problem 👑. Besides the traffic efficiency, ingress-based inter-subnet forwarding also optimizes packet forwarding rules and implementation at the egress nodes as well 💅. Details of these procedures are outlined in Sections 9.1 and 9.2.

9.1. Control-Plane Procedures 🦋

9.1.1. MAC Learning Options

The fundamental premise of [RFC7432] is a totally different approach to MAC address learning compared to traditional IEEE 802.1 bridge learning methods. Specifically, EVPN differentiates between data-driven and control-plane-driven learning mechanisms 🧚.

Data-driven learning means there is no separate communication channel used to advertise and propagate MAC addresses. Instead, MAC addresses are learned through IEEE-defined bridge learning procedures as well as by snooping on DHCP and ARP requests. As different MAC addresses show up on different ports, the Layer 2 (L2) FIB gets populated with the appropriate MAC addresses 🌸.

Control-plane-driven learning means there is a communication channel that could be either a control-plane protocol or a management-plane mechanism. In the context of EVPN, two different learning procedures are defined: local and remote procedures 💜.

  • Local learning defines the procedures used for learning the MAC addresses of network elements locally connected to a MAC-VRF. Local learning could be implemented through all three learning procedures: control plane, management plane, and data plane. However, the expectation is that for most use cases, local learning through the data plane should be sufficient 🎀.
  • Remote learning defines the procedures used for learning MAC addresses of network elements remotely connected to a MAC-VRF, i.e., far-end PEs. Remote learning procedures defined in [RFC7432] advocate using only control-plane learning, BGP specifically. Through the use of BGP EVPN NLRIs, the remote PE has the capability of advertising all the MAC addresses present in its local FIB 🫶.

9.1.2. Proxy ARP/ND

In EVPN, MAC addresses are advertised via the MAC/IP Advertisement route, as discussed in [RFC7432]. Optionally, an IP address can be advertised along with the MAC address advertisement. There are certain rules about IP address usage, though: if the MAC/IP Route contains an IP address, that particular IP address correlates directly with the advertised MAC address. This kind of advertisement lets us build a proxy ARP / Neighbor Discovery (ND) table populated with the IP<->MAC bindings received from all the remote nodes 🌈.

Based on these bindings, a local MAC-VRF can now provide proxy ARP/ND functionality for all ARP requests and ND solicitations directed to the IP address pool learned through BGP. So the amount of unnecessary L2 flooding (ARP/ND requests/solicitations in this case) can be further reduced by introducing proxy ARP/ND functionality across all EVI MAC-VRFs 🍓.

9.1.3. Unknown Unicast Flooding Suppression

Given that all locally learned MAC addresses are advertised through BGP to all remote PEs, suppressing flooding of unknown unicast traffic towards the remote PEs is a feasible network optimization 🦋.

The assumption in this use case is that any network device that appears on a remote MAC-VRF will somehow signal its presence to the network. This signaling can be done through, for example, gratuitous ARPs. Once the remote PE acknowledges the presence of the node in the MAC-VRF, it will do two things: install its MAC address in its local FIB, and advertise this MAC address to all other BGP speakers via EVPN NLRI. So we can assume that any active MAC address is propagated and learned through the entire EVI. Given that MAC addresses become prepopulated (once nodes are alive on the network), there is no need to flood any unknown unicast towards the remote PEs 🧚. If the owner of a given destination MAC is active, the BGP route will be present in the local RIB and FIB, assuming that the BGP import policies are successfully applied. Otherwise, the owner of such destination MAC is not present on the network 💀.

It is worth noting that unknown unicast flooding must not be suppressed unless (at least) one of the following two statements is given: a) control- or management-plane learning is performed throughout the entire EVI for all the MACs, or b) all the EVI-attached devices signal their presence when they come up (Gratuitous ARP (GARP) packets or similar). Don't skip this one, bestie 🙈⚠️.

Heads up: unknown unicast flooding must not be suppressed unless statement a) or statement b) above is given. Don't be the one who forgets 😭

9.1.4. Optimization of Inter-Subnet Forwarding

In a scenario where both L2 and L3 services are needed over the same physical topology, some interaction between EVPN and IP-VPN is required. A common way of stitching the two service planes together is through an Integrated Routing and Bridging (IRB) interface, which allows traffic to be either routed or bridged depending on its destination MAC address. If the destination MAC address is the one from the IRB interface, the traffic needs to pass through a routing module and potentially be either routed to a remote PE or forwarded to a local subnet. If the destination MAC address is not the one from the IRB interface, the MAC-VRF follows standard bridging procedures 🌸.

A typical example of EVPN inter-subnet forwarding would be a scenario in which multiple IP subnets are part of a single or multiple EVIs, and they all belong to a single IP-VPN. In such topologies, it's desired that inter-subnet traffic can be efficiently routed without any tromboning effects in the network. Due to the overlapping physical and service topology in such scenarios, all inter-subnet connectivity will be locally routed through the IRB interface 👑.

On top of optimizing the traffic patterns in the network, local inter-subnet forwarding also greatly optimizes the amount of processing needed to cross the subnets. Through EVPN MAC advertisements, the local PE learns the real destination MAC address associated with the remote IP address, so the inter-subnet forwarding can happen locally. When the packet is received at the egress PE, it's directly mapped to an egress MAC-VRF and bypasses any egress IP-VPN processing 💖.

Please refer to [EVPN-INTERSUBNET] for more information about the IP inter-subnet forwarding procedures in EVPN 📚.

9.2. Packet Walk-Through Examples 🎀

Assuming that the services are set up according to Figure 1 in Section 3, the following flow optimization processes will take place in terms of creating, receiving, and forwarding packets across the network ✨.

9.2.1. Proxy ARP Example for CE2-to-CE3 Traffic 💖

Okay bestie, let's bring back Figure 1 from Section 3 🎀. Picture EVI400 living on PE1, PE2, and PE3, connecting the CE2 and CE3 networks. PE1 and PE2 are both part of the all-active multihoming ES for CE2, and PE2 got elected designated forwarder for EVI400 👑. We're also assuming every PE implements proxy ARP in the MAC-VRF-400 context ✨.

Here's the tea: PE3 doesn't only advertise MAC addresses through the EVPN MAC Advertisement route. It also advertises the IP addresses of individual hosts behind CE3, aka the /32 prefixes 🧚. When PE1 and PE2 receive those EVPN routes, they install the MAC addresses in the MAC-VRF-400 FIB. Since they also have the associated IP addresses, PE1 and PE2 can now build a proxy ARP table within the MAC-VRF-400 context 🦋.

Now the forwarding part. When a node behind CE2 wants to send a frame to a node behind CE3, it first sends an ARP request to, for example, PE2 (whichever PE CE2's hashing picks) 🌸. If PE2 has filled its proxy ARP table with all the active nodes behind CE3, and the IP address in the ARP message matches an entry, PE2 responds to the ARP request with the actual MAC address, on behalf of the node behind CE3 😭. That's sneaky-helpful energy, and it's giving 💅.

Once the nodes behind CE2 learn the actual MAC addresses of the nodes behind CE3, all the MAC-to-MAC communication between the two networks is unicast. Iconic 🫶.

9.2.2. Flood Suppression Example for CE1-to-CE3 Traffic 🙈

Still using Figure 1 in Section 3, but now EVI500 lives on PE1 and PE3, connecting CE1 and CE3 🌈. Both PE1 and PE3 have switched off unknown unicast flooding for this specific EVI context 🚫.

Once the network devices behind CE3 come online, they learn their MAC addresses and create local FIB entries for them. Fun fact: local FIB entries can also be created through a control or management plane between PE and CE 💜. Either way, PE3 automatically creates EVPN Type 2 MAC Advertisement routes and advertises all the locally learned MAC addresses. The routes also include the corresponding MPLS Label 🍓.

PE1 automatically learns and installs all the MAC addresses behind CE3, so its MAC-VRF FIB is already prepopulated with the matching next-hops and label assignments 🧚. That means that as soon as traffic from CE1 to nodes behind CE3 lands in the EVI500 context, PE1 pushes the MPLS Label(s) onto the original Ethernet frame and sends the packet into the MPLS network 🚀. Once PE3 receives it, PE3 either does a next-hop lookup in the EVI500 context or forwards the traffic straight to CE3, depending on the forwarding model.

Now for the plot twist 💀. If PE1's MAC-VRF-500 doesn't have a MAC entry for a destination that CE1 is trying to reach, PE1 drops the frame, because unknown unicast flooding is disabled. Hard pass on flooding 🙅‍♀️.

This works because of an assumption: all the MAC entries behind the CEs are prepopulated through gratuitous ARP and/or DHCP requests. So if one specific MAC entry isn't in the MAC-VRF-500 FIB on PE1, the owner of that MAC is not alive on the network behind CE3. The traffic can be dropped at PE1 instead of flooded all over the place and eating up bandwidth 🦋.

9.2.3. Optimization of Inter-subnet Forwarding Example for CE3-to-CE2 Traffic 👑

Okay, same Figure 1 from Section 3, and this one is my favourite 🥰. There's an IP-VPN 666 context on PE1, PE2, and PE3, which connects CE1, CE2, and CE3 into one single IP-VPN domain 💅. There are also two EVIs on the PEs, EVI600 and EVI60 ✨.

Each IP subnet is tied to a different MAC-VRF context. So there's one subnet (subnet 600) between CE1 and CE3, established through EVI600. There's another subnet (subnet 60) between CE2 and CE3, established through EVI60 🌸. Since both subnets belong to the same IP-VPN, each EVI (or individual subnet) is mapped to a local IRB interface on all three PEs 🎀.

If a node behind CE2 wants to talk to a node on the same subnet, sitting behind CE3, the traffic follows the standard EVPN procedures. That means a FIB lookup within PE1 (or PE2) after the corresponding EVPN label is added to the MPLS Label stack. (That label comes from PE3's downstream label allocation for EVI60 🧚.)

Now crossing subnet boundaries is where it gets spicy 🌶️. The ingress PE does local inter-subnet forwarding. For example, when a node behind CE2 (EVI60) sends a packet to a node behind CE1 (EVI600), the destination IP address is in subnet 600. But the destination MAC address is the address of the source node's default gateway, which in this case is an IRB interface on PE1 that connects EVI60 to IP-VPN 666 💖. Once PE1 sees traffic destined for its own MAC address, it routes the packet into EVI600. It changes the source MAC address to the one of the IRB interface in EVI600, and changes the destination MAC address to the address of the node behind CE1. That address is already populated in the MAC-VRF-600 FIB, through either data-plane or control-plane learning 🦋.

The big optimization here is local inter-subnet forwarding instead of IP-VPN routing ✨. If the node from subnet 60 behind CE2 sends traffic to the remote end-node on subnet 600 behind CE3, the mechanism still honors the local inter-subnet (inter-EVI) forwarding. No detour needed 🙌.

So in our use case, when the node from subnet 60 behind CE2 sends traffic to the node on subnet 600 behind CE3, the destination MAC address is the PE1 MAC-VRF-60 IRB MAC address. Then, once the traffic locally crosses over to EVI600 via the IRB interface on PE1, the source MAC address becomes the IRB interface's MAC and the destination MAC becomes the one advertised by PE3 via EVPN and already installed in MAC-VRF-600. The rest of the forwarding through PE1 uses the MAC-VRF-600 forwarding context and label space 🌈.

One more very relevant optimization: traffic between PEs gets forwarded through EVPN rather than IP-VPN 🔥. In the example above, traffic from EVI60 on CE2 to EVI600 on CE3 doesn't need any IP-VPN processing on the egress PE3. The traffic either goes to the EVI600 context on PE3 for further MAC lookup and next-hop processing, or goes directly to the node behind CE3, depending on the egress forwarding model being used 💜.

10. Security Considerations 🔐

Please go check the "Security Considerations" section in [RFC7432] 🫶. Quick recap of what's out there: the SIDR Working Group standards cover secure route origin authentication (e.g., RFCs 6480 through 6493) and route advertisement security (e.g., RFCs 8205 through 8211) 🛡️. They protect the integrity and authenticity of IP address advertisements and ASN/IP prefix bindings.

Here's the plot twist 😭: this document and [RFC7432] use BGP to convey other info (e.g., MAC addresses). So the protections offered by the SIDR WG RFCs are not applicable in this context. Dramatic, I know, but it's the truth, and you need to know it 💀.

11. IANA Considerations 📝

This document has no IANA actions. Zero actions, zero drama 🌸.

12. References 📚

12.1. Normative References

12.2. Informative References

Acknowledgments 🙏

The authors want to thank Giles Heron for his detailed review of the document 🥹. We also thank Stefan Plug and Eric Wunan for their comments 💜.

Contributors 🫶

The following people contributed substantially to the content of this document and should be considered coauthors:

  • Florin Balus
  • Keyur Patel
  • Aldrin Isaac
  • Truman Boyes

Authors' Addresses ✉️

Jorge Rabadan (editor)
Nokia
777 E. Middlefield Road
Mountain View, CA 94043
United States America

Email: jorge.rabadan@nokia.com


Senad Palislamovic
Nokia

Email: senad.palislamovic@nokia.com


Wim Henderickx
Nokia
Copernicuslaan 50
2018 Antwerp
Belgium

Email: wim.henderickx@nokia.com


Ali Sajassi
Cisco

Email: sajassi@cisco.com


James Uttaro
AT&T

Email: uttaro@att.com
📦 where this page comes from