RFC 9014

Interconnect Solution for Ethernet VPN (EVPN) Overlay Networks

📅 published May 2021, from ietf

okay bestie, RFC 9014 is here and it's giving main character energy 🎀 Its real title is Interconnect Solution for Ethernet VPN (EVPN) Overlay Networks 💖. It's a Standards Track doc from the IETF, dated May 2021, with ISSN 2070-1721. J. Rabadan is the editor, and the other authors are S. Sathappan and W. Henderickx from Nokia, A. Sajassi from Cisco, and J. Drake from Juniper. The title block doesn't list any RFCs it obsoletes or updates, so this one stands on its own, no cap 💅

Abstract

okay so this doc is about hooking Network Virtualization Overlays (NVOs) up to a Wide Area Network (WAN) so tenants who need Layer 2 connectivity can stretch it across sites 🦋. It digs into how NVO networks running Ethernet Virtual Private Networks (EVPNs) interact with other Layer 2 VPN (L2VPN) techs that show up in the WAN, like Virtual Private LAN Services (VPLSs), VPLS extensions for Provider Backbone Bridging (PBB-VPLS), EVPN, and PBB-EVPN 🌈

It also explains how the existing specs apply to this kind of interconnect, and where EVPN procedures need extending in some cases ✨ The big focus is how EVPN routes get processed on Gateways (GWs) that connect EVPN-Overlay and EVPN-MPLS networks, plus the Interconnect Ethernet Segment (I-ES), which handles multihoming 👑 Last but not least, it covers the Unknown MAC Route (UMR), which helps avoid Media Access Control (MAC) scale problems on Data Center Network Virtualization Edge (NVE) devices 🫶

Status of This Memo

This is an Internet Standards Track document 🌸

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). More info on Internet Standards lives in Section 2 of RFC 7841 💜

Info about the current status of this doc, any errata, and how to send feedback is at https://www.rfc-editor.org/info/rfc9014 🍓

The copyright here belongs to the IETF Trust and the people listed as document authors, 2021, with all rights reserved 🌈

This doc is covered by BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read those, because they spell out your rights and restrictions for this document 🙈 Also, any Code Components pulled out of this document have to include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License itself 💅

Table of Contents

   1.  Introduction
   2.  Conventions and Terminology
   3.  Decoupled Interconnect Solution for EVPN-Overlay Networks
     3.1.  Interconnect Requirements
     3.2.  VLAN-Based Handoff
     3.3.  PW-Based Handoff
     3.4.  Multihoming Solution on the GWs
     3.5.  Gateway Optimizations
       3.5.1.  MAC Address Advertisement Control
       3.5.2.  ARP/ND Flooding Control
       3.5.3.  Handling Failures between GW and WAN Edge Routers
   4.  Integrated Interconnect Solution for EVPN-Overlay Networks
     4.1.  Interconnect Requirements
     4.2.  VPLS Interconnect for EVPN-Overlay Networks
       4.2.1.  Control/Data Plane Setup Procedures on the GWs
       4.2.2.  Multihoming Procedures on the GWs
     4.3.  PBB-VPLS Interconnect for EVPN-Overlay Networks
       4.3.1.  Control/Data Plane Setup Procedures on the GWs
       4.3.2.  Multihoming Procedures on the GWs
     4.4.  EVPN-MPLS Interconnect for EVPN-Overlay Networks
       4.4.1.  Control plane Setup Procedures on the GWs
       4.4.2.  Data Plane Setup Procedures on the GWs
       4.4.3.  Multihoming Procedure Extensions on the GWs
       4.4.4.  Impact on MAC Mobility Procedures
       4.4.5.  Gateway Optimizations
       4.4.6.  Benefits of the EVPN-MPLS Interconnect Solution
     4.5.  PBB-EVPN Interconnect for EVPN-Overlay Networks
       4.5.1.  Control/Data Plane Setup Procedures on the GWs
       4.5.2.  Multihoming Procedures on the GWs
       4.5.3.  Impact on MAC Mobility Procedures
       4.5.4.  Gateway Optimizations
     4.6.  EVPN-VXLAN Interconnect for EVPN-Overlay Networks
       4.6.1.  Globally Unique VNIs in the Interconnect Network
       4.6.2.  Downstream-Assigned VNIs in the Interconnect Network
   5.  Security Considerations
   6.  IANA Considerations
   7.  References
     7.1.  Normative References
     7.2.  Informative References
   Acknowledgments
   Contributors
   Authors' Addresses

1. Introduction ✨

[RFC8365] talks about using Ethernet Virtual Private Networks (EVPNs) [RFC7432] as the control plane for Network Virtualization Overlays (NVOs), and the possible data plane encapsulations there are VXLAN [RFC7348], NVGRE [RFC7637], or MPLS over GRE [RFC4023] 🦋

This model is super scalable and efficient for multitenant setups inside the Data Center 💖 But it's not always easy to stretch it out to the Wide Area Network (WAN), because of the requirements and the tech that's already deployed. For example, a Service Provider might already be running a Virtual Private LAN Service (VPLS) [RFC4761] [RFC4762], VPLS extensions for Provider Backbone Bridging (PBB-VPLS) [RFC7041], EVPN [RFC7432], or PBB-EVPN [RFC7623] network that has to be used to connect Data Centers and WAN VPN users 🌸 In those cases you need a Gateway (GW) function 🫶 Fun fact, [RFC8365] describes two main Data Center Interconnect (DCI) solution groups: "DCI using GWs" and "DCI using ASBRs" 👑 This document specifies the solutions that fall under the "DCI using GWs" group 💅

It's assumed that the NVO GW and the WAN Edge functions can either be decoupled into two separate systems or integrated into the same system. The first option is called the "decoupled interconnect solution" throughout the document, and the second is called the "integrated interconnect solution" 🎀

The specified procedures are local to the redundant GWs that connect a DC to the WAN. The document doesn't stop you from mixing options across different DCs for the same tenant. For instance, a "Decoupled" solution can run on GW1 and GW2 (for DC1), while an "Integrated" solution runs on GW3 and GW4 (for DC2) 🌈

Gateways and WAN Provider Edges (PEs) use existing specs in some cases, but the document also defines extensions that are specific to DCI. Those extensions are:

  • The Interconnect Ethernet Segment (I-ES), an Ethernet Segment that can be associated to a set of pseudowires (PWs) or other tunnels. The I-ES defined in this document is not associated with a set of Ethernet links, as per [RFC7432], but rather with a set of virtual tunnels (e.g., a set of PWs). This set of virtual tunnels is referred to as vES [VIRTUAL-ES].
  • The use of the Unknown MAC Route (UMR) in a DCI scenario.
  • The processing of EVPN routes on Gateways with MAC-VRFs connecting EVPN-Overlay and EVPN-MPLS networks, or EVPN-Overlay and EVPN-Overlay networks.

2. Conventions and Terminology 💅

Quick vibe check on the keywords 🔥 The BCP 14 key words are "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL". They work the way BCP 14 [RFC2119] [RFC8174] describes in this document, but only when they show up in all capitals, exactly like that 🙈

Here's the glossary, bestie 📚

  • AC: Attachment Circuit
  • ARP: Address Resolution Protocol
  • BUM: Broadcast, Unknown Unicast and Multicast (traffic)
  • CE: Customer Equipment
  • CFM: Connectivity Fault Management
  • DC: Data Center
  • DCI: Data Center Interconnect
  • DF: Designated Forwarder
  • EVI: EVPN Instance
  • EVPN: Ethernet Virtual Private Network, as in [RFC7432]
  • EVPN Tunnel binding: refers to a tunnel to a remote PE/NVE for a given EVI. Ethernet packets in these bindings are encapsulated with the Overlay or MPLS encapsulation and the EVPN label at the bottom of the stack.
  • ES: Ethernet Segment
  • ESI: Ethernet Segment Identifier
  • GW: Gateway or Data Center Gateway
  • I-ES and I-ESI: Interconnect Ethernet Segment and Interconnect Ethernet Segment Identifier. An I-ES is defined on the GWs for multihoming to/from the WAN.
  • MAC: Media Access Control
  • MAC-VRF: refers to an EVI instance in a particular node
  • MP2P and LSM tunnels: refer to multipoint-to-point and label switched multicast tunnels
  • ND: Neighbor Discovery
  • NDF: Non-Designated Forwarder
  • NVE: Network Virtualization Edge
  • NVGRE: Network Virtualization using Generic Routing Encapsulation
  • NVO: Network Virtualization Overlay
  • OAM: Operations, Administration, and Maintenance
  • PBB: Provider Backbone Bridging
  • PE: Provider Edge
  • PW: Pseudowire
  • RD: Route Distinguisher
  • RR: Route Reflector
  • RT: Route Target
  • S/C-TAG: refers to a combination of Service Tag and Customer Tag in a 802.1Q frame
  • TOR: Top-Of-Rack
  • UMR: Unknown MAC Route
  • vES: virtual Ethernet Segment
  • VNI/VSID: refers to VXLAN/NVGRE virtual identifiers
  • VPLS: Virtual Private LAN Service
  • VSI: Virtual Switch Instance or VPLS instance in a particular PE
  • VXLAN: Virtual eXtensible LAN

3. Decoupled Interconnect Solution for EVPN-Overlay Networks 🦋

This section covers the interconnect solution where the GW and WAN Edge functions live in different systems 🧚 Figure 1 shows the reference model for this section. Heads up, even though it isn't drawn in Figure 1, GWs can also have local Attachment Circuits (ACs) 💜

                                   +--+
                                   |CE|
                                   +--+
                                     |
                                  +----+
                             +----| PE |----+
           +---------+       |    +----+    |       +---------+
   +----+  |        +---+  +----+        +----+  +---+        |  +----+
   |NVE1|--|        |   |  |WAN |        |WAN |  |   |        |--|NVE3|
   +----+  |        |GW1|--|Edge|        |Edge|--|GW3|        |  +----+
           |        +---+  +----+        +----+  +---+        |
           |  NVO-1   |      |     WAN      |      |   NVO-2  |
           |        +---+  +----+        +----+  +---+        |
           |        |   |  |WAN |        |WAN |  |   |        |
   +----+  |        |GW2|--|Edge|        |Edge|--|GW4|        |  +----+
   |NVE2|--|        +---+  +----+        +----+  +---+        |--|NVE4|
   +----+  +---------+       |              |       +---------+  +----+
                             +--------------+

   |<-EVPN-Overlay-->|<-VLAN->|<-WAN L2VPN->|<--PW-->|<--EVPN-Overlay->|
                      handoff               handoff

Figure 1: Decoupled Interconnect Model

Next up, the interconnect requirements for this model 🎀

3.1. Interconnect Requirements ✨

okay so the decoupled interconnect is built for when the EVPN-Overlay folks and the WAN folks are different companies and you need a clean line between who's responsible for what 🫶 this solution has to nail these things:

  • A simple connectivity handoff between the EVPN-Overlay network provider and the WAN provider, so QoS and security enforcement are easy to set up 💅
  • Independence from whatever L2VPN technology the WAN uses 🦋
  • Multihoming between GW and WAN Edge routers, including per-service load balancing 🌸 per-flow load balancing isn't a strong requirement, because a deterministic path per service is usually needed for easy QoS and security enforcement
  • Ethernet OAM and Connectivity Fault Management (CFM) [IEEE.802.1AG] [Y.1731] between the GW and the WAN Edge router, so individual AC failures get detected 🔥
  • Some optimizations at the GW 👑:
    • Flooding reduction of unknown unicast traffic sourced from the DC Network Virtualization Edge (NVE) devices
    • Control of the WAN MAC addresses advertised to the DC
    • Address Resolution Protocol (ARP) and Neighbor Discovery (ND) flooding control for requests coming from the WAN

3.2. VLAN-Based Handoff 🎀

so in this option, the handoff between the GWs and the WAN Edge routers runs on VLANs [IEEE.802.1Q] 💖 check out Figure 1 (between the GWs in NVO-1 and the WAN Edge routers). each MAC-VRF in the GW hooks up to a different VSI/MAC-VRF instance in the WAN Edge router, using a different C-TAG VLAN ID or a different combination of S/C-TAG VLAN IDs that matches on both sides ✨

this option gives you the best possible demarcation between the DC and WAN providers, and it doesn't need any control plane interaction between the two providers 🫶 the catch is the provisioning overhead, because the service has to be mapped to a C-TAG or S/C-TAG VLAN ID combination at both the GW and the WAN Edge routers 😭

in this model, the GW acts as a regular Network Virtualization Edge (NVE) toward the DC. its control plane, data plane procedures, and interactions are described in [RFC8365] 💅

the WAN Edge router acts as a (PBB-)VPLS or (PBB-)EVPN PE with Attachment Circuits (ACs) to the GWs. its functions are described in [RFC4761], [RFC4762], [RFC6074], [RFC7432], and [RFC7623] 🌈

3.3. PW-Based Handoff 🦋

if MPLS is an option between the GW and the WAN Edge router, you can deploy a PW-based interconnect solution 🧚 here the handoff between both routers is based on FEC128-based PWs [RFC4762], or FEC129-based PWs (for a greater level of network automation) [RFC6074]. note that this model still gives a clear demarcation between DC and WAN, since there's a single PW between each MAC-VRF and peer VSI, and security/QoS policies can be applied on a per-PW basis 💜 it scales better than a C-TAG-based handoff and needs less provisioning than a combined C/S-TAG handoff. the PW-based handoff is shown in Figure 1 (between the NVO-2 GWs and the WAN Edge routers) ✨

in this model, on top of the usual MPLS procedures between the GW and the WAN Edge router [RFC3031], the GW MUST support an interworking function in each MAC-VRF that needs to extend to the WAN 👑

  • if a FEC128-based PW is used between the MAC-VRF (GW) and the VSI (WAN Edge), the corresponding Virtual Connection Identifier (VCID) MUST be provisioned on the MAC-VRF and MUST match the VCID used in the peer VSI at the WAN Edge router 🍓
  • if BGP Auto-discovery [RFC6074] and FEC129-based PWs are used between the GW MAC-VRF and the WAN Edge VSI, the provisioning of the VPLS-ID MUST be supported on the MAC-VRF and MUST match the VPLS-ID used in the WAN Edge VSI 💀

if you're doing a PW-based handoff, the GW's AC (or point of attachment to the EVPN instance) uses a combo of a PW label and VLAN IDs. PWs are treated as service interfaces, as defined in [RFC7432] 🌸

3.4. Multihoming Solution on the GWs 💅

EVPN single-active multihoming (aka per-service load-balancing multihoming) is required for this type of interconnect 🔥

the GWs get provisioned with a unique ES for each WAN interconnect, and the handoff attachment circuits or PWs between the GW and the WAN Edge router get assigned an ESI for each of those ESes 🫶 the ESI is administratively configured on the GWs following the procedures in [RFC7432]. from here on this I-ES is called "I-ES", and its identifier is called "I-ESI". the different ESI types are described in [RFC7432] 🦋 using Type 0 for the I-ESI is RECOMMENDED in this document 💖

the solution (on the GWs) MUST follow the single-active multihoming procedures described in [RFC8365] for the provisioned I-ESI. so Ethernet A-D routes per ES and per EVI get advertised to the DC NVEs for the multihoming functions, and ES routes get advertised so ES discovery and Designated Forwarder (DF) procedures can happen 🧚 the MAC addresses learned (in the data plane) on the handoff links get advertised with the I-ESI encoded in the ESI field ✨

3.5. Gateway Optimizations 🌈

these GW features are optional, and they optimize the control plane and data plane in the DC 💜

3.5.1. MAC Address Advertisement Control 🎀

EVPN in NVO networks brings a ton of benefits, as described in [RFC8365] 🙈 but if multiple DCs get interconnected into a single EVI, each DC has to import all the MAC addresses from every other DC 😭

even with optimized BGP techniques like RT constraint [RFC4684], the number of MAC addresses that a DC's GWs advertise (or withdraw, if something fails) could overwhelm the NVEs in that DC, especially when the NVEs live in hypervisors 💀

this solution uses the Unknown MAC Route (UMR) advertised into a given DC by each of the DC's GWs 👑 it's defined in [RFC7543], and it's a regular EVPN MAC/IP Advertisement route where the MAC Address Length is set to 48, the MAC address is set to 0, and the ESI field is set to the DC GW's I-ESI 🫶

an NVE in that DC that understands and processes the UMR will send unknown unicast frames to one of the DC's GWs, which then forwards the packet to the correct egress PE 🦋 since the ESI is set to the DC GW's I-ESI, all-active multihoming can be applied to unknown unicast MAC addresses 🔥 an NVE that doesn't understand the Unknown MAC Route handles unknown unicast as described in [RFC7432] 💅

this document proposes that local policy decide whether MAC addresses and/or the UMR get advertised into a given DC 🌸 for example, when all the DC MAC addresses are learned in the control/management plane, it may make sense to advertise only the UMR. advertising all the DC MAC addresses in the control/management plane is usually the case when the NVEs live in hypervisors ✨ see [RFC8365], Section 7 🧚

it's worth noting that the UMR usage in [RFC7543] and the UMR usage in this document are different 👀 in the former, a Virtual Spoke (V-spoke) doesn't necessarily learn all the MAC addresses for hosts in other V-spokes of the same network. communication between two V-spokes goes through the Default MAC Gateway (DMG) until the V-spokes learn each other's MAC addresses. in this document, two leaf switches in the same DC are recommended so V-spokes learn each other's MAC addresses for the same EVI. leaf-to-leaf communication is always direct and doesn't go through the GW 💖

3.5.2. ARP/ND Flooding Control 🌈

another optimization, naturally provided by EVPN in the GWs, is the Proxy ARP/ND function 🫶 the GWs should build a Proxy ARP/ND cache table, as per [RFC7432]. when the active GW gets an ARP/ND request/solicitation from the WAN, it does a Proxy ARP/ND table lookup and replies as long as the info is in its table 💅

this mechanism is especially recommended on the GWs, since it protects the DC network from external ARP/ND-flooding storms 🔥

3.5.3. Handling Failures between GW and WAN Edge Routers 💀

link/PE failures are handled on the GWs as specified in [RFC7432]. the GW that detects the failure withdraws the EVPN routes, as per [RFC7432] 😭

individual AC/PW failures may be detected by OAM mechanisms. for instance:

  • if the interconnect solution is based on a VLAN handoff, Ethernet-CFM [IEEE.802.1AG] [Y.1731] may be used to detect individual AC failures on both the GW and the WAN Edge router. an individual AC failure triggers the withdrawal of the corresponding A-D per EVI route, as well as the MACs learned on that AC 🦋
  • if the interconnect solution is based on a PW handoff, the Label Distribution Protocol (LDP) PW Status bits TLV [RFC6870] may be used to detect individual PW failures on both the GW and the WAN Edge router 🧚

4. Integrated Interconnect Solution for EVPN-Overlay Networks 👑

when the DC and the WAN are run by the same administrative entity, the Service Provider can decide to combine the GW and WAN Edge PE functions in the same router, for obvious reasons: saving on Capital Expenditure (CAPEX) and Operating Expenses (OPEX) 💅 this is shown in Figure 2. heads up, this model doesn't give you an explicit demarcation link between DC and WAN anymore 🙈 and although it isn't drawn in Figure 2, the GWs may have local ACs ✨

                             +--+
                             |CE|
                             +--+
                               |
                            +----+
                       +----| PE |----+
           +---------+ |    +----+    | +---------+
   +----+  |        +---+            +---+        |  +----+
   |NVE1|--|        |   |            |   |        |--|NVE3|
   +----+  |        |GW1|            |GW3|        |  +----+
           |        +---+            +---+        |
           |  NVO-1   |       WAN      |   NVO-2  |
           |        +---+            +---+        |
           |        |   |            |   |        |
   +----+  |        |GW2|            |GW4|        |  +----+
   |NVE2|--|        +---+            +---+        |--|NVE4|
   +----+  +---------+ |              | +---------+  +----+
                       +--------------+

   |<--EVPN-Overlay--->|<-----VPLS--->|<---EVPN-Overlay-->|
                       |<--PBB-VPLS-->|
     Interconnect  ->  |<-EVPN-MPLS-->|
      options          |<--EVPN-Ovl-->|*
                       |<--PBB-EVPN-->|

   * EVPN-Ovl stands for EVPN-Overlay (and it's an interconnect option).

                  Figure 2: Integrated Interconnect Model

4.1. Interconnect Requirements ✨

okay so the integrated interconnect solution has to hit these requirements, bestie 💅

  • Control plane and data plane interworking between the EVPN-Overlay network and the L2VPN technology supported in the WAN, no matter which one you pick. That's (PBB-)VPLS or (PBB-)EVPN, as shown in Figure 2 🦋
  • Multihoming, including single-active multihoming with per-service load balancing or all-active multihoming (aka per-flow load-balancing), as long as the technology deployed in the WAN supports it 🌸
  • Support for end-to-end MAC Mobility, Static MAC protection and other procedures (e.g., proxy-arp) described in [RFC7432], as long as EVPN-MPLS is the technology of choice in the WAN 💜
  • Independent inclusive multicast trees in the WAN and in the DC. In other words, the inclusive multicast tree type defined in the WAN doesn't have to match the one in the DC 🧚

4.2. VPLS Interconnect for EVPN-Overlay Networks 🎀

4.2.1. Control/Data Plane Setup Procedures on the GWs ✨

Regular MPLS tunnels and Targeted LDP (tLDP) / BGP sessions get set up to the WAN PEs and RRs as per [RFC4761], [RFC4762], and [RFC6074], and overlay tunnels and EVPN get set up as per [RFC8365]. Note that different route targets for the DC and the WAN are normally required (unless [RFC4762] is used in the WAN, in which case no WAN route target is needed). A single type-1 RD per service may be used 🌈

In order to support multihoming, the GWs get provisioned with an I-ESI (see Section 3.4), which is unique for each interconnection. In this case, the I-ES represents the group of PWs to the WAN PEs and GWs. All the [RFC7432] procedures are still followed for the I-ES. For example, any MAC address learned from the WAN gets advertised to the DC with the I-ESI in the ESI field 🔥

A MAC-VRF per EVI gets created in each GW. The MAC-VRF has two different types of tunnel bindings, instantiated in two different split-horizon groups:

  • VPLS PWs get instantiated in the WAN split-horizon group.
  • Overlay tunnel bindings (e.g., VXLAN, NVGRE) get instantiated in the DC split-horizon group.

Attachment circuits are also supported on the same MAC-VRF (although they aren't shown in Figure 2), but they aren't part of either of the split-horizon groups above 💖

Traffic received in a given split-horizon group will never be forwarded to a member of the same split-horizon group. No exceptions, periodt 🚫

As far as BUM flooding goes, a flooding list is made up of the sublist created by the inclusive multicast routes and the sublist created for VPLS in the WAN. BUM frames received from a local Attachment Circuit (AC) get forwarded to the flooding list. BUM frames received from the DC or the WAN also get forwarded to the flooding list, observing the split-horizon group rule described above 🌸

Heads up: the GWs are not allowed to have an EVPN binding and a PW to the same far end within the same MAC-VRF. This keeps loops and packet duplication away. If a GW can successfully establish both an EVPN binding and a PW to the same far-end PE, the EVPN binding wins, and the PW gets brought down operationally 😭

The optimization procedures described in Section 3.5 can also be applied to this model ✨

4.2.2. Multihoming Procedures on the GWs 🦋

This model supports single-active multihoming on the GWs. All-active multihoming isn't supported by VPLS, so it can't be used on the GWs 🙈

In this case, for a given EVI, all the PWs in the WAN split-horizon group get assigned to I-ES. All the single-active multihoming procedures described by [RFC8365] get followed for the I-ES 💜

The non-DF GW for the I-ES blocks the transmission and reception of all the PWs in the WAN split-horizon group for BUM and unicast traffic 🛑

4.3. PBB-VPLS Interconnect for EVPN-Overlay Networks 🧚

4.3.1. Control/Data Plane Setup Procedures on the GWs 🌈

Here there's no impact on the procedures described in [RFC7041] for the B-component. However, the I-component instances become EVI instances with EVPN-Overlay bindings and potentially local attachment circuits. A number of MAC-VRF instances can be multiplexed into the same B-component instance, which saves a ton of PWs to maintain in the WAN 🤑

The I-ESI concept described in Section 4.2.1 is also used for the PBB-VPLS-based interconnect 💅

B-component PWs and I-component EVPN-Overlay bindings established to the same far end get compared. The following rules are observed:

  • Attempts to set up a PW between the two GWs within the B-component context will never be blocked.
  • If a PW exists between two GWs for the B-component and an attempt is made to set up an EVPN binding on an I-component linked to that B-component, the EVPN binding stays down operationally. The BGP EVPN routes are still valid but not used 🙃
  • The EVPN binding only comes up and gets used as long as there's no PW to the same far end in the corresponding B-component. The EVPN bindings in the I-components get brought down before the PW in the B-component is brought up ⭐

The optimization procedures described in Section 3.5 can also be applied to this interconnect option 💖

4.3.2. Multihoming Procedures on the GWs 👑

This model supports single-active multihoming on the GWs. All-active multihoming isn't supported in this scenario 🙅

The single-active multihoming procedures described by [RFC8365] get followed for the I-ES for each EVI instance connected to the B-component. Note that in this case, for a given EVI, all the EVPN bindings in the I-component get assigned to the I-ES. The non-DF GW for the I-ES blocks the transmission and reception of all the I-component EVPN bindings for BUM and unicast traffic. When learning MACs from the WAN, the non-DF MUST NOT advertise EVPN MAC/IP routes for those MACs 🫶

4.4. EVPN-MPLS Interconnect for EVPN-Overlay Networks 🌸

If EVPN for MPLS tunnels (referred to as "EVPN-MPLS" hereafter) are supported in the WAN, an end-to-end EVPN solution can be deployed. The following sections describe the proposed solution and its impact on the procedures from [RFC7432] 🦋

4.4.1. Control plane Setup Procedures on the GWs 💅

The GWs MUST establish separate BGP sessions for sending and receiving EVPN routes to and from the DC, and to and from the WAN. Normally, each GW sets up one BGP EVPN session to the DC RR (or two BGP EVPN sessions if there are redundant DC RRs) and one session to the WAN RR (or two sessions if there are redundant WAN RRs) 🔥

To make separate BGP processes for the DC and WAN easier, EVPN routes sent to the WAN SHOULD carry a different Route Distinguisher (RD) than the EVPN routes sent to the DC. Also, although reusing the same value is possible, different route targets are expected to be handled for the same EVI in the WAN and the DC. Note that the EVPN service routes sent to the DC RRs will normally include a [RFC9012] BGP encapsulation extended community with a different tunnel type than the one sent to the WAN RRs 💜

As in the other options discussed, an I-ES and its assigned I-ESI get configured on the GWs for multihoming. This I-ES represents the WAN EVPN-MPLS PEs to the DC, but also the DC EVPN-Overlay NVEs to the WAN. Optionally, different I-ESI values can be configured for representing the WAN and the DC. If different EVPN-Overlay networks are connected to the same group of GWs, each EVPN-Overlay network MUST get assigned a different I-ESI 🧚

Received EVPN routes never get reflected on the GWs. Instead, they get consumed and re-advertised (if needed):

  • Ethernet A-D routes, ES routes, and Inclusive Multicast routes are consumed by the GWs and processed locally for the corresponding [RFC7432] procedures.
  • MAC/IP advertisement routes get received and imported. If they become active in the MAC-VRF, the information gets re-advertised as new routes with the following fields:
    • The RD will be the GW's RD for the MAC-VRF.
    • The ESI will be set to the I-ESI.
    • The Ethernet-tag value will be kept from the received NLRI.
    • The MAC length, MAC address, IP Length, and IP address values will be kept from the received NLRI.
    • The MPLS label will be a local 20-bit value (when sent to the WAN) or a DC-global 24-bit value (when sent to the DC for encapsulations using a VNI).
    • The appropriate Route Targets (RTs) and [RFC9012] BGP encapsulation extended community will be used according to [RFC8365].

The GWs also generate the following local EVPN routes, which get sent to the DC and WAN with their corresponding RTs and [RFC9012] BGP encapsulation extended community values 🌈

  • ES route(s) for the I-ESI(s).
  • Ethernet A-D routes per ES and EVI for the I-ESI(s). The A-D per-EVI routes sent to the WAN and the DC will have consistent Ethernet-Tag values.
  • Inclusive Multicast routes with an independent tunnel-type value for the WAN and DC. For example, a P2MP Label Switched Path (LSP) may be used in the WAN, whereas ingress replication may be used in the DC. The routes sent to the WAN and the DC will have a consistent Ethernet-Tag.
  • MAC/IP advertisement routes for MAC addresses learned in local attachment circuits. Note that these routes won't include the I-ESI value in the ESI field. These routes will include a zero ESI or a non-zero ESI for local multihomed Ethernet Segments (ES). The routes sent to the WAN and the DC will have a consistent Ethernet-Tag. ⭐

Assuming GW1 and GW2 are peer GWs of the same DC, each GW generates two sets of the above local service routes. Set-DC gets sent to the DC RRs and includes an A-D per EVI, Inclusive Multicast, and MAC/IP routes for the DC encapsulation and RT. Set-WAN gets sent to the WAN RRs and includes the same routes, but using the WAN RT and encapsulation. GW1 and GW2 receive each other's set-DC and set-WAN. This is the expected behaviour on GW1 and GW2 for locally generated routes 🫶

  • Inclusive multicast routes: When setting up the flooding lists for a given MAC-VRF, each GW includes its DC peer GW only in the EVPN-MPLS flooding list (by default), and not in the EVPN-Overlay flooding list. That is, GW2 imports two Inclusive Multicast routes from GW1 (from set-DC and set-WAN) but only considers one of the two, giving the set-WAN route higher priority. An administrative option MAY change this preference so that the set-DC route is selected first 👑
  • MAC/IP advertisement routes for local attachment circuits: As above, the GW selects only one, giving the route from the set-WAN a higher priority. As with the Inclusive multicast routes, an administrative option MAY change this priority 🎀

4.4.2. Data Plane Setup Procedures on the GWs ✨

Okay so the procedure we just covered is what keeps things loop-free and duplicate-free between the GWs of the same EVPN-Overlay network 🔁🚫 (for frames coming in from local ACs). Only one EVPN binding per EVI (or per Ethernet Tag when you're using VLAN-aware bundle services) gets set up in the data plane between the two nodes 💅. By default, that binding gets added to the EVPN-MPLS flooding list 🌸.

As for the rest of the EVPN tunnel bindings, they get sorted into one of the two flooding lists that each GW sets up for the same MAC-VRF 🎀:

  • EVPN-Overlay flooding list (made up of bindings to the remote NVEs, or the multicast tunnel to the NVEs) 🦋
  • EVPN-MPLS flooding list (made up of MP2P or LSM tunnels to the remote PEs) 🌐

Each flooding list is part of its own split-horizon group: either the WAN split-horizon group or the DC split-horizon group 👑. Traffic that comes from a local AC can get flooded to both split-horizon groups. Traffic from a binding in one split-horizon group can get flooded to the other split-horizon group and to local ACs, but never back to a member of its own split-horizon group 🙅‍♀️. Iconic behaviour, honestly 😭.

When GW1 or GW2 gets a BUM frame on an MPLS tunnel that has an ESI label at the bottom of the stack, it does an ESI label lookup and split-horizon filtering, as per [RFC7432] 🔍. It does this in case the ESI label points to a local ESI (I-ESI or any other nonzero ESI) 💜.

4.4.3. Multihoming Procedure Extensions on the GWs 🫶

This model supports single-active multihoming AND all-active multihoming 🔥.

All the [RFC7432] multihoming procedures for DF election on I-ES(s), plus the backup-path (single-active) and aliasing (all-active) procedures, get followed on the GWs 💖. Remote PEs in the EVPN-MPLS network follow regular [RFC7432] aliasing or backup-path procedures for MAC/IP routes they get from the GWs for the same I-ESI 🧚. NVEs in the EVPN-Overlay network do the same for MAC/IP routes received with that same I-ESI 🌈.

As far as the forwarding plane goes, the EVPN-Overlay network acts by default like the access ACs in [RFC7432] multihomed Ethernet Segments. Main character energy, no cap 💅✨.

Here's how forwarding works on the GWs:

  • Single-active multihoming (assuming a WAN split-horizon group made of EVPN-MPLS bindings, a DC split-horizon group made of EVPN-Overlay bindings, and local ACs on the GWs) 🌸:
    • Forwarding behaviour on the non-DF: the non-DF MUST block ingress and egress forwarding on the EVPN-Overlay bindings associated to the I-ES 🚫. The EVPN-MPLS network counts as the core network, so the EVPN-MPLS bindings to the remote PEs and GWs stay active 🌐.
    • Forwarding behaviour on the DF: the DF MUST NOT forward BUM or unicast traffic received from a given split-horizon group to a member of its own split-horizon group 🙅‍♀️. Forwarding to other split-horizon groups and local ACs is allowed (as long as the ACs aren't part of an ES where the node is non-DF) 🎀. As per [RFC7432], and for split-horizon purposes, when the DF GW receives BUM traffic on the EVPN-Overlay bindings associated to an I-ES, it SHOULD add the I-ESI label when forwarding to the peer GW over EVPN-MPLS 🏷️. (Ngl, this is the bit that keeps everyone honest 🫣.)
    • When the non-DF receives EVPN MAC/IP routes from the WAN, it MUST NOT reoriginate those EVPN routes and advertise them to the DC peers 🚧. Same vibe the other way: EVPN MAC/IP routes received from the DC MUST NOT be advertised to the WAN peers 🙅‍♀️. This is consistent with [RFC7432], and it lets the remote PE/NVEs figure out who the primary GW is, just from which MAC/IP routes they receive 👑🔍.
  • All-active multihoming (same setup: WAN split-horizon group of EVPN-MPLS bindings, DC split-horizon group of EVPN-Overlay bindings, and local ACs on the GWs) 🦋:
    • Forwarding behaviour on the non-DF: the non-DF follows the same behaviour as the non-DF in the single-active case, but only for BUM traffic 🙈. Unicast traffic received from a split-horizon group MUST NOT be forwarded to a member of its own split-horizon group, but it can be forwarded normally to the other split-horizon groups and local ACs 💖. If a known unicast packet gets identified as a "flooded" packet, the procedures for BUM traffic MUST be followed 🚨.
    • Forwarding behaviour on the DF: the DF follows the same behaviour as the DF in the single-active case, but only for BUM traffic 🌸. Unicast traffic received from a split-horizon group MUST NOT be forwarded to a member of its own split-horizon group, but it can be forwarded normally to the other split-horizon group and local ACs 🎀. If a known unicast packet gets identified as a "flooded" packet, the procedures for BUM traffic MUST be followed 😭. As per [RFC7432], and for split-horizon purposes, when the DF GW receives BUM traffic on the EVPN-Overlay bindings associated to an I-ES, the DF GW MUST add the I-ESI label when forwarding to the peer GW over EVPN-MPLS 🏷️✨.
    • Unlike the single-active case, both the DF and the non-DF reoriginate and advertise MAC/IP routes received from the WAN/DC peers 🌈. They add the corresponding I-ESI so the remote PE/NVEs can do regular aliasing, as per [RFC7432] 🫶.

The example in Figure 3 shows how BUM traffic that starts at an NVE moves through a pair of all-active multihoming GWs 🦋👀:

        |<--EVPN-Overlay--->|<--EVPN-MPLS-->|

                +---------+ +--------------+
         +----+ BUM       +---+             |
         |NVE1+----+----> |   +-+-----+     |
         +----+  | |   DF |GW1| |     |     |
                 | |      +-+-+ |     |    ++--+
                 | |        |   |     +--> |PE1|
                 | +--->X +-+-+ |          ++--+
                 |     NDF|   | |           |
         +----+  |        |GW2<-+           |
         |NVE2+--+        +-+-+             |
         +----+  +--------+ |  +------------+
                            v
                          +--+
                          |CE|
                          +--+

Figure 3: Multihoming BUM Forwarding

So GW2 is the non-DF for the I-ES and blocks the BUM forwarding 🚫. GW1 is the DF, so it forwards the traffic to PE1 and GW2 💁‍♀️. Packets sent to GW2 include the ESI label for the I-ES 🏷️. Based on that ESI label, GW2 knows they're I-ES-generated packets, so it only forwards them to local ACs (the CE in the example) and NOT back into the EVPN-Overlay network 🙅‍♀️💜.

4.4.4. Impact on MAC Mobility Procedures 🦋

MAC Mobility procedures described in [RFC7432] are not modified by this document 🌸. Easy peasy.

Note that an intra-DC MAC move still leaves the MAC attached to the same I-ES. So under the rules of [RFC7432], this does NOT count as a MAC Mobility event 🙌. Only when the MAC moves from the WAN domain to the DC domain (or from one DC to another) does the MAC get learned from a different ES, and that's when the MAC Mobility procedures kick in 🧚‍♀️.

The sticky-bit indication in the MAC Mobility extended community MUST be propagated between domains 🎀🔁.

4.4.5. Gateway Optimizations ⭐

All the Gateway optimizations described in Section 3.5 MAY be applied to the GWs when the interconnect is based on EVPN-MPLS 🌈. Optional but very cute 🍓.

In particular, using the Unknown MAC Route (as described in Section 3.5.1) solves some transient packet-duplication issues in all-active multihoming cases, as explained below 🔥.

Consider the diagram in Figure 2 for EVPN-MPLS interconnect with all-active multihoming, and the following sequence 🎬:

(a) MAC Address M1 is advertised from NVE3 in EVI-1.

(b) GW3 and GW4 learn M1 for EVI-1 and re-advertise M1 to the WAN with I-ESI-2 in the ESI field.

(c) GW1 and GW2 learn M1 and install GW3/GW4 as next hops, following the EVPN aliasing procedures.

(d) Before NVE1 learns M1, a packet arrives at NVE1 with destination M1. If the Unknown MAC Route had not been advertised into the DC, NVE1 would have flooded the packet throughout the DC, in particular to both GW1 and GW2. If the same VNI/VSID is used for both known unicast and BUM traffic (as is typically the case), there is no indication in the packet that it's a BUM packet, so both GW1 and GW2 would have forwarded it, creating packet duplication 😱. BUT because the Unknown MAC Route had been advertised into the DC, NVE1 will unicast the packet to either GW1 or GW2 🙌.

(e) Since both GW1 and GW2 know M1, the GW that receives the packet will forward it to either GW3 or GW4 💅.

4.4.6. Benefits of the EVPN-MPLS Interconnect Solution 💜

The "DCI using ASBRs" solution described in [RFC8365] and the GW solution with EVPN-MPLS interconnect might look similar, since they both keep the EVPN attributes between Data Centers and all the way through the WAN 🤝. But the EVPN-MPLS interconnect solution on the GWs has some big wins over the "DCI using ASBRs" solution 👑:

  • As in any of the described GW models, this solution supports connecting local attachment circuits on the GWs. That's NOT possible in a "DCI using ASBRs" solution 🙅‍♀️.
  • Different data plane encapsulations can be supported in the DC and the WAN, whereas the "DCI using ASBRs" solution needs a uniform encapsulation 🌈.
  • Optimized multicast, with independent inclusive multicast trees in the DC and the WAN 🌸.
  • MPLS label aggregation: for the case where MPLS labels are signalled from the NVEs for MAC/IP advertisement routes, this solution provides label aggregation 🏷️. A remote PE MAY receive a single label per GW MAC-VRF, as opposed to a label per NVE/MAC-VRF connected to the GW MAC-VRF 🎀. For instance, in Figure 2, PE would receive only one label for all the routes advertised for a given MAC-VRF from GW1, as opposed to a label per NVE/MAC-VRF 🧚‍♀️.
  • The GW will NOT propagate MAC Mobility for MACs moving within a DC. Intra-DC mobility is handled by all the NVEs in the DC 🙌. The MAC Mobility procedures on the GWs are only required in the case of mobility across DCs 🦋.
  • Proxy-ARP/ND function on the DC GWs can be leveraged to reduce ARP/ND flooding in the DC and/or the WAN 🌊.

4.5. PBB-EVPN Interconnect for EVPN-Overlay Networks 🌟

PBB-EVPN [RFC7623] is yet another interconnect option 🎀. It requires the use of GWs where I-components and associated B-components are part of EVI instances 💖.

4.5.1. Control/Data Plane Setup Procedures on the GWs ✨

Okay so both components run EVPN on their own 💅: the I-component MAC-VRF and the B-component MAC-VRF. Compared to [RFC7623], the DC customer MACs (C-MACs) are no longer learned in the data plane on the GW. They're learned in the control plane through EVPN running on the I-component 🧠. Remote C-MACs coming from remote PEs are still learned in the data plane 🌸. B-MACs in the B-component get assigned and advertised following the procedures in [RFC7623] 🎀.

An I-ES gets configured on the GWs for multihoming 🫶, but its I-ESI only gets used in the EVPN control plane for the I-component EVI. No unreserved ESIs get used in the control plane of the B-component EVI, per [RFC7623] 🙅‍♀️. So the I-ES shows up to the WAN PBB-EVPN PEs using shared or dedicated B-MACs 🦋.

The rest of the control plane stuff follows [RFC7432] for the I-component EVI and [RFC7623] for the B-component EVI 💜.

From the data plane side, the I-component and B-component EVPN bindings that go to the same far end get compared. The I-component EVPN-Overlay binding stays down following the rules in Section 4.3.1 💀. Chef's kiss 🤌.

4.5.2. Multihoming Procedures on the GWs

Okay this model supports single-active AND all-active multihoming 🙌. The DF and non-DF forwarding behavior changes based on what's described in Section 4.4.3. The WAN split-horizon group gets swapped in for the B-component 🔀, and [RFC7623] procedures handle the traffic sent or received on the B-component ✨.

4.5.3. Impact on MAC Mobility Procedures

C-MACs learned from the B-component get advertised in EVPN within the I-component EVI scope 📣. If the C-MAC was already known in the I-component database, EVPN advertises it with a higher sequence number, as per [RFC7432] 🔢. From the point of view of Mobility and the related procedures in [RFC7432], C-MACs learned from the B-component count as local 🏠.

4.5.4. Gateway Optimizations

Everything we covered in Section 4.4.5 applies to the PBB-EVPN interconnect option too 💖.

4.6. EVPN-VXLAN Interconnect for EVPN-Overlay Networks 🦋

If EVPN for Overlay tunnels is supported in the WAN and you need a GW function, you can deploy a full end-to-end EVPN solution 🌈. There are lots of overlay tunnel combos on the WAN and DC sides (MPLSoGRE, NVGRE, etc.), but we're focusing on VXLAN here because it's super popular 🌸. This section is about EVPN for VXLAN (EVPN-VXLAN from here on) and how it affects the [RFC7432] procedures 🧚.

The procedures in Section 4.4 apply here too 🔁. You just swap the EVPN-MPLS control plane specifics for EVPN-VXLAN ones, and use [RFC8365] "Local Bias" procedures instead of Section 4.4.3 💅. Since VXLAN has no ESI labels, GWs need to rely on "Local Bias" to apply split horizon on packets generated from the I-ES and sent to the peer GW 🙈.

So this setup assumes NVEs inside a DC use VNIs or VSIDs as globally unique identifiers 🆔. There's a Gateway at the DC edge that translates the VNI or VSID whenever traffic crosses a network boundary 🚪. That GW does VNI and tunnel-IP-address translation 🔄. If you want locally downstream-assigned VNIs or VSIDs (kinda like MPLS labels), that's covered in [RFC8365] 🫶.

While VNIs are globally significant within each DC, there are two possibilities in the interconnect network 👀:

  1. Globally unique VNIs in the interconnect network. The GWs and PEs in the interconnect network agree on a common VNI for a given EVI. The RT used in the interconnect network can be autoderived from the agreed-upon interconnect VNI. The VNI used inside each DC MAY be the same as the interconnect VNI.
  2. Downstream-assigned VNIs in the interconnect network. The GWs and PEs MUST use the proper RTs to import/export the EVPN routes. Even if the VNI is downstream assigned in the interconnect network, and unlike option (a), it only identifies the <Ethernet Tag, GW> pair and not the <Ethernet Tag, egress PE> pair. The VNI used inside each DC MAY be the same as the interconnect VNI. GWs SHOULD support multiple VNI spaces per EVI (one per interconnect network they are connected to).

In both options, NVEs inside a DC only have to be aware of a single VNI space 🧘‍♀️, and only GWs handle the complexity of managing multiple VNI spaces 💪. On top of the VNI translation above, the GWs translate the tunnel source IP for the packets generated from the NVEs, using their own IP address 🪞. GWs use that IP address as the BGP next hop in all the EVPN updates to the interconnect network 🛣️.

The following sections give you the deets on these two options 💖.

4.6.1. Globally Unique VNIs in the Interconnect Network

Considering Figure 2, say a host H1 in NVO-1 needs to talk to a host H2 in NVO-2 🙋‍♀️. Assume different VNIs get used in each DC for the same EVI (for example, VNI-10 in NVO-1 and VNI-20 in NVO-2). Then the VNIs MUST be translated to a common interconnect VNI (for example, VNI-100) on the GWs 🔁. Each GW gets provisioned with a VNI translation mapping, so it can translate the VNI in the control plane when sending BGP EVPN route updates to the interconnect network 🗺️. In other words, GW1 and GW2 MUST be configured to map VNI-10 to VNI-100 in the BGP update messages for H1's MAC route 🎯. This mapping also translates the VNI in the data plane in both directions 🔀. That's VNI-10 to VNI-100 when the packet is received from NVO-1, and the reverse mapping from VNI-100 to VNI-10 when the packet is received from the remote NVO-2 network and needs to be forwarded to NVO-1 💅.

The procedures described in Section 4.4 get followed, considering that the VNIs advertised/received by the GWs get translated accordingly ✨.

4.6.2. Downstream-Assigned VNIs in the Interconnect Network

Same vibe here 🌸. If a host H1 in NVO-1 needs to talk to a host H2 in NVO-2, and different VNIs are used in each DC for the same EVI (e.g., VNI-10 in NVO-1 and VNI-20 in NVO-2), then the VNIs MUST be translated as in Section 4.6.1. The twist is that there's no need to translate to a common interconnect VNI on the GWs 🙃. Each GW can translate the VNI received in an EVPN update to a locally assigned VNI advertised to the interconnect network. Each GW can use a different interconnect VNI, so that VNI doesn't need to be agreed upon across all the GWs and PEs of the interconnect network 🧩.

The procedures described in Section 4.4 get followed, taking into account the considerations above for the VNI translation 💜.

5. Security Considerations 🔐

This document applies existing specifications to a number of interconnect models 📖. The security considerations included in those documents, such as [RFC7432], [RFC8365], [RFC7623], [RFC4761], and [RFC4762], apply to this document whenever those technologies are used 🚨.

As discussed, [RFC8365] talks about two main DCI solution groups: "DCI using GWs" and "DCI using ASBRs" 🧐. This document specifies the solutions that correspond to the "DCI using GWs" group. It's super important to note that using GWs gives a superior level of security on a per-tenant basis, compared to using ASBRs 🛡️. That's because GWs need to perform a MAC lookup on the frames received from the WAN, and they apply security procedures like filtering undesired frames, filtering frames with a source MAC that matches a protected MAC in the DC, or applying MAC-duplication procedures defined in [RFC7432] 🙅‍♀️. On ASBRs, though, traffic is forwarded based on a label or VNI swap, and there is usually no visibility of the encapsulated frames, which can carry malicious traffic 💀.

The GW optimizations in this document also add extra protection for the DC tenant systems 🫶. For instance, the MAC-address advertisement control and Unknown MAC Route defined in Section 3.5.1 protect the DC NVEs from getting overwhelmed by an excessive number of MAC/IP routes learned on the GWs from the WAN 🧱. The ARP/ND flooding control described in Section 3.5.2 can reduce or suppress broadcast storms being injected from the WAN 🌪️.

Finally, the reader should be aware of the potential security implications of designing a DCI with the decoupled interconnect solution (Section 3) or the integrated interconnect solution (Section 4) 👑. In the decoupled interconnect solution, the DC is typically easier to protect from the WAN, since each GW has a single logical link to one WAN PE. In the integrated solution, the GW has logical links to all the WAN PEs that are attached to the tenant 🕸️. In either model, proper control plane and data plane policies should be put in place in the GWs in order to protect the DC from potential attacks coming from the WAN 🔒.

6. IANA Considerations 🧾

This document has no IANA actions 🙌.

7. References 📚

7.1. Normative References

okay so these are the must-read docs 📚 the normative ones are the stuff you literally can't skip if you want the spec to make sense 🫶

7.2. Informative References

these ones are the extra reading 🌸 nice to have, lowkey great for going deeper 💜

Acknowledgments ✨

the authors wanted to give a shoutout to Neil Hart, Vinod Prabhu, and Kiran Nagaraj for their really valuable comments and feedback 🙌 and also a big thank you to Martin Vigoureux and Alvaro Retana for their super detailed reviews and comments 💖

Contributors 🫶

in addition to the authors on the front page, these coauthors also helped write this doc 💅

Ravi Shekhar
Juniper Networks


Anil Lohiya
Juniper Networks


Wen Lin
Juniper Networks


Florin Balus
Cisco


Patrice Brissette
Cisco


Senad Palislamovic
Nokia


Dennis Cai
Alibaba

Authors' Addresses 💜

the people behind the doc, with where to find them 📬

Jorge Rabadan (editor)
Nokia
777 E. Middlefield Road
Mountain View, CA 94043
United States of America

Email: jorge.rabadan@nokia.com


Senthil Sathappan
Nokia

Email: senthil.sathappan@nokia.com


Wim Henderickx
Nokia

Email: wim.henderickx@nokia.com


Ali Sajassi
Cisco

Email: sajassi@cisco.com


John Drake
Juniper

Email: jdrake@juniper.net
📦 where this page comes from