RFC 7623
Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN)
RFC 7623 ✨ Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN)
- RFC number: 7623
- Title: Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN)
- Authors: A. Sajassi (Ed.) from Cisco, S. Salam from Cisco, N. Bitar from Verizon, A. Isaac from Juniper, and W. Henderickx from Alcatel-Lucent
- Category: Standards Track
- Organization: Internet Engineering Task Force (IETF)
- ISSN: 2070-1721
- Date: September 2015
- Obsoletes/Updates: The header doesn't list anything this RFC obsoletes or updates. 🫶
Abstract 💖
Okay so this doc is about combining Ethernet Provider Backbone Bridging (PBB) with Ethernet VPN (EVPN) 🎀. The goal is to shrink the number of BGP MAC Advertisement routes by bundling Customer/Client MAC (C-MAC) addresses under a Provider Backbone MAC (B-MAC) address. It also gives clients MAC address mobility through that C-MAC aggregation, keeps C-MAC learning limited to active flows, adds per-site policies, and stops C-MAC addresses from getting flushed when the topology changes. The combo is called PBB-EVPN ✨.
Status of This Memo 🌸
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 5741 📚.
Info about the current status of this document, any errata, and how to send feedback lives at http://www.rfc-editor.org/info/rfc7623.
Copyright Notice 🦋
Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date this document was published. Please read these documents carefully, because they spell out your rights and restrictions for this document. Any Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🔥.
Table of Contents 🧚
1. Introduction ....................................................3
2. Terminology .....................................................4
3. Requirements ....................................................4
3.1. MAC Advertisement Route Scalability ........................5
3.2. C-MAC Mobility Independent of B-MAC Advertisements .........5
3.3. C-MAC Address Learning and Confinement .....................5
3.4. Per-Site Policy Support ....................................6
3.5. No C-MAC Address Flushing for All-Active Multihoming .......6
4. Solution Overview ...............................................6
5. BGP Encoding ....................................................7
5.1. Ethernet Auto-Discovery Route ..............................7
5.2. MAC/IP Advertisement Route .................................7
5.3. Inclusive Multicast Ethernet Tag Route .....................8
5.4. Ethernet Segment Route .....................................8
5.5. ESI Label Extended Community ...............................8
5.6. ES-Import Route Target .....................................9
5.7. MAC Mobility Extended Community ............................9
5.8. Default Gateway Extended Community .........................9
6. Operation .......................................................9
6.1. MAC Address Distribution over Core .........................9
6.2. Device Multihoming .........................................9
6.2.1. Flow-Based Load-Balancing ...........................9
6.2.1.1. PE B-MAC Address Assignment ...............10
6.2.1.2. Automating B-MAC Address Assignment .......11
6.2.1.3. Split Horizon and Designated
Forwarder Election ........................12
6.2.2. Load-Balancing based on I-SID ......................12
6.2.2.1. PE B-MAC Address Assignment ...............12
6.2.2.2. Split Horizon and Designated
Forwarder Election ........................13
6.2.2.3. Handling Failure Scenarios ................13
6.3. Network Multihoming .......................................14
6.4. Frame Forwarding ..........................................14
6.4.1. Unicast ............................................15
6.4.2. Multicast/Broadcast ................................15
6.5. MPLS Encapsulation of PBB Frames ..........................16
7. Minimizing ARP/ND Broadcast ....................................16
8. Seamless Interworking with IEEE 802.1aq / 802.1Qbp .............17
8.1. B-MAC Address Assignment ..................................17
8.2. IEEE 802.1aq / 802.1Qbp B-MAC Address Advertisement .......17
8.3. Operation: ................................................17
9. Solution Advantages ............................................18
9.1. MAC Advertisement Route Scalability .......................18
9.2. C-MAC Mobility Independent of B-MAC Advertisements ........18
9.3. C-MAC Address Learning and Confinement ....................19
9.4. Seamless Interworking with 802.1aq Access Networks ........19
9.5. Per-Site Policy Support ...................................20
9.6. No C-MAC Address Flushing for All-Active Multihoming ......20
10. Security Considerations .......................................20
11. IANA Considerations ...........................................20
12. References ....................................................21
12.1. Normative References .....................................21
12.2. Informative References ...................................21
Acknowledgements ..................................................22
Contributors ......................................................22
Authors' Addresses ................................................231. Introduction 🌈
Okay so [RFC7432] brings in a solution for multipoint Layer 2 Virtual Private Network (L2VPN) services 💅. It comes with advanced multihoming and uses BGP to distribute customer/client MAC address reachability info across the core MPLS/IP network. Then there's [PBB], which is an architecture for Ethernet Provider Backbone Bridging. It uses MAC tunneling to boost service instance and MAC address scalability in both Ethernet and VPLS networks [RFC7080] 🧚.
In this doc, we're talking about how PBB and EVPN can team up to do a few things ✨:
- Shrink the number of BGP MAC Advertisement routes by aggregating Customer/Client MAC (C-MAC) addresses under a Provider Backbone MAC (B-MAC) address.
- Give clients MAC address mobility using C-MAC aggregation.
- Limit C-MAC learning to only active flows.
- Offer per-site policies.
- Avoid C-MAC address flushing on topology changes.
The combined solution is called PBB-EVPN 🦋.
2. Terminology 🍓
Let's go over the acronyms first, bestie 🎀:
ARP: Address Resolution Protocol
BEB: Backbone Edge Bridge
B-MAC: Backbone MAC
B-VID: Backbone VLAN ID
CE: Customer Edge
C-MAC: Customer/Client MAC
ES: Ethernet Segment
ESI: Ethernet Segment Identifier
EVI: EVPN Instance
EVPN: Ethernet VPN
I-SID: Service Instance Identifier (24 bits and global within a PBB
network see [RFC7080])
LSP: Label Switched Path
MP2MP: Multipoint to Multipoint
MP2P: Multipoint to Point
NA: Neighbor Advertisement
ND: Neighbor Discovery
P2MP: Point to Multipoint
P2P: Point to Point
PBB: Provider Backbone Bridge
PE: Provider Edge
RT: Route Target
VPLS: Virtual Private LAN ServiceNow for two redundancy modes that matter a lot here 👑.
Single-Active Redundancy Mode: Picture a group of PEs attached to one Ethernet segment. If only one of them is allowed to forward traffic to and from that segment, the Ethernet segment is operating in Single-Active redundancy mode 🔥.
All-Active Redundancy Mode: This time every PE attached to the Ethernet segment is allowed to forward traffic to and from it. When that's the case, the Ethernet segment is operating in All-Active redundancy mode 💜.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119]. Those capital words are the official BCP 14 vibes, so read them exactly as written 🫶.
3. Requirements ✨
The requirements for PBB-EVPN include all the requirements for EVPN that were described in [RFC7209], plus the following:
3.1. MAC Advertisement Route Scalability 🌸
Normally, an EVPN PE sends one BGP MAC Advertisement route per C-MAC address. That can be a real scalability problem in certain applications. Take data center interconnect (DCI) scenarios, where the number of virtual machines (VMs), and therefore the number of C-MAC addresses, can hit the millions 😭. In cases like that, it is required to reduce the number of BGP MAC Advertisement routes by relying on a 'MAC summarization' scheme, the kind PBB provides 🧚.
3.2. C-MAC Mobility Independent of B-MAC Advertisements 🦋
Some applications, like virtual machine mobility, need fast C-MAC address mobility 🏃♀️. For those apps using EVPN, the virtual machine MAC address has to be sent in a BGP MAC Advertisement route. If it isn't, traffic gets forwarded to the wrong segment once a virtual machine moves from one ES to another 💀. That also means MAC address prefixes can't be used in data center applications.
To get C-MAC address mobility and still keep the scalability perks of MAC summarization, PBB technology gets used 💖. PBB defines a B-MAC address space that's independent of the C-MAC address space, and it aggregates C-MAC addresses under a single B-MAC address ✨.
3.3. C-MAC Address Learning and Confinement 👑
Here's the tea 🍵. In EVPN, every PE node in the same EVPN instance gets exposed to all the C-MAC addresses learned by any one of those PEs. Why? Because a C-MAC learned on one PE gets advertised in BGP to the other PEs in that EVPN instance 🙈. That happens even when some of the PEs for that instance aren't forwarding traffic to or from those C-MAC addresses at all. Even if an implementation doesn't install hardware forwarding entries for C-MAC addresses that aren't part of active traffic flows on that PE, the device memory still gets eaten up by keeping track of those C-MAC addresses in the routing information base (RIB) table 😩. In network applications with millions of C-MAC addresses, that turns into a non-trivial waste of PE resources 💅. So it is required to confine the scope of visibility of C-MAC addresses to only the PE nodes that are actively involved in forwarding traffic to or from those addresses 🎀.
3.4. Per-Site Policy Support 💖
okay so in tons of apps you need to enforce connectivity policy at the granularity of a site (or segment) 🎀 🧚. that means controlling which PE nodes in the network can forward traffic to, or from, a given site. both EVPN and PBB-EVPN can give you this level of policy control ✨. and if the policy has to go all the way down to per C-MAC address, then the C-MAC address should be learned in the control plane (in BGP) per [RFC7432] 🫶.
3.5. No C-MAC Address Flushing for All-Active Multihoming 🌸
just like in [RFC7432], it is required to avoid C-MAC address flushing upon link, port, or node failure for All-Active multihomed segments. no flushing drama, bestie 💅✨.
4. Solution Overview 🧚
the solution is giving the EVPN PE nodes IEEE Backbone Edge Bridge (BEB) functionality, kinda like how PBB-VPLS puts BEB functionality on the VPLS PE nodes 🦋. so the PE devices take 802.1Q Ethernet frames from their attachment circuits, wrap them in the PBB header, and forward them over the IP/MPLS core 📦. then on the egress EVPN PE, the PBB header is removed after MPLS disposition, and the original 802.1Q Ethernet frame gets delivered to the customer equipment 💜.
BEB +--------------+ BEB
|| | | ||
\/ | | \/
+----+ AC1 +----+ | | +----+ +----+
| CE1|-----| | | | | |---| CE2|
+----+\ | PE1| | IP/MPLS | | PE3| +----+
\ +----+ | Network | +----+
\ | |
AC2\ +----+ | |
\| | | |
| PE2| | |
+----+ | |
/\ +--------------+
||
BEB
<-802.1Q-> <------PBB over MPLS------> <-802.1Q->
Figure 1: PBB-EVPN Networkthe PE nodes do the following things 🌈:
- Learn customer/client MAC addresses (C-MACs) over the attachment circuits in the data plane, per normal bridge operation. 🍓
- Learn remote C-MAC to B-MAC bindings in the data plane for traffic received from the core per the bridging operation described in [PBB]. 🍓
- Advertise local B-MAC address reachability information in BGP to all other PE nodes in the same set of service instances. ngl this one's important: every PE has a set of B-MAC addresses that uniquely identifies the device 👑. B-MAC address assignment is described in detail in Section 6.2.2 🔥.
- Build a forwarding table from remote BGP advertisements received, associating remote B-MAC addresses with remote PE IP addresses and the associated MPLS label(s) ✨.
5. BGP Encoding 📡
PBB-EVPN uses the same BGP routes and attributes defined in [RFC7432], just adapted as described below. it's giving reuse, no cap 🫶.
5.1. Ethernet Auto-Discovery Route 🦋
this route and all of its associated modes are not needed in PBB-EVPN, because PBB encapsulation already provides the required level of indirection for C-MAC addresses. like, an ES can be represented by a B-MAC address for the purpose of data-plane learning/forwarding 🧚💖.
the receiving PE knows that it doesn't need to wait for the Ethernet A-D (auto-discovery) route to resolve a route. it can use the reserved ESI encoded in the MAC Advertisement route for that, since the ESI values of 0 and MAX-ESI tell the receiving PE it can resolve the path without an Ethernet A-D route 🙈.
5.2. MAC/IP Advertisement Route 🎀
the EVPN MAC/IP Advertisement route is used to distribute the B-MAC addresses of the PE nodes, instead of the C-MAC addresses of end-stations/hosts. why? because the C-MAC addresses are learned in the data plane for traffic arriving from the core 🧠. the MAC Advertisement route is encoded as follows:
- The MAC address field contains the B-MAC address. 🍓
- The Ethernet Tag field is set to
0. 🍓 - The Ethernet Segment Identifier field must be set to either
0(for single-homed segments or multihomed segments with per-I-SID load-balancing) or toMAX-ESI(for multihomed segments with per-flow load-balancing). All other values are not permitted. 🚫💅 - All other fields are set as defined in [RFC7432]. 💜
this route is tagged with the RT corresponding to its EVI, and this EVI is analogous to a B-VID 🌸.
5.3. Inclusive Multicast Ethernet Tag Route 🌊
this route is used for multicast pruning per I-SID. it's how PEs participating in a given I-SID get auto-discovered, so that a multicast tunnel (MP2P, P2P, P2MP, or MP2MP LSP) can be set up for that I-SID 🧜♀️. [RFC7080] uses multicast pruning per I-SID based on [MMRP], which is a soft-state protocol. the glow-up here is that using this BGP route beats [MMRP] in two ways: a) it scales very well for a large number of PEs, and b) it works with any type of LSP (MP2P, P2P, P2MP, or MP2MP). [MMRP] only works over P2P pseudowires, so 😭👑.
the Inclusive Multicast Ethernet Tag route is encoded as follows:
- The Ethernet Tag field is set with the appropriate I-SID value. 🍓
- All other fields are set as defined in [RFC7432]. 💜
this route is tagged with an RT. this RT SHOULD be set to a value corresponding to its EVI (which is analogous to a B-VID). the RT for this route MAY also be auto-derived from the corresponding Ethernet Tag (I-SID) based on the procedure specified in Section 5.1.2.1 of [OVERLAY] ✨🎀.
5.4. Ethernet Segment Route 🌈
this route is used for auto-discovery of PEs belonging to a given redundancy group (e.g., attached to a given ES) per [RFC7432] 🫶.
5.5. ESI Label Extended Community 💀
this extended community is not used in PBB-EVPN. in [RFC7432], it goes along with the Ethernet A-D route to advertise an MPLS label for split-horizon filtering. but in PBB-EVPN, split-horizon filtering is done natively using the B-MAC source address, so there's no need for it 🙌.
5.6. ES-Import Route Target 🦋
this RT is used as defined in [RFC7432] 💖.
5.7. MAC Mobility Extended Community 🧚
this extended community is defined in [RFC7432], and it's used with a MAC route (a B-MAC route in the case of PBB-EVPN). the B-MAC route is tagged with the RT corresponding to its EVI (which is analogous to a B-VID). when this extended community is used along with a B-MAC route in PBB-EVPN, it indicates that all C-MAC addresses associated with that B-MAC address across all corresponding I-SIDs must be flushed 🧹💅.
when a PE first advertises a B-MAC, it MAY advertise it with this extended community, where the sticky/static flag is set to 1 and the sequence number is set to zero 🌸. in cases where the PE wants to signal the stickiness of a B-MAC, then when a flush indication is needed, the PE advertises the B-MAC along with the MAC Mobility extended community. the sticky/static flag remains set, and the sequence number is incremented 🔥.
5.8. Default Gateway Extended Community 🙈
this extended community is not used in PBB-EVPN 🫠.
6. Operation 🎀
this section talks about how PBB-EVPN operates, specifically in the areas where it differs from [RFC7432] 🧠✨.
6.1. MAC Address Distribution over Core 💜
in PBB-EVPN, host MAC addresses (i.e., C-MAC addresses) need not be distributed in BGP. instead, every PE independently learns the C-MAC addresses in the data plane via normal bridging operation 🦋. every PE has a set of one or more unicast B-MAC addresses associated with it, and those are the addresses distributed over the core in MAC Advertisement routes 📡.
6.2. Device Multihoming 🌷
6.2.1. Flow-Based Load-Balancing 🌊
this section describes the procedures for supporting device multihoming in an All-Active redundancy mode (i.e., flow-based load-balancing) 💅.
6.2.1.1. PE B-MAC Address Assignment 🍓
in [PBB], every BEB is uniquely identified by one or more B-MAC addresses, which are usually locally administered by the service provider. for PBB-EVPN, the choice of B-MAC address(es) for the PE nodes must be examined carefully, because it has implications on the proper operation of multihoming 🧐. in particular, for the scenario where a CE is multihomed to a number of PE nodes with All-Active redundancy mode, a given C-MAC address would be reachable via multiple PE nodes concurrently. any given remote PE will bind the C-MAC address to a single B-MAC address, so the various PE nodes connected to the same CE must share the same B-MAC address. otherwise, the MAC address table of the remote PE nodes will keep oscillating between the B-MAC addresses of the various PE devices 😵💫.
for example, consider the network of Figure 1, and assume that PE1 has B-MAC address BM1 and PE2 has B-MAC address BM2. also assume that both links from CE1 to the PE nodes are part of the same Ethernet link aggregation group. if BM1 is not equal to BM2, the consequence is that the MAC address table on PE3 will keep oscillating, such that the C-MAC address M1 of CE1 would flip-flop between BM1 or BM2, depending on the load-balancing decision on CE1 for traffic destined to the core 🙃🔥.
considering that there could be multiple sites (e.g., CEs) multihomed to the same set of PE nodes, it is required for all the PE devices in a redundancy group to have a unique B-MAC address per site. this way, it's possible to achieve fast convergence when a link or port failure impacts the attachment circuit connecting a single site to a given PE 🚀✨.
+---------+
+-------+ PE1 | IP/MPLS |
/ | |
CE1 | Network | PEr
M1 \ | |
+-------+ PE2 | |
/-------+ | |
/ | |
CE2 | |
M2 \ | |
\ | |
+------+ PE3 +---------+
Figure 2: B-MAC Address Assignmentin the example network shown in Figure 2 above, two sites, corresponding to CE1 and CE2, are dual-homed to PE1/PE2 and PE2/PE3, respectively. assume that BM1 is the B-MAC used for the site corresponding to CE1, and BM2 is the B-MAC used for the site corresponding to CE2 💖. on PE1, a single B-MAC address (BM1) is required for the site corresponding to CE1. on PE2, two B-MAC addresses (BM1 and BM2) are required, one per site. whereas on PE3, a single B-MAC address (BM2) is required for the site corresponding to CE2 🧚. all three PE nodes would advertise their respective B-MAC addresses in BGP using the MAC Advertisement routes defined in [RFC7432] 📡.
the remote PE, PEr, would learn via BGP that BM1 is reachable via PE1 and PE2, whereas BM2 is reachable via both PE2 and PE3. furthermore, PEr establishes, via the PBB bridge learning procedure, that C-MAC M1 is reachable via BM1 and C-MAC M2 is reachable via BM2 🦋. as a result, PEr can load-balance traffic destined to M1 between PE1 and PE2, as well as traffic destined to M2 between PE2 and PE3 ⚖️✨. in the case of a failure that causes, for example, CE1 to be isolated from PE1, PE1 can withdraw the route it advertised for BM1. this way, PEr updates its path list for BM1 and sends all traffic destined to M1 over to PE2 only 🥹💜.
6.2.1.2. Automating B-MAC Address Assignment
okay so the PE B-MAC for single-homed or Single-Active sites can be auto-derived from the hardware 🔧 (for example, the backplane's address and/or the PE's reserved MAC pool ✨). but the B-MAC for All-Active sites must be coordinated among the redundancy group members 🫶. to automate that one, the PE can take the MAC address portion of the CE's Link Aggregation Control Protocol (LACP) System Identifier and flip the 'Locally Administered' bit 🔄. this guarantees the B-MAC is unique within the network and that every PE connected to the same All-Active CE uses the same value 💖.
heads up 👀 with this automatic B-MAC provisioning for All-Active CEs, you can't support the uncommon scenario where a CE has multiple link bundles in the same LACP session towards the PE nodes AND the service hair-pins traffic from one bundle to another. the reason is that split-horizon filtering relies on B-MAC addresses instead of Site-ID Labels (spoiler: that's the next section 💅). the operator must explicitly configure the B-MAC for this fairly uncommon service scenario 🙈.
whenever a B-MAC is provisioned on the PE, either manually or automatically (as an outcome of CE auto-discovery), the PE MUST transmit a MAC Advertisement route for that B-MAC with a downstream assigned MPLS label that uniquely identifies that address on the advertising PE 📣. the route is tagged with the RTs of the associated EVIs, as described above 🏷️.
6.2.1.3. Split Horizon and Designated Forwarder Election
[RFC7432] relies on split-horizon filtering for a multi-homed ES, where the ES label is used for egress filtering on the attachment circuit to prevent forwarding loops 🔁. in PBB-EVPN, the B-MAC source address can do the same job, since it uniquely identifies the originating site of a given frame 🎯. so ES labels aren't used in PBB-EVPN, and egress split-horizon filtering is based on the B-MAC source address instead. worth noting: [PBB] already defines this B-MAC address-based filtering as part of the I-Component options, so no new functions are required for split-horizon beyond what [PBB] already covers 🧚.
the Designated Forwarder (DF) election procedures are defined in [RFC7432], so go check them out there 💅.
6.2.2. Load-Balancing based on I-SID
this section is all about device multihoming in a Single-Active redundancy mode with per-I-SID load-balancing 🎀.
6.2.2.1. PE B-MAC Address Assignment
okay so if you want per-I-SID load-balancing among the PE nodes in a given redundancy group, you allocate multiple unicast B-MAC addresses per multihomed ES 🦋. every PE connected to the multihomed segment gets its own unique B-MAC, and every PE advertises that B-MAC using the BGP MAC Advertisement route. in this mode, there are two B-MAC assignment models:
- Shared: the PE may use a shared B-MAC address for all its single-homed segments and/or all its multi-homed Single-Active segments (e.g., segments operating in per-I-SID load-balancing mode).
- Dedicated: the PE may use a dedicated B-MAC address for each ES operating in per-I-SID load-balancing mode.
a PE implementation MAY choose to support either the shared or the dedicated B-MAC model without causing any interoperability issues 🤝. the perk of dedicated over shared for multi-homed Single-Active segments is that when C-MAC flushing is needed, fewer C-MAC addresses get impacted 🧹. it also gives the disposition PE the power to skip C-MAC destination address lookup, though source C-MAC learning still happens in the data plane 🧠. the downside? there are more B-MAC advertisements in BGP 📬.
a remote PE initially floods traffic to a destination C-MAC address located in a given multihomed ES out to all the PE nodes configured with that I-SID 🌊. then, when reply traffic arrives, the remote PE learns (in the data path) the B-MAC address and the associated next-hop PE to use for that C-MAC 💫.
6.2.2.2. Split Horizon and Designated Forwarder Election
this is basically the same as the flow-based load-balancing case 🙌. the only difference is that DF filtering must be applied to unicast as well as multicast traffic, in both core-to-segment and segment-to-core directions.
6.2.2.3. Handling Failure Scenarios
when a PE connected to a multihomed ES loses connectivity to the segment (due to link or port failure), it needs to notify the remote PEs so they trigger C-MAC address flushing 🚨. this can be done in one of two ways, depending on the B-MAC assignment model:
- Shared B-MAC: if the PE uses a shared B-MAC address for multiple Ethernet segments, the C-MAC flushing is signaled by having the failed PE re-advertise the MAC Advertisement route for the associated B-MAC, tagged with the MAC Mobility extended community attribute. the Counter field in that attribute must be incremented prior to advertisement. this causes the remote PE nodes to flush all C-MAC addresses associated with that B-MAC 🧽. this is done across all I-SIDs mapped to the EVI of the withdrawn MAC route.
- Dedicated B-MAC: if the PE uses a dedicated B-MAC address for each ES operating under per-I-SID load-balancing mode, the failed PE simply withdraws the B-MAC route it previously advertised for that segment 📤. this causes the remote PE nodes to flush all C-MAC addresses associated with that B-MAC. this is done across all I-SIDs mapped to the EVI of the withdrawn MAC route.
when a PE connected to a multihomed ES fails (node failure) or becomes completely isolated from the EVPN network, the remote PEs start purging the MAC Advertisement routes that the failed PE advertised 💀. this happens either because the remote PEs detect that the BGP session to the failed PE went down, or because a Route Reflector withdraws all the routes that PE advertised. in this case, the remote PEs do C-MAC address flushing as a result of those MAC Advertisement route withdrawals.
for all failure scenarios (link/port failure, node failure, and PE node isolation), once the fault condition clears, the recovered PE re-advertises the associated ES route to the other members of its redundancy group 🔄. this triggers the backup PE(s) in the group to block the I-SIDs for which the recovered PE is the DF. when a backup PE blocks those I-SIDs, it sends a C-MAC flush notification to the remote PEs by re-advertising the MAC Advertisement route for the associated B-MAC, with the MAC Mobility extended community attribute. the Counter field in that attribute must be incremented prior to advertisement 📈. this causes the remote PE nodes to flush all C-MAC addresses associated with that B-MAC, across all I-SIDs mapped to the EVI of the withdrawn/re-advertised MAC route.
6.3. Network Multihoming
when an Ethernet network is multihomed to a set of PBB-EVPN PE nodes, the Single-Active redundancy model can be supported with per-service instance (i.e., I-SID) load-balancing 🌈. in this model, DF election makes sure a single PE node in the redundancy group is responsible for forwarding traffic for a given I-SID, so no forwarding loops get created 🔁. DF filtering applies to both unicast and multicast traffic, in both access-to-core and core-to-access directions, just like the Single-Active multihomed device scenario. (it's unlike the All-Active device scenario, where DF filtering is limited to multi-destination frames in the core-to-access direction 🙃.) similar to the Single-Active device scenario with I-SID load-balancing, a unique B-MAC address is assigned to each PE node connected to the multihomed network (segment) 🧁.
6.4. Frame Forwarding
the frame-forwarding jobs are split between the Bridge Module, which hosts the [PBB] BEB functionality, and the MPLS Forwarder, which handles the MPLS imposition/disposition 🧩. the details for unicast and multi-destination frames are coming right up 👇.
6.4.1. Unicast
known unicast traffic received from the Attachment Circuit (AC) gets PBB-encapsulated by the PE using the B-MAC source address of the originating site 📦. the unicast B-MAC destination address comes from a lookup of the C-MAC destination address (the binding between the two is done via transparent learning of reverse traffic 🔍). the resulting frame is then encapsulated with an LSP tunnel label and an EVPN label associated with the B-MAC destination address 🏷️. if per-flow load-balancing over ECMPs in the MPLS core is required, a flow label is added below the label associated with the B-MAC address in the label stack 🫧.
for unknown unicast traffic, the PE forwards these frames over the MPLS core 🌍. when these frames are forwarded, the same set of options used for forwarding multicast/broadcast frames (described in the next section) is used 🦋.
6.4.2. Multicast/Broadcast
multi-destination frames received from the AC get PBB-encapsulated by the PE using the B-MAC source address of the originating site 🎁. the multicast B-MAC destination address is selected based on the value of the I-SID, as defined in [PBB] 🔢. the resulting frame is then forwarded over the MPLS core using one of the following two options:
- Option 1: the MPLS Forwarder can perform ingress replication over a set of MP2P or P2P tunnel LSPs. the frame is encapsulated with a tunnel LSP label and the EVPN ingress replication label advertised in the Inclusive Multicast Ethernet Tag [RFC7432] 🪄.
- Option 2: the MPLS Forwarder can use P2MP tunnel LSP per the procedures defined in [RFC7432]. this includes either the use of Inclusive or Aggregate Inclusive trees. furthermore, the MPLS Forwarder can use MP2MP tunnel LSP if Inclusive trees are used 🌸.
note that the same procedures for advertising and handling the Inclusive Multicast Ethernet Tag defined in [RFC7432] apply here 💜.
6.5. MPLS Encapsulation of PBB Frames
the encapsulation for transporting PBB frames over MPLS is similar to that of classical Ethernet, just with the extra PBB header tucked in 🎀, as shown in the figure below:
+------------------+
| IP/MPLS Header |
+------------------+
| PBB Header |
+------------------+
| Ethernet Header |
+------------------+
| Ethernet Payload |
+------------------+
| Ethernet FCS |
+------------------+
Figure 3: PBB over MPLS Encapsulation7. Minimizing ARP/ND Broadcast 🦋
okay bestie, so the PE nodes MAY implement an ARP/ND-proxy function 💅 which is basically a way to stop ARP/ND traffic from getting broadcasted all over the MPLS network like it's a group chat that never ends 📢💀. For ARP proxy, each PE snoops on ARP request and response messages coming in over the access interfaces or the MPLS core 👀. From those snooped messages it builds a cache of IP/MAC address bindings 🗂️✨. Then it uses that cache to answer ARP requests coming in on access ports for target hosts that live in remote sites 🧚. If the PE finds a match for the IP address in its ARP cache, it replies right back to the requesting host and drops the request 🫶. If there's no match, the request gets flooded over the MPLS network using either ingress replication or P2MP LSPs 🌊. For ND proxy it's basically the same vibe, but with ND/NA messages per [RFC4389] 🌸.
8. Seamless Interworking with IEEE 802.1aq / 802.1Qbp 🎀
+--------------+
| |
+---------+ | MPLS | +---------+
+----+ | | +----+ +----+ | | +----+
|SW1 |--| | | PE1| | PE2| | |--| SW3|
+----+ | 802.1aq |---| | | |--| 802.1aq | +----+
+----+ | .1Qbp | +----+ +----+ | .1Qbp | +----+
|SW2 |--| | | Backbone | | |--| SW4|
+----+ +---------+ +--------------+ +---------+ +----+
|<------ IS-IS -------->|<-----BGP----->|<------ IS-IS ------>| CP
|<------------------------- PBB -------------------------->| DP
|<----MPLS----->|
Legend: CP = Control-Plane View
DP = Data-Plane View
Figure 4: Interconnecting 802.1aq / 802.1Qbp Networks with PBB-EVPNOkay so this is the picture of how the 802.1aq and 802.1Qbp networks hook up with PBB-EVPN 🦋. The control plane (CP) side is IS-IS on the edges and BGP across the middle, and the data plane (DP) side is PBB running over MPLS in the core 💖. Lowkey the diagram does all the work here, iykyk 😭.
8.1. B-MAC Address Assignment 💜
The B-MAC addresses need to be globally unique across all networks, including PBB-EVPN and IEEE 802.1aq / 802.1Qbp networks 🌈. The B-MAC addresses used for single-homed and Single-Active Ethernet segments SHOULD be unique, because they're typically auto-derived from the PE's pools of reserved MAC addresses that are unique 🍓. The B-MAC addresses used for All-Active Ethernet segments SHOULD also be unique, given that each network operator typically has its own assigned Organizationally Unique Identifier (OUI) and can therefore assign its own unique MAC addresses 🔥.
8.2. IEEE 802.1aq / 802.1Qbp B-MAC Address Advertisement 🫶
B-MAC addresses associated with 802.1aq / 802.1Qbp switches get advertised using the EVPN MAC/IP route advertisement that's already defined in [RFC7432] ✨.
8.3. Operation: ⚙️
When a PE receives a PBB-encapsulated Ethernet frame from the access side, it does a lookup on the B-MAC destination address to find the next hop 🔍. If that lookup says the next hop is a remote PE, the local PE encapsulates the PBB frame in MPLS 📦. The label stack is made up of the VPN label (advertised by the remote PE), followed by an LSP/IGP label 🏷️. From that point on, regular MPLS forwarding takes over 🚀.
On the disposition PE, assuming penultimate-hop-popping is employed, the PE receives the MPLS-encapsulated PBB frame with a single label, which is the VPN label 🎁. The value of that label tells the disposition PE that this is a PBB frame, so the label gets popped, the TTL field (in the 802.1Qbp F-Tag) is reinitialized, and normal PBB processing kicks in from there 💅✨.
9. Solution Advantages 👑
In this section we talk about all the ways the PBB-EVPN solution is giving, measured against the requirements in Section 3 💖.
9.1. MAC Advertisement Route Scalability 🌸
In PBB-EVPN, the number of MAC Advertisement routes depends on the number of Ethernet segments (like sites) instead of the number of hosts/servers 🏠. That's because the B-MAC addresses of the PEs get advertised in BGP, not the C-MAC addresses of the hosts/servers 🧠. As we talked about above, there's a one-to-one mapping between All-Active multihomed segments and their B-MAC addresses 💫. Single-Active multihomed segments can have either a one-to-one or a many-to-one mapping to B-MAC addresses 🦋. Single-homed sites have a many-to-one mapping to the B-MAC addresses on a given PE 🎀. So a single B-MAC can be tied to one or more segments, and each segment can have lots of C-MAC addresses 💜. The result? The volume of MAC Advertisement routes in PBB-EVPN may be multiple orders of magnitude less than in EVPN 🤯🔥.
9.2. C-MAC Mobility Independent of B-MAC Advertisements 🦋
As described above, a single B-MAC address in PBB-EVPN can aggregate lots of C-MAC addresses 💖. B-MAC addresses are tied to segments attached to a PE or to the PE itself, so their locations are fixed and not affected AT ALL by C-MAC mobility 🙅♀️. C-MAC mobility therefore does not affect B-MAC addresses, which means no re-advertisements of them 🫶. The reason is that the C-MAC to B-MAC association gets learned in the data plane, and C-MAC addresses are never advertised in BGP 🧚. Aggregation via B-MAC addresses in PBB-EVPN performs way better than EVPN 👑.
To see how this compares to EVPN, check out this example 👀:
If a PE running EVPN advertises reachability for N MAC addresses via a particular segment, and then 50% of the MAC addresses in that segment move to other segments (e.g., due to virtual machine mobility), then N/2 additional MAC Advertisement routes need to be sent for the MAC addresses that have moved. With PBB-EVPN, on the other hand, the B-MAC addresses that are statically associated with PE nodes are not subject to mobility. As C-MAC addresses move from one segment to another, the binding of C-MAC to B-MAC addresses is updated via data-plane learning in PBB-EVPN.
9.3. C-MAC Address Learning and Confinement 🌈
In PBB-EVPN, C-MAC address reachability info is built through data-plane learning 🧠. So PE nodes that aren't part of active conversations involving a particular C-MAC address will purge that address from their forwarding tables 🧹. Also, since C-MAC addresses aren't distributed in BGP, PE nodes won't keep any record of them in the control-plane routing table 🙈.
9.4. Seamless Interworking with 802.1aq Access Networks 🌸
Picture this: two access networks, one running MPLS and the other running 802.1aq, are connected through an MPLS backbone network 🧚♀️. The figure below shows an example of this setup 👇.
+--------------+
| |
+---------+ | MPLS | +---------+
+----+ | | +----+ +----+ | | +----+
| CE |--| | | PE1| | PE2| | |--| CE |
+----+ | 802.1aq |---| | | |--| MPLS | +----+
+----+ | | +----+ +----+ | | +----+
| CE |--| | | Backbone | | |--| CE |
+----+ +---------+ +--------------+ +---------+ +----+
Figure 5: Interoperability with 802.1aqIf the MPLS backbone network employs EVPN, then the 802.1aq data-plane encapsulation MUST be terminated on PE1 or the edge device connecting to PE1 😬. Either way, all the PE nodes in the associated service instances get exposed to all the C-MAC addresses of all hosts/servers on the access networks 😭. But if the MPLS backbone network employs PBB-EVPN, then the 802.1aq encapsulation can be extended over the MPLS backbone, which keeps C-MAC address transparency on PE1 🎉. And if PBB-EVPN is also extended over the MPLS access network on the right, then C-MAC addresses stay transparent to PE2 too 💅.
9.5. Per-Site Policy Support 💜
In PBB-EVPN, per-site policy can be supported through B-MAC addresses, by assigning a unique B-MAC address to every site/segment (usually multihomed, but it can also be single-homed) 🍓. Since the B-MAC addresses get sent in BGP MAC/IP route advertisements, it's possible to define per-site (i.e., B-MAC) forwarding policies, including policies for E-TREE service 🌈.
9.6. No C-MAC Address Flushing for All-Active Multihoming 🧚
Just like in [RFC7432], PBB-EVPN lets you avoid C-MAC address flushing when a topology change affects an All-Active multihomed segment 🫧. To see how, look at the example network in Figure 1 💡. Both PE1 and PE2 advertise the same B-MAC address (BM1) to PE3 📣. PE3 learns the C-MAC addresses of the servers/hosts behind CE1 through data-plane learning 🧠. If AC1 fails, PE3 doesn't need to flush any of the C-MAC addresses learned and associated with BM1 🙌. That's because PE1 withdraws the MAC Advertisement routes associated with BM1, which leaves PE3 with a single adjacency (to PE2) for this B-MAC address 🦋. So the topology change gets communicated to PE3, and no C-MAC flushing is needed ✨.
10. Security Considerations 🔐
All the security considerations in [RFC7432] apply directly to this document, because this document leverages the control plane described in [RFC7432] and its associated procedures, although not the complete set but rather a subset 😱.
This document does not introduce any new security considerations beyond those of [RFC7432] and [RFC4761], because advertisements and processing of B-MAC addresses follow [RFC7432] and processing of C-MAC addresses follows [RFC4761]. In other words, B-MAC addresses are learned in the control plane and C-MAC addresses are learned in the data plane 🛡️💖.
11. IANA Considerations 📋
There are no additional IANA considerations for PBB-EVPN beyond what is already described in [RFC7432] 🎀.
12. References 📚
12.1. Normative References 💜
[PBB] IEEE, "IEEE Standard for Local and metropolitan area
networks - Media Access Control (MAC) Bridges and Virtual
Bridged Local Area Networks", Clauses 25 and 26, IEEE Std
802.1Q, DOI 10.1109/IEEESTD.2011.6009146.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <http://www.rfc-editor.org/info/rfc7432>.12.2. Informative References
[MMRP] IEEE, "IEEE Standard for Local and metropolitan area networks - Media Access Control (MAC) Bridges and Virtual Bridged Local Area Networks", Clause 10, IEEE Std 802.1Q, DOI 10.1109/IEEESTD.2011.6009146. 📚
[OVERLAY] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Isaac, A., Uttaro, J., Henderickx, W., Shekhar, R., Salam, S., Patel, K., Rao, D., and S. Thoria, "A Network Virtualization Overlay Solution using EVPN", draft-ietf-bess-evpn-overlay-01, February 2015. 🦋
[RFC4389] Thaler, D., Talwar, M., and C. Patel, "Neighbor Discovery Proxies (ND Proxy)", RFC 4389, DOI 10.17487/RFC4389, April 2006, <http://www.rfc-editor.org/info/rfc4389>. 🌸
[RFC4761] Kompella, K., Ed., and Y. Rekhter, Ed., "Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling", RFC 4761, DOI 10.17487/RFC4761, January 2007, <http://www.rfc-editor.org/info/rfc4761>. 💜
[RFC7080] Sajassi, A., Salam, S., Bitar, N., and F. Balus, "Virtual Private LAN Service (VPLS) Interoperability with Provider Backbone Bridges", RFC 7080, DOI 10.17487/RFC7080, December 2013, <http://www.rfc-editor.org/info/rfc7080>. 🎀
[RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., Henderickx, W., and A. Isaac, "Requirements for Ethernet VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, <http://www.rfc-editor.org/info/rfc7209>. ✨
Acknowledgements
okay bestie, the authors wanna give a huge shoutout to Jose Liste and Patrice Brissette for reviewing and commenting on this doc 🫶 and also to Giles Heron, who did several rounds of reviews and gave super valuable input so the doc could get ready for IESG submission. literally iconic behaviour 👑
Contributors
these people also contributed to this doc, even though they weren't listed as authors. the whole crew 💅
Lizhong Jin, ZTE Sami Boutros, Cisco Dennis Cai, Cisco Keyur Patel, Cisco Sam Aldrin, Huawei Himanshu Shah, Ciena Jorge Rabadan, ALU
Authors' Addresses
here's where you can find the authors, in case you wanna slide into their inbox to talk protocols 🧚
Ali Sajassi, editor
Cisco
170 West Tasman Drive
San Jose, CA 95134
United States
Email: sajassi@cisco.comSamer Salam
Cisco
595 Burrard Street, Suite # 2123
Vancouver, BC V7X 1J1
Canada
Email: ssalam@cisco.comNabil Bitar
Verizon Communications
Email: nabil.n.bitar@verizon.comAldrin Isaac
Juniper
Email: aisaac@juniper.netWim Henderickx
Alcatel-Lucent
Email: wim.henderickx@alcatel-lucent.com📦 where this page comes from
- ietf: rfc7623 2015-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms