RFC 7024
Virtual Hub-and-Spoke in BGP/MPLS VPNs
RFC 7024 💖 Virtual Hub-and-Spoke in BGP/MPLS VPNs
- Organisation: Internet Engineering Task Force (IETF)
- Request for Comments: 7024
- Category: Standards Track
- ISSN: 2070-1721
- Date: October 2013
- Authors: H. Jeng (AT&T), J. Uttaro (AT&T), L. Jalil (Verizon), B. Decraene (Orange), Y. Rekhter (Juniper Networks), R. Aggarwal (Arktan)
- Obsoletes / Updates: none listed in the header
okay bestie, this is the big one 🎀 It's a Standards Track doc, so it's the real deal that implementers actually follow 👑 Let's get into it!
Table of Contents 🧚
1. Overview ........................................................3
2. Specification of Requirements ...................................4
3. Routing Information Exchange ....................................5
4. Forwarding Considerations .......................................7
5. Internet Connectivity ...........................................9
6. Deployment Considerations ......................................12
7. Multicast Considerations .......................................13
7.1. Terminology ...............................................14
7.2. Eligible Upstream Multicast Hop (UMH) Routes ..............14
7.3. Originating VPN-IP Default Route by a V-Hub ...............14
7.4. Handling C-Multicast Routes ...............................15
7.5. Originating I-PMSI/S-PMSI/SA A-D Routes by V-Spoke ........15
7.6. Originating I-PMSI/S-PMSI/SA A-D Routes by V-Hub ..........16
7.7. Receiving I-PMSI/S-PMSI/SA A-D Routes by V-Spoke ..........17
7.8. Receiving I-PMSI/S-PMSI/SA A-D Routes by V-Hub ............17
7.8.1. Case 1 .............................................17
7.8.2. Case 2 .............................................18
7.9. Use of Ingress Replication with I-PMSI A-D Routes .........20
8. An Example of RT Provisioning ..................................21
8.1. Unicast Routing ...........................................21
8.2. Multicast Routing .........................................22
9. Further Refinements ............................................23
10. Security Considerations .......................................23
11. Acknowledgements ..............................................23
12. References ....................................................24
12.1. Normative References .....................................24
12.2. Informative References ...................................24Abstract 🦋
With BGP/MPLS Virtual Private Networks (VPNs), giving any-to-any connectivity among the sites of one VPN usually means every Provider Edge (PE) router hooked up to one or more of those sites has to hold all the routes of that VPN 😩. This document lets the VPN service provider cut down how many PE routers carry all those routes, by having only a subset of them do it 💅.
Plus, when PE routers use ingress replication for VPN customers' multicast traffic, this approach may, under certain circumstances, cut down the bandwidth waste that ingress replication causes and spread the replication load across PE routers ✨.
Status of This Memo 📜
This is an Internet Standards Track document.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741.
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc7024.
Copyright Notice 🌸
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
1. Overview ✨
okay so with BGP/MPLS VPNs [RFC4364], giving any-to-any connectivity among the sites of a VPN is usually done by making every Provider Edge (PE) router that's connected to one or more of those sites hold all of that VPN's routes 🫶. The approach in this document lets the VPN service provider (SP) cut down how many PEs have to keep all those routes, by having only a subset of them do it 💅.
Picture a bunch of PEs that maintain VPN Routing and Forwarding tables (VRFs) for one VPN 🦋. For this VPN, we call one subset of those PEs "Virtual Spoke" PEs (or just Virtual Spokes), and some other subset that doesn't overlap with it "Virtual Hub" PEs (or just Virtual Hubs) 👑. Everything else in the set is a "vanilla" PE, meaning a PE that follows the procedures in [RFC4364] but doesn't implement the procedures in this document 🌈.
To keep it short, we'll say "V-hub" for a Virtual Hub and "V-spoke" for a Virtual Spoke 💖.
For a given VPN, its V-hubs can include not only PEs that have sites of that VPN connected, but also PEs with no sites of that VPN connected at all 😳. On those PEs, the VRF for that VPN can import routes from other VRFs of that VPN, even when that VRF has no sites connected to it.
Here's the twist: a PE can be a V-hub for one VPN and a V-spoke for a different VPN, and the other way around too 🔥. So a given PE can be a V-hub for only some of the VPNs living on it, not all of them. Same deal for being a V-spoke 🧚.
For a given VPN, each V-spoke is "associated" with one or more V-hubs of that VPN. You can use two V-hubs for redundancy so you don't have a single point of failure 🛡️. A V-hub is allowed to have no V-spokes associated with it at all 🙈. (For how a V-spoke and a V-hub actually become "associated," see Section 3 ✨.)
Let's say a set of V-spokes is associated with a V-hub called V-hub-1. If one of those V-spokes is also associated with another V-hub, V-hub-2, the other V-spokes in the set don't have to be associated with V-hub-2 too. They can be associated with other V-hubs instead, like V-hub-3, V-hub-4, and so on 💅.
This document defines a VPN-IP default route as a VPN-IP route whose VPN-IP prefix contains only a Route Distinguisher (RD). (For the definition of "VPN-IP route," see [RFC4364] 🍓.)
Okay, so a PE acting as a V-hub for a VPN keeps all the routes of that VPN. It imports routes from all the other V-hubs and V-spokes, plus from the "vanilla" PEs of that VPN 👑. A PE acting as a V-spoke for a VPN only needs to keep two kinds of routes: the routes of that VPN that come from the sites connected to that PE, and one or more VPN-IP default routes that come from the V-hub(s) associated with that V-spoke. A V-spoke only needs to import VPN-IP default routes from certain V-hubs 🎀. So only the subset of PEs that are V-hubs for that VPN has to hold all of its routes, while V-spokes only hold a (small) subset of them 🌸.
This document assumes that a given V-hub and its associated V-spoke(s) are in the same Autonomous System (AS). But if the PEs that maintain a VPN's VRFs span multiple ASes, this document doesn't force all of that VPN's V-hubs into the same AS. The V-hubs can be spread across those ASes 🌍.
You can model this approach as a two-level hierarchy: V-hubs are the top level and V-spokes are the bottom level 🪜. Generalising it to more than two levels is outside the scope of this document 🙅.
Now the multicast bit 💅. When PEs use ingress replication to carry VPN customers' multicast traffic, this approach may, under certain circumstances, reduce the bandwidth inefficiency that comes with ingress replication. It may also spread the replication load across the PEs 🦋. Why? A PE acting as a V-spoke for a VPN only needs to replicate multicast traffic to other V-hubs (and those V-hubs then replicate it to the V-spokes associated with them), instead of sending it to every PE in that VPN. A PE acting as a V-hub for a VPN needs to replicate multicast traffic to other V-hubs and to the V-spokes, but only the V-spokes associated with that V-hub, rather than to all PEs of that VPN ✨. Limiting replication like this could be especially helpful if the V-spoke PEs have limited replication capabilities and/or links with limited bandwidth 💜.
2. Specification of Requirements 💅
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 📖.
3. Routing Information Exchange ✨
okay so every PE in a given VPN has to follow these rules when they swap routing info 🫶
A PE with sites of a given VPN plugged into it has to hold onto the routing info it gets from those sites 💜 This is true whether the PE is a V-hub or a V-spoke for that VPN, and it still follows the rules in [RFC4364] while doing it.
Exporting works the same way 🦋 A PE with sites of that VPN exports the routes from those sites as VPN-IP routes, following [RFC4364], whether it's a V-hub or a V-spoke.
Now the big one 👑 A V-hub MUST export a VPN-IP default route for that VPN. This route MUST be exported to only the V-spokes associated with that V-hub. No spilling to the wrong spokes, bestie 🙈
For load sharing 🌸 each of the VPN's V-hubs MUST use a distinct RD (per V-hub, per VPN) when originating a VPN-IP default route. That way a V-spoke can spread its outbound traffic across the V-hubs it's linked to. Type 1 RDs may be an attractive option for these.
If a V-spoke imports several VPN-IP default routes, each one originated by its own V-hub, and they all have the same preference, then traffic from that V-spoke to other sites of the VPN gets load shared among the V-hubs ✨
Following [RFC4364], a V-hub of a given VPN imports all the non-default VPN-IP routes originated by all the other PEs with sites of that VPN 💅 That's true whether those PEs are V-hubs, V-spokes, or just "vanilla" PEs for that VPN, and whether or not those V-spokes are associated with the V-hub.
A V-hub of a given VPN MUST NOT import a VPN-IP default route unless the imported route is the Internet VPN-IP default route. (For the definition of "Internet VPN-IP default route," and for how to tell a regular VPN-IP default route apart from it, see Section 5 🧚)
Within a given VPN, a V-spoke MUST import all VPN-IP default routes that have been originated by the V-hubs associated with that V-spoke.
Also, a V-spoke of a given VPN MAY import VPN-IP routes for that VPN that were originated by some other V-spokes of that VPN, but only by the V-spokes associated with the same V-hub(s) as the V-spoke itself. Same squad only 🫶
The above rules are realized using Route Target (RT) extended communities [RFC4360] and VRF export/import policies based on those RTs. This document defines the following procedures for implementing the rules above.
Picture a "vanilla" any-to-any VPN 🌈 This document assumes that all the PEs of that VPN (or, to be more precise, all the VRFs of that VPN) are provisioned with the same export and import RT. We'll call this RT "RT-VPN." (Of course, for a given VPN service provider, each VPN uses its own RT-VPN, distinct from the RT-VPNs used by other VPNs.)
To evolve this VPN into V-hubs and V-spokes ✨ all the PEs (or, more precisely, all the VRFs) that are designated as either V-hubs or V-spokes of that VPN keep the same export RT-VPN. This RT-VPN is attached to all the VPN-IP routes originated by these PEs. All the V-hubs also keep the same import RT-VPN.
On top of that, each of a given VPN's V-hubs is provisioned with its own export RT, called RT-VH. This RT-VH MUST be different from the export RT (RT-VPN) provisioned on that V-hub. Also, for a given VPN service provider, no two VPNs can use the same RT-VH.
A given V-spoke becomes associated with a given V-hub by provisioning the V-spoke to import only the VPN-IP route(s) that carry the RT-VH provisioned on that V-hub. So adding a new V-spoke to a V-hub only needs provisioning on that V-spoke. No provisioning changes are required on the V-hub 🍓
Now, to avoid every V-spoke in a VPN being associated with every V-hub (in other words, to partition the V-spokes among the V-hubs) 💅 different V-hubs within that VPN MAY use different RT-VHs. At one extreme, every V-hub may use a distinct RT-VH, and IP-address-specific RTs may be an attractive option for this scenario. It's also possible for several V-hubs to use the same RT-VH, in which case all of those V-hubs are associated with the same set of V-spokes.
When a V-hub originates a (non-Internet) VPN-IP default route, the V-hub MUST attach RT-VH to that route. (The case where a V-hub originates the Internet VPN-IP default route is covered in Section 5.) This means the route gets imported by all the V-spokes associated with that V-hub 🦋
A V-spoke MAY be provisioned to export VPN-IP routes not just to the V-hubs but also to the V-spokes that import the same VPN-IP default route(s) as the V-spoke itself. The V-spoke does this by adding its import RT-VH(s) to the VPN-IP routes it exports 🫶
4. Forwarding Considerations 🎀
This section describes changes and modifications to the forwarding procedures specified in [RFC4364]. Let's get into it 💖
For a given VPN, the MPLS label that a V-hub of that VPN advertises with a VPN-IP default route MUST be the label mapped to a Next Hop Label Forwarding Entry (NHLFE) that identifies the VRF of the V-hub. So when the V-hub receives a packet carrying that label, it pops the label and works out what to do with the packet based on a lookup in the VRF. Main character energy 👑
Note that this document does not require the advertisement of labels mapped to an NHLFE that identifies a VRF for routes other than the VPN-IP default route.
When a V-hub of a given VPN originates a VPN-IP default route for that VPN, the V-hub MUST NOT install a default route in its VRF for that VPN, unless that route was originated as a result of:
- a) the V-hub receiving an IP default route from one of the VPN Customer Edge (CE) routers connected to it, or
- b) the V-hub receiving (and importing) the Internet VPN-IP default route (Section 5) from some other PE, or
- c) the VRF being provisioned with a default route pointing to the routing table that maintains the Internet routes.
When a multihomed site is connected to a V-hub and a V-spoke, the V-hub uses the following OPTIONAL procedures to support Internal BGP (IBGP) / External BGP (EBGP) load balancing for the site's inbound traffic that was originated by some other V-spoke associated with the V-hub. When the V-hub receives a packet from some other PE that carries an MPLS label the V-hub advertised in the VPN-IP default route, the V-hub uses that label to identify the VRF that should be used for further disposition of the packet. If the V-hub determines (using the information in the VRF) that the packet has to be forwarded using a non-default route present in the VRF, and this route indicates that the packet's destination is reachable either over one of the VRF attachment circuits (for the definition of "VRF attachment circuits," see [RFC4364]) or via some other (V-spoke) PE, the V-hub forwards the packet either over that attachment circuit or via that other PE. The choice between the two is a local matter to the V-hub 🧚
To illustrate this, let's walk through an example 🌸
<- RD:0/0 RD:0/0->
<- RD:192.0.2 <-192.0.2/24
CE1----PE-S-------------PE-H----------------PE-S1-------------CE2
/
| |
| | 192.0.2/24
| |
CE4 CE3A multihomed site (not shown in the figure above) is connected via CE2 and CE3. So both CE2 and CE3 advertise a route to 192.0.2/24. CE2 advertises this route (the route to 192.0.2/24) to PE-S1, which in turn originates a VPN-IP route RD:192.0.2. CE3 advertises this route to PE-H.
PE-H is a V-hub, while PE-S and PE-S1 are V-spokes associated with that V-hub. So PE-H originates a VPN-IP default route (RD:0/0), and both PE-S and PE-S1 import that route.
PE-H receives a VPN-IP route to RD:192.0.2 from PE-S1, and a plain IP route to 192.0.2 from CE3. So the VRF entry on PE-H has two possible next hops for 192.0.2: CE3 and PE-S1. (The latter is a next hop that is not directly connected to PE-H.) 🧚
Now consider what happens when CE1 originates a packet destined to 192.0.2.1 👀 When PE-S receives this packet, PE-S follows the VPN-IP default route and forwards the packet to PE-H. The MPLS label in the packet is the label that PE-H advertised to PE-S in the VPN-IP default route. So, following the rule specified above, PE-H may forward the packet either via CE3 or via PE-S1 (with PE-S1 subsequently forwarding the packet to CE2), which results in IBGP/EBGP load balancing 💖
Likewise, if CE4 originates a packet destined to 192.0.2.1, PE-H may forward the packet either via CE3 or via PE-S1 (with PE-S1 subsequently forwarding the traffic to CE2), which results in IBGP/EBGP load balancing ✨
Note, however, that if there is some other CE, CE5, connected to PE-S1, and CE5 sends a packet to 192.0.2.1, then (due to the IP longest match rule) PE-S1 will always forward this packet to CE2 💀
So for a multihomed site connected to a V-hub and a V-spoke, IBGP/EBGP load balancing is available for some but not all of the traffic destined to that site. Specifically, IBGP/EBGP load balancing is not available for the traffic destined to that site if this traffic was originated within some other site that is connected to the same V-spoke 🙈
Moreover, if CE3 advertises 192.0.2.0/25 and 192.0.2/24, while CE2 advertises 192.0.2.128/25 and 192.0.2/24 (which is yet another form of load balancing for a multihomed site), then when CE5 sends a packet to 192.0.2.1, the IP longest match rule means PE-S1 will always forward this packet to CE2. Even though the VPN customer would expect this traffic to flow via CE3 😭
This document proposes two options to address the issues raised in the previous two paragraphs. The first option is to disallow a given VPN from provisioning PEs that have multihomed sites of that VPN connected to them as V-spokes. (Such PEs could be provisioned as either V-hubs or plain "vanilla" PEs.) The second option is for the V-spoke, when it receives an IP route from a CE, to not install that route in its forwarding table, but instead just re-advertise the route as a VPN-IP route together with an MPLS label. The NHLFE [RFC3031] associated with that label MUST specify the CE that advertises the IP route as the next hop. As a result, when the PE receives data carrying that label, the PE just forwards the data to the CE without performing an IP lookup on the data. Note that doing this would force traffic between a pair of sites connected to the same V-spoke to go through the V-hub of that V-spoke 🦋
An implementation that supports IBGP/EBGP load balancing, as specified above, SHOULD support the second option. If the implementation does not support the second option, then deploying this implementation to support IBGP/EBGP load balancing, as specified above, would either (a) restrict the set of PEs that could be provisioned as V-spokes (any PE that has a multihomed site connected to it cannot be provisioned as a V-spoke) or (b) result in IBGP/EBGP load balancing not being available for certain scenarios (the scenarios that the second option is intended to cover) 🥺
5. Internet Connectivity ✨
okay so this section gives us TWO ways a VPN can get internet access 🌈 let's go bestie 💅
Option one is when a PE that already keeps Internet routes ALSO has a VRF for a given VPN. Here, the VPN can get Internet connectivity by setting up a default route in that VPN's VRF on that PE, pointing at the routing table on the same PE that holds the Internet routes 🧚 The PE MAY be set up this way. It MUST NOT be provisioned as a V-spoke for that VPN (it can be a V-hub or a "vanilla" PE, though 🫶). If the PE is a V-hub, it MUST originate a VPN-IP default route. That route MUST carry both RT-VPN and RT-VH of the V-hub (check Section 3 for what "RT-VPN" and "RT-VH" mean 📖). So that route gets imported by all the V-spokes tied to the V-hub, plus other V-hubs and "vanilla" PEs. Literally everyone gets it 💖 An implementation MUST support this first option.
Option two is when a site in a given VPN has Internet access, and a CE at that site advertises an IP default route to the PE it's connected to 🌸 This option has two subcases: (a) the PE is a V-hub, and (b) the PE is a V-spoke. An implementation MUST support subcase (a). An implementation MAY support subcase (b).
If the PE is a V-hub, it re-advertises that IP default route as a VPN-IP default route, and it installs an IP default route in its VRF whose next hop is the CE(s) advertising the IP default route to the PE 🦋 When re-advertising the VPN-IP default route, the route MUST carry both RT-VPN and RT-VH of the V-hub (again, Section 3 has the definitions 🎀). Same deal as before: all the V-spokes tied to that V-hub import it, and so do other V-hubs and "vanilla" PEs.
If the PE is a V-spoke, getting a default route from a CE MUST NOT make the V-spoke install an IP default route in its VRF 🙈 Instead, the V-spoke MUST originate a VPN-IP default route with a (non-null) MPLS label. That route MUST carry only RT-VPN, which means no other V-spokes import it, but V-hubs do ✨ The next hop of the NHLFE [RFC3031] tied to that label MUST specify the CE that advertises the IP default route. So when the V-spoke gets data carrying that label, it just forwards the data to the CE without doing an IP lookup on it 🚀 Here's the lowkey tricky bit: the VRF on the V-spoke ends up with an IP default route, but that route exists because the V-spoke got a VPN-IP default route from one of its V-hubs, NOT because it heard the IP default route from the CE 🤯 Also, if this V-spoke has other sites of that VPN hooked up, traffic from those sites headed to the Internet goes to the V-spoke, then to the V-hub the V-spoke picked, then back from that V-hub to the V-spoke, and finally to the CE advertising an IP default route to the V-spoke. It's a whole journey, periodt 🧳
If a PE is a V-spoke of a given VPN, and a CE of that VPN advertises an IP default route to it (because that CE's site provides the VPN's Internet connectivity), then the PE MUST NOT advertise an IP default route back to that CE 🙅♀️ But the CE still has to name that PE as the next hop for all the traffic going to other sites of that VPN. One way to make that work is to require the V-spoke to implement the procedures in Section 9 💅
Across every scenario in this section, we call the originated VPN-IP default route the "Internet VPN-IP default route" 🌟 To be exact, the Internet VPN-IP default route is a VPN-IP default route that a PE (a V-hub or a V-spoke) originates for either of two reasons: (a) it received an IP default route from a CE, or (b) the PE keeps Internet routes and has a default route in its VPN's VRF pointing at its own routing table that contains those Internet routes.
So what's the difference between an Internet VPN-IP default route and a non-Internet VPN-IP default route originated by a V-hub? It all comes down to the RTs 🎀 For a given VPN and a given V-hub in that VPN, the Internet VPN-IP default route carries both RT-VPN and RT-VH of that V-hub, while the non-Internet one carries only RT-VH of that V-hub.
When a V-hub originates the Internet VPN-IP default route, it MUST withdraw the non-Internet VPN-IP default route it had been originating. And when that V-hub withdraws the Internet VPN-IP default route it had been originating, it MUST originate a non-Internet VPN-IP default route. Basically, at any given moment, a V-hub is originating either the Internet VPN-IP default route OR a non-Internet VPN-IP default route, never both 🫧
Because of those rules, if a V-hub originates the Internet VPN-IP default route, then all the V-spokes tied to that V-hub MUST import it. Unlike a non-Internet VPN-IP default route, other V-hubs MAY import it too. A V-hub MAY also import Internet VPN-IP default routes that V-spokes originate. But a V-spoke MUST NOT import the Internet VPN-IP default route originated by any other V-spoke, and that route MAY be imported only by V-hubs 👑
Here's a fun one: if the Internet VPN-IP default route from one V-hub has the same preference as the (non-Internet) VPN-IP default route from another V-hub, then a V-spoke that imports VPN-IP default routes from both of those V-hubs ends up load sharing its outgoing Internet traffic between them 🍓 So some of that V-spoke's outgoing Internet traffic goes first to the V-hub that does NOT originate the Internet VPN-IP default route, and then from there over to the V-hub that DOES 😭
If taking that extra hop through another V-hub for Internet traffic is seen as undesirable, then it is RECOMMENDED that the Internet VPN-IP default route have a higher preference than a (non-Internet) VPN-IP default route originated by some other V-hub 💜 Heads up though: in that case, traffic from the V-spokes to other sites of that VPN won't get load shared between those two V-hubs.
6. Deployment Considerations 🧚
For a given VPN, you should pick a V-hub and a set of V-spokes tied to that V-hub in a way that keeps the extra network distance/latency penalty as small as possible, given the VPN's geographic footprint 🗺️
For a given VPN, some or all of its V-spokes can be grouped into geographically based clusters (for example, V-spokes in the same cluster might be physically close to each other) with any-to-any connectivity inside each cluster 🌸 The V-spokes in one cluster don't have to be tied to the same V-hub(s). Same goes the other way: not every V-spoke tied to a given V-hub has to be in the same cluster 🫶 Picture a big retail chain where data traffic goes hub-and-spoke between each store and the central datacenters, but the stores in the same area also need direct Voice over IP (VoIP) traffic between each other. That's exactly this kind of setup 🎀
Using constrained route distribution for BGP/MPLS IP VPNs ("RT constrains") [RFC4684] may further help facilitate and optimize routing exchange for V-hubs and V-spokes 💖
Adding a V-spoke PE to a VPN may bring the following changes for that VPN's customer:
- Traceroute from a CE connected to a V-spoke may show an extra hop: the V-hub PE 🔍
- Latency for traffic sent from a CE connected to a V-spoke may go up, depending on where the V-hub sits in the SP's layer 3 and layer 1 network topology 🐢
7. Multicast Considerations 🌈
This section describes procedures for supporting Multicast VPN (MVPN) when Virtual Hub-and-Spoke is in play 📡 The procedures rely on the MVPN specifications defined in [RFC6513], [RFC6514], and [RFC6625].
The procedures assume that, to make sure nothing gets duplicated, both V-hubs and V-spokes can discard packets from a "wrong" PE, as specified in Section 9.1.1 of [RFC6513] 🫧 The existing procedures for Selective Provider Multicast Service Interface (S-PMSI) auto-discovery (A-D) routes [RFC6513] [RFC6514] [RFC6625] are enough to discard packets coming from a "wrong" PE for every type of provider tunnel (P-tunnel) specified in [RFC6514], including Ingress Replication. The existing procedures for Inclusive Provider Multicast Service Interface (I-PMSI) A-D routes [RFC6513] [RFC6514] are enough to discard packets coming from a "wrong" PE for every type of P-tunnel specified in [RFC6514], EXCEPT for Ingress Replication 😭 Section 7.9 of this document changes the procedures in [RFC6514] so that packets from a "wrong" PE can be discarded when Ingress Replication is used for I-PMSI P-tunnels 💅
The V-hub/V-spoke architecture, as this document specifies it, affects certain multicast scenarios 👀 In particular, it affects scenarios where the source of a multicast flow is at a site attached to a V-hub and a receiver of that flow is at a site attached to a V-spoke that is NOT tied to that same V-hub. It also affects scenarios where the source is at a site attached to a V-spoke, a receiver is at a site attached to a different V-spoke, and the set intersection between the V-hub(s) tied to the first V-spoke and the V-hub(s) tied to the second V-spoke is empty. It may also affect scenarios where the source is at a site connected to a V-spoke, a receiver is at a site attached to a different V-spoke, and that set intersection is non-empty (these scenarios get affected if the I-PMSI/S-PMSI A-D routes originated by the first V-spoke aren't imported by the second V-spoke) 🦋
Using Virtual Hub-and-Spoke together with seamless MPLS multicast [MPLS-MCAST] is outside the scope of this document 🙅♀️
7.1. Terminology 📖
Here's the vocab: we'll say a P-tunnel is "bound" to a particular I-PMSI/S-PMSI A-D route if that P-tunnel is specified in that route's PMSI Tunnel attribute 🎀
When Ingress Replication is used, the P-tunnel bound to a particular I-PMSI/S-PMSI A-D route is actually a set of unicast tunnels (the procedures differ from [RFC6514] for I-PMSI and are specified in Section 7.9 of this document 🧚). The PE that originates the I-PMSI/S-PMSI A-D route uses these unicast tunnels to carry traffic to the PEs that import that route. The PEs that import the route advertise labels for the unicast tunnels in Leaf A-D routes that they originate in response to the I-PMSI/S-PMSI A-D route. When we say traffic was received by a PE on a P-tunnel "bound" to a particular I-PMSI/S-PMSI A-D route that PE imported, we mean the unicast tunnel whose label was advertised in a Leaf A-D route by the PE that imported the I-PMSI/S-PMSI route. The PE that originated that route uses this tunnel to send traffic to the PE that imported it 💜
7.2. Eligible Upstream Multicast Hop (UMH) Routes 🦋
Okay so on a V-spoke, the Eligible UMH routes are basically every unicast VPN-IP route the V-spoke receives, and that includes the default VPN-IP routes it gets from its V-hub(s) 💖. Ngl, the V-spoke can also have routes that came from other V-spokes, and those MAY be in the mix too 👀. Those can be "vanilla" VPN-IP routes (the ones using the IPv4 or IPv6 AFI with SAFI set to 128 "MPLS-labeled VPN address" [IANA-SAFI]), or routes that use the IPv4 or IPv6 AFI (as appropriate) but with SAFI 129 "Multicast for BGP/MPLS IP Virtual Private Networks (VPNs)" [IANA-SAFI] instead 🌸.
The default VPN-IP routes from the V-hub(s) can be either Internet default VPN-IP routes or non-Internet default VPN-IP routes 🌈.
7.3. Originating VPN-IP Default Route by a V-Hub 👑
When a V-hub originates a VPN-IP default route, it follows Section 3 like always, but it also follows Sections 6 and 7 of [RFC6514] (see also Section 5.1 of [RFC6513]) ✨. Specifically, the V-hub MUST add the VRF Route Import Extended Community that embeds the V-hub's IP address. The route also MUST include the Source AS extended community 💅.
7.4. Handling C-Multicast Routes 💅
In this part, "C-multicast routes" just means BGP routes that carry customer multicast routing information [RFC6514] 🫶.
Originating C-multicast routes follows the procedures in [RFC6514], no matter if a V-hub or a V-spoke is the one originating them 🧚.
When a V-spoke receives a C-multicast route, it follows the procedures described in [RFC6514] 🌸.
When a V-hub receives a C-multicast route, it checks whether the customer Rendezvous Point (C-RP) or the customer source (C-S) of the route is reachable via one of its VRF interfaces. If yes, the V-hub follows the procedures described in [RFC6514] ✨.
Otherwise, the C-RP/C-S is reachable via some other PE. This is the case where the route was originated by a V-spoke that sees the V-hub as the "upstream PE" for a given source, but the V-hub sees some other PE (either a V-hub or a V-spoke) as the "upstream PE" for that source 🙈. In this case, the V-hub uses the type (Source Tree Join vs Shared Tree Join), the Multicast Source, and Multicast Group from the received C-multicast route to build a new route of the same type, with the same Multicast Source and Multicast Group 🦋. The hub builds the rest of the new route following the procedures in Section 11.1.3 of [RFC6514]. The hub also creates the appropriate (C-*, C-G) or (C-S, C-G) state in its MVPN Tree Information Base (MVPN-TIB) 💜.
7.5. Originating I-PMSI/S-PMSI/SA A-D Routes by V-Spoke 🌈
When a V-spoke originates an I-PMSI, an S-PMSI, or Source Active (SA) A-D route, it follows the procedures in [RFC6514] (or, for a wildcard S-PMSI A-D route, the procedures in [RFC6625]), including the procedures for constructing RT(s) carried by the route 🎀. Note that as a result, such a route will be imported by the V-hubs. For an I-PMSI/S-PMSI A-D route, the P-tunnel bound to it is used to carry traffic from the sites connected to the V-spoke over to these V-hubs 💖.
If the V-spoke exports its (unicast) VPN-IP routes not just to the V-hubs but also to some other V-spokes (as described in Section 3), then the I-PMSI/S-PMSI/SA A-D route it originates will be imported by the V-hubs AND by the other V-spokes 😭. Why? Because following the procedures in [RFC6514] (or, for a wildcard S-PMSI A-D route, the procedures in [RFC6625]) means the route carries more than one RT. One of those RTs, RT-VPN, imports the route into the V-hubs, while the other RT(s) import it into the V-spokes (the other RT(s) are the RT(s) the V-spoke uses for importing the VPN-IP default route) 🧚. In this case, the P-tunnel bound to this I-PMSI/S-PMSI A-D route is also used to carry traffic from the sites connected to the originating V-spoke to these other V-spokes 🫶.
7.6. Originating I-PMSI/S-PMSI/SA A-D Routes by V-Hub 👑
When a V-hub originates an I-PMSI/S-PMSI/SA A-D route, it follows the procedures in [RFC6514] (or, for a wildcard S-PMSI A-D route, the procedures in [RFC6625]). The one twist is that, on top of the RT(s) built by those procedures, the route MUST also carry the RT of the VPN-IP default route advertised by the V-hub (RT-VH) 💅. So this route gets imported by other V-hubs AND by the V-spokes, but only the V-spokes associated with that V-hub (the V-spokes that import the VPN-IP default route the V-hub originated) 🦋. For an I-PMSI/S-PMSI A-D route, the P-tunnel bound to it carries traffic from the sites connected to the originating V-hub over to those other V-hubs and V-spokes 🌸.
Also, if a V-hub originates an I-PMSI A-D route following the procedures in [RFC6514], the V-hub MUST originate another I-PMSI A-D route too. We'll call this one an "Associated-V-spoke-only I-PMSI A-D route" 🙈. The RT carried by this route MUST be the RT that is carried in the VPN-IP default route advertised by the V-hub (RT-VH). That means this route only gets imported by the V-spokes associated with the V-hub (the V-spokes that import the VPN-IP default route advertised by this V-hub) 🔥. The P-tunnel bound to this route carries traffic from the sites connected to (a) other V-hubs, (b) other V-spokes, including the V-spokes that import the VPN-IP default route from the V-hub, or (c) "vanilla" PEs over to these V-spokes 💜.
More details on using this P-tunnel are in Section 7.8 ✨.
So, a V-hub ends up originating not one, but TWO I-PMSI A-D routes: a "vanilla" I-PMSI A-D route and an Associated-V-spoke-only I-PMSI A-D route 👯. Each of these routes MUST have a distinct RD 💖.
When a V-hub gets traffic from one of its connected sites, and it decides (using some local policies) that this traffic should go out using an I-PMSI, it forwards that traffic on the P-tunnel bound to the "vanilla" I-PMSI A-D route. It MUST NOT forward it on the P-tunnel bound to the Associated-V-spoke-only I-PMSI A-D route 🚫💅
7.7. Receiving I-PMSI/S-PMSI/SA A-D Routes by V-Spoke 🧚
When a V-spoke receives an I-PMSI/S-PMSI/SA A-D route, it follows the procedures in [RFC6514] (or, for a wildcard S-PMSI A-D route, the procedures in [RFC6625]) 🌈. So a V-spoke that's associated with a given V-hub (the V-spoke that imports the VPN-IP default route that V-hub originated) will also import the I-PMSI/S-PMSI/SA A-D routes that V-hub originates. To be specific, the V-spoke imports BOTH the "vanilla" I-PMSI A-D route AND the Associated-V-spoke-only I-PMSI A-D route from that V-hub 🎀.
And if a V-spoke imports the (unicast) VPN-IP routes that some other V-spokes originated (as described in Section 3), then it will also import the I-PMSI/S-PMSI/SA A-D routes from those other V-spokes too 🦋.
7.8. Receiving I-PMSI/S-PMSI/SA A-D Routes by V-Hub 💜
The following describes the procedures a V-hub MUST follow when it receives an I-PMSI/S-PMSI/SA A-D route ✨.
7.8.1. Case 1 🌸
This is the case where a V-hub receives an I-PMSI/S-PMSI/SA A-D route, and one of the RT(s) carried in the route is the RT that the V-hub uses for advertising its VPN-IP default route (RT-VH) 👀.
In this case, the received route was originated either:
- by a V-spoke associated with the V-hub (the V-spoke that imports the VPN-IP default route originated by the V-hub), or
- by some other V-hub that uses the same RT as the receiving V-hub for advertising the VPN-IP default route.
So the received I-PMSI/S-PMSI/SA A-D route carries more than one RT. One of those RTs results in importing the route into the V-hubs. Another one is the RT the V-hub uses when advertising its VPN-IP default route (RT-VH). That RT imports the received route into ALL the V-spokes associated with the V-hub (the V-spokes that import the VPN-IP default route originated by the V-hub) 🫶.
When handling this kind of route, the V-hub simply follows the procedures in [RFC6514] (or, for a wildcard S-PMSI A-D route, the procedures in [RFC6625]) 🌷.
Specifically, the V-hub MUST NOT reoriginate this route the way Case 2 below does 🙅.
Here are the rules the V-hub MUST follow when handling traffic it receives on a P-tunnel bound to this I-PMSI/S-PMSI A-D route 🔥. The V-hub may forward this traffic only to the sites connected to that V-hub (forwarding it to those sites follows the procedures in [RFC6514] or, for a wildcard S-PMSI A-D route, the procedures in [RFC6625]) 💖. The V-hub MUST NOT forward the traffic received on this P-tunnel to any other V-hubs or V-spokes, including the V-spokes that import the VPN-IP default route originated by the V-hub (the V-spokes associated with the V-hub) 😭. Specifically, the V-hub MUST NOT forward the traffic it receives on the P-tunnel advertised in the received I-PMSI A-D route over the P-tunnel that the V-hub binds to its Associated-V-spoke-only I-PMSI A-D route 🛑💅
7.8.2. Case 2 ✨
Okay so this is Case 2 💅. A V-hub receives an I-PMSI/S-PMSI/SA A-D route, and that route does not carry the RT the V-hub uses when it advertises its VPN-IP default route (RT-VH).
The route came from either some other V-hub or a V-spoke. The V-hub imports it (and so do the other V-hubs), but none of the V-spokes tied to this V-hub import it. Those are the V-spokes that import the VPN-IP default route this V-hub originated. 🦋
In this case the V-hubs follow the procedures in [RFC6514] (or the procedures in [RFC6625] if it's a wildcard S-PMSI A-D route), with these extras:
Once a V-hub accepts an I-PMSI A-D route, and data shows up on the P-tunnel bound to that route, the V-hub uses the procedures in [RFC6513] and [RFC6514] to decide whether to accept that data. If the data is accepted, the V-hub forwards it over the P-tunnel bound to the Associated-V-spoke-only I-PMSI A-D route it originated. 🌸
Quick heads up on that forwarding decision: the V-hub SHOULD take into account the (multicast) state in its MVPN-TIB that came from C-multicast routes received from its associated V-spokes. If the MVPN-TIB shows that none of those V-spokes have receivers for the data, the V-hub SHOULD NOT forward the data over the P-tunnel bound to the Associated-V-spoke-only I-PMSI A-D route it originated. 💀
Whenever a V-hub imports an S-PMSI A-D route (or an SA A-D route), unlike in Case 1, the V-hub MUST originate an S-PMSI A-D route (or SA A-D route) aimed at its V-spokes 👑. To pull this off, it swaps the RT(s) on the route for the RT it uses when originating its VPN-IP default route (RT-VH). It changes the RD to the RD it uses for its Associated-V-spoke-only I-PMSI A-D route, and it sets Next Hop to the IP address it puts in the Global Administrator field of the VRF Route Import Extended Community on its advertised VPN-IP routes. For S-PMSI A-D routes, it also changes the Originating Router's IP address in the MCAST-VPN NLRI (Network Layer Reachability Information) to match the Next Hop. Before advertising the new route, the V-hub also tweaks its PMSI Tunnel attribute as needed (for example, by swapping the P-tunnel rooted at the route's originator for one rooted at the V-hub). ✨
Heads up: a V-hub in a given MVPN may receive and accept multiple (C-*, C-*) wildcard S-PMSI A-D routes [RFC6625], each one from its own PE. Even so, the V-hub re-advertises just one (C-*, C-*) S-PMSI A-D route, which means it aggregates the ones it received. The same goes for (C-*, C-G) S-PMSI A-D routes. 🧚
Whenever a V-hub receives data on the P-tunnel bound to a received S-PMSI A-D route, it follows [RFC6513] and [RFC6514] (or [RFC6625] for a wildcard S-PMSI A-D route) to decide whether to accept the data. If the data is accepted, the V-hub forwards it over the P-tunnel bound to the S-PMSI A-D route it re-advertised. 🫶
If the V-hub got multiple S-PMSIs that were aggregated into the same P-tunnel, then before forwarding that data to its associated V-spokes, the V-hub MAY de-aggregate and then re-aggregate it (in a different way). It can use the MVPN-TIB state built from C-multicast routes received from those V-spokes to do this. 🎀
Even if each S-PMSI the V-hub received has its own P-tunnel, the V-hub MAY still aggregate those S-PMSIs before forwarding the data to the V-spokes. It uses the same MVPN-TIB state for that too. 🌈
7.9. Use of Ingress Replication with I-PMSI A-D Routes
These tweaks to the [RFC6514] procedures for originating and receiving I-PMSI A-D routes let a PE discard packets from a "wrong" PE when Ingress Replication is used for I-PMSI P-tunnels. For other P-tunnel types, [RFC6513] and [RFC6514] are already enough. 🔥
The modifications are required to be implemented (by all the PEs of a given MVPN) only under these conditions:
- At least one of those PEs is a V-hub or V-spoke PE for the given MVPN.
- The given MVPN is configured to use the optional procedure of using Ingress Replication to instantiate an I-PMSI.
If Ingress Replication is used with I-PMSI A-D routes, then when a PE advertises such a route, the Tunnel Type in the PMSI Tunnel attribute MUST be set to Ingress Replication; the Leaf Information Required flag MUST be set to 1; and the attribute MUST carry no MPLS labels. 💖
A PE that receives such an I-PMSI A-D route MUST respond with a Leaf A-D route. The PMSI Tunnel attribute of that Leaf A-D route is built like this:
- The Tunnel Type is set to Ingress Replication.
- The Tunnel Identifier MUST carry a routable address of the PE that originates the Leaf A-D route.
- The PMSI Tunnel attribute MUST carry a downstream-assigned MPLS label that is used to demultiplex the traffic received over a unicast tunnel by the PE.
- The receiving PE MUST assign the label in such a way as to enable the receiving PE to identify (a) the VRF on that PE that should be used to process the traffic received with this label and (b) the PE that sends the traffic with this label.
This document assumes that for a given MVPN, all the PEs that have sites of that MVPN connected to them implement the procedures specified in this section. 🙈
8. An Example of RT Provisioning ✨
Okay, time for an example! 🎀 Picture VPN A with 9 sites, site-1 through site-9, and each site is hooked up to its own PE, PE-1 through PE-9.
We make PE-3, PE-6, and PE-9 the V-hubs 👑.
To keep things simple, this example has each V-spoke associated with just one V-hub. But like we said earlier, in practice each V-spoke should be associated with two or more V-hubs.
PE-1 and PE-2 are V-spokes for PE-3. PE-4 and PE-5 are V-spokes for PE-6. PE-7 and PE-8 are V-spokes for PE-9. 🦋
8.1. Unicast Routing
All the PEs (V-hubs and V-spokes alike) are set up to export routes with RT-A, just like a "vanilla" any-to-any VPN. 💅
All the V-hubs (PE-3, PE-6, and PE-9) import routes with RT-A, same as a "vanilla" any-to-any VPN. 🌸
PE-3 also originates a VPN-IP default route with RT-A-VH-1 (but not with RT-A), and PE-1 and PE-2 import routes with RT-A-VH-1. 🧚
Same deal for PE-6: it originates a VPN-IP default route with RT-A-VH-2 (but not with RT-A), and PE-4 and PE-5 import routes with RT-A-VH-2. ✨
PE-9 originates a VPN-IP default route with RT-A-VH-3 (but not with RT-A), and PE-7 and PE-8 import routes with RT-A-VH-3. 🫶
Now let's spice it up: site-3 has Internet connectivity. So site-3 advertises an IP default route to PE-3, and PE-3 originates a VPN-IP default route in response. This time that route carries RT-A and RT-A-VH-1 (instead of just RT-A-VH-1, like before). That means it gets imported by PE-6 and PE-9, as well as PE-1 and PE-2. 🌈
If PE-7 and PE-8 also want to import a VPN-IP default route from PE-9, plus each other's VPN-IP routes, then PE-7 and PE-8 export their VPN-IP routes with two RTs: RT-A and RT-A-VH-3. 💜
8.2. Multicast Routing
All the PEs designated as V-spokes (PE-1, PE-2, PE-4, PE-5, PE-7, and PE-8) are set up to export their I-PMSI/S-PMSI/SA A-D routes with RT-A, just like a "vanilla" any-to-any MVPN. That means all the V-hubs (PE-3, PE-6, and PE-9) can import these routes. 🎀
The V-hub on PE-3 exports its I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-1. So those routes can be imported by the other V-hubs (PE-6 and PE-9) and by the V-spokes, but only the V-spokes tied to the V-hub on PE-3 (PE-1 and PE-2) actually get them. On top of that, the V-hub on PE-3 originates the Associated-V-spoke-only I-PMSI A-D route with RT-A-VH-1. Only the V-spokes tied to the V-hub on PE-3 (PE-1 and PE-2) can import that one. 🔥
The V-hub on PE-6 exports its I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-2. So those routes can be imported by the other V-hubs (PE-3 and PE-9) and by the V-spokes, but only the V-spokes tied to the V-hub on PE-6 (PE-4 and PE-5) get them. The V-hub on PE-6 also originates the Associated-V-spoke-only I-PMSI A-D route with RT-A-VH-2, which only the V-spokes tied to the V-hub on PE-6 (PE-4 and PE-5) can import. 💖
The V-hub on PE-9 exports its I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-3. So those routes can be imported by the other V-hubs (PE-3 and PE-6) and by the V-spokes, but only the V-spokes tied to the V-hub on PE-9 (PE-7 and PE-8) get them. The V-hub on PE-9 also originates the Associated-V-spoke-only I-PMSI A-D route with RT-A-VH-3, which only the V-spokes tied to the V-hub on PE-9 (PE-7 and PE-8) can import. 👑
If PE-7 and PE-8 also want to import a VPN-IP default route from PE-9 and each other's VPN-IP routes, they export their I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and RT-A-VH-3. 🌸
If the V-hub on PE-9 imports an S-PMSI A-D route or SA A-D route that was originated by either another V-hub (PE-3 or PE-6) or by a V-spoke that isn't associated with this V-hub (PE-1, or PE-2, or PE-4, or PE-5), then the V-hub originates an S-PMSI A-D route (respectively, SA A-D route). It builds this route from the imported one following the procedures in Section 7.8.2. Specifically, it replaces the RT(s) carried in the imported route with just one RT -- RT-A-VH-3. So the originated route can be imported only by the V-spokes tied to the V-hub on PE-9 (PE-7 and PE-8). 🦋
9. Further Refinements ✨
Okay so, sometimes a VPN customer doesn't wanna rely only on an (IP) default route being advertised from a V-spoke to a CE 🥺. They might want their CEs to get all the VPN routes, like for faster detection of VPN connectivity failures and spinning up some backup connectivity 💅.
In that case, an OPTIONAL approach is to install only the VPN-IP default route advertised by the V-hub associated with the V-spoke into the V-spoke's data plane, even if the V-spoke gets an IP default route from the CE 🧚. Then you keep all the VPN-IP routes in the V-spoke's control plane, so the V-spoke can still advertise them as IP routes to the CEs 🎀. Granted, this doesn't change control-plane resource consumption at all, but it does cut down on forwarding state in the data plane 🦋.
10. Security Considerations 🔒
Okay, this one is short but SO important 😭. This document introduces no new security considerations above and beyond those already specified in [RFC4364]. So all the security tea lives over in RFC 4364, bestie 🫶.
11. Acknowledgements 💖
Huge shoutout to Han Nguyen (AT&T) for his contributions to this document 👑. Thank you to Eric Rosen (Cisco) for his review and comments 🌸. We also love Samir Saad (AT&T), Jeffrey (Zhaohui) Zhang (Juniper), and Thomas Morin (Orange) for their review and comments 🥰.
12. References 📚
12.1. Normative References
[IANA-SAFI] IANA Subsequent Address Family Identifiers (SAFI)
Parameters,
<http://www.iana.org/assignments/safi-namespace/>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3031] Rosen, E., Viswanathan, A., and R. Callon,
"Multiprotocol Label Switching Architecture", RFC 3031,
January 2001.
[RFC4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
Communities Attribute", RFC 4360, February 2006.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, February 2006.
[RFC4684] Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk,
R., Patel, K., and J. Guichard, "Constrained Route
Distribution for Border Gateway Protocol/MultiProtocol
Label Switching (BGP/MPLS) Internet Protocol (IP)
Virtual Private Networks (VPNs)", RFC 4684, November
2006.
[RFC6513] Rosen, E., Ed., and R. Aggarwal, Ed., "Multicast in
MPLS/BGP IP VPNs", RFC 6513, February 2012.
[RFC6514] Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
Encodings and Procedures for Multicast in MPLS/BGP IP
VPNs", RFC 6514, February 2012.
[RFC6625] Rosen, E., Ed., Rekhter, Y., Ed., Hendrickx, W., and R.
Qiu, "Wildcards in Multicast VPN Auto-Discovery Routes",
RFC 6625, May 2012.12.2. Informative References
[MPLS-MCAST] Rekhter, Y., Aggarwal, R., Morin, T., Grosclaude, I.,
Leymann, N., and S. Saad, "Inter-Area P2MP Segmented
LSPs", Work in Progress, May 2013.Authors' Addresses 💌
Huajin Jeng
AT&T
EMail: hj2387@att.com
James Uttaro
AT&T
EMail: ju1738@att.com
Luay Jalil
Verizon
EMail: luay.jalil@verizon.com
Bruno Decraene
Orange
EMail: bruno.decraene@orange.com
Yakov Rekhter
Juniper Networks, Inc.
EMail: yakov@juniper.net
Rahul Aggarwal
Arktan
EMail: raggarwa_1@yahoo.com📦 where this page comes from
- ietf: rfc7024 2013-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms