RFC 6517
Mandatory Features in a Layer 3 Multicast BGP/MPLS VPN Solution
Mandatory Features in a Layer 3 Multicast BGP/MPLS VPN Solution 🎀
okay bestie, this is RFC 6517 and it's giving informational energy 💅 It's not a standards-track spec, it's published just to share info. The title is "Mandatory Features in a Layer 3 Multicast BGP/MPLS VPN Solution" ✨
- Authors: T. Morin (Ed.) from France Telecom - Orange, B. Niven-Jenkins (Ed.) from BT, Y. Kamite from NTT Communications, R. Zhang from Alcatel-Lucent, N. Leymann from Deutsche Telekom, and N. Bitar from Verizon 🌸
- Category: Informational
- ISSN: 2070-1721
- Date: February 2012
- Obsoletes / Updates: the title block doesn't list either one 🦋
Abstract
Okay so, there's more than one set of ways to do multicast in a layer 3 BGP/MPLS VPN, and each one was written up in its own doc as an optional building block 🧩 The problem is that optional means implementations can pick different pieces and then they won't play nice together 😭
So this doc picks out a subset of features that's treated as mandatory for a multicast BGP/MPLS VPN implementation. The goal is to help implementers and deployers see which L3VPN multicast requirements each option handles best. Super helpful, no cap 💖
Status of This Memo
This document is not an Internet Standards Track specification; it is published for informational purposes. Lowkey, that's the whole vibe 🌈
It's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is up for any level of Internet Standard, though. Check Section 2 of RFC 5741 for that 🧚
For the current status, any errata, and how to send feedback, head to http://www.rfc-editor.org/info/rfc6517 🔥
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. Please read these carefully, because they spell out your rights and restrictions for this doc 🙈 Code Components pulled out of this document must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🍓
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Examining Alternative Mechanisms for MVPN Functions . . . . . 4
3.1. MVPN Auto-Discovery . . . . . . . . . . . . . . . . . . . 4
3.2. S-PMSI Signaling . . . . . . . . . . . . . . . . . . . . . 5
3.3. PE-PE Exchange of C-Multicast Routing . . . . . . . . . . 7
3.3.1. PE-PE C-Multicast Routing Scalability . . . . . . . . 7
3.3.2. PE-CE Multicast Routing Exchange Scalability . . . . . 10
3.3.3. Scalability of P Routers . . . . . . . . . . . . . . . 10
3.3.4. Impact of C-Multicast Routing on Inter-AS Deployments 10
3.3.5. Security and Robustness . . . . . . . . . . . . . . . 11
3.3.6. C-Multicast VPN Join Latency . . . . . . . . . . . . . 12
3.3.7. Conclusion on C-Multicast Routing . . . . . . . . . . 14
3.4. Encapsulation Techniques for P-Multicast Trees . . . . . . 15
3.5. Inter-AS Deployments Options . . . . . . . . . . . . . . . 16
3.6. BIDIR-PIM Support . . . . . . . . . . . . . . . . . . . . 19
4. Co-Located RPs . . . . . . . . . . . . . . . . . . . . . . . . 20
5. Avoiding Duplicates . . . . . . . . . . . . . . . . . . . . . 21
6. Existing Deployments . . . . . . . . . . . . . . . . . . . . . 21
7. Summary of Recommendations . . . . . . . . . . . . . . . . . . 22
8. Security Considerations . . . . . . . . . . . . . . . . . . . 22
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 23
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23
10.1. Normative References . . . . . . . . . . . . . . . . . . . 23
10.2. Informative References . . . . . . . . . . . . . . . . . . 23
Appendix A. Scalability of C-Multicast Routing Processing Load . 25
A.1. Scalability with an Increased Number of PEs . . . . . . . 26
A.1.1. SSM Scalability . . . . . . . . . . . . . . . . . . . 26
A.1.2. ASM Scalability . . . . . . . . . . . . . . . . . . . 35
A.2. Cost of PEs Leaving and Joining . . . . . . . . . . . . . 37
Appendix B. Switching to S-PMSI . . . . . . . . . . . . . . . . . 401. Introduction ✨
Okay so the specs for multicast in BGP/MPLS [RFC6513] come with several alternative mechanisms for some of the required building blocks. But they never say which ones you MUST have to get interop 😭 Without a mandatory-to-implement set, one implementation might support a different set of optional mechanisms than another, and they just won't work together. That's a real headache for the many operators running multi-vendor backbones 💀
This doc builds on the requirements that were already written down [RFC4834]. It studies how each approach behaves, looking for mechanisms that are good candidates for a core mandatory set. That core set can be the foundation for interoperable solutions, and honestly that's iconic 👑
It goes through the different building blocks of the solution and works out which mechanisms an implementation is required to implement. Section 7 sums all of this up 🦋
Given the history of multicast VPN proposals and implementations, it's also worth talking about how existing deployments of early implementations [RFC6037] [MVPN] can be accommodated, and the doc offers some suggestions on that 🫶
2. Terminology 💅
Please refer to [RFC6513] and [RFC4834]. As a reminder, Section 3.1 of [RFC6513] uses the "C-" and "P-" notations to tell apart the provider scope from a specific VPN customer's scope. So "C-PIM" means a PIM protocol instance inside a VPN or VRF, while "P-PIM" means the instance of PIM the provider eventually runs across its core, between P and PE routers 🌸
Other acronyms used in this document include the following:
- LSP: Label Switched Path
- P2MP: Point to Multipoint
- MP2MP: Multipoint to Multipoint
- GRE: Generic Routing Encapsulation
- mLDP: Multicast LDP
- I-PMSI: Inclusive Provider Multiservice Interface
- MI-PMSI: Multidirectional Inclusive Provider Multiservice Interface
- S-PMSI: Selective Provider Multiservice Interface
- SSM: Source-Specific Multicast
- ASM: Any-Source Multicast
- PIM-SM: PIM Sparse Mode
- PIM-SSM: PIM Sparse Mode in SSM Mode
- BIDIR-PIM: Bidirectional PIM
- AS: Autonomous System
- ASBR: Autonomous System Border Router
- VRF: VPN Routing and Forwarding
- PE: Provider Edge
- CE: Customer Edge
- RPA: Rendezvous Point Address
- RPL: Rendezvous Point Link
Additionally, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. These are the official RFC 2119 key words, so the capitals matter 🔥
3. Examining Alternative Mechanisms for MVPN Functions 🦋
3.1. MVPN Auto-Discovery 💖
The current solution doc [RFC6513] offers two different mechanisms for Multicast VPN (MVPN) auto-discovery:
- BGP-based auto-discovery
- "PIM/shared P-tunnel": discovery done through the exchange of PIM Hellos by C-PIM instances, across an MI-PMSI implemented with one shared P-tunnel per VPN (using ASM or MP2MP LDP)
Both solutions address Section 5.2.10 of [RFC4834], which says "the operation of a multicast VPN solution SHALL be as light as possible, and providing automatic configuration and discovery SHOULD be a priority when designing a multicast VPN solution. Particularly, the operational burden of setting up multicast on a PE or for a VR/ VRF SHOULD be as low as possible". So the goal is to keep things easy to set up and run 🧁
Here's the key thing 👀 PIM-based discovery only works for deployments that use a shared P-tunnel to instantiate an MI-PMSI. It doesn't work if you're only using P2P, PIM-SSM, and P2MP LDP/RSVP-TE P-tunnels, because unlike ASM and MP2MP LDP, building those kinds of P-tunnels can't happen until auto-discovery has been done. BGP-based auto-discovery, on the other hand, doesn't put any constraint on which type of P-tunnel you use. It's independent of the P-tunnel type, which meets the requirement in Section 5.2.4.1 of [RFC4834] that "a multicast VPN solution SHOULD be designed so that control and forwarding planes are not interdependent" ✨
Also, a bunch of service providers already use SSM-based P-tunnels for their default multicast distribution trees in current deployments, so they're already relying on some BGP-based auto-discovery 🌈
Another perk: when shared P-tunnels are in use, BGP auto-discovery lets you catch inconsistencies in the addresses or identifiers used for the shared P-tunnel. For example, the same shared P-tunnel identifier showing up in different VPNs with distinct BGP route targets. That's especially attractive for inter-AS VPNs, where one misconfiguration can get magnified and a single service provider might not run all the ASes 😳 Heads up though: this detection technique may not work during a transition period from shared-P-tunnel auto-discovery over to BGP-based auto-discovery 🙈
So the recommendation is that BGP-based auto-discovery is mandated and should be supported by all MVPN implementations. Main character energy, honestly 👑
3.2. S-PMSI Signaling 🦋
okay so the current solution doc, [RFC6513], gives us two ways to signal that multicast flows are about to get switched over to a Selective PMSI (S-PMSI) 💅
- a UDP-based TLV protocol that's built just for S-PMSI signaling (check out Section 7.4.2 for the deets) 🎀
- a BGP-based mechanism for S-PMSI signaling (that one's in Section 7.4.1) 🌸
so [RFC4834] has this whole vibe in Section 5.2.10 where it says "as far as possible, the design of a solution SHOULD carefully consider the number of protocols within the core network: if any additional protocols are introduced compared with the unicast VPN service, the balance between their advantage and operational burden SHOULD be examined thoroughly". ngl that's a super fair thing to care about 🧐✨
so the UDP-based option would add a brand new protocol to the MVPN stack, and the BGP-based S-PMSI switching signaling doesn't, because (a) BGP is already required for auto-discovery, and (b) the BGP-based S-PMSI switching procedures look a LOT like the auto-discovery procedures. it's giving "one protocol to rule them all" 👑
also the UDP-based approach needs an MI-PMSI, but the BGP-based one doesn't. in practice that means a PE with the UDP-based protocol has to join all the P-tunnels of every PE in the MVPN 😭. with BGP-based signaling, though, a PE can wait to join a P-tunnel rooted at another PE until traffic from that PE is actually needed. that's way less state sitting on the P routers 🙌
S-PMSI switching signaling can also be compared in an inter-AS context (see Section 3.5). the BGP-based approach is a really good fit with both the segmented and non-segmented inter-AS approaches (again, Section 3.5). the UDP-based approach looks like it's usable with segmented inter-AS tunnels, BUT some major perks of the segmented approach get lost 💔
- ASes are no longer independent in their ability to choose when S-PMSI tunnels will be triggered in their AS (and so they lose control over how much state gets created on their P routers).
- ASes are no longer independent in their ability to choose the tunneling technique for the P-tunnels used for an S-PMSI.
- in an inter-AS option B context, ASes stay isolated because PEs in one AS don't have (direct) exchange of routing information with PEs in other ASes. that property does NOT survive if UDP-based S-PMSI switching signaling is used. BGP-based C-multicast switching signaling, on the other hand, does keep it intact 🥹
so with all that said, the authors recommend that BGP be the preferred solution for S-PMSI switching signaling, and that it should be supported by all implementations. iykyk, it's the clear winner 🏆
if nothing stops S-PMSIs from being created super fast, then S-PMSI switching signaling with BGP could possibly hit the route reflectors (RRs) that handle MVPN routes. BUT a fast-paced situation like that would hit P and PE routers from the S-PMSI tunnel signaling too, and that impact is the same no matter which S-PMSI signaling approach you pick. so the real move is to avoid it entirely by setting up proper mechanisms 🛡️
the UDP-based S-PMSI switching signaling protocol can still be considered as an option, since it's been deployed for a while now. implementations that support both protocols should offer a per-VRF (VPN Routing and Forwarding) configuration knob. that lets an implementation use the UDP-based TLV protocol for S-PMSI switching signaling on specific VRFs, so both protocols can co-exist (for example, during migration scenarios) 🦋. apart from those migration-friendly mechanisms, though, the authors specifically do not recommend extending the already proposed UDP-based TLV protocol to new types of P-tunnels. no new P-tunnel types for UDP, bestie 🙅♀️
3.3. PE-PE Exchange of C-Multicast Routing 🌈
okay so [RFC6513] also proposes multiple ways for PEs to swap customer multicast routing info (C-multicast routing) with each other:
- full per-MVPN PIM peering across an MI-PMSI (described in Section 3.4.1.1) 💜
- lightweight PIM peering across an MI-PMSI (described in Section 3.4.1.2) 🧚
- unicasting PIM C-Join/Prune messages (described in Section 3.4.1.3) 🍓
- using BGP to carry C-multicast routing (described in Section 3.4.2) 👑
3.3.1. PE-PE C-Multicast Routing Scalability ✨
scalability is one of the core requirements for multicast VPN, so it's super useful to compare the proposed C-multicast routing mechanisms from this angle. [RFC4834] Section 4.2.4 recommends that "a multicast VPN solution SHOULD support several hundreds of PEs per multicast VPN, and MAY usefully scale up to thousands", and Section 4.2.5 says "a solution SHOULD scale up to thousands of PEs having multicast service enabled" 🫶
scaling with more VPNs per PE, or with more multicast state per VPN, also matters a lot, but this section doesn't dig into it because we didn't find differences between the approaches on that front. everything else being equal, the load on a PE from C-multicast routing goes up roughly linearly with the number of VPNs per PE and with the amount of multicast state per VPN 📈
this section has the conclusions about PE-PE C-multicast routing scalability. Appendix A has the deeper dive on how the PIM-based approaches and the BGP-based approach handle the C-multicast routing load differently, plus quantified numbers on the state and messages for each approach. a bunch of the points made here are detailed in Appendix A.1 🤓
at high multicast scales, the first and third mechanisms make PEs keep a big pile of PIM adjacencies with other PEs in the same multicast VPN (which means regularly exchanging PIM Hellos with each other). they also have to periodically refresh C-Join/Prune states. so the processing cost goes up as the number of PEs grows (more in Appendix A.1). the second approach is less affected by this, and the fourth approach isn't affected by it at all 🙌
the third mechanism would cut down the amount of C-Join/Prune processing for a given multicast flow on PEs that aren't the upstream neighbor for that flow. the catch is it needs "explicit tracking" state maintained by the upstream PE 🥺. it also isn't compatible with the "Join suppression" mechanism. one way to reduce the signaling with this approach is a PIM refresh-reduction mechanism. that mechanism, which is based on TCP, is being specified by the PIM IETF Working Group ([PIM-PORT]). its use in a multicast VPN context isn't described in [RFC6513], but this approach is expected to give scalability similar to the BGP-based approach without RRs 💅
the second mechanism would work similarly to full per-MVPN PIM peering, except PIM Hello messages aren't sent and PIM C-Join/Prune refresh-reduction gets used. that improves scalability, but this approach hasn't been fully described yet. either way, it only improves one of the things that impact scalability as the number of PEs grows 🤔
the first and second mechanisms can take advantage of "Join suppression" and so reduce the processing burden on an upstream PE. the upside is that the upstream PE doesn't have to process a Join refresh message for each remote PE joined to a multicast stream 💖. this improvement needs all PEs in a multicast VPN to process all PIM Join and Prune messages sent by any other PE in the same multicast VPN, whether they're the upstream PE or not.
the fourth mechanism (using BGP to carry C-multicast routing) has a similar drawback: all PEs have to process a BGP C-multicast route that only matters to one specific upstream PE 😬. that's why Section 16 of [RFC6514] recommends the Route Target constrained BGP distribution [RFC4684] mechanisms. those fix this by making sure only the interested upstream PE receives a BGP C-multicast route. specifically, when Route Target constrained BGP distribution is used, the fourth mechanism cuts the total C-multicast routing processing load on PEs, because "unrelated" PEs (the ones that are neither the joining PE nor the upstream PE) don't process customer multicast routing info at all 🎉
the fourth mechanism also cuts the total message processing load even more. it skips periodic refreshes and inherits BGP features that are expected to help scalability (for instance, it can offload some of the processing burden tied to customer multicast routing onto one or many BGP route reflectors 🦋). the perks come at a cost, though: state that grows linearly with the number of PEs joined to a stream. the good news is route reflectors let that cost get spread across multiple route reflectors, so a single route reflector doesn't have to hold all of that state 💜
BUT the fourth mechanism is special because it can offload customer multicast routing processing onto one or more BGP route reflector(s). when that happens, there's a drawback: the processing load on the route reflector infrastructure goes up. at higher scales, it may be necessary to adapt the route reflector infrastructure to the MVPN routing load, for example by:
- separating resources for unicast and multicast VPN routing: using dedicated MVPN route reflector(s) (or using dedicated MVPN BGP sessions or dedicated MVPN BGP instances), and
- deploying additional route reflector resources, for example by increasing the processing resources on existing route reflectors or deploying additional route reflectors.
the most straightforward approach is to introduce route reflectors dedicated to the MVPN service and size them for what that service needs (doing so is not required, though, and it's left up to the operator's engineering call) 🧑💻🌸
3.3.2. PE-CE Multicast Routing Exchange Scalability 🫧
the overhead from the PE-CE exchange of C-multicast routing doesn't depend on which mechanism you pick for PE-PE C-multicast routing. so the impact of that PE-CE overhead on overall system scalability is independent of the protocol used for PE-PE signaling, which means it's not relevant when comparing the different PE-PE approaches proposed here. this holds even if, in some operational setups, PE-CE C-multicast routing overhead is a big chunk of the total system overhead 💀
3.3.3. Scalability of P Routers ✨
okay so the first and second mechanisms are only allowed to be used in multicast VPNs that use an MI-PMSI 🫶 which means you end up needing either:
- a P-tunnel technique that allows shared P-tunnels (for example, PIM-SM in ASM mode or MP2MP LDP), or
- one P-tunnel per PE per VPN, even for PEs that don't have sources in their directly attached sites for that VPN 💀
the fourth mechanism is giving main character energy here because it doesn't impose either of those restrictions 💅 and when P2MP P-tunnels are used, it only needs one P-tunnel per VPN per PE attached to a site with a multicast source or Rendezvous Point (RP) (or with a candidate Bootstrap Router (BSR), if BSR is used). iconic 🦋
when there are fewer PEs connected with sources than the total number of PEs, the fourth mechanism cuts down the amount of state the P routers have to maintain, compared to what's needed to build an MI-PMSI with P2MP P-tunnels 📉✨ and these cases are expected to be super common for multicast VPN deployments (see Section 4.2.4.1 of [RFC4834]) 🌸
3.3.4. Impact of C-Multicast Routing on Inter-AS Deployments 🌈
so co-existence with unicast inter-AS VPN options, plus an equal level of security for multicast and unicast (including in an inter-AS context), are specifically mentioned in Sections 5.2.6 and 5.2.8 of [RFC4834] 📚
in an inter-AS option B context, you get isolation of ASes because PEs in one AS don't have (direct) exchange of routing information with PEs of other ASes 🧚 but that property is NOT preserved if PIM-based PE-PE C-multicast routing is used 😭 the fourth option (BGP-based C-multicast routing) does preserve it, so it's giving reliable 👑
the authors also point out that the proposed BGP-based approach for C-multicast routing is a good fit with both the segmented and non-segmented inter-AS approaches 🍓 meanwhile, PIM-based C-multicast routing is usable with segmented inter-AS tunnels, but the inter-AS scalability advantage gets lost, since PEs in an AS will see the C-multicast routing activity of all the other PEs of all the other ASes. rip scalability 🙈
3.3.5. Security and Robustness 🔐
BGP supports MD5 authentication of its peers for extra security 🔒 which could directly benefit multicast VPN customer multicast routing, whether it's intra-AS or inter-AS. with a PIM-based approach, though, no mechanism giving a comparable level of security to authenticate communications between remote PEs has been fully described yet [RFC5796] 😤 and even then, it would take a lot of extra operations for the provider to be usable in a multicast VPN context. not a vibe 💀
the robustness of the infrastructure is key, especially the existing infrastructure providing unicast VPN connectivity 🫶 the C-multicast routing function, especially under load, is going to compete with the unicast routing infrastructure. with the PIM-based approaches, unicast and multicast VPN routing are only expected to compete in the PE, for control plane processing resources. with the BGP-based approach, they'll compete on the PE for processing resources AND in the route reflectors (supposing they're used for MVPN routing) 🌀 in both cases, mechanisms will be needed to arbitrate resources (e.g., processing priorities) ⚖️ for PIM-based procedures, that arbitration happens between the different control plane routing instances in the PE. for the BGP-based approach, it's likely to require using distinct BGP sessions for multicast and unicast (e.g., through dedicated MVPN BGP route reflectors, or a distinct session with an existing route reflector) 🎀
multicast routing is dynamic by nature, and multicast VPN routing has to follow the VPN customers' multicast routing events 🦋 you can compare the different approaches on how they're expected to behave when multicast routing in the VPNs is under intense activity. the scalability of each approach under that kind of load is detailed in Appendix A.2 ✨ and the fourth approach (BGP-based), used together with the Route Target Constraint mechanisms [RFC4684], is the only one whose cost for join/leave operations is independent of the number of PEs in the VPN (with one exception, detailed in Appendix A.2) and whose state maintenance isn't concentrated on the upstream PE 👑
on the other hand 🙃 the BGP-based approach is likely to slow down when the load is greater than the available processing resources (because of possibly congested TCP sockets). the PIM-based approaches would instead react to that load by dropping messages, with failure recovery coming from message refreshes 😬 so the BGP-based approach could degrade join/leave latency in a way that's typically spread evenly across all the multicast streams being joined in that period, while the PIM-based approach could increase join/leave latency for some random streams by a multiple of the time between refreshes (e.g., tens of seconds). and in some states, the adjacency may time out, which would disrupt multicast streams 🥺
the behavior of the PIM-based approach under that kind of load is also harder to predict, because the performance of the "Join suppression" mechanism (an important mechanism for this approach to scale) will itself get impeded by delays in Join processing 🌪️ for these reasons, the BGP-based approach would be able to provide a smoother degradation and more predictable behavior under a highly dynamic load ✨
in fact, both an "evenly spread degradation" and an "unevenly spread larger degradation" can be problematic 😵💫 what really matters is the ability for the VPN backbone operator to (a) limit the amount of multicast routing activity that can be triggered by a multicast VPN customer, and (b) provide the best possible independence between distinct VPNs 💜 both of these can be addressed through local implementation improvements, and both the BGP-based and PIM-based approaches could be engineered to provide (a) and (b). it can be noted though that the BGP approach proposes ways to dampen C-multicast route withdrawals and/or advertisements, so it already describes a way to provide (a), while nothing comparable has been described yet for the PIM-based approaches (even though it doesn't appear difficult). the PIM-based approaches rely on a per-VPN data plane to carry the MVPN control plane, so they may benefit from this first level of separation to solve (b) 🧁
3.3.6. C-Multicast VPN Join Latency ⏱️
okay so Section 5.1.3 of [RFC4834] says the "group join delay [...] is also considered one important QoS parameter. It is thus RECOMMENDED that a multicast VPN solution be designed appropriately in this regard". 🫡 in a multicast VPN context, the "group join delay" we care about is the time between a CE sending a PIM Join to its PE and the first packet of the corresponding multicast stream being received by the CE ⏳
the C-multicast routing procedures will only impact the group join latency of a given multicast stream for the first receiver that's located across the provider backbone from the multicast source-connected PE (or for the first <n> receivers in the specific case where a specific Upstream Multicast Hop selection algorithm is used, which allows <n> distinct PEs to be selected as the Upstream Multicast Hop by distinct downstream PEs) 🦩
the different approaches seem to have different characteristics for how they impact join latency 🫧
- the PIM-based approaches minimize the number of control plane processing hops between a new receiver-connected PE and the source-connected PE. because they're datagram-based, they introduce minimal delay, so their join latency could be as good as possible, depending on implementation efficiency 🚀
- under degraded conditions (packet loss, congestion, or high control plane load), the PIM-based approach may impact latency for a given multicast stream in an all-or-nothing manner: if a C-multicast routing PIM Join packet is lost, latency can reach a high time (a multiple of the periodicity of PIM Join refreshes) 😱
- the BGP-based approach uses TCP exchanges, which may add some delay depending on BGP and TCP implementation. under degraded conditions (such as packet loss, congestion, or high control plane load), it would typically result in a comparably lower increase of latency, spread more evenly across the streams 🌊
- as shown in Appendix A, the BGP-based approach is particular in that it removes load from all the PEs (without putting that load on the upstream PE for a stream). this reduction in background load can bring improved performance when a PE acts as the upstream PE for a stream, and so it benefits join latency 💖
this qualitative comparison shows that the BGP-based approach is designed for a smoother degradation of latency under degraded conditions such as packet loss, congestion, or high control plane load 🌈 on the other hand, the PIM-based approaches seem structurally able to reach the shorter "best-case" group join latency (especially compared to a deployment of the BGP-based approach where route reflectors are used) 🏎️
doing a quantitative comparison of latencies isn't possible without referring to specific implementations and benchmarking procedures, and it could expose different conclusions 📊 this is especially true for best-case group join latency, which is expected to vary with PIM and BGP implementations. we can also note that improving a BGP implementation for reduced latency of route processing wouldn't only benefit multicast VPN group join latency, it would benefit the whole BGP-based routing too. so the need for good BGP/RR performance isn't specific to multicast VPN routing 🧠
last, C-multicast join latency will be impacted by the overall load put on the control plane, so the scalability of the C-multicast routing approach has to be taken into account 📈 as explained in Section 3.3.1 and Appendix A, the BGP-based approach will provide the best scalability with an increased number of PEs per VPN, which benefits group join latency in those higher-scale scenarios 🎉
3.3.7. Conclusion on C-Multicast Routing 💅
okay so bestie, the first and fourth approaches are the two real contenders for C-multicast routing 🔥 🫶. From a theory standpoint, comparing them shows some advantages, plus some possible drawbacks in the fourth approach 🤔. From a practical standpoint it's harder, ngl 😭. The fourth approach only has reduced deployment and implementation info so far, so the first approach gets the edge, because it's been run in multiple deployments and proven operationally viable 💖.
Also, the first mechanism (full per-MVPN PIM peering across an MI-PMSI) is the one [RFC6037] uses 🦋. That means it's already deployed and running in MVPNs today 🌈. The fourth approach may or may not end up being the preferred pick for any given deployment. But since the first approach has been around for a while, supporting it will still be super helpful for an eventual migration away from a deployment that uses something close to the first approach 🧚.
So right now, implementations are recommended to support both the fourth (BGP-based) and the first (full per-MVPN PIM peering) mechanisms ✨. Honestly, the fourth approach needs more real-world experience before anyone can define some best practices 🙈. Until then, supporting both lets a service provider pick between the first and fourth mechanisms without vendor implementation choices making the decision for them 👑. A service provider can also weigh the factors that matter for its own deployment context, which is very much a main character move 💜.
3.4. Encapsulation Techniques for P-Multicast Trees 🎀
Quick heads up: the authors aren't making any restricting recommendations in this section 🙌. Whether a specific provider core data plane technology is right depends on tons of factors, like the provider's currently deployed unicast data plane, and a lot of those are specific to each provider 🌸.
That said, implementations should not unreasonably restrict which data plane technology can be used, and they should not force the same technology on different VPNs attached to a single PE 🍓. Initial implementations might only support a smaller set of encapsulation techniques and data plane technologies, which is fine 💅. It just shouldn't become a blocker that stops future support for other encapsulations, data plane technologies, or interoperability ✨.
Section 5.2.4.1 of [RFC4834] says, and I'm quoting it exactly because it's important 💖:
"In a multicast VPN solution extending a unicast layer 3 PPVPN solution, consistency in the tunneling technology has to be favored: such a solution SHOULD allow the use of the same tunneling technology for multicast as for unicast. Deployment consistency, ease of operation, and potential migrations are the main motivations behind this requirement".
Current unicast VPN deployments use a mix of LDP, RSVP-TE, and GRE/IP-Multicast to encapsulate customer packets for transport across the provider core 🌈. So that the same encapsulations can be used for unicast and multicast VPN traffic, multicast VPN standards should recommend that implementations support multicast VPNs and all the P2MP variants of the encapsulations and signaling protocols they support for unicast, as long as a multipoint extension is defined for them 🦋. Examples are mLDP, P2MP RSVP-TE, and GRE/IP-multicast 🧚.
All three of those encapsulation techniques can build P2MP multicast P-tunnels 🔥. On top of that, mLDP and GRE/IP-ASM-Multicast implementations may also build MP2MP multicast P-tunnels 💜. MP2MP P-tunnels can be a scaling win for the provider, because only a single MP2MP P-tunnel needs to be deployed per VPN. That cuts the multicast state P routers have to maintain by about an order of magnitude 🤯. The tradeoff is bandwidth: sites without multicast receivers for streams sourced behind a given PE group will still get packets from those streams, so the VPN core isn't using bandwidth optimally 😭. One more thing to keep in mind is that MP2MP P-tunnels need extra configuration so every PE uses the same P-tunnel identifier or multicast ASM group address 🎀. Some auto-configuration might be possible for MP2MP P-tunnels, but the auto-discovery procedures don't currently support it 🙈. (C-multicast routing schemes not covered in [RFC6513] could show off different advantages of MP2MP multicast P-tunnels, but that's out of scope for this document 🌸.)
MVPN services can also run over a unicast VPN core using ingress PE replication 🍓. With this, the ingress PE copies multicast traffic over the P2P tunnels already used for unicast traffic ✨. It's nice because the provider doesn't have to change its existing P routers' protocol support, and it doesn't need multicast-specific state on those P routers to deploy a multicast VPN service 💖. But the bandwidth usage is obviously non-optimal, so it's unlikely to be the long-term solution providers pick 😭. It can still be useful during some migration scenarios, or when a provider thinks its multicast traffic is too low to justify multicast-specific support in its VPN core 🫶.
Every proposed control plane and data plane approach can aggregate multicast groups within a VPN and across different multicast VPNs, which can reduce the state P routers have to hold 👑. The across-VPN aggregation, though, needs support for upstream-assigned labels on the PEs 🧚. Supporting upstream-assigned labels might mean changing the PEs' data plane processing 🔥. So any provider thinking about aggregate PMSI tunnels should check this against the specific platforms it has deployed 💅✨.
3.5. Inter-AS Deployments Options ✨
Okay bestie, so there are a few different situations where you end up needing inter-AS multicast VPNs 🌸 Here are the main ones:
- A service provider may have a large network that it has segmented into a number of ASes.
- A service provider's multicast VPN may consist of a number of ASes because of acquisitions and mergers with other service providers 💅
- A service provider may wish to interconnect its multicast VPN platform with that of another service provider 🦋
Scenario 1 is the "simplest" one, honestly 🧚 The network is fully managed by one service provider under one strategy, so it's probably using a consistent set of technologies across every AS.
Scenario 2 is a bit messier 😭 Each AS may have picked its strategy and technology for different reasons because of its own history, and the service provider may not have unified those choices (or may be unwilling to unify them). Not a vibe, but it's real.
Scenario 3 is the most complex, no cap 💀 It has all the drama of scenario 2, plus the ASes are run by different service providers, so they may be subject to a different trust model than in the other scenarios. Trust issues, in networking form 🙈
Section 5.2.6 of [RFC4834] says that "a solution MUST support inter-AS multicast VPNs, and SHOULD support inter-provider multicast VPNs", that "considerations about co-existence with unicast inter-AS VPN Options A, B, and C (as described in Section 10 of [RFC4364]) are strongly encouraged", and that "a multicast VPN solution SHOULD provide inter-AS mechanisms requiring the least possible coordination between providers, and keep the need for detailed knowledge of providers' networks to a minimum -- all this being in comparison with corresponding unicast VPN options". So basically, keep it simple and don't make everyone learn each other's whole network 🫶
Section 8 of [RFC6513] takes on these requirements and proposes two approaches for MVPN inter-AS deployments ✨
- Non-segmented inter-AS tunnels, where the multicast tunnels are end-to-end across ASes. So even though the PEs belonging to a given MVPN may be in different ASes, the ASBRs play no special role and just function as P routers (described in Section 8.1).
- Segmented inter-AS tunnels, where each AS builds its own separate multicast tunnels, and those get "stitched" together by the ASBRs (described in Section 8.2).
(Quick aside 🎀: an inter-AS deployment can alternatively rely on Option A, the so-called "back-to-back" VRFs. That option isn't covered in this section, because it can be used without any inter-AS-specific mechanism.)
Section 5.2.6 of [RFC4834] also says, "Within each service provider, the service provider SHOULD be able on its own to pick the most appropriate tunneling mechanism to carry (multicast) traffic among PEs (just like what is done today for unicast)." The segmented approach is the only one that can meet this requirement 👑
Now, the segmented inter-AS solution looks like it gives operators the most deployment flexibility 🌈 But the non-segmented one can simplify things in a limited number of scenarios. [RFC6037] only supports the non-segmented inter-AS solution, so the non-segmented one is likely to be useful to some operators for backward compatibility and during migration from [RFC6037] to [RFC6513] 🍓
Here's the comparison matrix between the "segmented inter-AS P-tunnels" and "non-segmented inter-AS P-tunnels" approaches. Bestie, take notes 📝
- Scalability for I-PMSIs: The "segmented inter-AS P-tunnels" approach is more scalable, because an ASBR can aggregate multiple intra-AS P-tunnels used for I-PMSI within its own AS into one inter-AS P-tunnel to be used by other ASes. Note that the I-PMSI scalability improvement brought by the "segmented inter-AS P-tunnels" approach is higher when segmented P-tunnels have a granularity of source AS (see item below).
- Scalability for S-PMSIs: The "segmented inter-AS P-tunnels" approach, when used with the BGP-based C-multicast routing approach, gives flexibility in how the bandwidth/state trade-off is handled, which helps with scalability 🔥 In that case, the trade-off made for a said
(C-S,C-G)in a downstream AS can be made more in favor of scalability than the trade-off made by the neighbor upstream AS, thanks to the ability to aggregate one or more S-PMSIs of the upstream AS in one I-PMSI tunnel in a downstream AS. - Configuration at ASBRs: Depending on whether segmented P-tunnels have a granularity of source ASBR or source AS, the "segmented inter-AS P-tunnels" approach would require, respectively, the same or additional configuration on ASBRs as the "non-segmented inter-AS P-tunnels" approach.
- Independence of tunneling technology from one AS to another: The "segmented inter-AS P-tunnels" approach provides this, and the "non-segmented inter-AS P-tunnels" approach does not 🦋
- Facilitated coexistence with, and migration from, existing deployments and lighter engineering in some scenarios: The "non-segmented inter-AS P-tunnels" approach provides this, and the "segmented inter-AS P-tunnels" approach does not 🌸
Whether segmented or non-segmented inter-AS tunnels fit a given deployment or inter-provider interconnect depends on a bunch of factors specific to each service provider 💜 But given the different elements we just went over, the authors recommend that all implementations should support the segmented inter-AS model. They also recommend that implementations should consider supporting the non-segmented inter-AS model, to facilitate coexistence with and migration from existing deployments, and to provide lighter engineering in a restricted set of scenarios. It's recognized that initial implementations may only support one or the other, and that's okay ✨
3.6. BIDIR-PIM Support 🌈
Okay, BIDIR-PIM is iconic 👑 Its packet-forwarding rules are improved over PIM-SM, allowing traffic to be passed up the shared tree toward the RPA. To avoid multicast packet looping, BIDIR-PIM uses a mechanism called the designated forwarder (DF) election, which establishes a loop-free tree rooted at the RPA. This ensures that only one copy of every packet is sent to an RPA, even if there are parallel equal cost paths to the RPA. To avoid loops, the DF election process enforces a consistent view of the DF on all routers on the network segment. During periods of ambiguity or routing convergence, traffic forwarding is suspended 🫶
In the context of a multicast VPN solution, a BIDIR-PIM support solution has to keep this property of avoiding packet loops too 🔥 That includes the case where multicast VRFs in a given MVPN don't have a consistent view of the routing to C-RPL/C-RPA (Customer-RPL/Customer-RPA, i.e., RPL/RPA of a Bidir customer PIM instance). Main character energy, but make it loop-free 💅
Section 11 of the current MVPN specification [RFC6513] defines three methods to support BIDIR-PIM, which are RECOMMENDED in [RFC4834]:
- Standard DF election procedure over an MI-PMSI
- VPN Backbone as the RPL (Section 11.1)
- Partitioned Sets of PEs (Section 11.2)
Method (1) fits naturally with deployments using "Full per-MVPN PIM peering across an MI-PMSI" for C-multicast routing. But, as [RFC6513], Section 11 points out, the DF election may not work well in an MVPN environment, so an alternative to DF election would be desirable 🙈
Methods (2) and (3) have the advantage that they don't require running the DF election procedure among PEs ✨
Method (2) takes advantage of the fact that in BIDIR-PIM, running the DF election procedure isn't needed on the RPL. This makes it simpler to implement, especially where BGP-based C-multicast routing is used. The drawback is that it puts constraints on how BIDIR-PIM is deployed, which may not always match what MVPN customers need 🥺
Method (3) treats an MVPN as a collection of sets of multicast VRFs. All PEs in a set have the same reachability information toward C-RPA, but it's different from the PEs in the other sets. So with this method, C-Bidir packet loops in an MVPN are resolved by partitioning the VPN into disjoint sets of VRFs, each with a distinct view of the converged network 🦋 The partitioning approach requires either upstream-assigned MPLS labels (to denote the partition) or a unique MP2MP LSP per partition. The former is based on PE Distinguisher Labels that have to be distributed using auto-discovery BGP routes, and handling them requires support for upstream assigned labels and context label lookups [RFC5331]. The latter, using an MP2MP LSP per partition, doesn't have these constraints, but it's restricted to P-tunnel types that support MP2MP connectivity (such as mLDP [RFC6388]).
This approach to C-Bidir can work with PIM-based or BGP-based C-multicast routing procedures. It's also generic, in the sense that it doesn't impose any requirements on the BIDIR-PIM service offering 🌸
Given all of the above, method (3) "Partitioned Sets of PEs" is the RECOMMENDED approach 👑
If method (3) isn't applicable (because of a lack of support for upstream assigned labels or for a P-tunnel type providing MP2MP connectivity), then method (1) "Standard DF election procedure over an MI-PMSI" and method (2) "VPN Backbone as the RPL" are RECOMMENDED as interim solutions. Method (1) has the advantage over method (2) of not putting constraints on how BIDIR-PIM is deployed. Its drawbacks are that it's only applicable when PIM-based C-multicast is used, and that it may not work well in an MVPN environment 💜
4. Co-Located RPs ✨
okay so RFC 4834 Section 5.1.10.1 says, "In the case of PIM-SM in ASM mode, engineering of the RP function requires the deployment of specific protocols and associated configurations. A service provider may offer to manage customers' multicast protocol operation on their behalf. This implies that it is necessary to consider cases where a customer's RPs are outsourced (e.g., on PEs). Consequently, a VPN solution MAY support the hosting of the RP function in a VR or VRF". 🎀
BUT (and this is the tea 🍵) a lot of customers already have multicast running in their own networks with their own internal RPs, and they're honestly kind of reluctant to hand control of those RPs over to their provider and go co-located. 🙈 Also, putting an RP on a PE means turning on Multicast Source Discovery Protocol (MSDP) or processing PIM Registers on the PE, and locking down PE routers for that takes special care and extra work. It will probably lean on specific features the routers themselves have to provide. 💅
So whether co-located RPs even make sense for a given MVPN depends on a bunch of stuff specific to each customer and each provider. ✨
Here's the recommendation, bestie: implementations should support a co-located RP model 🫶, but supporting it shouldn't force anyone into using it. Implementations MUST support deployments where activating a PIM RP function (PIM Register processing and RP-specific PIM procedures) or a VRF MSDP instance is not required on any PE router, and where all the RPs live inside the customers' networks or CEs. Main character energy for customer-owned RPs 👑
5. Avoiding Duplicates
It is recommended that implementations support the procedures in Section 9.1.1 of [RFC6513], "Discarding Packets from Wrong PE", which lets you fully avoid duplicates. No duplicate drama 🦋✨
6. Existing Deployments
Okay this part is so practical 🧚. Some of the suggestions in this document can be used to gradually modify deployments that already exist, without breaking them and without messing up the consistency of the standardized solution. The trick is adding optional per-VRF configuration knobs that support modes of operation compatible with what's already deployed, while implementations that support the standard use the recommended approach. 🌸
If that's not easy to pull off, the recommended approach is a per-VRF configuration knob that allows incremental, per-VPN migration of the mechanisms a PE uses. The catch is that this kind of migration comes with some per-VPN interruption of service, so you'd schedule it for something like a maintenance window. 🕰️
"Live" migration, where multiple alternatives run concurrently for a given PE and a given VPN, is not a priority 🙅♀️. The implementation complexity would be a lot, honestly. BUT if there are cases where it could be done relatively simply, those mechanisms might help make migration easier to manage. 💖
7. Summary of Recommendations
Here's the recap of the conclusions on the mechanisms that make up the set of mandatory-to-implement mechanisms in the context of [RFC6513]. 📝
Note well that implementing the non-mandatory alternative mechanisms is not precluded. So you're totally free to do more! 🌈
The recommendations are:
- that BGP-based auto-discovery be the mandated solution for auto-discovery;
- that BGP be the mandated solution for S-PMSI switching signaling;
- that implementations support both the BGP-based and the full per-MVPN PIM peering solutions for PE-PE exchange of customer multicast routing until further operational experience is gained with both solutions;
- that implementations use the "Partitioned Sets of PEs" approach for BIDIR-PIM support;
- that implementations implement the P2MP variants of the P2P protocols that they already implement, such as mLDP, P2MP RSVP-TE, and GRE/IP-Multicast;
- that implementations support segmented inter-AS tunnels and consider supporting non-segmented inter-AS tunnels (in order to maintain backward compatibility and for migration);
- that implementations MUST support deployments when the activation of a PIM RP function (PIM Register processing and RP-specific PIM procedures) or VRF MSDP instance is not required on any PE router; and
- that implementations support the procedures described in Section 9.1.1 of [RFC6513].
8. Security Considerations
This document does not by itself raise any particular security considerations. 🔒✨ No drama here, periodt 💅
9. Acknowledgements
We would like to thank Adrian Farrel, Eric Rosen, Yakov Rekhter, and Maria Napierala for their feedback that helped shape this document. 🫶 Big thanks to them, they're iconic.
Additional credit is due to Maria Napierala for co-authoring Section 3.6 on BIDIR-PIM Support. 👑
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC6513] Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/
BGP IP VPNs", RFC 6513, February 2012.
[RFC6514] Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP
Encodings and Procedures for Multicast in MPLS/BGP IP
VPNs", RFC 6514, February 2012.10.2. Informative References
[MVPN] Aggarwal, R., "Base Specification for Multicast in BGP/
MPLS VPNs", Work in Progress, June 2004.
[PIM-PORT] Farinacci, D., Wijnands, I., Venaas, S., and M.
Napierala, "A Reliable Transport Mechanism for PIM", Work
in Progress, October 2011.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, February 2006.
[RFC4684] Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk,
R., Patel, K., and J. Guichard, "Constrained Route
Distribution for Border Gateway Protocol/MultiProtocol
Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual
Private Networks (VPNs)", RFC 4684, November 2006.
[RFC4834] Morin, T., Ed., "Requirements for Multicast in Layer 3
Provider-Provisioned Virtual Private Networks (PPVPNs)",
RFC 4834, April 2007.
[RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream
Label Assignment and Context-Specific Label Space",
RFC 5331, August 2008.
[RFC5796] Atwood, W., Islam, S., and M. Siami, "Authentication and
Confidentiality in Protocol Independent Multicast Sparse
Mode (PIM-SM) Link-Local Messages", RFC 5796, March 2010.
[RFC6037] Rosen, E., Cai, Y., and IJ. Wijnands, "Cisco Systems'
Solution for Multicast in BGP/MPLS IP VPNs", RFC 6037,
October 2010.
[RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas,
"Label Distribution Protocol Extensions for Point-to-
Multipoint and Multipoint-to-Multipoint Label Switched
Paths", RFC 6388, November 2011.Appendix A. Scalability of C-Multicast Routing Processing Load ✨
Okay so the main job of multicast routing is letting routers figure out when they should start or stop forwarding a given multicast stream on a given link. 🦋 In an MVPN context, that has to happen for every MVPN, so it gets its own name: "customer-multicast routing" or "C-multicast routing" 💅. Its job is to help PE routers decide when they should start or stop forwarding a multicast stream's traffic toward remote PEs, over some PMSI tunnel.
When a PE gets a Join message, it knows it should be sending traffic for that multicast group in that MVPN 🌸. But a Prune message from a remote PE isn't enough on its own for the PE to know it can stop forwarding. It also has to make sure no other PEs still have receivers for that traffic 😭.
There are lots of ways to design the "C-multicast routing" building block, and they differ in (among other things) how a PE decides it can stop forwarding a given multicast stream toward other PEs 💖:
PIM LAN Procedures, by default 🎀
By default, when PIM LAN procedures are used and a PE on a LAN prunes itself from a multicast tree, all the other PEs on that LAN check their own state to know whether they're on the tree. If they are, they send a PIM Join message on that LAN to override the Prune ✨. So for every PIM Prune message, all the PE routers on the LAN work together to let the upstream PE answer the "did the last receiver leave?" question 🙈.
BGP-based C-multicast routing 👑
When BGP-based procedures are used for C-multicast routing and no BGP route reflector is involved, the upstream PE answers "did the last receiver leave?" by keeping an up-to-date list of the PEs joined to the tree. That way it can instantly know the answer whenever a PE leaves the tree 🍓.
But when a BGP route reflector is used (which is expected to be the recommended approach 💜), the job of keeping an updated list of PEs in a multicast tree belongs to the route reflector(s) 🧚. With BGP procedures, a route reflector that had previously advertised a C-multicast Source Tree Join route for a given (C-S,C-G) to other route reflectors will withdraw that route once none of its client PEs are advertising it anymore. Similarly, a route reflector that had advertised this route to its client PEs will withdraw it when its other client PEs and its route reflector peers are no longer advertising it either 🌈. In this setup, the route reflector(s) can be said to answer the "did the last receiver leave?" question ⭐.
The BGP route distribution can also use more than one route reflector. If there are multiple route reflectors, with PEs (as clients) spread among them, the "did the last receiver leave?" question is partly answered by each of those route reflectors 🫶.
The "last receiver leaves" question is part of the work the C-multicast routing building block has to handle, and the different approaches really do differ 💅. They can lead to different amounts of processing, and to that processing being spread differently across the functions. These differences are easier to estimate if we quantify the amount of message processing and state maintenance 🔥.
Even though the types of processing, messages, and states vary between approaches, we give a rough estimate here of PE load, in terms of the number of messages processed and the number of control plane states maintained 🦋. A "message processed" is a message being parsed, a lookup being done, and some action being taken (for example, updating a control plane or data plane state, or discarding the information in the message) 🌸. A "state maintained" is a multicast state kept in a PE's control plane memory, related to an interface or to a PE subscribing to a multicast stream. A state gets counted on a piece of equipment once for each protocol it appears in, so it counts two times when it's present both as a PIM state and as a BGP route 🧁. We don't compare data plane states on PE routers here, since those wouldn't vary between the options anyway 💖.
A.1. Scalability with an Increased Number of PEs 🌈
The following sections estimate the processing and state maintenance load as the number of PEs in a VPN grows 📈.
A.1.1. SSM Scalability ✨
The following subsections make this estimate for each proposed C-multicast routing approach, across different phases of this scenario 💅:
- One SSM multicast stream is considered.
- Only the intra-AS case is covered. (With segmented inter-AS tunnels and BGP-based C-multicast routing,
#mvpn_PEand#R_PErefer to the PEs of the MVPN in the AS, not to all PEs of the MVPN 🎀.) - The scenario goes like this:
- One PE joins the multicast stream (because a new receiver-connected site has sent a Join on the PE-CE link), followed by a number of additional PEs that also join the same stream, one after another. We evaluate the processing needed to add each PE 🦋.
- A period of time T passes with no PE joining or leaving (this is the baseline) ⭐.
- All PEs leave, one after another, until the last one leaves. We evaluate the processing needed for each PE's leave 😭.
- The parameters are:
#mvpn_PE: the number of PEs in the MVPN#R_PE: the number of PEs joining the multicast stream#RR: the number of route reflectorsT_PIM_r: the time between two refreshes of a PIM Join (default is 60s) 💜
The estimation unit is the "message.equipment" (or "m.e"): one "message.equipment" means "one equipment processing one message" 🔥. So "10 m.e" could be "10 equipments processing each one message", "5 messages each processed by 2 equipments", or "1 message processed by 10 equipment", and so on 🍓. For control plane state, the unit is "state.equipment" or "s.e" 🧚. That's because a message (or a state) can be processed (or maintained) by more than one node 🫶.
We distinguish three types of equipment: the upstream PE for the multicast stream in question, the RR (if there is one), and the other PEs (meaning the ones that aren't the upstream PE) 🌸.
The numbers and orders of magnitude in the tables of the following subsections are totals across all equipment of the same type, for each type of equipment, in the "m.e" and "s.e" units defined above 📊.
Also, keep in mind:
- For PIM, only Join and Prune messages are counted 🎀.
- The load from PIM Hellos can be computed separately and only depends on the number of PEs in the VPN 💅.
- Message processing for the PIM Assert mechanism is also left out, for simplicity 🙈.
- For BGP, all advertisements and withdrawals of C-multicast Source Tree Join routes are counted (Source-Active auto-discovery routes aren't used in an SSM context). Following the recommendation in Section 16 of [RFC6514], the case where the Route Target Constraint mechanism [RFC4684] is not used is not covered 🌈.
(Note that for all the C-multicast routing options, the procedures for setting up and maintaining a shortest path tree toward an SSM group's source are the same as those for setting up and maintaining a shortest path tree toward an RP or a non-SSM source. So the results in this section get reused in Appendix A.1.2 👑.)
A.1.1.1. PIM LAN Procedures, by Default 💖
upstream other PEs RR total across
PE (1) (total across (none) all
(#mvpn_PE-1 | equipments
PEs) | |
+------------+------------+---------------+----------+--------------+
| first PE | 1 m.e | #mvpn_PE-1 | / | #mvpn_PE m.e |
| joins | | m.e | | |
+------------+------------+---------------+----------+--------------+
| for *each* | 1 m.e | #mvpn_PE-1 | / | #mvpn_PE m.e |
| additional | | m.e | | |
| PE joining | | | | |
+------------+------------+---------------+----------+--------------+
| baseline | T/T_PIM_r | (T/T_PIM_r) . | / | (T/T_PIM_r) |
| processing | m.e | (#mvpn_PE-1) | | x #mvpn_PE |
| over a | | m.e | | m.e |
| period T | | | | |
+------------+------------+---------------+----------+--------------+
| for *each* | 2 m.e | 2(#mvpn_PE-1) | / | 2 x #mvpn_PE |
| PE leaving | | m.e | | m.e |
+------------+------------+---------------+----------+--------------+
| the last | 1 m.e | #mvpn_PE-1 | / | #mvpn_PE m.e |
| PE leaves | | m.e | | |
+------------+------------+---------------+----------+--------------+
| total for | #R_PE x 2 | (#mvpn_PE-1) | 0 | #mvpn_PE x ( |
| #R_PE PEs | + | x (#R_PE) x 2 | | 3 x #R_PE + |
| | T/T_PIM_r | + T/T_PIM_r) | | T/T_PIM_r ) |
| | m.e | . | | m.e |
| | | (#mvpn_PE-1) | | |
| | | m.e | | |
+------------+------------+---------------+----------+--------------+
| total | 1 s.e | #R_PE s.e | 0 | #R_PE+1 s.e |
| state | | | | |
| maintained | | | | |
+------------+------------+---------------+----------+--------------+
Messages Processing and State Maintenance - PIM LAN Procedures, by
DefaultWe're assuming here that the PIM Join suppression and Prune Override mechanisms work perfectly, meaning a Join or Prune message sent by a PE is instantly seen by the other PEs 🫶. Honestly, that's not strictly true 😅. Depending on network delays and timing, more messages could get exchanged in some cases, so the numbers in this table are a lower bound on the number of PIM messages exchanged ⭐.
A.1.1.2. BGP-Based C-Multicast Routing ✨
Okay so this analysis assumes BGP route reflectors (RRs) are being used, with no hierarchy of RRs 👑, and it also assumes Route Target Constraint mechanisms are in play 💅 (the ones from [RFC4684]).
Given all that, a message carrying a C-multicast route from a downstream PE has to be processed by the RRs that have that PE as a client 🫶. Because of Route Target Constraint [RFC4684], those RRs only pass the message along to the RRs that have the upstream PE as a client. Nobody else gets it. None of the other RRs and none of the other PEs 🙈. So for a message tied to a given MVPN, the number of RRs that have to process it depends only on how many RRs keep C-multicast routes for that MVPN and have either the receiver-connected PE or the source-connected PE as a client. It's independent of the total number of RRs or PEs. Iconic, right? 🔥
In practice, a PE is a client of just 2 RRs (an RR cluster typically has 2 RRs, for redundancy 🎀). So the message ends up processed by at most 4 RRs, or 2 if the downstream PE and the upstream PE are clients of the same RRs. Basically, at most 4 RRs process any given message. Since RRs in different RR clusters have a full Internal BGP (iBGP) mesh among themselves, each RR in the cluster with the upstream PE receives the message from each RR in the cluster with the downstream PE. With 2 RRs per cluster, the total number of messages processed by all the RRs is 6 🧚.
Also, once there's a receiver-connected PE in each RR cluster, the number of RRs processing a C-multicast route tends quickly toward 2 💜 (this counts the fact that a PE peering to RRs gets made redundant).
Here's the whole breakdown in one table, bestie:
+------------+----------+--------------+-----------+----------------+
| | upstream | other PEs | RRs (#RR) | total across |
| | PE (1) | (total | | all equipments |
| | | across | | |
| | | (#mvpn_PE-1) | | |
| | | PEs) | | |
+------------+----------+--------------+-----------+----------------+
| first PE | 2 m.e | 2 m.e | 6 m.e | 10 m.e |
| joins | | | | |
+------------+----------+--------------+-----------+----------------+
| for *each* | between | 2 m.e | (at most) | (at most) 10 |
| additional | 0 and 2 | | 6 m.e | m.e tending |
| PE joining | m.e | | tending | toward 4 m.e |
| | | | toward 2 | |
| | | | m.e | |
+------------+----------+--------------+-----------+----------------+
| baseline | 0 | 0 | 0 | 0 |
| processing | | | | |
| over a | | | | |
| period T | | | | |
+------------+----------+--------------+-----------+----------------+
| for *each* | between | 2 m.e | (at most) | (at most) 10 |
| PE leaving | 0 and 2 | | 6 m.e | m.e tending |
| | m.e | | tending | toward 4 m.e |
| | | | toward 2 | |
+------------+----------+--------------+-----------+----------------+
| the last | 2 m.e | 2 m.e | 6 m.e | 10 m.e |
| PE leaves | | | | |
+------------+----------+--------------+-----------+----------------+
| total for | at most | #R_PE x 4 | (at most) | at most 10 x |
| #R_PE PEs | 2 x #RRs | m.e | 6 x #R_PE | #R_PE + 2 x |
| | m.e (see | | m.e | #RRs m.e |
| | note | | (tending | (tending |
| | below) | | toward 2 | toward 6 x |
| | | | x #R_PE | #R_PE + #RRs |
| | | | m.e) | m.e ) |
+------------+----------+--------------+-----------+----------------+
| total | 4 s.e | 2 x #R_PE | approx. 2 | approx. 4 |
| state | | s.e | #R_PE + | #R_PE + #RRx |
| maintained | | | #RR x | #clusters + 4 |
| | | | #clusters | m.e |
| | | | s.e | |
+------------+----------+--------------+-----------+----------------+
Message Processing and State Maintenance - BGP-Based ProceduresNote on the total of m.e on the upstream PE: 🍓
- There are as many "message.equipment"s on the upstream PE as the number of times the RRs of the upstream PE's cluster need to re-advertise the C-multicast (C-S,C-G) route. That re-advertisement isn't useful for the upstream PE, because its behavior for a said (VPN associated to the Route Target, C-S,C-G) doesn't depend on the exact attributes on the route (other than the Route Target, obviously). It still happens sometimes, though, because of how BGP processes these routes. A BGP peer may re-advertise a route when its current best path changes for the said NLRI, if the set of attributes to advertise also changes.
- Let's look at the attributes and when they can trigger a re-advertisement of a C-multicast route 👀:
- next-hop and originator-id: A new PE joining doesn't automatically mean a C-multicast route needs re-advertising. The RR aggregates C-multicast routes with the same NLRI received from PEs in its own cluster (Section 11.4 of [RFC6514]), and the RR rewrites these attributes. However, the advertisements from different RRs peering with the RRs in the upstream PE's cluster can lead to updates of these values.
- cluster-list: This value only varies between clusters. Changes to it don't "follow" PE advertisements, and only advertisements from different RRs may lead to updates of it.
- local-pref: This value is determined locally. That's true both for routes advertised by each PE (which could all be configured with the same value) and for the route an RR creates by aggregating the routes with the same NLRI advertised by the PEs in its cluster (the RRs could also be configured with a local pref independent of the local_pref of the routes advertised to them). So this attribute can cause a need to re-advertise a C-multicast route.
- Other BGP attributes: These don't have a particular reason to be set for C-multicast routes in intra-AS. If they were, an RR (or, for attributes relevant to inter-AS, an ASBR) would overwrite these values when aggregating the routes anyway.
- So, for a given C-multicast Source Tree Join (S,G) NLRI, what might make an RR re-advertise the route with different attributes to the upstream PE is an RR from another cluster advertising a route that's better than its current best route. That happens because of values specific to that RR (next-hop, originator-id, cluster-list), not because of anything about the PEs behind it 🙃. If we consider our (#R_PE -1) PEs joining a said (C-S,C-G), one after another following the first PE joining, some of these events may lead to a re-advertisement to the upstream PE. The number of times this can happen is at worst the number of RRs in clusters that have receivers, plus one for the possible advertisement of the same route by a PE in the local cluster.
- Since we're looking at scalability with an increased number of PEs in this section, we need to consider that all clusters may have a client PE with a receiver. We also need to consider that the two RRs of the upstream PE's cluster may need to re-advertise the route. With this in mind, 2x#RRs is an upper bound on the number of updates RRs make to the upstream PE for the considered C-multicast route 🔥
A.1.1.3. Side-by-Side Orders of Magnitude Comparison ✨
okay bestie, this section picks up right where the last one left off 🧚 It looks at what happens to the orders of magnitude when a VPN keeps adding PEs 💅
+------------+--------------------------------+---------------------+
| | PIM LAN Procedures | BGP-based |
+------------+--------------------------------+---------------------+
| first PE | O(#mvpn_PE) | O(1) |
| joins (in | | |
| m.e) | | |
+------------+--------------------------------+---------------------+
| for *each* | O(#mvpn_PE) | O(1) |
| additional | | |
| PE joining | | |
| (in m.e) | | |
+------------+--------------------------------+---------------------+
| baseline | (T/T_PIM_r) x O(#mvpn_PE) | 0 |
| processing | | |
| over a | | |
| period T | | |
| (in m.e) | | |
+------------+--------------------------------+---------------------+
| for *each* | O(#mvpn_PE) | O(1) |
| PE leaving | | |
| (in m.e) | | |
+------------+--------------------------------+---------------------+
| the last | O(#mvpn_PE) | O(1) |
| PE leaves | | |
| (in m.e) | | |
+------------+--------------------------------+---------------------+
| total for | O(#mvpn_PE x #R_PE) + | O(#R_PE) |
| #R_PE PEs | O(#mvpn_PE x T/T_PIM_r) | |
| (in m.e) | | |
+------------+--------------------------------+---------------------+
| states (in | O(#R_PE) | O(#R_PE) |
| s.e) | | |
| notes | (processing and state | (processing and |
| | maintenance are essentially | state maintenance |
| | done by, and spread amongst, | is essentially done |
| | the PEs of the MVPN; | by, and spread |
| | non-upstream PEs have | amongst, the RRs) |
| | processing to do) | |
+------------+--------------------------------+---------------------+
Comparison of Orders of Magnitude for Message Processing and State
Maintenance (Totals across All Equipments)So the big takeaways are these 👀
- ngl, in the PIM-based approach, every message gets processed by all the PEs, even the ones that are neither upstream nor downstream for it 💀 That means the total is
O(#mvpn_PE x #R_PE), which works out toO(#mvpn_PE ^ 2)if the proportion of receiver PEs stays constant as the number of PEs grows. The Join refreshes add a linear factor. It doesn't change the order of magnitude, but it can get seriously big for long-lived streams 🥹 - The BGP-based approach needs message processing in
O(#R_PE), which is lower than the PIM-based one 🔥 And that amount doesn't depend on how long the streams last. - State maintenance is the same order of magnitude for every approach,
O(#R_PE), but it's spread out differently 🦋- The PIM-based approach spreads the state all over and keeps it minimal, with one state per PE 🌸
- The BGP-based procedures put all the state on the set of route reflectors 🎀
A.1.2. ASM Scalability
okay so the conclusions from Appendix A.1.1 get reused here, for the parts that overlap when you're setting up and maintaining state for a source tree or a shared tree 🫶
When PIM-SM runs in a VPN and an ASM multicast group gets joined by some PEs (#R_PEs), with some sources sending toward that group address, here's the tea 🍓
PEs will usually have to maintain one shared tree, plus one source tree for each source sending toward G. Each of those trees brings a similar amount of processing and state maintenance to what's described in Appendix A.1.1, with the same differences in order of magnitude between the approaches once the number of PEs gets high 💅
There's one exception. If, for a given group in a VPN among the PIM instances in the customer routers and VRFs, nobody ever switches to the shortest path tree (SPT) (SwitchToSptDesired always false), then the processing and state maintenance load is just what's needed to maintain the shared tree 🌈 It's important to note that this scenario depends on customer policy 🙈 To compare the load in that case between the PIM-based approaches and the BGP-based approach configured to use inter-site shared trees, you can use the scenario from Appendix A.1.1, with #R_PEs joining a (C-*, C-G) ASM group instead of an SSM group. The same differences in order of magnitude still hold ✨ For the BGP-based approach used without inter-site shared trees, there's extra load to account for, because each PE has to join the source tree to each source in order to build the C-PIM shared tree 😭 That extra load (total across all equipments) is proportional to #R_PEs and the number of sources. The order of magnitude as the number of PEs increases stays the same, and so do the differences in order of magnitude 💜
Beyond maintaining the trees, PEs also have to handle some processing and state maintenance for individual sources sending to a multicast group. How those procedures and behaviors work can vary a lot, depending on which C-multicast routing protocol is used, how it's configured, how the multicast source discovery mechanism is used in the customer VPN, and which SwitchToSptDesired policy is in play. Still, here's what can be observed 👑
- When BGP-based C-multicast routing is used:
- Each PE might have to process and maintain a BGP Source-Active auto-discovery route for some or all of the sources of an ASM group. Usually there's just one Source-Active auto-discovery route, but the count can depend on the number of upstream PEs in these cases: when inter-site shared trees are used and more than one PE is used as the upstream PE for SPT (
C-S,C-G) trees at the same time, and when inter-site shared trees are used and there are multiple PEs that could be upstream for this (S,G) 🧚 - This leads to message processing and state maintenance (total across all the equipments) that scales linearly with the number of PEs in the VPN (
#mvpn_PE) for each source. It doesn't depend on how many PEs are joined to the group 🦋 - Whether this happens for all (
S,G) of an ASM group or only some of them depends on whether inter-site shared trees are used, on theSwitchToSptDesiredpolicy in the PIM instances in the customer routers and VRFs, and on where the sources and RPs sit relative to each other. It happens in parallel with maintaining the shared and/or source trees, or at the first join of a PE on a source tree 💖
- Each PE might have to process and maintain a BGP Source-Active auto-discovery route for some or all of the sources of an ASM group. Usually there's just one Source-Active auto-discovery route, but the count can depend on the number of upstream PEs in these cases: when inter-site shared trees are used and more than one PE is used as the upstream PE for SPT (
- When PIM-based C-multicast routing is used, depending on the
SwitchToSptDesiredpolicy in the PIM instances in the customer routers and VRFs, and on where the sources and RPs are relative to each other, there are:- Possible control plane state transitions triggered by the reception of (
S,G) packets. Those events would make every PE joined toGdo some processing 🔥 - Possible PIM Assert messages specific to (
S,G). Each PIM Assert message would make every PE in the VPN do some message processing 🙈
- Possible control plane state transitions triggered by the reception of (
So, all in all, the extra processing that can happen for each individual source sending to the group, on top of maintaining the source and shared trees, doesn't change the order of magnitude we already identified above 🎀
A.2. Cost of PEs Leaving and Joining 💅
okay so the message-processing numbers in Appendix A.1.1 come from one specific scenario: every PE with receivers joins once and leaves once 🫶. to figure out what happens with other patterns of PEs coming and going, we just compare the cost of two things: "a new PE joining" and "a PE leaving" ✨.
For the "PIM LAN Procedure" approach, if there's a single SSM or SPT tree, the total processing across all nodes grows linearly with the number of PEs in the VPN whenever a PE joins that tree 📈.
And for the "BGP-based" approach:
- Single SSM tree: the total processing across all nodes doesn't depend on how many PEs there are, whether a new PE is joining or a PE is leaving 🦋. It does depend on how the route reflectors are meshed, but that dependency isn't linear.
- ASM group with SPT trees: BGP picks up extra processing from possible Source-Active auto-discovery routes 👀.
- When BGP-based C-multicast routing uses inter-site shared trees, the first PE to join (and the last PE to leave) a given SPT triggers processing of the Source-Active auto-discovery routes, and that cost is linear in the number of PEs in the VPN. PEs joining after the first (and PEs leaving that aren't the last) cause no processing from advertising or withdrawing Source-Active auto-discovery routes 🌸.
- When BGP-based C-multicast routing is used without inter-site shared trees, processing Source-Active auto-discovery routes for an (S,G) happens no matter which PEs are joining or leaving the SPT for (S,G) 🙈.
Now say a new PE has to join a shared tree for an ASM group G. Here's the tea ☕:
- The processing from the PE joining the shared tree itself is the same as setting up an SSM tree, like we covered before. (Fun fact: this doesn't happen with BGP-based C-multicast routing that doesn't use inter-site shared trees 💀.)
- For each source the PE joins the SPT for, the cost is the same as one SPT tree, as described before.
- The conditions for a PE to join the SPT for a given (C-S,C-G) are the same for the BGP-based approach with inter-site shared trees and the PIM-based approach. They depend solely on the SwitchToSptDesired policy in the PIM instances in the customer routers at the sites connected to the PE and/or in the VRF 🎀.
- The conditions for a PE to join the SPT for a given (C-S,C-G) differ between the BGP-based approach without inter-site shared trees and the PIM-based approach.
- The SPT for a given (S,G) can be joined by the PE in these cases:
- as soon as one router, or the VPN VRF on the PE, has SwitchToSptDesired(S,G) being true
- when BGP-based routing is used and configured not to use inter-site shared trees
- Said differently, the PE will only skip joining the SPT for (S,G) when all routers in the sites of the VPN connected to the PE, or the VPN VRF itself, will never have SwitchToSptDesired(S,G) being true. The extra condition is that inter-site shared trees are used when BGP-based C-multicast routing is used.
So when one PE joins a group G that n sources are sending to, here's how the cost (total processing across all equipment) depends on the number of PEs 🧚:
- In the general case (where any router in the site of the VPN connected to the PE, or the VRF itself, may have SwitchToSptDesired(S,G) being true):
- For the "PIM LAN Procedure" approach, the cost is linear in the number of PEs in the VPN and linear in the number of sources.
- For the "BGP-based" approach, the cost is linear in the number of sources. For the sub-case using inter-site shared trees, it also depends on the number of PEs in the VPN, but only if the PE is the first to join the group or the SPT for some source sending to the group.
- Otherwise, assuming routers in the sites of the VPN connected to the PE, and the VPN VRF itself, will never have the policy function SwitchToSptDesired(S,G) being possibly true:
- For the PIM-based approach, the cost is linear in the number of PEs in the VPN, with no dependency on the number of sources.
- For the BGP-based approach with inter-site shared trees, the cost is linear in the number of RRs, with no dependency on the number of sources.
- For the BGP-based approach without inter-site shared trees, the cost is linear in the number of RRs and in the number of sources.
So the big takeaway: with the PIM-based approach, the overall cost across all equipment of any PE joining an ASM group G always depends on the number of PEs (same for a PE leaving), while the BGP-based approach has a cost that's independent of the number of PEs. The exception is the first PE joining the ASM group under BGP-based routing without inter-site shared trees. In that case, there is a dependency on the number of PEs 👑.
On the number of sources: without making any assumption about the SwitchToSptDesired policy on PIM routers and VRFs of a VPN, a PE joining an ASM group may induce a processing cost that's linear in the number of sources 😭. Outside that general case, if SwitchToSptDesired is always false on all PIM routers and VRFs of the VPN, then the PIM-based approach and the BGP-based approach with inter-site shared trees both process a number of messages that's independent of the number of sources. (Heads up: that condition depends on customer policy 🙈.)
Appendix B. Switching to S-PMSI 🦋
(The following point was fixed in a draft version of the document that became [RFC6513] and is here for reference only.)
Okay, so in early versions of the document that became [RFC6513], there were two proposed approaches for how a source PE decides when to start sending customer multicast traffic on an S-PMSI 🌈:
- The source PE sends multicast packets for the (C-S,C-G) on both the I-PMSI P-multicast tree and the S-PMSI P-multicast tree at the same time for a pre-configured period of time. This lets the receiver PEs pick the new tree for reception before the source switches to only the S-PMSI.
- The source PE waits for a pre-configured period of time after advertising the (C-S,C-G) entry bound to the S-PMSI before fully switching the traffic onto the S-PMSI-bound P-multicast tree.
The first alternative had basically two drawbacks:
- (C-S,C-G) traffic gets sent twice for some period of time. That seems pretty at odds with the whole reason for switching to an S-PMSI, which is to optimize the bandwidth used by the multicast tree for that stream 💀.
- The switchover is unlikely to happen without packet loss or duplication if the transit delays of the I-PMSI P-multicast tree and the S-PMSI P-multicast tree differ 😭.
The second alternative has none of these drawbacks. It satisfies the requirement in Section 5.1.3 of [RFC4834], which says that "a multicast VPN solution SHOULD as much as possible ensure that client multicast traffic packets are neither lost nor duplicated, even when changes occur in the way a client multicast data stream is carried over the provider network". Bonus: the second alternative is also the one used in existing deployments 💖.
Consistent with all of this, only the second alternative is discussed in [RFC6513] ✨.
Authors' Addresses 🫶
Thomas Morin (editor)
France Telecom - Orange
2 rue Pierre Marzin
Lannion 22307
France
EMail: thomas.morin@orange.com
Ben Niven-Jenkins (editor)
BT
208 Callisto House, Adastral Park
Ipswich, Suffolk IP5 3RE
UK
EMail: ben@niven-jenkins.co.uk
Yuji Kamite
NTT Communications Corporation
Granpark Tower
3-4-1 Shibaura, Minato-ku
Tokyo 108-8118
Japan
EMail: y.kamite@ntt.com
Raymond Zhang
Alcatel-Lucent
777 Middlefield Rd.
Mountain View, CA 94043
USA
EMail: raymond.zhang@alcatel-lucent.com
Nicolai Leymann
Deutsche Telekom
Winterfeldtstrasse 21-27
10781 Berlin
Germany
EMail: n.leymann@telekom.de
Nabil Bitar
Verizon
60 Sylvan Road
Waltham, MA 02451
USA
EMail: nabil.n.bitar@verizon.com📦 where this page comes from
- ietf: rfc6517 2012-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms