RFC 8220
Protocol Independent Multicast (PIM) over Virtual Private LAN Service (VPLS)
RFC 8220 ๐
Protocol Independent Multicast (PIM) over Virtual Private LAN Service (VPLS) is the RFC we're looking at today ๐ It was written by O. Dornon, J. Kotalwar, V. Hemige, R. Qiu, and Z. Zhang. The block lists Nokia next to V. Hemige, mistnet.io next to R. Qiu, and Juniper Networks, Inc. next to Z. Zhang. It gives no affiliation for O. Dornon or J. Kotalwar ๐ซถ The category is Informational, the ISSN is 2070-1721, and it's dated September 2017. The title block has no Obsoletes or Updates lines, so nothing is listed as obsoleted or updated here โจ
Abstract ๐ธ
Okay so this doc is all about how VPLS Provider Edges (PEs) can send multicast traffic only out of the ports that actually want it ๐ The ports that want it are the ones with interested Protocol Independent Multicast (PIM) routers or Internet Group Management Protocol (IGMP) hosts behind them. The trick is PIM snooping and PIM proxying ๐ง
With PIM snooping, PEs quietly listen to certain PIM control messages. They use those to build their control and forwarding states, and they keep passing the messages along untouched ๐ With PIM proxying, PEs don't flood PIM Join/Prune messages. Instead they make their own and send them out of certain ports, based on the control states built from the downstream Join/Prune messages. Proxying is required when PIM Join suppression is enabled on the Customer Edge (CE) devices, and it's super useful for cutting down PIM control traffic in a VPLS domain ๐ฅ
The doc also describes PIM relay, which is basically lightweight proxying ๐ฆ Every downstream Join/Prune message just gets forwarded out of certain ports, and nothing gets flooded. That way PIM Join suppression doesn't get triggered on the CE devices. Iconic behavior honestly ๐
Status of This Memo ๐
This document is not an Internet Standards Track specification. It's published for informational purposes, so no vibe check needed ๐
It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It went through public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every document the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you want the full story ๐
For the current status, any errata, and how to give feedback, go to https://www.rfc-editor.org/info/rfc8220 โจ
Copyright Notice ๐
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. ยฉ๏ธ๐
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can find at https://trustee.ietf.org/license-info. The version in effect on the publication date is the one that counts. Please read these carefully, because they spell out your rights and restrictions for this document ๐ Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come without warranty, as described in the Simplified BSD License ๐
Table of Contents ๐งพ
1. Introduction ....................................................4
1.1. Multicast Snooping in VPLS .................................5
1.2. Assumptions ................................................6
1.3. Definitions ................................................6
1.4. Requirements Language ......................................7
2. PIM Snooping for VPLS ...........................................7
2.1. PIM Protocol Background ....................................7
2.2. General Rules for PIM Snooping in VPLS .....................8
2.2.1. Preserving Assert Triggers ..........................8
2.3. Some Considerations for PIM Snooping .......................9
2.3.1. Scaling .............................................9
2.3.2. IPv4 and IPv6 ......................................10
2.3.3. PIM-SM (*,*,RP) ....................................10
2.4. PIM Snooping vs. PIM Proxying .............................10
2.4.1. Differences between PIM Snooping, Relay,
and Proxying .......................................10
2.4.2. PIM Control Message Latency ........................11
2.4.3. When to Snoop and When to Proxy ....................12
2.5. Discovering PIM Routers ...................................13
2.6. PIM-SM and PIM-SSM ........................................14
2.6.1. Building PIM-SM States .............................15
2.6.2. Explanation for Per-(S,G,N) States .................17
2.6.3. Receiving (*,G) PIM-SM Join/Prune Messages .........18
2.6.4. Receiving (S,G) PIM-SM Join/Prune Messages .........20
2.6.5. Receiving (S,G,rpt) Join/Prune Messages ............22
2.6.6. Sending Join/Prune Messages Upstream ...............23
2.7. Bidirectional PIM (BIDIR-PIM) .............................24
2.8. Interaction with IGMP Snooping ............................24
2.9. PIM-DM ....................................................25
2.9.1. Building PIM-DM States .............................25
2.9.2. PIM-DM Downstream Per-Port PIM(S,G,N) State
Machine ............................................25
2.9.3. Triggering Assert Election in PIM-DM ...............26
2.10. PIM Proxy ................................................26
2.10.1. Upstream PIM Proxy Behavior .......................26
2.11. Directly Connected Multicast Source ......................26
2.12. Data-Forwarding Rules ....................................27
2.12.1. PIM-SM Data-Forwarding Rules ......................28
2.12.2. PIM-DM Data-Forwarding Rules ......................29
3. IANA Considerations ............................................29
4. Security Considerations ........................................30
5. References .....................................................30
5.1. Normative References ......................................30
5.2. Informative References ....................................31
Appendix A. BIDIR-PIM Considerations ..............................32
A.1. BIDIR-PIM Data-Forwarding Rules ............................32
Appendix B. Example Network Scenario ..............................33
B.1. PIM Snooping Example .......................................33
B.2. PIM Proxy Example with (S,G) / (*,G) Interaction ...........36
Acknowledgements ..................................................42
Contributors ......................................................42
Authors' Addresses ................................................43
1. Introduction ๐ธ
In VPLS, the Provider Edge (PE) devices give you a logical interconnect, so the Customer Edge (CE) devices in a specific VPLS instance look like they're all hooked up to one single LAN ๐ง The Forwarding Information Base (FIB) for a VPLS instance fills itself in dynamically through Media Access Control (MAC) address learning. Once a unicast MAC address is learned and tied to a particular Attachment Circuit (AC) or pseudowire (PW), a frame aimed at that MAC only has to go out on that AC or PW. Efficient, bestie ๐
Now for a frame that isn't addressed to a known unicast MAC address, flooding is the only option. That covers the "BUM" (Broadcast, Unknown Unicast, and Multicast) traffic, which breaks down like this:
- B: The destination MAC address is a broadcast address.
- U: The destination MAC address is unknown (has not been learned).
- M: The destination MAC address is a multicast address.
Multicast frames get flooded because a PE has no way of knowing where the corresponding multicast group members live ๐ญ VPLS solutions (RFC 4762 [VPLS-LDP] and RFC 4761 [VPLS-BGP]) do the replication for multicast traffic at the ingress PE devices. The VPLS Multicast Requirements document (RFC 5501 [VPLS-MCAST-REQ]) says there are two issues with VPLS multicast today:
- Multicast traffic is replicated to non-member sites.
- Multicast traffic may be replicated when several PWs share a physical path.
Issue 1 can be solved with multicast snooping. PEs learn which sites have multicast group members by snooping multicast protocol control messages on ACs, and then they forward IP multicast traffic only to member sites ๐ฏ This doc describes how to make that work when CE devices are PIM adjacencies of each other. Issue 2 is outside the scope of this document and is discussed in RFC 7117 [VPLS-MCAST] ๐
Even though this doc talks about VPLS, the procedures also apply to regular Layer 2 switches that are connected by physical connections. The only difference is that the PW-related concepts and procedures don't apply in that case ๐ฆ
1.1. Multicast Snooping in VPLS ๐
IGMP snooping, described in RFC 4541 [IGMP-SNOOP], makes sure IP multicast traffic only goes out on these:
- ACs connecting to hosts that report related group membership
- ACs connecting to routers that join related multicast groups
- PWs connecting to remote PEs that have the above-described ACs
Here's the catch ๐ Traffic always gets sent on ports that have point-to-point connections to routers that are attached to a LAN where there's at least one other router. IGMP snooping alone can't tell whether there are interested receivers beyond those routers. So we always have to send traffic to those ports, even when there are no snooped group memberships. To cut down the traffic going to those routers even more, PIM snooping can be used. This document describes the procedures for PIM snooping, including the rules for when both IGMP and PIM snooping are enabled in a VPLS instance. See Sections 2.8 and 2.11 for details ๐ง
Also, for both IGMP and PIM, the term "snooping" is used loosely. It just means a Layer 2 device peeks into Layer 3 routing protocol messages to build relevant control and forwarding states โจ Depending on whether the control messages are transparently flooded, selectively forwarded, or aggregated, the processing may get called "snooping" or "proxying" depending on the context ๐
We'll use the term "PIM snooping" in this document. Unless it's explicitly noted otherwise, though, the procedures apply equally to PIM snooping and PIM proxying ๐ The procedures specific to PIM proxying are described in Section 2.6.6. The differences you need to keep in mind while implementing one or the other, plus recommendations on which method to use in different scenarios, are in Section 2.4 ๐
This document also describes PIM relay, which you can think of as lightweight PIM proxying. Unless it's explicitly noted otherwise, proxying in the rest of this document implicitly includes relay as well. For an overview of how snooping, proxying, and relay differ, check Section 2.4.1 ๐
1.2. Assumptions
okay so this doc assumes you already know the PIM protocols really well ๐ bestie, we are not explaining PIM from zero. to make everything easier to follow, this doc is written in the same style as these PIM RFCs:
- RFC 3973 [PIM-DM] ๐ธ
- RFC 4607 [PIM-SSM] ๐ธ
- RFC 5015 [BIDIR-PIM] ๐ธ
- RFC 5384 [JOIN-ATTR] ๐ธ
- RFC 7761 [PIM-SM] ๐ธ
we also don't copy-paste the PIM handling stuff from those RFCs (copy-paste is so not it ๐). instead this doc cross-references the matching definitions and procedures in them. any PIM snooping-specific deviations in protocol handling get spelled out right here โจ
1.3. Definitions
there are a bunch of definitions that are already nicely explained in these PIM RFCs: RFC 3973 [PIM-DM], RFC 5015 [BIDIR-PIM], and RFC 7761 [PIM-SM]. these are the extra ones used all through this doc:
- a port is either an AC or a PW ๐ฆ
- when we say a PIM message is received on a PE port, it means the PE is processing that message for snooping/proxying or relaying ๐
and here are the abbreviations you'll see everywhere ๐ง
- S: IP address of the multicast source.
- G: IP address of the multicast group.
- N: Upstream Neighbor field in a Join/Prune/Graft message.
- Port(N): Port on which neighbor N is learned, i.e., the port on which N's Hellos are received.
- rpt: Rendezvous Point Tree.
- PIM-DM: Protocol Independent Multicast - Dense Mode.
- PIM-SM: Protocol Independent Multicast - Sparse Mode.
- PIM-SSM: Protocol Independent Multicast - Source-Specific Multicast.
1.4. Requirements Language
the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be read as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they show up in all capitals, as shown here. ๐ฅ
2. PIM Snooping for VPLS โจ
2.1. PIM Protocol Background
PIM is a multicast routing protocol that runs between routers, which are CE devices in a VPLS ๐ง. it uses the unicast routing table to provide reverse-path info for building multicast trees. there are a few PIM flavors. per RFC 3973 [PIM-DM], multicast datagrams get pushed toward downstream neighbors, kinda like a broadcast, but in parts of the network with no group members, routers prune back the tree branches toward the source. unlike PIM-DM, the other flavors (RFC 7761 [PIM-SM], RFC 4607 [PIM-SSM], and RFC 5015 [BIDIR-PIM]) use a pull approach with explicit Joins instead of push-and-prune. that's giving "ask before you get" ๐
PIM routers swap Hello messages regularly to discover neighbors and keep stateful sessions going ๐ซถ. once neighbors are found, PIM routers can signal that they want to join or prune specific multicast groups. they do this by having downstream routers send an explicit Join/Prune message to their upstream router (for the sake of simplicity, think of Graft messages for PIM-DM as Join messages). the Join/Prune message can be group specific (*,G) or group and source specific (S,G) ๐
2.2. General Rules for PIM Snooping in VPLS
the following rules for the correct operation of PIM snooping MUST be followed ๐ญ
- PIM snooping MUST NOT affect the operation of customer Layer 2 protocols or Layer 3 protocols.
- PIM messages and multicast data traffic forwarded by PEs MUST follow the split-horizon rule for mesh PWs, as defined in RFC 4762 [VPLS-LDP].
- PIM states in a PE MUST be per VPLS instance.
- PIM Assert triggers MUST be preserved to the extent necessary to avoid sending duplicate traffic to the same PE (see Section 2.2.1).
2.2.1. Preserving Assert Triggers
in PIM-SM / PIM-DM, there are scenarios where several routers could be forwarding the same multicast traffic on a LAN ๐. when that happens, they kick off the PIM Assert election by sending PIM Assert messages, so that only the Assert winner forwards multicast traffic on the LAN. the Assert election is data-driven, so it only happens if a router sees traffic on the interface it should be forwarding that traffic to. in a VPLS with PIM snooping, two routers may forward the same multicast datagrams at the same time, but each copy may reach a different set of PEs, and that's acceptable from the point of view of avoiding duplicate traffic. if the two copies may reach the same PE, though, the sending routers must be able to see each other's traffic so they can trigger Assert election and stop the duplicate traffic. ๐ to achieve that, PEs enabled with PIM-SSM / PIM-SM snooping MUST forward multicast traffic for an (S,G) / (,G) not only on the ports on which they snooped Join(S,G) / Join(,G), but also towards the upstream neighbor(s). in other words, the ports on which the upstream neighbors are learned must be added to the outgoing port list, along with the ports on which Joins are snooped. check Section 2.6.1 for the rules that determine the set of upstream neighbors for a particular (x,G).
similarly, PIM-DM snooping SHOULD make sure that Asserts can be triggered (Section 2.9.3).
all of this logic has to work without breaking VPLS split-horizon forwarding rules. that means traffic should not be forwarded on the port on which it was received, and traffic arriving on a PW MUST NOT be forwarded onto other PW(s). ๐ฆ
2.3. Some Considerations for PIM Snooping
the PIM snooping solution described here needs a PE to examine and operate on only PIM Hello and PIM Join/Prune packets ๐. the PE does not need to examine any other PIM packets.
most of the PIM snooping procedures for handling Hello/Join/Prune messages are very similar to what runs in a PIM router โจ. however, the PE doesn't need any routing tables like the ones PIM routing requires. it only knows how to forward Join/Prune messages by looking at the Upstream Neighbor field in those packets, as described in Section 2.12.
the PE does not need to know about Rendezvous Points (RPs), and it doesn't have to maintain any RP Set. all of that is transparent to a PIM snooping PE ๐
in the subsections below, we list some considerations and observations for implementing PIM snooping in the VPLS ๐
2.3.1. Scaling
PIM snooping needs to be employed on ACs at the downstream PEs (PEs receiving multicast traffic across the VPLS core) to keep traffic from being sent out of ACs unnecessarily. PIM snooping techniques can also be employed on PWs at the upstream PEs (PEs receiving traffic from local ACs in a hierarchical VPLS) to keep traffic from being sent to PEs unnecessarily. this may work well for small-scale or medium-scale deployments. however, if there are a large number of VPLS instances with a large number of PEs per instance, the amount of snooping required at the upstream PEs can overwhelm the upstream PEs. ๐ญ
there are two methods to reduce the burden on the upstream PEs. one is to use PIM proxying, as described in Section 2.6.6, which reduces the control messages forwarded by a PE. the other is not to snoop on the PWs at all, but to have PEs signal the snooped states to other PEs out of band via BGP, as described in RFC 7117 [VPLS-MCAST]. in this document, it is assumed that snooping is performed on PWs. ๐ธ
2.3.2. IPv4 and IPv6
in the VPLS, PEs forward Ethernet frames received from CEs, and as such they are agnostic of the Layer 3 protocol used by the CEs ๐. however, as a PIM snooping PE, the PE would have to look deeper into the IP and PIM packets and build snooping state based on that. the PIM protocol specifications handle both IPv4 and IPv6. the specification for PIM snooping in this document can be applied to both IPv4 and IPv6 payloads. ๐ง
2.3.3. PIM-SM (,,RP)
this document does not address (,,RP) states in the VPLS network, as they have been removed from the PIM protocol as described in RFC 7761 [PIM-SM]. ๐
2.4. PIM Snooping vs. PIM Proxying
this document has previously alluded to PIM snooping/relay/proxying ๐. details on the PIM relay/proxying solution are discussed in Section 2.6.6. in this section, you get a brief description and comparison. ๐
2.4.1. Differences between PIM Snooping, Relay, and Proxying
okay so let's line up all three side by side ๐ Differences between PIM snooping and relay/proxying look like this:
+--------------------+---------------------+-----------------------+
| PIM snooping | PIM relay | PIM proxying |
+====================|=====================|=======================+
| Join/Prune messages| Join/Prune messages | Join/Prune messages |
| snooped and flooded| snooped; forwarded | consumed. Regenerated|
| according to VPLS | as is out of certain| ones sent out of |
| flooding procedures| upstream ports | certain upstream ports|
+--------------------+---------------------+-----------------------+
| Hello messages | Hello messages | Hello messages |
| snooped and flooded| snooped and flooded | snooped and flooded |
| according to VPLS | according to VPLS | according to VPLS |
| flooding procedures| flooding procedures | flooding procedures |
+--------------------+---------------------+-----------------------+
| No PIM packets | No PIM packets | New Join/Prune |
| generated | generated | messages generated |
+--------------------+---------------------+-----------------------+
| CE Join suppression| CE Join suppression | CE Join suppression |
| not allowed | allowed | allowed |
+--------------------+---------------------+-----------------------+
Other than that chart, most of the procedures are shared between PIM snooping and PIM relay/proxying, unless it's specifically called out otherwise ๐ซถ
Pure PIM snooping PEs just peek at PIM packets as they zoom through the VPLS ๐ They're giving transparent LAN service fr, because no customer packets get changed or eaten, and no new packets get introduced into the VPLS โจ It's also simpler to build than proxying. BUT (big but ๐), for PIM snooping to work correctly, CE routers MUST disable Join suppression in the VPLS. If they don't, most of the CE routers that are interested in a given multicast data stream will fail to send Join/Prune messages for that stream, and then the PEs can't tell which ACs and/or PWs have listeners for it ๐ญ
Here's the tea: a lot of existing CE deployments don't support turning off Join suppression, and managing that setting across the VPLS is a lot of operational hassle for the provider ๐ So it becomes a tricky solution to actually deploy. Another downside of PIM snooping is that it doesn't scale as well as PIM proxying. If there are tons of CEs in a VPLS, then every CE sees every other CE's Join/Prune messages. Not it ๐
PIM relay/proxying has the advantage that it doesn't need Join suppression disabled in the VPLS. Multicast over a VPLS can be set up super easily without touching the CE routers at all ๐ฆ PIM relay/proxying also helps VPLS multicast scale, since Join/Prune messages only get sent out certain upstream ports instead of being flooded, and in the full proxying case (vs. relay), the PEs smartly generate only one Join/Prune message for a given multicast stream ๐
PIM proxying, however, loses the transparency vibe, since Join/Prune packets could get modified or even consumed at a PE. New packets could also get introduced into the VPLS. BUT this loss of transparency is limited to PIM Join/Prune packets only. It's worth it for optimizing multicast in the VPLS and helping the VPLS network scale way better, for both the provider and the customer ๐ Data traffic is still completely transparent โจ
2.4.2. PIM Control Message Latency โฑ๏ธ
A PIM snooping/relay/proxying PE snoops on PIM Hello packets while transparently flooding them in the VPLS. So there's no latency added by the VPLS when delivering PIM Hello packets to remote CEs in the VPLS ๐ธ
A PIM snooping PE snoops on PIM Join/Prune packets while transparently flooding them in the VPLS. There's no latency added by the VPLS when delivering PIM Join/Prune packets when PIM snooping is used ๐
A PIM relay/proxying PE doesn't just flood PIM Join/Prune packets. This can cause extra latency for a downstream CE to start receiving multicast traffic after it has sent a Join ๐ฅฒ When a downstream CE prunes a multicast stream, the traffic SHOULD stop flowing to the CE with no additional latency introduced by the VPLS.
Only proxying Join/Prune messages (and not Hello messages) keeps the PE's behavior very similar to a PIM router's, without adding too much complexity ๐ง It keeps the PIM proxying solution fairly simple. Join/Prune messages get forwarded by a PE along the slow path, and all other PIM packet types get forwarded along the fast path. So it's very likely that packets on the fast path will arrive "ahead" of Join/Prune packets at a CE router. (Note the stress on this: fast-path messages will never arrive after Join/Prune packets.) Hello packets on the fast path are the big ones to watch ๐ We can build a bunch of scenarios where Hellos and Join/Prune messages get delivered out of order. Still, there should be no deviation from normal expected behavior at the CE router receiving those messages out of order ๐
2.4.3. When to Snoop and When to Proxy
From everything above, the factors that decide between snooping, relay, and proxying are:
- Whether CEs do Join suppression or not
- Whether Join/Prune latency is critical or not
- Whether the scale of PIM protocol messages/states in a VPLS requires the scaling benefit of proxying
Of these factors, Join suppression is the big one ๐ฅ Pure snooping can only be used when Join suppression is disabled on all CEs. The latency that comes with relay/proxying depends on the implementation, and it might not be a concern at all with a particular implementation ๐ The scaling benefit might not matter much either, because on a real LAN with Explicit Tracking (ET), a PIM router also needs to receive and process all PIM Join/Prune messages โจ
A PIM router says Join suppression is disabled if the T-bit is set in the LAN Prune Delay option of its Hello message. If every PIM router on a LAN sets the T-bit, then ET is possible, which lets an upstream router track all the downstream neighbors that have Join states for any (S,G) or (*,G). That gives two perks:
- No need for the Prune-Pending process. The upstream router can immediately stop forwarding data when it gets a Prune from the last downstream neighbor, and it can immediately prune to its upstream neighbor.
- For management purposes, the upstream router knows exactly which downstream routers exist for a particular Join state.
Full proxying can be used with or without Join suppression on CEs, and it doesn't get in the way of an upstream CE skipping the Prune-Pending process ๐ BUT it proxies all its downstream CEs as one single CE to the upstream neighbors, so that second perk above goes out the window ๐ญ
So the general rule is: if Join suppression is enabled on one or more CEs, then proxying or relay MUST be used. But if Join suppression is known to be disabled on all CEs, then snooping, relay, or proxying MAY be used, while snooping or relay SHOULD be used ๐
An implementation MAY choose to figure out which mode to use dynamically, by tracking the T-bit mentioned above in all snooped PIM Hello messages. Or it MAY just require static provisioning ๐งธ
2.5. Discovering PIM Routers
A PIM snooping PE MUST snoop on PIM Hellos received on ACs and PWs. That means the PE transparently floods the PIM Hello while snooping on it ๐ฆ PIM Hellos are how the snooping PE discovers PIM routers and their characteristics.
For each neighbor a PE discovers, it adds an entry to the PIM Neighbor Database with these fields:
- Layer 2 encapsulation for the router sending the PIM Hello.
- IP address and address family of the router sending the PIM Hello.
- Port (AC/PW) on which the PIM Hello was received.
- Hello Option fields.
The PE should be able to interpret and act on Hello Option fields as currently defined in RFC 7761 [PIM-SM]. The Option fields of particular interest in this document are:
- Hello-Hold-Time
- Tracking Support
- Designated Router (DR) Priority
Please refer to RFC 7761 [PIM-SM] for a list of the Hello Option fields ๐ When a PIM Hello is received, the PE MUST reset the neighbor-expiry-timer to Hello-Hold-Time. If a PE doesn't receive a Hello message from a router within Hello-Hold-Time, the PE MUST remove that neighbor from its PIM Neighbor Database. And if a PE receives a Hello message from a router with the Hello-Hold-Time value set to zero, the PE MUST remove that router from the PIM snooping state immediately ๐
From the PIM Neighbor Database, a PE MUST be able to use the procedures defined in RFC 7761 [PIM-SM] to identify the PIM DR in the VPLS instance. It should also be able to determine if tracking support is active in the VPLS instance โจ
2.6. PIM-SM and PIM-SSM
The key characteristic of PIM-SM and PIM-SSM is explicit Join behavior ๐ In this model, multicast traffic is only forwarded to locations that specifically asked for it. Everything in this section applies to both PIM-SM and PIM-SSM, except that there's no (*,G) state in PIM-SSM ๐ธ
2.6.1. Building PIM-SM States โจ
okay so PIM-SM and PIM-SSM states get built by snooping on the PIM-SM Join/Prune messages that come in on ACs/PWs ๐
the downstream state machine of a PIM-SM snooping PE is giving very PIM-SM router energy. the downstream state looks like this:
Per downstream (Port,*,G):
DownstreamJPState: One of{"NoInfo" (NI), "Join" (J), "Prune-Pending" (PP)}
Per downstream (Port,*,G,N):
- Prune-Pending Timer
(PPT(N)) - Join Expiry Timer
(ET(N))
Per downstream (Port,S,G):
DownstreamJPState: One of{"NoInfo" (NI), "Join" (J), "Prune-Pending" (PP)}
Per downstream (Port,S,G,N):
- Prune-Pending Timer
(PPT(N)) - Join Expiry Timer
(ET(N))
Per downstream (Port,S,G,rpt):
DownstreamJPRptState: One of{"NoInfo" (NI), "Pruned" (P), "Prune-Pending" (PP)}
Per downstream (Port,S,G,rpt,N):
- Prune-Pending Timer
(PPT(N)) - Join Expiry Timer
(ET(N))
where S is the address of the multicast source, G is the group address, and N is the Upstream Neighbor field in the Join/Prune message.
not gonna lie, this part is a little different from routers ๐ a PIM-SM router keeps the PPT and ET per (Interface,S,G), but a PIM snooping PE has to keep the PPT and ET per (Port,S,G,N). the reasons are explained in Section 2.6.2, so stay tuned ๐ง
on top of the states above, we also get a few state summarization macros:
UpstreamNeighbors(*,G): If there are one or more Join(,G)s received on any port with upstream neighbor N and ET(N) is active, then N is added toUpstreamNeighbors(*,G). This set is used to determine if a Join(,G) or a Prune(*,G) with upstream neighbor N needs to be sent upstream.UpstreamNeighbors(S,G): If there are one or more Join(S,G)s received on any port with upstream neighbor N and ET(N) is active, then N is added toUpstreamNeighbors(S,G). This set is used to determine if a Join(S,G) or a Prune(S,G) with upstream neighbor N needs to be sent upstream.UpstreamPorts(*,G): This is the set of all Port(N) ports where N is in the setUpstreamNeighbors(*,G). Multicast streams forwarded using a (*,G) match MUST be forwarded to these ports. So,UpstreamPorts(*,G)MUST be added toOutgoingPortList(*,G).UpstreamPorts(S,G): This is the set of all Port(N) ports where N is in the setUpstreamNeighbors(S,G).UpstreamPorts(S,G)MUST be added toOutgoingPortList(S,G).InheritedUpstreamPorts(S,G): This is the union ofUpstreamPorts(S,G)andUpstreamPorts(*,G).UpstreamPorts(S,G,rpt): If PruneDesired(S,G,rpt) becomes TRUE, then this set is set toUpstreamPorts(*,G). Otherwise, this set is empty.UpstreamPorts(*,G)(-)UpstreamPorts(S,G,rpt)MUST be added toOutgoingPortList(S,G).UpstreamPorts(G): This set is the union of all theUpstreamPorts(S,G)andUpstreamPorts(*,G)for a given G. Proxy (S,G) Join/Prune and (*,G) Join/Prune messages MUST be sent to a subset ofUpstreamPorts(G)as specified in Section 2.6.6.1.PWPorts: This is the set of all PWs.OutgoingPortList(*,G): This is the set of all ports to which traffic needs to be forwarded on a (*,G) match.OutgoingPortList(S,G): This is the set of all ports to which traffic needs to be forwarded on an (S,G) match.
see Section 2.12 ("Data-Forwarding Rules") for how OutgoingPortList gets calculated ๐
NumETsActive(Port,*,G): This is the number of(Port,*,G,N)entries that have the Expiry Timer running. This macro keeps track of the number of Join(*,G)s that are received on this Port with different upstream neighbors.NumETsActive(Port,S,G): This is the number of(Port,S,G,N)entries that have the Expiry Timer running. This macro keeps track of the number of Join(S,G)s that are received on this Port with different upstream neighbors.JoinAttributeTlvs(*,G): Join Attributes (RFC 5384 [JOIN-ATTR]) are TLVs that may be present in received Join(*,G) messages. An example would be Reverse Path Forwarding (RPF) Vectors (RFC 5496 [RPF-VECTOR]). If present, they must be copied toJoinAttributeTlvs(*,G).JoinAttributeTlvs(S,G): Join Attributes (RFC 5384 [JOIN-ATTR]) are TLVs that may be present in received Join(S,G) messages. If present, they must be copied toJoinAttributeTlvs(S,G).
since a PIM snooping PE and a PIM-SM router have a few differences in their downstream state machines, we spell out the details of the PIM-SM snooping PE's downstream state machine, even if that means repeating a lot of what RFC 7761 [PIM-SM] already says ๐ซถ
2.6.2. Explanation for Per-(S,G,N) States โจ
in PIM routing protocols, states get built per (S,G). on a router, an (S,G) has only one RPF-Neighbor. but a PIM snooping PE doesn't have the Layer 3 routing info that the routers have, so it can't figure out the RPF-Neighbor for a multicast flow on its own. it just discovers it by snooping the Join/Prune message ๐ต๏ธโโ๏ธ
here's where it gets messy: a PE could snoop two or more different Join/Prune messages for the same (S,G), and those could carry different Upstream Neighbor fields. that can happen during transient network conditions or because of dual-homed sources. a PE cannot make assumptions about which one to pick. instead, it must allow the CE routers to decide which upstream neighbor gets elected as the RPF-Neighbor. and for that to work, the PE has to track downstream and upstream Joins and Prunes per (S,G,N) ๐
2.6.3. Receiving (*,G) PIM-SM Join/Prune Messages โจ
okay so a Join(,G) or Prune(,G) counts as "received" if one of the following conditions is met:
- The port on which it arrived is not Port(N), where N is the upstream neighbor N of the Join/Prune(*,G).
- If both Port(N) and the arrival port are PWs, then there exists at least one other
(*,G,Nx)or(Sx,G,Nx)state with an AC UpstreamPort.
the case where both Port(N) and the arrival port are PWs gets nicknamed "PW-only Join/Prune" in this doc, for simplicity ๐ the PW-only Join/Prune handling exists so the Port(N) PW can be added to the related forwarding entries' OutgoingPortList to trigger an Assert. but that's only needed for those states with AC UpstreamPorts. also, in the PW-only case, it's OK for the arrival port and Port(N) to be the same. check Appendix B for examples ๐
when a router receives a Join(,G) or a Prune(,G) with upstream neighbor N, it must process the message as defined in the state machine below. the macro computations that come out of this state machine transition are exactly as specified in RFC 7761 [PIM-SM], so no surprises there โจ
we define the following per-port (*,G,N) macro to help with the state machine below:
+---------------++-------------------------------------------------+
| || Previous State |
| ++-------------+--------------+--------------------+
| Event || NoInfo (NI) | Join (J) | Prune-Pending (PP) |
+---------------++-------------+--------------+--------------------+
| Receive || -> J state | -> J state | -> J state |
| Join(*,G) || Action | Action | Action |
| || RxJoin(N) | RxJoin(N) | RxJoin(N) |
+---------------++-------------+--------------+--------------------+
|Receive || - | -> PP state | -> PP state |
|Prune(*,G) and || | Start PPT(N) | |
|NumETsActive<=1|| | | |
+---------------++-------------+--------------+--------------------+
|Receive || - | -> J state | - |
|Prune(*,G) and || | Start PPT(N) | |
|NumETsActive>1 || | | |
+---------------++-------------+--------------+--------------------+
|PPT(N) expires || - | -> J state | -> NI state |
| || | Action | Action |
| || | PPTExpiry(N) | PPTExpiry(N) |
+---------------++-------------+--------------+--------------------+
|ET(N) expires || - | -> NI state | -> NI state |
|and || | Action | Action |
|NumETsActive<=1|| | ETExpiry(N) | ETExpiry(N) |
+---------------++-------------+--------------+--------------------+
|ET(N) expires || - | -> J state | - |
|and || | Action | |
|NumETsActive>1 || | ETExpiry(N) | |
+---------------++-------------+--------------+--------------------+
Figure 1: Downstream Per-Port (*,G) State Machine in Tabular Form
Action RxJoin(N):
If ET(N) is not already running, then start ET(N). Otherwise, restart ET(N). If N is not already in UpstreamNeighbors(*,G), then add N to UpstreamNeighbors(*,G) and trigger a Join(,G) with upstream neighbor N to be forwarded upstream. If there are Join Attribute TLVs in the received (,G) message and if they are different from the recorded JoinAttributeTlvs(*,G), then copy them into JoinAttributeTlvs(*,G). In the case of conflicting attributes, the PE will need to perform conflict resolution per (N) as described in RFC 5384 [JOIN-ATTR].
Action PPTExpiry(N):
Same as Action ETExpiry(N) below, plus send a Prune-Echo(*,G) with upstream neighbor N on the downstream port.
Action ETExpiry(N):
Disable timers ET(N) and PPT(N). Delete Neighbor state (Port,*,G,N). If there are no other (Port,*,G) states with NumETsActive(Port,*,G) > 0, transition DownstreamJPState (RFC 7761 [PIM-SM]) to NoInfo. If there are no other (Port,*,G,N) states (different ports but for the same N), remove N from UpstreamPorts(*,G) -- this will also trigger the Upstream Finite State Machine (FSM) with "JoinDesired(*,G,N) to FALSE".
2.6.4. Receiving (S,G) PIM-SM Join/Prune Messages ๐ธ
okay so a Join(S,G) or Prune(S,G) only counts as "received" ๐ซถ if one of these is true:
- The port it arrived on is not Port(N), where N is the upstream neighbor N of the Join/Prune(S,G).
- Both Port(N) and the arrival port are PWs, and there's at least one other (*,G,Nx) or (S,G,Nx) state with an AC UpstreamPort.
For simplicity, the case where both Port(N) and the arrival port are PWs gets called "PW-only Join/Prune" in this doc ๐ . The PW-only handling exists so the Port(N) PW can be added to the related forwarding entries' OutgoingPortList to trigger an Assert, but that's only needed for states with AC UpstreamPorts. In the PW-only case, it's OK for the arrival port and Port(N) to be the same. See Appendix B for examples โจ
When a router receives a Join(S,G) or a Prune(S,G) with upstream neighbor N, it must process the message as defined in the state machine below. The macro computations from this state machine transition are exactly as specified in RFC 7761 [PIM-SM], so no surprises there ๐
+---------------++-------------------------------------------------+
| || Previous State |
| ++-------------+--------------+--------------------+
| Event || NoInfo (NI) | Join (J) | Prune-Pending (PP) |
+---------------++-------------+--------------+--------------------+
| Receive || -> J state | -> J state | -> J state |
| Join(S,G) || Action | Action | Action |
| || RxJoin(N) | RxJoin(N) | RxJoin(N) |
+---------------++-------------+--------------+--------------------+
|Receive || - | -> PP state | - |
|Prune(S,G) and || | Start PPT(N) | |
|NumETsActive<=1|| | | |
+---------------++-------------+--------------+--------------------+
|Receive || - | -> J state | - |
|Prune(S,G) and || | Start PPT(N) | |
|NumETsActive>1 || | | |
+---------------++-------------+--------------+--------------------+
|PPT(N) expires || - | -> J state | -> NI state |
| || | Action | Action |
| || | PPTExpiry(N) |PPTExpiry(N) |
+---------------++-------------+--------------+--------------------+
|ET(N) expires || - | -> NI state | -> NI state |
|and || | Action | Action |
|NumETsActive<=1|| | ETExpiry(N) | ETExpiry(N) |
+---------------++-------------+--------------+--------------------+
|ET(N) expires || - | -> J state | - |
|and || | Action | |
|NumETsActive>1 || | ETExpiry(N) | |
+---------------++-------------+--------------+--------------------+
Figure 2: Downstream Per-Port (S,G) State Machine in Tabular Form
Action RxJoin(N): If ET(N) isn't already running, start ET(N). Otherwise, restart ET(N) ๐ฅ
If N isn't already in UpstreamNeighbors(S,G), add N to UpstreamNeighbors(S,G) and trigger a Join(S,G) with upstream neighbor N to be forwarded upstream. If there are Join Attribute TLVs in the received (S,G) message, and they're different from the recorded JoinAttributeTlvs(S,G), copy them into JoinAttributeTlvs(S,G). In cases of conflicting attributes, the PE will need to perform conflict resolution per (N) as described in RFC 5384 [JOIN-ATTR] ๐ง
Action PPTExpiry(N): Same as Action ETExpiry(N) below, plus send a Prune-Echo(S,G) with upstream neighbor N on the downstream port ๐
Action ETExpiry(N): Disable timers ET(N) and PPT(N). Delete Neighbor state (Port,S,G,N). If there are no other (Port,S,G) states with NumETsActive(Port,S,G) > 0, transition DownstreamJPState to NoInfo. If there are no other (Port,S,G,N) states (different ports but for the same N), remove N from UpstreamPorts(S,G). This will also trigger the Upstream FSM with "JoinDesired(S,G,N) to FALSE" ๐
2.6.5. Receiving (S,G,rpt) Join/Prune Messages ๐ฆ
A Join(S,G,rpt) or Prune(S,G,rpt) is "received" when the port it came in on isn't also the port where the upstream neighbor N of the Join/Prune(S,G,rpt) was learned.
It's important to make sure the (S,G) and (*,G) state machines can handle per-(S,G,N) states, but it's not as important for (S,G,rpt) states. It's enough to say that the downstream (S,G,rpt) state machine is the same as what's defined in Section 4.5.3 of RFC 7761 [PIM-SM] โจ
2.6.6. Sending Join/Prune Messages Upstream ๐
This section applies only to a PIM relay/proxying PE, not to a PIM snooping PE ๐
A full PIM proxying (not relay) PE MUST implement the Upstream FSM along the lines of the procedure described in Section 4.5.4 of RFC 7761 [PIM-SM] ๐
For the purposes of the Upstream FSM, a Join or Prune message with upstream neighbor N is "seen" on a PIM relay/proxying PE if the port it was received on is also Port(N) and that port is an AC. The AC requirement exists because a Join received on the Port(N) PW must not suppress this PE's Join on that PW ๐ซถ
A PIM relay PE doesn't implement the Upstream FSM. It just forwards received Join/Prune messages out of the same set of upstream ports as in the PIM proxying case ๐
To correctly facilitate Asserts among the CE routers, these Join/Prune messages need to be sent not only towards the upstream neighbor but also on certain PWs, as described below โญ
- If
JoinAttributeTlvs(*,G)is not empty, it must be encoded in a Join(*,G) message sent upstream. - If
JoinAttributeTlvs(S,G)is not empty, it must be encoded in a Join(S,G) message sent upstream.
2.6.6.1. Where to Send Join/Prune Messages ๐
The following rules apply to both (1) forwarded (in the case of PIM relay) and (2) refreshed and triggered (in the case of PIM proxying) (S,G) / (*,G) Join/Prune messages.
- The Upstream Neighbor field in the Join/Prune to be sent is set to the N in the corresponding Upstream FSM.
- If Port(N) is an AC, send the message to Port(N).
- Additionally, if
OutgoingPortList(x,G,N)contains at least one AC, then the message MUST be sent to at least all the PWs inUpstreamPorts(G)(for (*,G)) orInheritedUpstreamPorts(S,G)(for (S,G)). Alternatively, the message MAY be sent to all PWs.
Sending to a subset of PWs as described above guarantees that if traffic (of the same flow) from two upstream routers were to reach this PE, then the two routers will receive from each other, triggering an Assert ๐ฅ
Sending to all PWs guarantees that if two upstream routers both send traffic for the same flow (even if it's to different sets of downstream PEs), then the two routers will receive from each other, triggering an Assert ๐ฆ
2.7. Bidirectional PIM (BIDIR-PIM) ๐ธ
Bidirectional PIM (BIDIR-PIM) is a variation of PIM-SM. The main differences between PIM-SM and BIDIR-PIM are:
- There are no source-based trees, and SSM isn't supported (i.e., no (S,G) states) in BIDIR-PIM.
- Multicast traffic can flow up the shared tree in BIDIR-PIM.
- To avoid forwarding loops, one router on each link is elected as the Designated Forwarder (DF) for each RP in BIDIR-PIM.
The main advantage of BIDIR-PIM is that it scales well for many-to-many applications. However, the lack of source-based trees means multicast traffic is forced to stay on the shared tree ๐ง
As described in RFC 5015 [BIDIR-PIM], parts of a BIDIR-PIM-enabled network may forward traffic without exchanging Join/Prune messages. For instance, that happens between DFs and the Rendezvous Point Link (RPL) ๐
Since the described procedures for PIM snooping rely on Join/Prune messages being present, turning on PIM snooping in BIDIR-PIM networks could break BIDIR-PIM functionality. Deploying PIM snooping on BIDIR-PIM-enabled networks will need some further study. Some thoughts on this topic are discussed in Appendix A ๐
2.8. Interaction with IGMP Snooping ๐ซถ
Whenever IGMP snooping is enabled in conjunction with PIM snooping in the same VPLS instance, the PE SHOULD follow these rules:
- To maintain the list of multicast routers and the ports they're attached to, the PE SHOULD NOT use the rules described in RFC 4541 [IGMP-SNOOP], but SHOULD rely on the neighbors discovered by PIM snooping. This list SHOULD then be used to apply the first forwarding rule (rule 1) listed in Section 2.1.1 of RFC 4541 [IGMP-SNOOP] ๐
- If the PE supports proxy reporting, an IGMP membership learned only on a port to which a PIM neighbor is attached (i.e., not learned elsewhere) SHOULD NOT be included in the summarized upstream report sent to that port โจ
2.9. PIM-DM ๐
The key characteristic of PIM-DM is flood-and-prune behavior. Shortest-path trees get built as a multicast source starts transmitting ๐
2.9.1. Building PIM-DM States ๐
PIM-DM states are built by snooping on the PIM-DM Join, Prune, Graft, and State Refresh messages received on ACs/PWs, and on State Refresh messages sent on ACs/PWs. By snooping on these PIM-DM messages, a PE builds the following states per (S,G,N), where S is the address of the multicast source, G is the group address, and N is the upstream neighbor to which Prunes/Grafts are sent by downstream CEs:
Per PIM(S,G,N):
Port PIM(S,G,N) Prune State:
* DownstreamPState(S,G,N,Port): One of {"NoInfo" (NI),
"Pruned" (P), "Prune-Pending" (PP)}
* Prune-Pending Timer (PPT)
* Prune Timer (PT)
* Upstream Port (valid if the PIM(S,G,N) Prune state is "Pruned")
2.9.2. PIM-DM Downstream Per-Port PIM(S,G,N) State Machine ๐
The downstream per-port PIM(S,G,N) state machine is as defined in Section 4.4.2 of RFC 3973 [PIM-DM], with a few changes relevant to PIM snooping. When reading Section 4.4.2 of RFC 3973 [PIM-DM], keep in mind that for PIM snooping, the downstream states are built per (S,G,N,Downstream-Port) and not per (Downstream-Interface,S,G) like in a PIM-DM router. As noted in Section 2.9.1, the states (DownstreamPState) and timers (PPT and PT) are per (S,G,N,Port) ๐ง
2.9.3. Triggering Assert Election in PIM-DM ๐ฅ
Since PIM-DM is a flood-and-prune protocol, traffic is flooded to all routers unless it's explicitly pruned. Since PIM-DM routers don't prune on non-RPF interfaces, PEs should typically not receive Prunes on Port(RPF-Neighbor). So the asserting routers should typically be in pim_oiflist(S,G). In most cases, Assert election should occur naturally without any special handling, since data traffic will be forwarded to the asserting routers ๐ฆ
However, there are some scenarios where a Prune might be received on a port that's also an upstream port. If we prune the port from pim_oiflist(S,G), it wouldn't be possible for the asserting routers to determine whether traffic arrived on their downstream port. This can be fixed by adding pim_iifs(S,G) to pim_oiflist(S,G) so that data traffic flows to the upstream ports ๐
2.10. PIM Proxy ๐
okay so remember how PIM snooping only works correctly if Join suppression is disabled in the VPLS? ๐ if Join suppression is enabled, then PEs MUST do PIM relay/proxying or VPLS multicast just won't work right, bestie ๐ญ this section is only about full proxying, not relay ๐ฆ
2.10.1. Upstream PIM Proxy Behavior
a PIM proxying PE eats the Join/Prune messages and makes brand new PIM Join/Prune messages to send upstream ๐ it does this by running the Upstream FSM from Section 4.5.4 of RFC 7761 [PIM-SM]. that's literally the only difference from PIM relay โจ
the source IP address in PIM packets sent upstream SHOULD be the address of a PIM downstream neighbor in the matching Join/Prune state ๐ธ but the chosen address MUST NOT be the Upstream Neighbor field that gets encoded in the packet, no mixing those up ๐ โโ๏ธ and the Layer 2 encapsulation for the picked source IP address MUST be the encapsulation recorded in the PIM Neighbor Database for that IP address ๐ง
2.11. Directly Connected Multicast Source ๐
okay this one is a whole plot twist ๐ญ PIM snooping/relay/proxying could be turned on for a LAN that connects a multicast source AND a PIM First-Hop Router (FHR) ๐ but the FHR won't send any downstream Join/Prune messages, so we can't set up any forwarding state for that source ๐
so if there's a source in the CE network plugged straight into the VPLS instance, then multicast traffic from that source MUST be sent to all PIM routers on the VPLS instance too, on top of the IGMP receivers in the VPLS โจ if there's already (S,G) or (*,G) snooping state on any PE, this won't happen under the current forwarding rules and guidelines, which is the drama ๐ค so a PE has to be able to tell whether the traffic came from a host on that LAN, and there are three ways to handle it ๐
- the PE would have to do IPv4 ARP snooping and/or IPv6 Neighbor Discovery snooping to figure out if a source is directly connected.
- another option is to configure all PEs to say there are CE sources directly connected to the VPLS instance, and turn off snooping for the groups that source is going to send traffic to. that way, traffic from that source to those groups will always be flooded within the provider network ๐ซถ
- a third option is to require that sources of CE multicast traffic must be behind a router.
this document recommends the third option -- sources of traffic must be behind a router ๐
2.12. Data-Forwarding Rules ๐ฆ
first things first, we define the rules that PIM-SM and PIM-DM PEs both share ๐ each protocol's own forwarding rules are in the subsections below โจ
if there's no matching forwarding state, then the PE SHOULD discard the packet, i.e., the UserDefinedPortList (Sections 2.12.1 and 2.12.2) SHOULD be empty ๐๏ธ
these general rules MUST be followed when forwarding multicast traffic in a VPLS ๐
- traffic arriving on a port MUST NOT be forwarded back onto the same port. no boomerangs ๐ช
- due to VPLS split-horizon rules, traffic ingressing on a PW MUST NOT be forwarded to any other PW. that's giving strict bouncer energy ๐ช
2.12.1. PIM-SM Data-Forwarding Rules ๐
per the rules in RFC 7761 [PIM-SM] and per the additional rules specified in this document, we get these formulas ๐งฎ
OutgoingPortList(*,G) = immediate_olist(*,G) (+)
UpstreamPorts(*,G) (+)
Port(PimDR)
OutgoingPortList(S,G) = inherited_olist(S,G) (+)
UpstreamPorts(S,G) (+)
(UpstreamPorts(*,G) (-)
UpstreamPorts(S,G,rpt)) (+)
Port(PimDR)
RFC 7761 [PIM-SM] is the one that explains how immediate_olist(*,G) and inherited_olist(S,G) get built ๐ and PimDR is the IP address of the PIM DR in the VPLS ๐ซก
the PIM-SM snooping data-forwarding rules are written out below in pseudocode ๐ค
BEGIN
iif is the incoming port of the multicast packet.
S is the source IP address of the multicast packet.
G is the destination IP address of the multicast packet.
If there is (S,G) state on the PE
Then
OutgoingPortList = OutgoingPortList(S,G)
Else if there is (*,G) state on the PE
Then
OutgoingPortList = OutgoingPortList(*,G)
Else
OutgoingPortList = UserDefinedPortList
Endif
If iif is an AC
Then
OutgoingPortList = OutgoingPortList (-) iif
Else
## iif is a PW
OutgoingPortList = OutgoingPortList (-) PWPorts
Endif
Forward the packet to OutgoingPortList.
END
so first, if there's (S,G) state on the PE, the outgoing ports are OutgoingPortList(S,G) ๐ otherwise, if there's (,G) state on the PE, the outgoing ports are OutgoingPortList(,G) โจ
the packet gets forwarded to whichever set of outgoing ports got picked, while still following the general rules up in Section 2.12 ๐
2.12.2. PIM-DM Data-Forwarding Rules ๐ธ
the PIM-DM snooping data-forwarding rules are written out below in pseudocode ๐
BEGIN
iif is the incoming port of the multicast packet.
S is the source IP address of the multicast packet.
G is the destination IP address of the multicast packet.
If there is (S,G) state on the PE
Then
OutgoingPortList = olist(S,G)
Else
OutgoingPortList = UserDefinedPortList
Endif
If iif is an AC
Then
OutgoingPortList = OutgoingPortList (-) iif
Else
## iif is a PW
OutgoingPortList = OutgoingPortList (-) PWPorts
Endif
Forward the packet to OutgoingPortList.
END
if there's forwarding state for (S,G), then forward the packet to olist(S,G) while following the general rules above in Section 2.12 ๐ฆ
RFC 3973 [PIM-DM] is where you find out how olist(S,G) gets constructed ๐
3. IANA Considerations ๐ง
this document does not require any IANA actions โจ nothing to do here, bestie ๐ซถ
4. Security Considerations ๐ฅ
security considerations from the VPLS solution documents (i.e., RFC 4762 [VPLS-LDP] and RFC 4761 [VPLS-BGP]) apply to this document as well ๐ญ they're taking this part seriously, so should you ๐
5. References ๐
5.1. Normative References
[BIDIR-PIM] Handley, M., Kouvelas, I., Speakman, T., and L. Vicisano, "Bidirectional Protocol Independent Multicast (BIDIR-PIM)", RFC 5015, DOI 10.17487/RFC5015, October 2007, <https://www.rfc-editor.org/info/rfc5015>.
[JOIN-ATTR] Boers, A., Wijnands, I., and E. Rosen, "The Protocol Independent Multicast (PIM) Join Attribute Format", RFC 5384, DOI 10.17487/RFC5384, November 2008, <https://www.rfc-editor.org/info/rfc5384>.
[PIM-DM] Adams, A., Nicholas, J., and W. Siadak, "Protocol Independent Multicast - Dense Mode (PIM-DM): Protocol Specification (Revised)", RFC 3973, DOI 10.17487/RFC3973, January 2005, <https://www.rfc-editor.org/info/rfc3973>.
[PIM-SM] Fenner, B., Handley, M., Holbrook, H., Kouvelas, I., Parekh, R., Zhang, Z., and L. Zheng, "Protocol Independent Multicast - Sparse Mode (PIM-SM): Protocol Specification (Revised)", STD 83, RFC 7761, DOI 10.17487/RFC7761, March 2016, <https://www.rfc-editor.org/info/rfc7761>.
[PIM-SSM] Holbrook, H. and B. Cain, "Source-Specific Multicast for IP", RFC 4607, DOI 10.17487/RFC4607, August 2006, <https://www.rfc-editor.org/info/rfc4607>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RPF-VECTOR] Wijnands, IJ., Boers, A., and E. Rosen, "The Reverse Path Forwarding (RPF) Vector TLV", RFC 5496, DOI 10.17487/RFC5496, March 2009, <https://www.rfc-editor.org/info/rfc5496>.
5.2. Informative References
[IGMP-SNOOP] Christensen, M., Kimball, K., and F. Solensky, "Considerations for Internet Group Management Protocol (IGMP) and Multicast Listener Discovery (MLD) Snooping Switches", RFC 4541, DOI 10.17487/RFC4541, May 2006, <https://www.rfc-editor.org/info/rfc4541>.
[VPLS-BGP] Kompella, K., Ed., and Y. Rekhter, Ed., "Virtual Private LAN Service (VPLS) Using BGP for Auto-Discovery and Signaling", RFC 4761, DOI 10.17487/RFC4761, January 2007, <https://www.rfc-editor.org/info/rfc4761>.
[VPLS-LDP] Lasserre, M., Ed., and V. Kompella, Ed., "Virtual Private LAN Service (VPLS) Using Label Distribution Protocol (LDP) Signaling", RFC 4762, DOI 10.17487/RFC4762, January 2007, <https://www.rfc-editor.org/info/rfc4762>.
[VPLS-MCAST] Aggarwal, R., Ed., Kamite, Y., Fang, L., Rekhter, Y., and C. Kodeboniya, "Multicast in Virtual Private LAN Service (VPLS)", RFC 7117, DOI 10.17487/RFC7117, February 2014, <https://www.rfc-editor.org/info/rfc7117>.
[VPLS-MCAST-REQ] Kamite, Y., Ed., Wada, Y., Serbest, Y., Morin, T., and L. Fang, "Requirements for Multicast Support in Virtual Private LAN Services", RFC 5501, DOI 10.17487/RFC5501, March 2009, <https://www.rfc-editor.org/info/rfc5501>.
Appendix A. BIDIR-PIM Considerations ๐
this appendix has some guidelines you can use to keep BIDIR-PIM working alongside PIM snooping ๐ฆ
to preserve BIDIR-PIM snooping, routers need to set up forwarding states so that:
- on the RPL, all traffic is forwarded to all Port(N) ports.
- on any other interface, traffic is always forwarded to the DF.
the info needed to set these states up can be gotten by:
- determining the mapping between the group (range) and the RP.
- snooping and storing DF election information.
- determining where the RPL is. this could be done with static configuration or by combining the info from the two bullets above.
A.1. BIDIR-PIM Data-Forwarding Rules โจ
the BIDIR-PIM snooping data-forwarding rules are written out below in pseudocode ๐ค
BEGIN
iif is the incoming port of the multicast packet.
G is the destination IP address of the multicast packet.
If there is forwarding state for G
Then
OutgoingPortList = olist(G)
Else
OutgoingPortList = UserDefinedPortList
Endif
If iif is an AC
Then
OutgoingPortList = OutgoingPortList (-) iif
Else
## iif is a PW
OutgoingPortList = OutgoingPortList (-) PWPorts
Endif
Forward the packet to OutgoingPortList.
END
if there's forwarding state for G, then forward the packet to olist(G) while following the general rules above in Section 2.12 ๐
RFC 5015 [BIDIR-PIM] is the one that explains how olist(G) gets constructed ๐ซถ
Appendix B. Example Network Scenario โจ
Okay bestie, let's look at the example network in Figure 3 ๐ซถ
+------+ AC3 +------+
| PE2 |-----| CE3 |
/| | +------+
/ +------+ |
/ | |
/ | |
/PW12 | |
/ | /---\
/ |PW23 | S |
/ | \---/
/ | |
/ | |
/ | |
+------+ / +------+ |
+------+ | PE1 |/ PW13 | PE3 | +------+
| CE1 |-----| |-------------| |-----| CE4 |
+------+ AC1 +------+ +------+ AC4 +------+
|
|AC2
+------+
| CE2 |
+------+
Figure 3: An Example Network for Triggering an Assert ๐ฆ
In the examples below, JT(Port,S,G,N) is the downstream Join Expiry Timer on the specified Port for the (S,G) with upstream neighbor N. It's the timer keeping the whole vibe alive ๐
B.1. PIM Snooping Example โจ
In the network depicted in Figure 3, S is the source of a multicast stream (S,G). CE1 and CE2 both have two ECMP routes to reach the source. Two routes, very dramatic ๐
- CE1 sends a
Join(S,G)withUpstreamNeighbors(S,G) = CE3. - PE1 snoops on the
Join(S,G)and builds forwarding state, since it is received on an AC. It also floods theJoin(S,G)in the VPLS. PE2 snoops on theJoin(S,G)and builds forwarding state, since theJoin(S,G)is targeting a neighbor residing on an AC. PE3 does not create forwarding state for (S,G) because this is a PW-only Join and there is neither an existing (*,G) state with an AC inUpstreamPorts(*,G)nor an existing (S,G) state with an AC inUpstreamPorts(S,G). Both PE2 and PE3 will also flood theJoin(S,G)in the VPLS. PE3 is literally sitting this one out ๐
The resulting states at the PEs are as follows:
PE1 states:
JT(AC1,S,G,CE3) = JP_HoldTime
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW12 }
OutgoingPortList(S,G) = { AC1, PW12 }
PE2 states:
JT(PW12,S,G,CE3) = JP_HoldTime
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { AC3 }
OutgoingPortList(S,G) = { PW12, AC3 }
PE3 states:
No (S,G) state
- The multicast stream (S,G) flows along CE3 -> PE2 -> PE1 -> CE1. ๐
- Now CE2 sends a
Join(S,G)withUpstreamNeighbors(S,G) = CE4. - All PEs snoop on the
Join(S,G), build forwarding state, and flood theJoin(S,G)in the VPLS. Note that for PE2, even though this is a PW-only Join, forwarding state is built on thisJoin(S,G), since PE2 has an existing (S,G) state with an AC inUpstreamPorts(S,G). PE2 is the main character here, no cap โจ
The resulting states at the PEs are as follows:
PE1 states:
JT(AC1,S,G,CE3) = active
JT(AC2,S,G,CE4) = JP_HoldTime
UpstreamNeighbors(S,G) = { CE3, CE4 }
UpstreamPorts(S,G) = { PW12, PW13 }
OutgoingPortList(S,G) = { AC1, PW12, AC2, PW13 }
PE2 states:
JT(PW12,S,G,CE4) = JP_HoldTime
JT(PW12,S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3, CE4 }
UpstreamPorts(S,G) = { AC3, PW23 }
OutgoingPortList(S,G) = { PW12, AC3, PW23 }
PE3 states:
JT(PW13,S,G,CE4) = JP_HoldTime
UpstreamNeighbors(S,G) = { CE4 }
UpstreamPorts(S,G) = { AC4 }
OutgoingPortList(S,G) = { PW13, AC4 }
- The multicast stream (S,G) flows into the VPLS from two of the CEs, CE3 and CE4. PE2 forwards the stream received from CE3 to PW23, and PE3 forwards the stream to AC4. This helps the CE routers to trigger Assert election. Let us say that CE3 becomes the Assert winner. Drama unlocked ๐ฅ
- CE3 sends an Assert message to the VPLS. The PEs flood the Assert message without examining it.
- CE4 stops sending the multicast stream to the VPLS.
- CE2 notices an RPF change due to the Assert and sends a
Prune(S,G)with upstream neighbor = CE4. CE2 also sends aJoin(S,G)with upstream neighbor = CE3. ๐ - All the PEs start a Prune-Pending timer on the ports on which they received the
Prune(S,G). When the Prune-Pending timer expires, all PEs will remove the downstream (S,G,CE4) states.
The resulting states at the PEs are as follows:
PE1 states:
JT(AC1,S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW12 }
OutgoingPortList(S,G) = { AC1, AC2, PW12 }
PE2 states:
JT(PW12,S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { AC3 }
OutgoingPortList(S,G) = { PW12, AC3 }
PE3 states:
JT(PW13,S,G,CE3) = JP_HoldTime
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW23 }
OutgoingPortList(S,G) = { PW13, PW23 }
Note that at this point at PE3, since there is no AC in OutgoingPortList(S,G) and no (*,G) or (S,G) state with an AC in UpstreamPorts(*,G) or UpstreamPorts(S,G), respectively, the existing (S,G) state at PE3 can also be removed. So, finally:
PE3 states:
No (S,G) state
Note that at the end of the Assert election, there should be no duplicate traffic forwarded downstream, and traffic should flow only on the desired path. Also note that there are no unnecessary (S,G) states on PE3 after the Assert election. Clean state, clean vibes ๐
B.2. PIM Proxy Example with (S,G) / (*,G) Interaction ๐ฆ
okay so this is the spicy one bestie ๐ we're keeping the same network, but this time CE4 is the upstream neighbor towards the RP for G ๐
JPST(S,G,N) is the JP sending timer for the (S,G) with upstream neighbor N. โฐ
- CE1 sends a Join(S,G) with UpstreamNeighbors(S,G) = CE3. ๐ธ
- PE1 consumes the Join(S,G) and builds forwarding state, since the Join(S,G) is received on an AC. โจ
PE2 consumes the Join(S,G) and builds forwarding state, since the Join(S,G) is targeting a neighbor residing on an AC. ๐
PE3 consumes the Join(S,G) but does not create forwarding state for (S,G), since this is a PW-only Join and there is neither an existing (,G) state with an AC in UpstreamPorts(,G) nor an existing (S,G) state with an AC in UpstreamPorts(S,G). ๐ not me PE3 being like "nah, I'm good" lol
The resulting states at the PEs are as follows:
PE1 states:
JT(AC1,S,G,CE3) = JP_HoldTime
JPST(S,G,CE3) = t_periodic
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW12 }
OutgoingPortList(S,G) = { AC1, PW12 }
PE2 states:
JT(PW12,S,G,CE3) = JP_HoldTime
JPST(S,G,CE3) = t_periodic
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { AC3 }
OutgoingPortList(S,G) = { PW12, AC3 }
PE3 states:
No (S,G) state
Joins are triggered as follows:
PE1 triggers a Join(S,G) targeting CE3. Since the Join(S,G) was received on an AC and is targeting a neighbor that is residing across a PW, the triggered Join(S,G) is sent on all PWs. ๐
PE2 triggers a Join(S,G) targeting CE3. Since the Join(S,G) is targeting a neighbor residing on an AC, it only sends the Join on AC3. ๐ฏ
PE3 ignores the Join(S,G), since this is a PW-only Join and there is neither an existing (,G) state with an AC in UpstreamPorts(,G) nor an existing (S,G) state with an AC in UpstreamPorts(S,G). ๐ bye bye
- The multicast stream (S,G) flows along CE3 -> PE2 -> PE1 -> CE1. ๐๐
- Now let us say that CE2 sends a Join(,G) with UpstreamNeighbors(,G) = CE4. ๐
- PE1 consumes the Join(,G) and builds forwarding state, since the Join(,G) is received on an AC. โจ
PE2 consumes the Join(,G); although this is a PW-only Join, forwarding state is built on this Join(,G), since PE2 has an existing (S,G) state with an AC in UpstreamPorts(S,G). ๐ซถ However, since this is a PW-only Join, PE2 only adds the PW towards PE3 (PW23) into UpstreamPorts(,G) and hence into OutgoingPortList(,G). It does not add the PW towards PE1 (PW12) into OutgoingPortList(*,G). ๐ โโ๏ธ lowkey so precise
PE3 consumes the Join(,G) and builds forwarding state, since the Join(,G) is targeting a neighbor residing on an AC. ๐
The resulting states at the PEs are as follows:
PE1 states:
JT(AC1,*,G,CE4) = JP_HoldTime
JPST(*,G,CE4) = t_periodic
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(*,G) = { PW13 }
OutgoingPortList(*,G) = { AC2, PW13 }
JT(AC1,S,G,CE3) = active
JPST(S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW12 }
OutgoingPortList(S,G) = { AC1, PW12, PW13 }
PE2 states:
JT(PW12,*,G,CE4) = JP_HoldTime
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(G) = { PW23 }
OutgoingPortList(*,G) = { PW23 }
JT(PW12,S,G,CE3) = active
JPST(S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { AC3 }
OutgoingPortList(S,G) = { PW12, AC3, PW23 }
PE3 states:
JT(PW13,*,G,CE4) = JP_HoldTime
JPST(*,G,CE4) = t_periodic
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(*,G) = { AC4 }
OutgoingPortList(*,G) = { PW13, AC4 }
Joins are triggered as follows:
PE1 triggers a Join(,G) targeting CE4. Since the Join(,G) was received on an AC and is targeting a neighbor that is residing across a PW, the triggered Join(S,G) is sent on all PWs. ๐ฃ
PE2 does not trigger a Join(*,G) based on this Join, since this is a PW-only Join. ๐ซ
PE3 triggers a Join(,G) targeting CE4. Since the Join(,G) is targeting a neighbor residing on an AC, it only sends the Join on AC4. ๐
- If traffic is not flowing yet (i.e., step 3 is delayed so that it occurs after step 6) and in the interim JPST(S,G,CE3) on PE1 expires, causing it to send a refresh Join(S,G) targeting CE3, since the refresh Join(S,G) is targeting a neighbor that is residing across a PW, the refresh Join(S,G) is sent on all PWs. โณ this is the "wait, what if" moment ๐ญ
- Note that PE1 refreshes its JT based on reception of refresh Joins from CE1 and CE2. ๐
PE2 consumes the Join(S,G) and refreshes the JT(PW12,S,G,CE3) timer. ๐ฅฐ
PE3 consumes the Join(S,G). It also builds forwarding state on this Join(S,G), even though this is a PW-only Join, since now PE2 has an existing (,G) state with an AC in UpstreamPorts(,G). ๐ง However, since this is a PW-only Join, PE3 only adds the PW towards PE2 (PW23) into UpstreamPorts(S,G) and hence into OutgoingPortList(S,G). It does not add the PW towards PE1 (PW13) into OutgoingPortList(S,G). ๐
PE3 states:
JT(PW13,*,G,CE4) = active
JPST(S,G,CE4) = active
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(*,G) = { AC4 }
OutgoingPortList(*,G) = { PW13, AC4 }
JT(PW13,S,G,CE3) = JP_HoldTime
UpstreamNeighbors(*,G) = { CE3 }
UpstreamPorts(*,G) = { PW23 }
OutgoingPortList(*,G) = { PW13, AC4, PW23 }
Joins are triggered as follows:
PE2 already has (S,G) state, so it does not trigger a Join(S,G) based on reception of this refresh Join. ๐
PE3 does not trigger a Join(S,G) based on this Join, since this is a PW-only Join. ๐
- The multicast stream (S,G) flows into the VPLS from two of the CEs -- CE3 and CE4. โก PE2 forwards the stream received from CE3 to PW12 and PW23. At the same time, PE3 forwards the stream received from CE4 to PW13 and PW23. ๐
The stream received over PW12 and PW13 is forwarded by PE1 to AC1 and AC2. ๐ซ
The stream received by PE3 over PW23 is forwarded to AC4. The stream received by PE2 over PW23 is forwarded to AC3. Either of these helps the CE routers to trigger Assert election. ๐ณ๏ธ
- CE3 and/or CE4 send(s) Assert message(s) to the VPLS. The PEs flood the Assert message(s) without examining it. ๐จ
- CE3 becomes the (S,G) Assert winner, and CE4 stops sending the multicast stream to the VPLS. ๐ winner winner
- CE2 notices an RPF change due to the Assert and sends a Prune(S,G,rpt) with upstream neighbor = CE4. ๐ฅฒ
- PE1 consumes the Prune(S,G,rpt), and since PruneDesired(S,G,Rpt,CE4) is TRUE, it triggers a Prune(S,G,rpt) to CE4. Since the Prune is targeting a neighbor across a PW, it is sent on all PWs. โ๏ธ
PE2 consumes the Prune(S,G,rpt) and does not trigger any Prune based on this Prune(S,G,rpt), since this was a PW-only Prune. ๐
PE3 consumes the Prune(S,G,rpt), and since PruneDesired(S,G,rpt,CE4) is TRUE, it sends the Prune(S,G,rpt) on AC4. ๐
PE1 states:
JT(AC2,*,G,CE4) = active
JPST(*,G,CE4) = active
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(*,G) = { PW13 }
OutgoingPortList(*,G) = { AC2, PW13 }
JT(AC2,S,G,CE4) = JP_HoldTime with S,G,rpt prune flag
JPST(S,G,CE4) = none, since this is sent along
with the Join(*,G) to CE4 based
on JPST(*,G,CE4) expiry
UpstreamPorts(S,G,rpt) = { PW13 }
UpstreamNeighbors(S,G,rpt) = { CE4 }
JT(AC1,S,G,CE3) = active
JPST(S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW12 }
OutgoingPortList(S,G) = { AC1, PW12, AC2 }
PE2 states:
JT(PW12,*,G,CE4) = active
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(*,G) = { PW23 }
OutgoingPortList(*,G) = { PW23 }
JT(PW12,S,G,CE4) = JP_HoldTime with S,G,rpt prune flag
JPST(S,G,CE4) = none, since this was created
off a PW-only Prune
UpstreamPorts(S,G,rpt) = { PW23 }
UpstreamNeighbors(S,G,rpt) = { CE4 }
JT(PW12,S,G,CE3) = active
JPST(S,G,CE3) = active
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { AC3 }
OutgoingPortList(*,G) = { PW12, AC3 }
PE3 states:
JT(PW13,*,G,CE4) = active
JPST(*,G,CE4) = active
UpstreamNeighbors(*,G) = { CE4 }
UpstreamPorts(*,G) = { AC4 }
OutgoingPortList(*,G) = { PW13, AC4 }
JT(PW13,S,G,CE4) = JP_HoldTime with S,G,rpt prune flag
JPST(S,G,CE4) = none, since this is sent along
with the Join(*,G) to CE4 based
on JPST(*,G,CE4) expiry
UpstreamNeighbors(S,G,rpt) = { CE4 }
UpstreamPorts(S,G,rpt) = { AC4 }
JT(PW13,S,G,CE3) = active
JPST(S,G,CE3) = none, since this state is
created by a PW-only Join
UpstreamNeighbors(S,G) = { CE3 }
UpstreamPorts(S,G) = { PW23 }
OutgoingPortList(S,G) = { PW23 }
Even in this example, at the end of the (S,G) / (*,G) Assert election, there should be no duplicate traffic forwarded downstream, and traffic should flow only to the desired CEs. ๐ฅ slay
However, we don't have duplicate traffic because one of the CEs stops sending traffic due to the Assert, not because we don't have any forwarding state in the PEs to do this forwarding. ๐ the PEs are NOT the reason, the Assert is the hero of this story ๐ฆธโโ๏ธ
Acknowledgements ๐ซถ
Many members of the former L2VPN and PIM working groups have contributed to, and provided valuable comments and feedback on, this document, including Vach Kompella, Shane Amante, Sunil Khandekar, Rob Nath, Marc Lasserre, Yuji Kamite, Yiqun Cai, Ali Sajassi, Jozef Raets, Himanshu Shah (Ciena), and Himanshu Shah (Cisco). ๐ the whole squad showed up for this one
Contributors ๐
Yetik Serbest and Suresh Boddapati coauthored earlier draft versions of this document. โ๏ธ
Karl (Xiangrong) Cai and Princy Elizabeth made significant contributions to bring the specification to its current state, especially in the area of Join forwarding rules. ๐ the Join forwarding rules were literally carried by these two
Authors' Addresses ๐ฌ
Olivier Dornon
Nokia
Copernicuslaan 50
B-2018 Antwerp
Belgium
Email: olivier.dornon@nokia.com
Jayant Kotalwar
Nokia
701 East Middlefield Rd.
Mountain View, CA 94043
United States of America
Email: jayant.kotalwar@nokia.com
Venu Hemige
Nokia
Email: vhemige@gmail.com
Ray Qiu
mistnet.io
Email: ray@mistnet.io
Jeffrey Zhang
Juniper Networks, Inc.
10 Technology Park Drive
Westford, MA 01886
United States of America
Email: zzhang@juniper.net
๐ฆ where this page comes from
- ietf: rfc8220 2017-09
- ๐ girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms