RFC 2121

Issues affecting MARS Cluster Size 🦋

okay so this one is RFC 2121 💖, and it's written by G. Armitage from Bellcore, dated March 1997. It's in the Informational category, which means it's giving information for the Internet community, not a standard 🎀. The title block doesn't list anything it obsoletes or updates, so there's no drama there ✨.

Status of this Memo

This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 📜

Abstract

okay bestie, so IP multicast over ATM uses the MARS model [1] to manage ATM pt-mpt SVCs for IP multicast packet forwarding 🧚. The scope of any MARS service is the MARS Cluster, which is usually the same as an IPv4 Logical IP Subnet (LIS) 🌸. Current IP/ATM networks are usually built around unicast routing and forwarding, and those issues are what set how big each LIS gets. But as IP multicast becomes a real service, a LIS can only be as big as its MARS Cluster can be 👑.

This doc takes a qualitative look at what's limiting a MARS Cluster's size. That includes the impact of VC limits in switches and NICs, where cluster members are located geographically, and whether VC Mesh or MCS modes are used to support multicast groups 💅.

1. Introduction

A MARS Cluster is the set of IP/ATM interfaces that are willing to engage in direct, ATM level pt-mpt SVCs to perform IP multicast packet forwarding [1] 🦋. Each IP/ATM interface (a MARS Client) has to keep state information about the ATM addresses of each leaf node (recipient) of each pt-mpt SVC it has open 📝. On top of that, each MARS Client gets MARS_JOIN and MARS_LEAVE messages from the MARS whenever Clients around the Cluster need to update their pt-mpt SVCs for a given IP multicast group 🫶.

"Cluster size" can mean two things, bestie: how many MARS Clients use a given MARS, and how far apart those MARS Clients are geographically 🌍. The number of MARS Clients affects how much state any single client might have to store while it manages its outgoing pt-mpt SVCs 💜. It also affects the average rate of JOIN/LEAVE traffic that the MARS passes along on ClusterControlVC, plus how many pt-mpt VCs might need changing each time a MARS_JOIN or MARS_LEAVE shows up on ClusterControlVC 🔥.

Where the clients are located affects the latency between a client sending a MARS_JOIN and that client finally being added onto the pt-mpt VCs of the other MARS Clients sending to that multicast group ⏱️. (This latency is made up of both the time to propagate the MARS_JOIN, and the delay in the underlying ATM cloud's reaction to the ensuing ADD_PARTY messages 😭.)

When you're building an IP/ATM network, it's really important to understand the worst case scaling limits for your Clusters 💀. This doc is mostly a qualitative look at the design choices that put the biggest constraints on Cluster size 🍓. Because the focus is on worst-case scenarios, most of the analysis assumes multicast groups that are VC Mesh based and have every cluster member as both a source and a receiver 👯. Engineering against those worst-case boundary conditions, and then applying optimisations like Multicast Servers (MCS), gives the Cluster a margin of safety 🛡️. The hope is that this document will inspire more detailed, quantitative analysis of Cluster sizing limits 🌈.

Here's the roadmap: Section 2 talks about the VC state requirements of the MARS model. Sections 3 and 4 cover the group change processing load and latency characteristics of a cluster as a function of its size 📊. Section 5 looks at how Multicast Routers (both conventional and combination router/switch architectures) increase the scale of a multicast capable IP/ATM network 🚀. Finally, Section 6 discusses how using Multicast Servers (MCS) might affect the worst case Cluster size limits ✨.

2. VC state limitations.

Two characteristics of ATM NICs and switches will limit how many members a Cluster can contain 🎀. They are:

• The maximum number of VCs that can be originated from, or terminate on, a port (VCmax).
• The maximum number of leaf nodes supportable by a root node (LEAFmax).

We'll assume the MARS node has similar VCmax and LEAFmax values to Cluster members 💅. VCmax affects Cluster size for the following reasons:

• The MARS terminates a pt-pt control VC from each cluster member, and originates a VC for ClusterControlVC and ServerControlVC.
• When a multicast group is VC Mesh based, a group member terminates a VC from every sender to the group, per group.
• When a multicast group is MCS based, the MCS terminates a VC from every sender to the group.

LEAFmax affects Cluster size for these reasons:

• ClusterControlVC from the MARS. It has a leaf node per cluster member (MARS Client).
• Packet forwarding SVCs out of each MARS Client for each IP multicast group being sent to. It has a leaf node for each group member when a group is VC Mesh based.
• Packet forwarding SVCs out of each MCS for each IP multicast group being sent to. It has a leaf node for each group member when a group is MCS based.

Say we have N cluster members and M multicast groups active (using VC Mesh mode, and densely populated, meaning all receivers are senders) 🧁. Here's what we can observe:

ClusterControlVC has N leaf nodes, so:

N <= LEAFmax.

The MARS terminates a pt-pt VC from each cluster member, and originates ClusterControlVC and ServerControlVC, so:

(N+2) <= VCmax.

Each Cluster Member sources 1 VC per group, terminates (N-1) VC per group, originates a pt-pt VC to the MARS, and terminates 1 VC as a leaf on ClusterControlVC, so:

(M*N) + 2 <= VCmax.

The VC sourced by each Cluster member per group goes to all other cluster members, so:

(N-1) <= LEAFmax.

All of the above conditions have to be true at the same time, so the most constraining requirement is either:

(M*N) + 2 <= VCmax.

or

N <= LEAFmax.

🙈 The limit involving VCmax is basically controlled by the VC consumption of group members using a VC Mesh for data forwarding, rather than by the termination of pt-pt control VCs on the MARS. (In practice, it's going to depend a lot on how multicast group membership is distributed within the cluster.)

The LEAFmax limit comes from ClusterControlVC, and it doesn't depend on how dense group membership is, or on the ratios of senders to receivers, for active multicast groups within the cluster 🌸.

Under UNI 3.0/3.1, the most obvious limit on LEAFmax is 2^15, because the leaf node ID is 15 bits wide 🔢. But the signaling driver software for most ATM NICs may impose a much lower limit. That depends on how much per-leaf node state information they need to store (and are able to store) for pt-mpt SVCs 😩.

VCmax is constrained either by the ATM NIC hardware (for available segmentation or reassembly instances), or by the VC capacity of the switch port the NIC is attached to. Whichever one is smaller is the VCmax 💅.

A MARS Client can also impose its own state storage limitations. This means the combined memory use of a MARS Client and the ATM NIC's driver in a given host can push both LEAFmax and VCmax lower than the ATM NIC alone might have been able to support 🥺.

It may be possible to work around LEAFmax limits by spreading the leaf nodes across multiple pt-mpt SVCs running in parallel. However, that approach needs further study, and it doesn't solve the VCmax limitation of a node terminating too many VCs 🙃.

There's also a related observation: the number of MARS Clients in a Cluster may be limited by the MARS's own memory constraints. The MARS is required to keep state on all the groups that each one of its MARS Clients has joined 🧠. For a given memory limit, the maximum number of MARS Clients has to drop if the average number of groups joined per Client goes up. Depending on the level of group memberships, this limitation may be more severe than LEAFmax 🫠.

3. Signaling load.

In any given cluster there will be an 'ambient' level of MARS_JOIN/LEAVE activity 🌊. The dynamic characteristics of this activity depend on the types of multicast applications running within the cluster 🎮. For a constant relative distribution of multicast applications, we can assume that as the number of MARS Clients in a given cluster goes up, so does the ambient level of MARS_JOIN/LEAVE activity 📈. That increases how often the MARS processes and propagates MARS_JOIN/LEAVE messages 🏃‍♀️.

MARS_JOIN/LEAVE traffic also has a knock-on effect on signaling activity at the ATM level, across the UNI and {P}NNI boundaries 🌐. For groups that are VC Mesh supported, each MARS_JOIN or MARS_LEAVE propagated on ClusterControlVC results in an ADD_PARTY or DROP_PARTY message sent across the UNIs of all MARS Clients that are transmitting to a given group 📣. As a cluster's membership goes up, so does the average number of MARS Clients that trigger ATM signaling activity in response to MARS_JOIN/LEAVEs 💥.

The size of a cluster needs to be chosen to provide some level of containment for this ambient level of MARS and UNI/NNI signaling. This is important, so don't skip it 🚨.

Some refinements to MARS Client behaviour may also be explored to smooth out UNI signaling transients 🧈. MARS Clients are currently required to start revalidating group memberships only when the Client next sends a packet to an invalidated group SVC. A Client could apply a similar algorithm to decide when it should issue ADD_PARTYs 🤔. For example, after seeing a MARS_JOIN, it could wait until it actually has a packet to send, send the packet, and then initiate the ADD_PARTY. As a result, actively transmitting Clients would update their SVCs sooner than intermittently transmitting Clients ⭐.

4. Group change latencies 🕰️✨

okay so group change latency is basically how long it takes for every sender in a group to update their forwarding SVCs after the MARS sends out a MARS_JOIN or MARS_LEAVE. 💅 It's shaped by two things: how many Cluster members there are, and how spread out they are geographically. Groups that are MCS based get the lowest impact when people join or leave, because only the MCS has to update its forwarding SVC. Lowkey genius. 🧚 Sometimes, especially in modelling or simulation environments, keeping group change latencies inside a cluster under control is a really important thing to care about. 🔥

As we talked about in the last section, the ADD_PARTY/DROP_PARTY signaling load from membership changes in VC Mesh based groups keeps climbing as the number of cluster members goes up. 📈 This assumes the worst case, where every single cluster member is also a sender to the group. The more the UNI load grows, the more the ATM network itself can start processing those requested events more slowly. 😭

Wide geographic spread is also a delay problem. 🌍 It slows down how fast MARS_JOIN/MARS_LEAVE traffic and ATM UNI/NNI messages travel between members. The further apart everyone is, the longer it takes for them to receive MARS_JOIN/MARS_LEAVE traffic on ClusterControlVC, and the longer the ATM network takes to react to ADD_PARTY and DROP_PARTY requests. If those long distance paths go through lots of ATM switches, the per-switch processing delays add a ton on top of the speed of light delays. 🐢

⚠️ Heads up bestie: the smoothing mechanisms for the transient ATM signaling load from section 3 come with a catch. They make group change latency go up, because the whole point is that some senders deliberately wait before updating their forwarding SVCs. 😩 This is a situation-specific trade-off, so the system architect has to make the call. 💜

It's not clear how much the MARS's own internal processing affects group change latency, or how cluster size changes that. 🤷‍♀️ Part of the MARS processing latency depends on the specific database implementation and search algorithms, and also on how many group members the group being modified has at any given instant. The maximum number of group members in a given group is equal to the number of cluster members. So there's an indirect link (even if it's a small one) between worst case MARS processing latencies and cluster size. 🎀

---

5. Large IP/ATM networks using Mrouters 🌐🦋

Building a big multicast capable IP over ATM network is a tradeoff between Cluster sizes and the number of Mrouters. 💖 For a fixed number of hosts, the number of clusters goes up as individual clusters get smaller. And since Mrouters are the topological intersections between clusters, the number of Mrouters goes up as clusters shrink too. 📊 How many Mrouters you actually end up with depends mostly on the logical IP topology you pick, since one physical Mrouter can interconnect more than two Clusters at once. 🧠 The best mix of Clusters and Mrouters is a local deployment question, so you get to decide what works for your setup. 🫶

Right now there are two broad classes of Mrouters you can spot:

• Those that originate unique VCs into target Clusters, and forward/interleave data at the IP packet level (the Conventional Mrouter).
• Those that originate unique VCs into target Clusters, but create internal, cell level 'cut through' paths between VCs from different Clusters (e.g. the Cell Switch Router).

How these Mrouters set up and manage the associations between VCs and IP traffic flows is out of scope for this document. 🙈 Still, it's worth taking a quick look at how they affect VC consumption and ATM signaling load. 👀

5.1 Impact of the Conventional Mrouter 🧺

A conventional Mrouter is an aggregation point for both signaling and data plane loads. 💅 It hides host specific group membership changes in one cluster from senders in other clusters. It also protects group members (receivers) in one cluster from having to be leaf nodes on SVCs from senders in other Clusters. Iconic protection. 👑

When a conventional Mrouter acts as an ingress point into a cluster, it sets up a single forwarding SVC for IP packets. ✨ That one SVC carries data from other clusters, interleaved at the IP packet level. Only this single SVC has to change when group memberships change inside the target cluster. So sources in other clusters don't need to know about, or react to, MARS_JOIN/MARS_LEAVE traffic in the target cluster. 🦋 The UNI signaling load from section 3 stays localized inside the target Cluster too. 🫧

MARS Clients in the target cluster get a win from this data path aggregation as well. 🌸 They only terminate one SVC from the Mrouter (per group), instead of multiple SVCs coming from actual senders in other Clusters. Way less to manage. 😌

Conventional Mrouters help control the limiting factors from sections 2, 3, and 4. 🎀 Take a hypothetical 10000 node Cluster. It could be split into two 5000 node Clusters, or four 2500 node Clusters, and so on, to cut down on VC consumption. Or you could notice that 200 nodes out of the 10000 are known to join and leave groups rapidly, while the other 9800 are pretty steady. Then you deploy clusters of 200, 2500, 2500, 2500, and 2300 hosts respectively. Big brain move. 🧠💖

5.2. Impact of the Cell Switch Router (CSR). 🔥

Another class of Mrouter is the Cell Switch Router (CSR). It tries to use IP level flow information to dynamically manage how data gets switched through the device below the IP level. 🌈 Once the CSR has identified a flow of IP traffic and linked it with an inbound and an outbound SVC, it starts working as an ATM cell level device instead of a packet level device. Main character energy. 👑

Even in this mode, the CSR still isolates attached Clusters from each other's MARS_JOIN/LEAVE activities, the same way a conventional Mrouter does. 🙌 That's because the CSR manages its forwarding SVCs just like a normal MARS Client. It responds to MARS_JOIN/MARS_LEAVE messages within the target cluster by updating the pt-mpt trees rooted on its own ATM ports. 🧚

Here's the tea though. ☕ AAL5 AAL_SDUs can't be interleaved at the cell level on a single SVC. So a CSR can't do cell level cut-through and also aggregate IP packet flows from multiple senders onto a single SVC into a target Cluster at the same time. 😭 That means the CSR has to build a separate forwarding SVC into a target cluster for each SVC it is a leaf of in a source Cluster. This makes sure cells from individual sources don't get interleaved before they reach the re-assembly engines of the group members in the target cluster. Protecting the reassembly, periodt. 🛡️💜

Now the wild part. 🙃 The UNI signaling load the CSR puts on the target Cluster can be bigger than a conventional Mrouter's. If there are N senders in the source Cluster, the CSR ends up having built N identical pt-mpt SVCs out to the group members in the target Cluster. When a new MARS_JOIN is issued in the target Cluster, the CSR has to issue N ADD_PARTYs to update those N SVCs into the target Cluster. A conventional Mrouter, in the same situation, would only issue one ADD_PARTY for its single SVC into the target Cluster. 💀

So, without internal cut-through forwarding that keeps the AAL_SDU boundaries intact, the CSR only gives you isolation of MARS_JOIN/LEAVE traffic within clusters. 🥺 It can't give you the data path aggregation that a conventional Mrouter does. Not me making the trade-off sound this dramatic. 😤🎀

6. The impact of Multicast Servers (MCSs) ✨

okay so this doc is mostly about worst-case scenarios, which means most of the analysis assumes VC Mesh based groups where every cluster member is both a source and a receiver 🫶 an MCS can be a total game changer for one group's resource use, but it's less dramatic when you zoom out to overall cluster size limits 💅

the per-group, intra-cluster effect of an MCS is kinda like the inter-cluster effect of a conventional Mrouter 🦋 the MCS aggregates the data flows (only 1 SVC terminates on each group member, no matter how many senders there are) and isolates MARS_JOIN/LEAVE traffic, which gets shifted over to ServerControlVC instead of ClusterControlVC 💜 the UNI signaling traffic and load go down too, because only the forwarding SVC coming out of the MCS has to be modified for every membership change in the MCS supported group. literally so efficient 😭

mixing MCS based and VC Mesh based groups will definitely improve resource utilization 🌈 but how much better it gets (and so how big the cluster can actually be) depends a lot on how your typical apps behave and which limits from sections 2, 3, and 4 are your main bottlenecks 🧚

for example, if VCmax or LEAFmax (section 2) are your main problem, keep in mind that each MCS has the same NIC limits as the MARS and MARS Clients 💀 using an MCS dramatically cuts the number of VCs per MARS Client per group, but each MCS still has to terminate 1 SVC per sender, so potentially up to 1 SVC from each Cluster member. (This may become 1 SVC per member per group if the MCS supports multiple groups at once.) 🙈

let's imagine a Cluster where every group is MCS based, each MCS supports only one group, and both VCmax and LEAFmax apply equally to MCS nodes and to MARS and MARS Client nodes 🎀 say we have N cluster members, M groups, and all receivers are senders for a given MCS supported group. these are the observations:

Each MCS forwarding SVC has N leaf nodes, so:

N <= LEAFmax.

Each MCS terminates an SVC from N senders, originates 1 SVC forwarding path, originates a pt-pt control SVC to the MARS, and terminates 1 SVC as a leaf on ServerControlVC, so:

N + 3 <= VCmax.

MARS ClusterControlVC has N leaf nodes, so:

N <= LEAFmax.

MARS ServerControlVC has M leaf nodes, so:

M <= LEAFmax.

The MARS terminates a pt-pt VC from each cluster member, a pt-pt VC from each MCS, originates ClusterControlVC, and originates ServerControlVC, so:

N + M + 2 <= VCmax.

Each Cluster Member sources 1 VC per group, terminates 1 VC per group, originates a pt-pt VC to the MARS, and terminates 1 VC as a leaf on ClusterControlVC, so:

2*M + 2 <= VCmax.

all of the above have to be true at the same time 🔥 so the most constraining requirements are:

N + M + 2 <= VCmax   (if M <= N)

2*M + 2 <= VCmax     (if M >= N)

or

N <= LEAFmax.

(assuming that in general M+2 > 3, so the VCmax constraint at each MCS isn't a limiting factor 👑)

to get a feel for how VC Mesh groups and MCS based groups compare, picture a cluster where M1 is the number of VC Mesh based groups and M2 is the number of MCS based groups. again we assume worst case group density (all N cluster members are group members, and all receivers are also senders) 💖

as noted in section 2, the VCmax constraint in VC Mesh mode comes from each MARS Client, and it's:

N*M1 <= VCmax - 2

for the MCS case there are two scenarios: M2 <= N and M2 >= N.

if M2 <= N, the VC consumption by VC Mesh based groups becomes the constraint on cluster size N when:

N + M2 <= N*M1

i.e.

M1 >= 1 + (M2/N)

so if there's more than 1 VC Mesh based group and fewer MCS based groups than cluster members (M2 < N), the cluster size constraint is set by the VC Mesh characteristics: N*M1 <= VCmax - 2. (If M2 == N, then there may be 2 VC Mesh based groups before the VC Mesh characteristics become the deciding factor 🫠)

now, if M2 > N (more MCS based groups, and so more MCSes, than cluster members) the math gets messier 🌀 in this case VCmax at the MARS Client is the limiting parameter for both the VC Mesh and MCS cases. the limit becomes:

N*M1 + 2*M2 <= VCmax - 2

that said, on face value this is kinda an odd situation anyway, since it implies more MCS entities than hosts or router interfaces in the cluster (given the assumption of one group per MCS) 🙈

the impact of MCS entities that support multiple groups at once is left for future study 🔮

---

7. Open Issues 🧐

there's a wide range of qualitative analysis you can pull out of typical MARS deployment scenarios ✨ this document doesn't try to build any numerical models for VC consumption, end to end latencies, etc. that's a whole other era bestie 💅

8. Conclusion 👑

okay recap time 🎀 this doc gave a high level, qualitative overview of the parameters that affect how big MARS Clusters can get. the limits on the number of leaf nodes a pt-mpt SVC can support, the sizes of the MARS database, the propagation delays of MARS and UNI messages, and how often MARS and UNI control messages get sent are all identified as things that will constrain Clusters 🌸 conventional Mrouters are useful aggregators of IP multicast traffic and signaling information 🦋 Cell Switch Routers only offer some of the aggregation attributes of conventional Mrouters 🙈 large scale IP multicasting over ATM needs a combo of Mrouters and appropriately sized MARS Clusters 🧚 and finally, it's been shown that in a simple cluster with fewer MCS based groups than cluster members, two or more VC Mesh based groups are enough to make Multicast Servers irrelevant to the worst case cluster size limit 🔥😭

Security Considerations 🔐

Security issues are not discussed in this memo. 💀

Acknowledgments 🫶

Thanks must go to Rajesh Talpade (Georgia Tech) for specific input on aspects of the VC Mesh vs MCS tradeoffs, and Joel Halpern (Newbridge) for general input on the document's focus. 🌈

Author's Address 📮

Grenville Armitage
Bellcore, 445 South Street
Morristown, NJ, 07960
USA

EMail: gja@thumper.bellcore.com
Phone +1 201 829 2635

References 📚

[1] Armitage, G., "Support for Multicast over UNI 3.0/3.1 based ATM Networks.", Bellcore, RFC 2022, November 1996.