RFC 5439
An Analysis of Scaling Issues in MPLS-TE Core Networks
RFC 5439: An Analysis of Scaling Issues in MPLS-TE Core Networks 🎀
okay bestie, this one is about MPLS-TE (traffic engineered Multiprotocol Label Switching) and whether it can actually handle the huge networks providers want to build 🌐✨
Title block: it's an Informational RFC from February 2009 📅. The authors are S. Yasukawa (NTT), A. Farrel (Old Dog Consulting), and O. Komolafe (Cisco Systems) 🫶. It came out of the Network Working Group, and its header doesn't list any RFCs it obsoletes or updates, so it stands on its own 💅
Status of This Memo 📝
This memo gives information to the Internet community. It is not an Internet standard of any kind, and it's free to distribute with no limits 🦋
Copyright Notice ©️
Copyright (c) 2009 IETF Trust and the people listed as document authors. All rights reserved 💜
This document falls under BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as they stand on the day it was published. Read those carefully, because they spell out your rights and restrictions for this document 📜
Abstract 🌸
Providers are planning to grow their MPLS-TE core networks, so they need to know whether the current protocols and implementations can handle the network sizes they have in mind 👀
This document looks at some of the scaling worries around the number of Label Switching Paths (LSPs) in MPLS-TE core networks. It also checks how well two techniques, LSP hierarchies and multipoint-to-point LSPs, help with scaling 🔥 The goal is to push for better deployment techniques and protocol extensions so MPLS-TE can work in big networks 🚀
This document only looks at scalability for point-to-point MPLS-TE LSPs. Point-to-multipoint MPLS-TE LSPs are for future study 🧚
Table of Contents 🗂️
1. Introduction ....................................................3
1.1. Overview ...................................................3
1.2. Glossary of Notation .......................................5
2. Issues of Concern for Scaling ...................................5
2.1. LSP State ..................................................5
2.2. Processing Overhead ........................................6
2.3. RSVP-TE Implications .......................................6
2.4. Management .................................................7
3. Network Topologies ..............................................8
3.1. The Snowflake Network Topology .............................9
3.2. The Ladder Network Topology ...............................11
3.3. Commercial Drivers for Selected Configurations ............14
3.4. Other Network Topologies ..................................15
4. Required Network Sizes .........................................16
4.1. Practical Numbers .........................................16
5. Scaling in Flat Networks .......................................16
5.1. Snowflake Networks ........................................17
5.2. Ladder Networks ...........................................18
6. Scaling Snowflake Networks with Forwarding Adjacencies .........22
6.1. Two-Layer Hierarchy .......................................22
6.1.1. Tuning the Network Topology to Suit the
Two-Layer Hierarchy ................................23
6.2. Alternative Two-Layer Hierarchy ...........................24
6.3. Three-Layer Hierarchy .....................................25
6.4. Issues with Hierarchical LSPs .............................26
7. Scaling Ladder Networks with Forwarding Adjacencies ............27
7.1. Two-Layer Hierarchy .......................................27
7.2. Three-Layer Hierarchy .....................................28
7.3. Issues with Hierarchical LSPs .............................29
8. Scaling Improvements through Multipoint-to-Point LSPs ..........30
8.1. Overview of MP2P LSPs .....................................30
8.2. LSP State: A Better Measure of Scalability ................31
8.3. Scaling Improvements for Snowflake Networks ...............32
8.3.1. Comparison with Other Scenarios ....................33
8.4. Scaling Improvements for Ladder Networks ..................34
8.4.1. Comparison with Other Scenarios ....................36
8.4.2. LSP State Compared with LSP Numbers ................37
8.5. Issues with MP2P LSPs .....................................37
9. Combined Models ................................................39
10. An Alternate Solution .........................................39
10.1. Pros and Cons of the Alternate Solution ..................40
11. Management Considerations .....................................42
12. Security Considerations .......................................42
13. Recommendations ...............................................42
14. Acknowledgements ..............................................43
15. Normative References ..........................................43
16. Informative References ........................................431. Introduction ✨
Network operators and service providers are looking hard at scaling as they try to deploy even bigger traffic engineered MPLS-TE networks 📈 People are worried about how many Label Switched Paths (LSPs) have to be supported at the edge and in the core 😬 All that control plane and management plane load could cancel out the benefits that made MPLS-TE so popular, and the physical limits of the routers could box in what deployments are possible 🫠
Historically, people assumed all MPLS-TE scaling problems could be solved with hierarchical LSPs [RFC4206] 🙃 But the analysis shows that LSP hierarchies don't help as much as expected in every topology or at every point in the network 📉 They also bring extra management headaches, like figuring out where the endpoints of the hierarchical LSPs go and how to run them 😩 That doesn't make hierarchies useless, but it does mean more techniques may be needed to really scale MPLS-TE networks 💅
This document checks the scaling properties of two generic MPLS-TE network topologies and looks at the benefits of two scaling techniques 🦋
1.1. Overview 🌈
Physical scaling worries are handled by building networks that aren't fully meshed 🕸️ Core networks tend to be meshed while the edges are tree-shaped, which gives you a snowflake design ❄️ Or the core can look more like a ladder with tree-shaped edges 🪜
MPLS-TE sets up a logical full mesh between all the edge points in the network, and that's exactly where the scaling problems start 😭 The structure of the network tends to pile a huge number of LSPs into the core 🌀
This document presents two generic network topologies, the snowflake and the ladder, and tries to parameterize them by making some general assumptions 📐 It introduces terminology for the different scaling parameters and looks at how many LSPs might need to be carried in the core of a network 🧮
Two techniques, hierarchical LSPs and multipoint-to-point LSPs, get introduced here, and the document looks at how much they help with scaling as well as some of the concerns that come with using them ⚖️
Of necessity, this document makes a lot of generalizations 🫣 One big one is a set of assumptions about the symmetry and connectivity of the physical network. The hope is that these generalizations won't get in the way of the overview of scaling properties this document is trying to give 🙏 Honestly, the symmetry of the example topologies really highlights the scaling issues of the different solution models, which can be useful for exposing the worst case scenarios 👀
Protection mechanisms like Fast Reroute (FRR) [RFC4090] get a brief mention, but the main body of this document considers stable network cases 🧘 Also note that make-before-break re-optimisation after a link failure may leave behind a significant number of 'duplicate' LSPs 🙈 This document does not address that issue 🚫
Next, deployment models that run separate traffic engineered LSPs to deliver different services, like layer 3 Virtual Private Networks (VPNs) [RFC4110] or pseudowires [RFC3985], or different classes of service [RFC3270], can end up with 'duplicate' or 'parallel' LSPs between any pair of provider edge nodes (PEs) 👯♀️ This scaling factor isn't considered here either, but a reader can easily apply it as a linear factor 🧠
Security mechanisms in MPLS-TE networks [MPLS-SEC] can affect how well the network scales 🔐 For example, they may increase both the size and the number of control plane messages 📨 They can also add processing overhead, since control plane messages go through processing algorithms like encryption, and security keys have to be managed 🔑 Deployers will need to weigh scaling goals against security goals, and they should resist the temptation to switch off security techniques they've already decided are necessary just because scaling got worse 🚨 Further analysis of how security measures affect scalability is not covered in this document 🙅♀️
This document is designed to help service providers find out whether existing protocols and implementations can support the network sizes they're planning 🗺️ To do that, it analyzes some of the scaling concerns for MPLS-TE core networks and checks how well two techniques improve scaling 💖 That should motivate better deployment techniques and protocol extensions so MPLS-TE can be used in large networks 🎀
This document only considers scalability for point-to-point MPLS-TE LSPs 🧚 Point-to-multipoint MPLS-TE LSPs are for future study 🔮
1.2. Glossary of Notation 🧚
Okay so this doc uses a consistent notation for the network parameters it analyzes 💅 Each term gets defined the moment it shows up in the text, but we've lined them all up here so you can check back fast. For the full explanations, go read the actual text, bestie 🫶
n A network level. n = 1 is the core of the network.
See Section 3 for more details on the definition of a level.
P(n) A node at level n in the network.
S(n) The number of nodes at level n. That is, the number of P(n)
nodes.
L(n) The number of LSPs seen by a P(n) node.
X(n) The number of LSP segment states held by a P(n) node.
M(n) The number of P(n+1) nodes subtended to a P(n) node.
R The number of rungs in a ladder network.
E The number of edge nodes (PEs) subtended below (directly or
indirectly) a spar-node in a ladder network.
K The cost-effectiveness of the network expressed in terms of
the ratio of the number of PEs to the number of network nodes.2. Issues of Concern for Scaling 🔥
This section is all about the stuff that can get in the way when a Label Switching Router (LSR) or a whole network has to support a bunch of LSPs ✨ These issues might mean there's a limit on how many LSPs can be supported, so we're gonna dig into each one 🦋
2.1. LSP State 💖
LSP state is the data an LSR has to store in order to keep an LSP alive 🧠 That covers the info needed to maintain forwarding plane state, plus any extra info needed when LSPs get set up through control plane protocols. The exact size is up to each implementation, but literally every implementation is gonna need some data to hold LSP state, so no escaping that 😭
So here's the scaling problem: as an LSR picks up more LSPs, the memory it needs grows in direct proportion, no cap 📈 And since an LSR's memory is limited, there's a matching limit on how many LSPs it can support. It's giving finite resources, babe 💀
Quick side note: techniques for cutting memory needs (like data compression) can let an LSR support more LSPs, but only by a moderate multiplier 🌈 Plus they may seriously slow down how fast the state can get processed. Not a free lunch!
Now, this document defines X(n) as "the number of LSP segment states held by a P(n) node." 👑 Here's why it's set up this way: an LSR at the end of an LSP only has to keep state in one direction (into the network), but a transit LSR has to keep state in both directions (toward both ends of the LSP) 🫣 It gets even more chaotic in multipoint-to-point (MP2P) LSPs (see Section 8). There, a transit LSR might need to hold LSP state for one downstream segment (toward the destination) and multiple upstream segments (from multiple sources). So we define LSP segment state as the state needed to keep an LSP going in one direction, to one adjacent node 🎀
2.2. Processing Overhead ⏱️
Depending a lot on implementation details, the number of LSPs going through an LSR can hit how fast it processes each LSP 🐢 For example, control block search times can grow as the number of control blocks to search grows, and even excellent implementations can't completely fix that. So since CPU power is limited in every LSR, there may be a practical limit on how many LSPs it can support 💅
More processing overhead depends on things specific to the control plane protocols, which we'll get into in the next section ✨
2.3. RSVP-TE Implications 🌸
Like a lot of connection-oriented signaling protocols, RSVP-TE (Resource Reservation Protocol - Traffic Engineering) needs state to be held inside the network to keep LSPs running 🫶 We already covered the impact of that in Section 2.1. Also, RSVP-TE needs separate info kept for upstream and downstream relationships, but it doesn't force any specific way of implementing that state, so you've got options 💜
Now here's the soft-state part, which is kinda the core drama 🎭 RSVP-TE is a soft-state protocol. That means protocol messages (refresh messages) have to be exchanged regularly between signaling neighbors to keep each LSP's state alive between them. A common refresh interval is 30 seconds, so every LSR has to send and receive one message in each direction (upstream and downstream) every 30 seconds for every LSP it supports 😩 That can be a huge constraint on network scaling! But various improvements [RFC2961] mean refresh processing can be cut down a lot, so an implementation can be tuned to remove nearly all worries about soft-state scaling in a stable network 🙌
Observations of existing implementations suggest there may be a threshold of around 50,000 LSPs. Above that, an LSR can struggle to get enough processing done to maintain LSP state 🥲 Refresh reduction [RFC2961] may help a lot here, but it's also been seen that in these conditions the size of the Srefresh can get really big, and the processing needed might still cause serious disruption to an LSR. Yikes 🙈
Another option is to increase the refresh time 🕰️ There's a correlation between how much you bump up the refresh time and how much the LSR's performance improves. BUT (lowercase, no cap, just vibes) RSVP-TE's soft-state nature depends on those regular refresh messages, so bumping the refresh time means losing some functionality 😬 That loss can be partially softened by using the RSVP-TE Hello message, and it can also be reduced by using various GMPLS extensions [RFC3473], like using [RFC2961] message acknowledgements on all messages. Main character energy in the fixes 💅
RSVP-TE also needs signaling adjacencies to be maintained through Hello message exchanges. Now, [RFC3209] suggests Hello messages should be retransmitted every 5 ms, but in practice, values around 3 seconds are more common 🤏 Either way, Hello support can be a scaling limit for an RSVP-TE implementation, since one message has to be sent to and received from each signaling adjacency every time period. That can limit how many neighbors (physical or logical) an LSR supports, but it doesn't affect how many LSPs the LSR can handle 🦋
2.4. Management 🗂️
Another practical worry for scaling large MPLS-TE networks is being able to manage the network 🧐 That can be limited by the available tools, by how practical it is to manage huge numbers of LSPs, and by the management protocols being used.
Management tools are software, so yeah, they have limits too 💀 Those implementations shouldn't constrain the control plane protocols, but it's realistic to accept that deployments will be capped by how well the tools scale. In practice, most existing tools have a max number of LSPs they can handle. A Network Management System (NMS) might be able to support a big number of LSPs, but the number an Element Management System (EMS) can handle (or the number an NMS can handle per-LSR) is more likely to be limited 📉
Management protocols can also put practical constraints on things. For example, an LSR might get swamped with management protocol requests asking to read info about the LSPs it supports, and that could affect its ability to keep those LSPs up in the control plane 😱 OAM (Operations, Administration, and Management), alarms, and notifications can pile even more on an LSR and limit how many LSPs it can support.
So all of this points toward reducing the number of LSPs supported within the network and at any particular LSR 💜
3. Network Topologies 🌸
Okay, so to give some generic analysis of the potential scaling issues for MPLS-TE networks, this doc looks at two network topology models 🧚 They were picked partly because of their symmetry, which makes them easier to tackle with a formulaic approach, and partly because they're generalizations of real deployment models. Section 3.3 talks about the commercial drivers behind deployed topologies and digs into why it makes sense to consider these two, so stay tuned ✨
Model one is the snowflake model ❄️ In this kind of network, only the very core is meshed, and the edges are formed as trees rooted in the core.
Model two is the ladder model 🪜 Here, the core is shaped and meshed like a ladder, and trees get attached, rooted at the edge of the ladder.
The sections that follow examine these topologies in detail so we can parameterize them 💅
3.1. The Snowflake Network Topology ❄️
okay so the snowflake topologies in this doc are built on a hierarchy of connectivity inside the core network 🫶 PE nodes connect to P-nodes like in Figure 1, and there is no direct connectivity between the PEs 🙅♀️ dual homing of PEs to multiple P-nodes isn't covered here, even though it could be a super valuable addition to a network config 💅
P
/|\
/ | \
/ | \
/ | \
PE PE PE
Figure 1 : PE to P-Node Connectivitythe P-nodes are also layered, bestie 🌸 like in Figure 2, multiple P-nodes at one level hook up to a P-node at a higher level 👑 we number the levels so level 1 is the top (the top of our figure, closest to the core of the network) and level (n) sits immediately above level (n+1). a P-node at level n is written as P(n) 🦋
P(n)
/|\
/ | \
/ | \
/ | \
P(n+1) P(n+1) P(n+1)
Figure 2 : Relationship between P-Nodessame deal as with PEs: there's no direct connectivity between P(n+1) nodes 🙈 and again, dual homing of P(n+1) nodes to multiple P(n) nodes isn't considered in this document, but it could be a valuable addition to a network configuration ✨
at the top level, the P(1) nodes form a full mesh 🕸️ real life, the level 1 part of the network might be a little less well-connected than this, but assuming a full mesh keeps things general 💖 so the snowflake topology is basically a clique, with topologically equivalent trees hanging off every node in the clique. iconic ❄️
the key multipliers for scalability are the number of P(1) nodes, plus the multiplier relationship between P(n) and P(n+1) at each level, all the way down to and including the PEs 🔥
we define the multiplier M(n) as the number of P(n+1) nodes at level (n+1) attached to any one P(n) 🧚 we assume M(n) is constant for all nodes at level n. nodes at the same level aren't interconnected (except at the top level), and each P(n+1) node is connected to precisely one P(n) node. so M(n) is one less than the degree of the node at level n: the P(n) node is attached to M(n) nodes at level (n+1) and to 1 node at level (n-1) 💜
we define S(n) as the number of nodes at level (n) 🍓
so, thus:
S(n) = S(1)*M(1)*M(2)*...*M(n-1)the number of PEs can be written as:
S(PE) = S(1)*M(1)*M(2)*...*M(n)where the network has (n) layers of P-nodes 🌈
here's an example snowflake network, shown in Figure 3. in this case:
S(1) = 3
M(1) = 3
S(2) = S(1)*M(1) = 9
M(2) = 2
S(PE) = S(1)*M(1)*M(2) = 18 PE PE PE PE PE PE
\ \/ \/ /
PE--P(2) P(2) P(2) P(2)--PE
\ | | /
\| |/
PE--P(2)---P(1)------P(1)---P(2)--PE
/ \ / \
PE \ / PE
\/
P(1)
/|\
/ | \
/ | \
PE--P(2) P(2) P(2)--PE
/ /\ \
PE PE PE PE
Figure 3 : An Example Snowflake Network3.2. The Ladder Network Topology 🪜
the ladder networks in this section are built on an arrangement of routers in the core network that looks like a ladder ✨
ladder networks usually have long, thin cores that are arranged like conventional ladders. that means one or more spars connected by rungs 🪜 each node on a spar may have:
- connection to one or more other spars,
- connection to a tree of other core nodes,
- connection to customer nodes.
Figure 4 shows a simplified example of a ladder network. a core of twelve nodes makes up two spars connected by six rungs 💅
PE PE PE PE
PE PE PE | PE | PE PE PE | PE | PE
\| \|/ |/ | \| \|/
PE-P-----P-----P-----P------P-----P--PE
| | | | | |\
| | | | | | PE
| | | | | |
PE-P-----P-----P-----P------P-----P
/| /|\ |\ |\ |\ \
PE PE PE | PE | PE | PE | PE PE
PE PE PE PE
Figure 4 : A Simplified Ladder Networkin practice, not every node on a spar (call them spar-nodes 🫧) needs subtended PEs. they can just exist to give connectivity along the spar to other spar-nodes, or across a rung to another spar 🌸 the connectivity between spars can also get more complex, with multiple connections from one spar-node to another spar. and the network might get more complicated by including more than two spars (or get simpler by reducing it to a single spar) 🙃
these variables make the ladder network non-trivial to model, so for simplicity we're going to make these restrictions:
- There are precisely two spars in the core network.
- Every spar-node connects to precisely one spar-node on the other spar. That is, each spar-node is attached to precisely one rung.
- Each spar-node connects to either one (end-spar) or two (core-spar) other spar-nodes on the same spar.
- Every spar-node has the same number of PEs subtended. This does not mean there are no P-nodes subtended to the spar-nodes, but it does mean the edge tree subtended to each spar-node is identical.
from these restrictions, we can quantify a ladder network like this 🧮
- R - The number of rungs. That is, the number of spar-nodes on each spar.
- S(1) - The number of spar-nodes in the network. S(1)=2*R.
- E - The number of subtended edge nodes (PEs) to each spar-node.
the number of rungs can vary a lot. fewer than 3 is unlikely (that wouldn't be a significantly connected network), and more than 100 seems improbable (that would be a very long, thin network) 🤏
E can be treated the same way as for the snowflake network. so we can consider a number of levels of attachment from P(1) nodes (which are the spar-nodes) through P(i) down to P(n) (which are the PEs) 🌷 practically, we only need to consider n=2 (PEs attached directly to the spar-nodes) and n=3 (one level of P-nodes between the PEs and the spar-nodes) 💖
let M(i) be the ratio of P(i) nodes to P(i-1) nodes, i.e., the connectivity between levels of P-node as defined for the snowflake topology. so the number of nodes at any level (n) is:
S(n) = S(1)*M(1)*M(2)*...*M(n-1)the number of PEs subtended to a spar-node is:
E = M(1)*M(2)*...*M(n)and the number of PEs can be written as:
S(PE) = S(1)*M(1)*M(2)*...*M(n)
= S(1)*Ehere's an example ladder network, shown in Figure 5. in this case:
R = 5
S(1) = 10
M(1) = 2
S(2) = S(1)*M(1) = 20
M(2) = 2
E = M(1)*M(2) = 4
S(PE) = S(1)*E = 40 PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE
\| \| \| \| |/ |/ |/ |/
P(2) P(2) P(2) P(2) P(2) P(2) P(2) P(2)
\ \ | \ / | / /
PE \ \ | \ / | / / PE
\ \ \| \/ |/ / /
PE-P(2)---P(1)---P(1)---P(1)---P(1)---P(1)---P(2)-PE
| | | | |
| | | | |
| | | | |
PE-P(2)---P(1)---P(1)---P(1)---P(1)---P(1)---P(2)-PE
/ / / | /\ |\ \ \
PE / / | / \ | \ \ PE
/ / | / \ | \ \
P(2) P(2) P(2) P(2) P(2) P(2) P(2) P(2)
/| /| /| /| |\ |\ |\ |\
PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE PE
Figure 5 : An Example Ladder Network3.3. Commercial Drivers for Selected Configurations 💸
it's totally fair to ask why these two particular topologies were picked 🤔
the most important consideration is physical scalability 🏗️ each node (Label Switching Router - LSR) can only support a limited number of physical interfaces. that naturally limits how far you can fully mesh a network, and it's why the network ends up tree-shaped as you move toward the PEs 🌳
there's also a real commercial angle for an operator: the only revenue-generating nodes in the network are the PEs 🤑 the other nodes are only needed for connectivity and scalability. so operators want to maximize S(PE) while minimizing the sum of S(n) for all values of (n). this could be done by using fewer levels and maximizing the connectivity at each layer, M(n). but honestly, that would end up as a network of just interconnected PEs, which clearly clashes with the physical scaling situation 😭
so the solution calls for a "few" levels with "relatively large" connectivity at each level ✨ we might say the cost-effectiveness of the network can be stated as:
K = S(PE)/(S(1)+S(2) + ... + S(n)) where n is the level above the PEswe should observe, though, that this equation may be naive. the cost of a network isn't really a function of the number of routers (a router chassis is often free or low cost). it's really a function of the cost of the line cards, which is itself a product of the capacity of those line cards 💅 so relatively high connectivity decreases cost-effectiveness, while a topology that channels data through a network core tends to demand higher capacity (and so, more expensive) line cards 💀
one more thing: availability of connectivity (usually fibers) between LSR sites 🧵 laying new fiber is always possible, but it may not be cost-effective or timely. the physical shape and topography of the country the network is in can be just as much of a problem. if the country is "long and thin", a ladder network is likely to be used 🇮🇹
this document looks at the control plane and data plane scalability implications of this type of network when MPLS-TE LSPs are used to provide full connectivity between all PEs 🦋
3.4. Other Network Topologies 🌍
as explained in Section 1, this document uses two symmetrical and generalized network topologies, just to keep the modelling simple 🧸 in practice, there are two other topological considerations:
a. Multihoming
it's pretty common for a node at level (n) to be attached to more than one node at level (n-1). this is especially common at PEs that may be connected to more than one P(n) 🫶
b. Meshing within a level
a level in the network will often include links between P-nodes at the same level, including the possibility of links between PEs 🕸️ this can result in a network that looks like a series of concentric circles with spokes 🎯
both of these features are likely to affect the scaling of the networks 👀 but for the purposes of setting the ground rules for scaling, this document sticks to the symmetrical networks described in Sections 2.1 and 2.2. discussing other network formats is for future study 🔮
4. Required Network Sizes 🎀
okay so the big question for this whole evaluation is how big the network actually needs to be 👀. How many PEs do we need? What ratio of P to PE is acceptable? How many ports do devices have for physical connectivity? And what kind of MPLS-TE connectivity between PEs is required? bestie, these are the questions that decide everything 💅
Figures for desired network sizes have to be taken with a grain of salt, because networks grow beyond all projections every single time 😭. Still, it's useful to set some acceptable lower bounds, so we can say we're interested in networks of at least a certain size 🌸.
The most important features are:
- The network should have at least 1000 PEs.
- Each pair of PEs should be connected by at least one LSP in each direction.
4.1. Practical Numbers 🧚
In practice, the reasonable target numbers look like this ✨
S(PE) >= 1000
Number of levels is 3. That is: 1, 2, and PE.
M(2) <= 20
M(1) <= 20
S(1) <= 1005. Scaling in Flat Networks 💜
Before we look at any potential scaling improvements, we need to check how well the flat networks from the previous sections actually scale 🔥.
Consider the requirement for a full mesh of LSPs linking all PEs. That means each PE has an LSP to and from every other LSP (well, every other PE, lol 💀). So if there are S(PE) PEs in the network, there are S(PE)*(S(PE) - 1) LSPs.
Define L(n) as the number of LSPs handled by a level (n) LSR.
L(PE) = 2*(S(PE) - 1)5.1. Snowflake Networks ❄️
There are a total of S(PE) PEs in the network. Since each PE sets up an LSP with every other PE, you'd expect S(PE) - 1 LSPs incoming to each PE and the same number outgoing from that same PE, which gives a total of 2(S(PE) - 1) on the incident link. So in a snowflake topology (see Figure 3), since there are M(2) PEs attached to each P(2) node, it may be tempting to think that L(2) (the number of LSPs traversing each P(2) node) is simply 2*(S(PE) - 1)*M(2). BUT (and this is the twist 🙈), it should be noted that out of the S(PE) - 1 LSPs incoming to each PE, M(2) - 1 came from nodes attached to the same P(2) node. That means this value counts the LSPs between the M(2) PEs attached to each P(2) node twice: once when they're outgoing from the M(2) - 1 other nodes, and once when they're incoming into a particular PE.
There are a total of M(2)(M(2) - 1) LSPs between these M(2) PEs. Since this value gets erroneously included twice in 2(S(PE) - 1)*M(2), the correct value is:
L(2) = 2*M(2)*(S(PE) - 1) - M(2)*(M(2) - 1)
= M(2)*(2*S(PE) - M(2) - 1)There's also an alternative way of looking at this, and it turns out to be extensible for the calculation of L(1) 🦋. Each PE subtended to a P(2) node has an LSP in each direction to all S(PE) - M(2) PEs in the rest of the system, and there are M(2) such locally subtended PEs. That gives 2*M(2)(S(PE) - M(2)). Additionally, there are M(2)(M(2) - 1) LSPs between the locally subtended PEs. So:
L(2) = 2*M(2)*(S(PE) - M(2)) + M(2)*(M(2) - 1)
= M(2)*(2*S(PE) - M(2) - 1)L(1) can be computed the same way as this second evaluation of L(2). Each PE subtended below a P(1) node has an LSP in each direction to all PEs that are not below that P(1) node. There are M(1)M(2) PEs below the P(1) node, so this accounts for 2M(1)M(2)(S(PE) - M(1)M(2)) LSPs. To this, we need to add the number of LSPs that pass through the P(1) node and run between the PEs subtended below it. Consider each P(2): it has M(2) PEs, and each one has an LSP going to all of the PEs subtended to the other P(2) nodes that hang off the same P(1). There are M(1) - 1 such other P(2) nodes, so there are M(2)(M(1) - 1) other such PEs. That means the number of LSPs from the PEs below a P(2) node is M(2)M(2)(M(1) - 1). And since there are M(1) P(2) nodes below the P(1), that gives a total of M(2)M(2)M(1)*(M(1) - 1) LSPs. Thus:
L(1) = 2*M(1)*M(2)*(S(PE) - M(1)*M(2)) + M(2)*M(2)*M(1)*(M(1) - 1)
= M(1)*M(2)*(2*S(PE) - M(2)*(M(1) + 1))So, for example, with S(1) = 5, M(1) = 10, and M(2) = 20, we see:
S(PE) = 1000
L(PE) = 1998
L(2) = 39580
L(1) = 356000Alternatively, with S(1) = 10, M(1) = 10, and M(2) = 20, we see:
S(PE) = 2000
L(PE) = 3998
L(2) = 79580
L(1) = 756000In both examples, the number of LSPs at the core (P(1)) nodes is probably unacceptably large, even though there are only a relatively modest number of PEs. In fact, L(2) may even be too large in the second example 😬.
5.2. Ladder Networks 🪜
In ladder networks, L(PE) stays the same at 2*(S(PE) - 1).
L(2) can be computed with the same mechanism as for the snowflake topology, because the subtended tree is the same format. Hence,
L(2) = 2*M(2)*(S(PE) - 1) - M(2)*(M(2) - 1)But L(1) needs a different computation, because each P(1) doesn't just see LSPs for its subtended PEs. It's also a transit node for some of the LSPs that cross the core (the core is not fully meshed) 🌈.
Each P(1) sees:
- all of the LSPs between locally attached PEs,
- less those LSPs between locally attached PEs that can be served exclusively by the attached P(2) nodes,
- all LSPs between locally attached PEs and remote PEs, and
- LSPs in transit that pass through the P(1).
The first three numbers are easy to determine and match what we saw from the snowflake network. They are:
- E*(E-1)
- M(1)M(2)(M(2)-1) = E*(M(2) - 1)
- 2*E*E*(S(1) - 1)
The number of LSPs in transit is more complicated to compute. It's simplified by not looking at the ends of the ladders, but by examining an arbitrary segment of the middle of the ladder, as shown in Figure 6. We want to compute and generalize the number of LSPs traversing each core link (labeled a and b in Figure 6), and from that determine the number of transit LSPs seen by each P(1).
: : : : : :
: : : : : :
P(2) P(2) P(2) P(2) P(2) P(2)
\ | \ / | /
\ | \ / | /
\| \/ |/
......P(1)---P(1)---P(1)......
| a | |
| |b |
| | |
......P(1)---P(1)---P(1)......
/| /\ |\
/ | / \ | \
/ | / \ | \
P(2) P(2) P(2) P(2) P(2) P(2)
: : : : : :
: : : : : :Figure 6 : An Arbitrary Section of a Ladder Network
Of course, the number of LSPs carried on links a and b in Figure 6 depends on how LSPs are routed through the core network. But if we assume a symmetrical routing policy and an even distribution of LSPs across all shortest paths, the result is the same ✨.
Now we can see that each P(1) sees half of 2a+b LSPs (since each LSP would otherwise be counted twice as it passed through the P(1)), except that some of the LSPs are locally terminated and so are only included once in the sum 2a+b.
So L(1) = a + b/2 - (locally terminated transit LSPs)/2 + (locally contained LSPs)
Thus:
L(1) = a + b/2 -
2*E*E*(S(1) - 1)/2 +
E*(E-1) - E*(M(2) - 1)
= a + b/2 +
E*E*(2 - S(1)) - E*M(2)So all we have to do is work out a and b.
Recall that the ladder length R = S(1)/2, and define X = E*E.
Consider the contribution made by all of the LSPs that make n hops on the ladder to the totals of each of a and b. If the ladder were unbounded, then for a, there are n*2X LSPs along the spar only, and n(n-1)*2X/n = 2X(n-1) LSPs use a rung and the spar. Thus, the LSPs that make n hops on the ladder contribute (4n-2)X LSPs to a. Note that the edge cases are special, because LSPs that make only one hop on the ladder cannot transit a P(1) but only start or end there.
So with a ladder of length R = S(1)/2, we could say:
R
a = SUM[(4i-2)*X] + 2RX
i=2
= 2*X*R*(R+1)And similarly, considering b in an unbounded ladder: the LSPs that only travel one hop on the ladder are a special case, contributing 2X LSPs, and every other LSP that traverses n hops on the ladder contributes 2n*2X/n = 4X LSPs. So:
R+1
b = 2X + SUM[4X]
i=2
= 2*X + 4*X*RIn fact, the ladders are bounded, so the number of LSPs is reduced because of the effect of the ends of the ladders. The links that see the most LSPs are in the middle of the ladder. Consider a ladder of length R: a node in the middle of the ladder is R/2 hops away from the end of the ladder. So the formula for the contribution to the count of spar-only LSPs for a is only valid up to n=R/2, and for spar-and-rung LSPs, up to n=1+R/2. Above these limits, the contribution made by spar-only LSPs decays as (n-R/2)*2X.
However, for a first-order approximation, we will use the values of a and b as computed above. This gives us an upper bound on the number of LSPs without using a more complex formula for the reduction made by the effect of the ends of the ladder.
From this:
L(1) = a + b/2 +
E*E*(2 - S(1)) - E*M(2)
= 2*X*R*(R+1) +
X + 2*X*R +
E*E*(2 - S(1)) - E*M(2)
= E*E*S(1)*(1 + S(1)/2) +
E*E + E*E*S(1) +
2*E+E - E*E*S(1) - E*M(2)
= E*E*S(1)*(1 + S(1)/2) + 3*E+E - E*M(2)
= E*E*S(1)*S(1)/2 + E*E*S(1) + 3*E*E - E*M(2)So, for example, with S(1) = 6, M(1) = 10, and M(2) = 17, we see:
E = 170
S(PE) = 1020
L(PE) = 2038
L(2) = 34374
L(1) = 777410Alternatively, with S(1) = 10, M(1) = 10, and M(2) = 20, we see:
E = 200
S(PE) = 2000
L(PE) = 3998
L(2) = 79580
L(1) = 2516000In both examples, the number of LSPs at the core (P(1)) nodes is probably unacceptably large, even though there are only a relatively modest number of PEs. In fact, L(2) may even be too large in the second example 💀.
Compare the L(1) values with the total number of LSPs in the system, S(PE)*(S(PE) - 1), which is 1039380 and 3998000, respectively 🫶.
6. Scaling Snowflake Networks with Forwarding Adjacencies 🦋
One of the purposes of LSP hierarchies [RFC4206] is to improve the scaling properties of MPLS-TE networks. LSP tunnels (sometimes known as Forwarding Adjacencies (FAs)) may be established to provide connectivity over the core of the network, and multiple edge-to-edge LSPs may be tunneled down a single FA LSP.
In our network we consider a mesh of FA LSPs between all core nodes at the same level. We look at two possibilities here. In the first, all P(2) nodes are connected to all other P(2) nodes by LSP tunnels, and the PE-to-PE LSPs are tunneled across the core of the network. In the second, an extra layer of LSP hierarchy is introduced by connecting all P(1) nodes in an LSP mesh and tunneling the P(2)-to-P(2) tunnels through these 🌸.
6.1. Two-Layer Hierarchy ✨
okay so in this hierarchy model, the P(2) nodes are connected by a mesh of tunnels 🦋 which means the P(1) nodes never see the PE-to-PE LSPs 🙈
L(PE) = 2*(S(PE) - 1)L(2) gets a lil bit bigger though 😭. You get it by adding up all the LSPs for every attached PE, including the LSPs between the attached PE (this figure is unchanged from Section 5.1, i.e., M(2)(2S(PE) - M(2) - 1)), plus the number of FA LSPs that make the mesh to the other P(2) nodes. Since the number of P(2) nodes is S(2), each P(2) node sees 2*(S(2) - 1) FA LSPs. So:
L(2) = M(2)*(2*S(PE) - M(2) - 1) + 2*(S(2) - 1)L(1) though? It's significantly reduced 💖 and you can work it out as the sum of the FA LSPs to and from each attached P(2) to each other P(2) in the network, including (but counting only once) the FA LSPs between attached P(2) nodes. The problem is literally identical to the L(2) computation in Section 5.1. So:
L(1) = M(1)*(2*S(2) - M(1) - 1)So, for example, with S(1) = 5, M(1) = 10, and M(2) = 20, we get:
S(PE) = 1000
S(2) = 50
L(PE) = 1998
L(2) = 39678
L(1) = 890Alternatively, with S(1) = 10, M(1) = 10, and M(2) = 20, we get:
S(PE) = 2000
S(2) = 100
L(PE) = 3998
L(2) = 79778
L(1) = 1890So in both examples, potential problems at the core (P(1)) nodes caused by an excessive number of LSPs can be avoided 🎀 but any problem with L(2) gets slightly worse, as you can see in the table below 🥺
Example| Count | Unmodified | 2-Layer
| | (Section 5.1) | Hierarchy
-------+-------+---------------+----------
A | L(2) | 39580 | 39678
| L(1) | 356000 | 890
-------+-------+---------------+----------
B | L(2) | 79580 | 79778
| L(1) | 756000 | 18906.1.1. Tuning the Network Topology to Suit the Two-Layer Hierarchy
Clearly, we can bring L(2) down by picking good values of S(1), M(1), and M(2) 🧚 We can do this with no negative consequences, because no change will affect L(PE), and a big percentage increase in L(1) is totally sustainable now that L(1) is so small 💅
Observe that:
L(2) = M(2)*(2*S(PE) - M(2) - 1) + 2*(S(2) - 1)where S(PE) = S(1)*M(1)*M(2) and S(2) = S(1)*M(1). So L(2) scales with M(2)^2, and we get the most impact by reducing M(2) while keeping S(PE) constant 👑
For example, with S(1) = 10, M(1) = 10, and M(2) = 10, we get:
S(PE) = 1000
S(2) = 100
L(PE) = 1998
L(2) = 20088
L(1) = 1890And similarly, with S(1) = 20, M(1) = 20, and M(2) = 5, we get:
S(PE) = 2000
S(2) = 400
L(PE) = 3998
L(2) = 20768
L(1) = 15580These amazing scaling benefits have to be balanced against the cost-effectiveness of the network 💀 Recall from Section 3.3 that:
K = S(PE)/(S(1)+S(2) ... + S(n))where n is the level above the PEs, so for our network:
K = S(PE) / (S(1) + S(2))So in the first example the cost-effectiveness gets cut in half, from 1000/55 to 1000/110 😭 In the second example, it drops to roughly one quarter, going from 2000/110 to 2000/420.
So even though the tuning changes may be needed to hit the network size you want, they come at a considerable cost to the operator 🫶
6.2. Alternative Two-Layer Hierarchy
An alternative to the two-layer hierarchy from Section 6.1 is to give you a full mesh of FA LSPs between P(1) nodes ✨ This technique is only useful for nodes in the core of the level 1 network, so it's giving very specific vibes 🌸 It makes no difference to the PE and P(2) nodes, since they keep seeing only the PE-to-PE LSPs. Also, this approach piles more work onto the P(1) nodes, because they have to support all of the PE-to-PE LSPs like in the flat model, plus the additional 2*(S(1) - 1) P(1)-to-P(1) FA LSPs 😩 So this approach should only be considered where there is a mesh of P-nodes within the ring of P(1) nodes, and it isn't considered any further in this document 🙅♀️
6.3. Three-Layer Hierarchy
As Section 6.2 showed, putting a mesh of FA LSPs at the top level (P(1)) does nothing for us 💔 But if we add another level to the network (P(3) between P(2) and PE) to make a four-level snowflake, we can add a new layer of FA LSPs. That way we get a full mesh of FA LSPs between all P(3) nodes to carry the PE-to-PE LSPs, and a full mesh of FA LSPs between all P(2) nodes to carry the P(3)-to-P(3) LSPs 🦋
The number of PEs is S(PE) = S(1)*M(1)*M(2)*M(3), and the number of PE-to-PE LSPs at a PE stays at L(PE) = 2*(S(PE) - 1).
The number of LSPs at a P(3) can be worked out from Section 6.1. It's the sum of all LSPs for all attached PEs, including the LSPs between the attached PE, plus the number of FA LSPs making a mesh to the other P(3) nodes.
L(3) = M(3)*(2*S(PE) - M(3) - 1) + 2*(S(3) - 1)The number of LSPs at P(2) can also be worked out from Section 6.1, since it's the sum of all LSPs for all attached P(3) nodes, including the LSPs between the attached PE, plus the number of FA LSPs making a mesh to the other P(2) nodes.
L(2) = M(2)*(2*S(3) - M(2) - 1) + 2*(S(2) - 1)Finally, L(1) can be copied straight over from 6.1.
L(1) = M(1)*(2*S(2) - M(1) - 1)For example, with S(1) = 5, M(1) = 5, M(2) = 5, and M(3) = 8, we get:
S(PE) = 1000
S(3) = 125
S(2) = 25
L(PE) = 1998
L(3) = 16176
L(2) = 1268
L(1) = 220Similarly, with S(1) = 5, M(1) = 5, M(2) = 8, and M(3) = 10, we get:
S(PE) = 2000
S(3) = 200
S(2) = 25
L(PE) = 3998
L(3) = 40038
L(2) = 3184
L(1) = 220Clearly, this three-layer hierarchy gives us huge scaling wins, and all of the numbers (even L(3) in the second example) are totally manageable 🎉
Of course, the extra level in the network makes the cost-effectiveness of these networks drop, going from K = 1000/55 and K = 2000/110 down to K = 1000/155 and K = 2000/230 for the examples above 😵 That's a reduction by a factor of 3 in the first case and 2 in the second case. A change like that in cost-effectiveness has to be weighed against how badly you want to deploy such a large network 🤔 If LSP hierarchies are the only scaling tool you've got, and networks this size are required, you may have to sacrifice cost-effectiveness 💅
6.4. Issues with Hierarchical LSPs
A basic observation about hierarchical scaling techniques is that it's hard to have any impact on the number of LSPs that the level of P(n) nodes next to the PEs has to support (for example, it's hard to reduce L(3) in Section 6.3) 🥲 In fact, the only way to change the number of LSPs these nodes support is to change the scaling ratio M(n) in the network. In other words, you change the number of PEs subtended to any P(n). But that kind of change has a direct effect on the number of PEs in the network, so the cost-effectiveness takes a hit 📉
Another concern with the hierarchical approach is that it must be configured and managed. This might not seem like a big burden, but remember that the P(n) nodes aren't at the edge of the network. They're a set of nodes that must be identified so the FA LSPs can be configured and provisioned 🧐 Basically, the operator has to plan and build a layered network with a ring of P(n) nodes giving access to the level (n) network. This design work carries considerable risk, because failing to close the ring (i.e., allowing a node to be at both level (n+1) and at level (n)) may cause operational confusion 😬
Protocol techniques (such as IGP automesh [RFC4972]) have been developed to cut down on the configuration needed to build this type of multi-level network 🌈 With automesh, the routing protocol is used to advertise membership of a 'mesh group', and all members of the mesh group can discover each other and connect with LSP tunnels. So the P(n) nodes giving access to level (n) can advertise their existence to each other, and you don't have to configure each one with info about all of the others 🙌 Although this process can reduce the configuration overhead, it doesn't eliminate it, because each member of the mesh group still has to be planned and configured for membership.
An important consideration for using hierarchical LSPs is how they can be protected using MPLS Fast Reroute (FRR) [RFC4090]. FRR may provide link protection either by protecting the tunnels as they traverse a broken link, or by treating each level (n) tunnel LSP as a link in level (n+1) and providing protection for the level (n+1) LSPs (although in this model, fault detection and propagation time may be an issue) 🫣 Node protection may be performed in a similar way, but protecting the first and last nodes of a hierarchical LSP is particularly difficult 😭 On top of that, the whole notion of scaling with regard to FRR brings up separate concerns that are outside the scope of this document as currently formulated.
Finally, observe that we've been explaining these techniques using conveniently symmetrical networks. Think about how you'd arrange the hierarchical LSPs in a network where some PEs are connected closer to the center of the network than others 🧠✨
7. Scaling Ladder Networks with Forwarding Adjacencies ✨
7.1. Two-Layer Hierarchy
In Section 6.2, we saw that there's no value in placing FA LSPs between the P(1) nodes of our example snowflake topologies 🙃 That's because those LSPs would be just one hop long, and they would only increase the burden at the P(1) nodes. However, in the ladder model, this approach is actually valuable 💖 The P(1) nodes are the spar-nodes of the ladder, and they aren't all mutually adjacent. That means the P(1)-to-P(1) hierarchical LSPs can create a full mesh of P(1) nodes where one doesn't exist in the physical topology 🦋
The number of LSPs seen by a P(1) node is then:
- all of the tunnels terminating at the P(1) node,
- any transit tunnels, and
- all of the LSPs due to subtended PEs.
This is a substantial reduction. All of the transit LSPs get cut down to just one per remote P(1) that causes any transit LSP 🎉 So:
L(1) = 2*(S(1) - 1) +
O(S(1)*S(1)/2) +
2*E*E*(S(1) - 1) + E*(E-1) - E*(M(2) - 1)where O(S(1)*S(1)/2) gives an upper bound order of magnitude. So:
L(1) = S(1)*S(1)/2 + 2*S(1) + 2*E*E*(S(1) - 1) - E*M(2) - 2So, in our two examples:
With S(1) = 6, M(1) = 10, and M(2) = 17, we get:
E = 170
S(PE) = 1020
L(PE) = 2038
L(2) = 34374
L(1) = 286138Alternatively, with S(1) = 10, M(1) = 10, and M(2) = 20, we get:
E = 200
S(PE) = 2000
L(PE) = 3998
L(2) = 79580
L(1) = 716060Both of these are a big improvement on the previous L(1) figures of 777410 and 2516000 🎀 But the numbers are still too large to manage 😩 and there's no improvement in the L(2) figures either.
7.2. Three-Layer Hierarchy 🧚✨
okay so we can ALSO apply the three-layer hierarchy to the ladder model 💅 here the number of LSPs between P(1) nodes doesn't get reduced, BUT tunnels get set up between all the P(2) nodes too 🎀 so the number of LSPs a P(1) node sees is:
- all of the tunnels terminating at the P(1) node,
- any transit tunnels between P(1) nodes, and
- all of the LSPs due to subtended P(2) nodes.
no PE-to-PE LSPs are seen at the P(1) nodes 🙈 bestie, that's the vibe!
L(1) = 2*(S(1) - 1) +
O(S(1)*S(1)/2) +
2*(S(1) - 1)*M(1)*M(1) + M(1)*(M(1) - 1)where O(S(1)*S(1)/2) gives an upper bound order of magnitude 🔢 so:
L(1) = S(1)*S(1)/2 + 2*S(1) + 2*M(1)*M(1)*S(1) - M(1)(M(1) + 1) - 2ngl, there's a small increase in the number of LSPs seen by the P(2) nodes 😭 each P(2) still sees all of the PE-to-PE LSPs it saw before, and it's ALSO an end-point for a set of P(2)-to-P(2) tunnels. so L(2) goes up to:
L(2) = 2*M(2)*(S(PE) - 1) - M(2)*(M(2) - 1) + 2*(S(1)*M(1) - 1)so, in our two examples 👀 with S(1) = 6, M(1) = 10, and M(2) = 17, we get:
E = 170
S(PE) = 1020
L(PE) = 2038
L(2) = 34492
L(1) = 1118alternatively, with S(1) = 10, M(1) = 10, and M(2) = 20, we get:
E = 200
S(PE) = 2000
L(PE) = 3998
L(2) = 79778
L(1) = 1958this is a VERY dramatic decrease in LSPs across the core 🔥👑
7.3. Issues with Hierarchical LSPs 💜
the same issues exist for hierarchical LSPs as described in Section 6.4 😮💨 yes the scaling numbers for LSPs at core nodes can get dramatically better, but ONLY at the cost of configuring P(2) to P(2) tunnels. the mesh of P(1) tunnels just isn't enough on its own 🥲
and the sheer number of P(2) to P(2) tunnels that have to be configured is a significant management burden 🫠 it can only be eased by using a technique like automesh [RFC4972] 🎀
but here's the tea ☕ the scaling problem at the P(2) nodes can't be improved by using tunnels. the only way to ease it in the hierarchical approach would be to institute another layer of hierarchy (that is, P(3) nodes) between the P(2) nodes and the PEs 🧱 and of course, that's a significant expense 💸
8. Scaling Improvements through Multipoint-to-Point LSPs 🌈
an alternative (or complementary!) scaling technique has been proposed using multipoint-to-point (MP2P) LSPs 🦋 the fundamental improvement here comes from reducing the number of LSPs toward the destination, because LSPs toward the same destination get merged 🫶
this section gives an overview of MP2P LSPs and describes their applicability and scaling benefits ✨
8.1. Overview of MP2P LSPs 🍓
quick heads up 🚨 the MP2P LSPs discussed here are for MPLS-TE, and they are NOT the same concept as the one familiar from the Label Distribution Protocol (LDP) described in [RFC5036] 🙅♀️
traffic flows generally converge toward their destination, and MPLS can use that to build an MP2P LSP 🌸 with one of these, the Label Forwarding Information Base (LFIB) mappings at each LSR are many-to-one, so multiple pairs {incoming interface, incoming label} get mapped to a single pair {outgoing interface, outgoing label} 🧩 obviously, if per-platform labels are used, this mapping may be optimized within an implementation 💅
it's important to note that with MP2P MPLS-TE LSPs, the traffic flows are MERGED 🌀 that means some additional form of identifier is required if de-merging is needed. for example, if the payload is IP traffic belonging to the same client network, no extra de-merging info is required since the IP packet already has enough data 📦 but if the data comes from a variety of VPN client networks, then the flows need to be labeled in their own right as point-to-point (P2P) flows, so the traffic can be disambiguated at the egress of the MP2P LSPs 🕵️♀️
techniques for establishing MP2P MPLS-TE LSPs, and for assigning the correct bandwidth downstream of LSP merge points, are out of the scope of this document 🙏
8.2. LSP State: A Better Measure of Scalability 🔍
let's look at the network topology in Figure 3 🗺️ say we establish MP2P LSP tunnels so there's one tunnel terminating at each PE, and that tunnel has every other PE as an ingress. so a PE-to-PE MP2P LSP tunnel would have S(PE)-1 ingresses and one egress, and there would be S(PE) such tunnels 🎀
note that there still remain 2*(S(PE) - 1) PE-to-PE P2P LSPs carried through these tunnels 🫠
let's consider the number of LSPs handled at each node in the network 👀
the PEs keep handling the same number of PE-to-PE P2P LSPs, and they ALSO have to handle the MP2P LSPs. so:
L(PE) = 2*(S(PE) - 1) + S(PE)but all P(n) nodes in the network only handle the MP2P LSP tunnels. nominally, that means L(n) = S(PE) for all values of n 🤩 this would appear to be a great success, with the number of LSPs cut to completely manageable levels!
however 🫢 the number of LSPs is not the only issue (although it may have some impact for some of the scaling concerns listed in Section 4). we're more interested in the amount of LSP state maintained by an LSR. that reflects the amount of storage required at the LSR, the amount of protocol processing, and the amount of information that needs to be managed 🧠
fun fact: we were also interested in this measure of scalability in the earlier sections of this document 💖 but in those cases we could see a direct correlation between the number of LSPs and the amount of LSP state, since transit LSPs had two pieces of state information (one on the incoming and one on the outgoing interface), and ingress or egress LSPs had just one piece of state 🧃
we can quantify the amount of LSP state according to the number of LSP segments managed by an LSR 🔢 so (as above), for a P2P LSP, an ingress or egress has one segment to maintain, while a transit has two segments. similarly, for an MP2P LSP, an LSR has to maintain one set of state information for each upstream segment (which we can assume is in a one-to-one relationship with the number of upstream neighbors) and exactly one downstream segment. ingresses obviously have no upstream neighbors, and egresses have no downstream segments 🌟
so we can start again on our examination of the scaling properties of MP2P LSPs, using X(n) to represent the amount of LSP state held at each P(n) node 🦋
8.3. Scaling Improvements for Snowflake Networks ❄️
at the PEs, there's only connectivity to one other network node: the P(2) node 🥰 but if P2P LSPs need to be used to allow disambiguation of data at the MP2P LSP egresses, then those P2P LSPs are tunneled within the MP2P LSPs 🪆 so X(PE) is:
X(PE) = 2*(S(PE) - 1) if no disambiguation is required,
and
X(PE) = 4*(S(PE) - 1) if disambiguation is required.each P(2) node has M(2) downstream PEs 🏠 the P(2) sees a single MP2P LSP targeted at each downstream PE, with one downstream segment (to that PE) and M(2) - 1 upstream segments from the other subtended PEs. on top of that, each of these LSPs has an upstream segment from the one upstream P(1). this gives a total of M(2)*(1 + M(2)) LSP segments 🧮
there are also LSPs running from the subtended PEs to every other PE in the network 🌐 there are S(PE) - M(2) such PEs, and the P(2) sees one upstream segment for each of these from each subtended PE. it also has one downstream segment for each of these LSPs. this gives (M(2) + 1)*(S(PE) - M(2)) LSP segments 💫
so:
X(2) = M(2)*(1 + M(2)) + (M(2) + 1)*(S(PE) - M(2))
= S(PE)*(M(2) + 1)similarly, at each P(1) node there are M(1) downstream P(2) nodes, so a total of M(1)*M(2) downstream PEs 🪞 each P(1) is connected in a full mesh with the other P(1) nodes, so it has (S(1) - 1) neighbors 🕸️
the P(1) sees a single MP2P LSP targeted at each downstream PE. this has one downstream segment (to the P(2) the PE is connected to) and M(1) - 1 upstream segments from the other subtended P(2) nodes. additionally, each of these LSPs has an upstream segment from each of the P(1) neighbors. this gives a total number of LSP segments of M(1)M(2)(M(1) + S(1) - 1) 🧁
there are also LSPs running from each of the subtended PEs to every other PE in the network 🌍 there are S(PE) - M(1)M(2) such PEs, and the P(1) sees one upstream segment for each of these from each subtended P(2) (since the aggregation from the subtended PEs has already happened at the P(2) nodes). it also has one downstream segment to the appropriate next-hop P(1) neighbor for each of these LSPs. this gives (M(1) + 1)*(S(PE) - M(1)*M(2)) LSP segments 🎀
so:
X(1) = M(1)*M(2)*(M(1) + S(1) - 1) +
(M(1) + 1)*(S(PE) - M(1)*M(2))
= M(1)*M(2)*(S(1) - 2) + S(PE)*(M(1) + 1)for example, with S(1) = 10, M(1) = 10, and M(2) = 10, we get:
S(PE) = 1000
S(2) = 100
X(PE) = 3996 (or 1998)
X(2) = 11000
X(1) = 11800and similarly, with S(1) = 20, M(1) = 20, and M(2) = 5, we get:
S(PE) = 2000
S(2) = 400
X(PE) = 5996 (or 2998)
X(2) = 12000
X(1) = 398008.3.1. Comparison with Other Scenarios 🔬
for comparison with the examples in Sections 5 and 6, we need to convert those LSP-based figures to our new measure of LSP state 🔄
observe that each LSP in Sections 5 and 6 generates two state units at a transit LSR and one at an ingress or egress. so we can provide conversions like this:
Section 5 (flat snowflake network)
L(PE) = 2*(S(PE) - 1)
L(2) = M(2)*(2*S(PE) - M(2) - 1)
L(1) = M(1)*M(2)*(2*S(PE) - M(2)*(M(1) + 1))
X(PE) = 2*(S(PE) - 1)
X(2) = 2*M(2)*(2*S(PE) - M(2) - 1)
X(1) = 2*M(1)*M(2)*(2*S(PE) - M(2)*(M(1) + 1))for the example with S(1) = 10, M(1) = 10, and M(2) = 10, this gives a comparison table:
Count | Unmodified | MP2P
------+-------------+----------
X(PE) | 1998 | 3996
X(2) | 39780 | 11000
X(1) | 378000 | 11800clearly, this technique is a significant improvement over the flat network within the core of the network 🚀 although the PEs are more heavily stressed if disambiguation is required 😬
Section 6.1 (two-layer hierarchy snowflake network)
L(PE) = 2*(S(PE) - 1)
L(2) = M(2)*(2*S(PE) - M(2) - 1) + 2*(S(2) - 1)
L(1) = M(1)*(2*S(2) - M(1) - 1)
X(PE) = 2*(S(PE) - 1)
X(2) = 2*M(2)*(2*S(PE) - M(2) - 1) + 2*(S(2) - 1)
X(1) = 2*M(1)*(2*S(2) - M(1) - 1)note that in the computation of X(2), the hierarchical LSPs only add one state at each P(2) node 🤓
for the same example with S(1) = 10, M(1) = 10, and M(2) = 10, this gives a comparison table:
Count | 2-Layer | MP2P
| Hierarchy |
------+-------------+----------
X(PE) | 1998 | 3996
X(2) | 39978 | 11000
X(1) | 3780 | 11800we can observe that the MP2P model is better at P(2), but the hierarchical model is better at P(1) 💅
in fact, this comparison can be generalized: the MP2P model produces its best effects toward the edge of the network, while the hierarchical model makes most of its impression at the core 🫶 however, the requirement for disambiguation of P2P LSPs tunneled within the MP2P LSPs does cause a double burden at the PEs 😭💖
8.4. Scaling Improvements for Ladder Networks ✨
okay so MP2P only inside the ladder? barely moves the needle 😭 but slap MP2P on every LSP at every node and it's giving massive glow-up, and you don't have to configure anything extra 💅
the LSP state at a spar-node splits into two buckets 🧚. there are the segments that come in or go out because of the PEs hanging off it (local LSP segments), and the ones that come in or go out because of PEs far away (remote segments) 🌸
the local segments can be counted like this:
- E LSPs targeting local PEs
- (S(1)-1)*E*M(1) LSPs targeting remote PEs
the remote segments can be counted like this:
- (S(1)-1)*E outgoing LSPs targeting remote PEs
- <= 3*S(1)*E incoming LSPs targeting any PE (there are precisely P(1) nodes attached to any other P(1) node)
so if we use X(1) as a measure of LSP state (not a plain count of LSPs), we get:
X(1) <= E + (S(1)-1)*E*M(1) + (S(1)-1)*E + 3*S(1)*E
<= (4 + M(1))*S(1)*E - M(1)*Ethe P(2) nodes get a glow-up too 🦋 we can count their LSP state the same way, so there are:
- M(2) LSPs targeting local PEs,
- M(2)*(S(1)*E) LSPs from local PEs to all other PEs, and
- S(1)*E - M(2) LSPs to remote PEs.
so using X(2) as a measure of LSP state and not a count of LSPs, we have:
X(2) = M(2) + M(2)*(S(1)*E) + S(1)*E - M(2)
= (M(2) + 1)*S(1)*Elet's plug in the examples from Section 5.2 🎀
With S(1) = 6, M(1) = 10, and M(2) = 17, we see:
E = 170
S(PE) = 1020
X(PE) = 2038
X(2) = 18360
X(1) <= 12580Alternatively, with S(1) = 10, M(1) = 10, and M(2) = 20, we see:
E = 200
S(PE) = 2000
X(PE) = 3998
X(2) = 42000
X(1) <= 260008.4.1. Comparison with Other Scenarios
MP2P absolutely ate across every scaling scenario 🔥 it's the only technique that can actually shrink X(2), and it does it by almost a factor of two. for X(1), it beats everything except the three-level hierarchy 👑
here's a quick cross-reference table for the ladder example figures 🍓 heads up: the earlier numbers got tweaked so they count LSP state instead of LSP numbers. same deal as before: each LSP contributes one state at its endpoints and two states at transit nodes.
so for all the cases:
X(PE) = 2*(S(PE) - 1) or
X(PE) = 4*(S(PE) - 1) if disambiguation is required.for the unmodified (flat) case:
X(2) = 2*(M(2)*(2*S(PE) - M(2) - 1))
X(1) = 2*(M(1)*M(2)*(2*S(PE) - M(2)*(M(1) + 1)))for the two-level hierarchy:
X(2) = 2*(2*M(2)*(S(PE) - 1) - M(2)*(M(2) - 1))
X(1) = S(1)*S(1) + 2*S(1) + 4*E*E*(S(1) - 1) - 2*E*M(2) - 2for the three-level hierarchy:
X(2) = 2*(2*M(2)*(S(PE) - 1) - M(2)*(M(2) - 1)) + 2*(S(1)*M(1) - 1)
X(1) = S(1)*S(1) + 2*S(1) + 4*M(1)*M(1)*S(1) - 2*M(1)(M(1) + 1) - 2example A: S(1) = 6, M(1) = 10, and M(2) = 17 example B: S(1) = 10, M(1) = 10, and M(2) = 20
Example| Count | Unmodified | 2-Level | 3-Level | MP2P
| | | Hierarchy | Hierarchy |
-------+-------+------------+------------+-------------+-------
A | X(2) | 68748 | 68748 | 68866 | 18360
| X(1) | 1554820 | 572266 | 2226 | 12580
-------+-------+------------+------------+-------------+-------
B | X(2) | 159160 | 159160 | 159358 | 42000
| X(1) | 5032000 | 1433998 | 3898 | 260008.4.2. LSP State Compared with LSP Numbers
remember Section 8.3? we said the real MP2P win is in LSP segment state, not the raw LSP count 🧠 because MP2P LSPs need state on every branch, the savings don't scale in a straight line with the drop in LSP numbers. it's giving nuance 💜
you could run the same analysis for the ladder network. the net effect is that the state goes up by an order of two for all transit LSPs in the P2P models, and by a multiplier equal to the degree of a node in the MP2P model.
a rough estimate says that, just like with snowflake networks, MP2P scales better than the one-level hierarchical model and is way better at the core. but honestly MP2P compares less well with the two-level hierarchy, especially in the core 🥲
8.5. Issues with MP2P LSPs
okay, the biggest challenges for MP2P LSPs are getting support into the control and data planes 🛠️ and the management plane needs some love too 🫶
control plane support is mostly about defining the protocols and procedures [MP2P-RSVP]. just know this will add some complexity to the control plane 🌀
hardware can get a little trickier 😬 for example, the capacity of the upstream segments must never (allowing for statistical over-subscription) exceed the capacity of the downstream segment. and data planes need enough buffers to handle incoming packet collisions 💀
the management plane gets hit in a few ways. first, management apps will need to handle LSPs with multiple senders. so even though they have fewer LSPs to process, they get more complicated and will actually need to process the same number of ingresses and egresses 🙈 diagnostics and OAM also need upgrades to support MP2P, but they could borrow a lot from LDP networks 🌸
last thing: with the MP2P solution, the receiver (the single egress PE of an MP2P tunnel) can't use the incoming label to tell where the data came from. that's different from P2P LSPs 🦋 depending on the deployment, this might not matter, since PE-PE connectivity may already be a tunnel with inner labels that tell the data flows apart ✨
in other deployments, you might want extra PE-PE P2P LSPs tunneled through the MP2P LSPs. that means PEs would need to support twice as many LSPs. PEs aren't usually as beefy as P-routers, so that could be a concern 😅 though using penultimate hop popping on the MP2P LSPs might help with that.
in every case, don't mix up fewer LSPs with less LSP state 🚫 honestly the discussion in Section 8.3 is a bit optimistic, because LSP state toward the destination will probably need to include sender information, so it grows with the number of senders on the MP2P LSP. Section 8.4, on the other hand, counts LSP state rather than LSPs. how this plays out depends on the protocol solution for MP2P RSVP-TE, which is out of scope for this document 📝
MPLS Fast Reroute (FRR) [RFC4090] is a really attractive option for rapid local protection from node or link failures 💖 but FRR hasn't been designed for MP2P yet (at the time of writing), so we'll have to wait and see how practical it is, especially if a merge node fails 👀 early looks suggest FRR wouldn't be a problem for MP2P, since each flow can be handled separately.
also, the MP2P scenario in this document may be a bit optimistic 🫠 MP2P LSP merging can be hard to pull off between LSPs with very different traffic and Quality of Service (QoS) parameters. so you might need more MP2P LSPs arriving at an egress.
9. Combined Models 🌈
nothing stops you from mixing hierarchical and MP2P solutions in the same network 💅
but heads up: if MP2P LSPs get tunneled through P2P FA LSPs across the core, none of the LSP merging benefit shows up for the hops where they're tunneled 😭
on the flip side, you can build solutions where MP2P FA LSPs are constructed within the network, and that gets you savings from both modes of operation 👑
10. An Alternate Solution 🧚
there's a simple solution for cutting down the number of LSP tunnels any node has to handle, and it's been proposed 💖 the idea is that part of the problem is just the total number of LSPs in the network, which depends on the number of PEs, since a full mesh of PE-PE LSPs is required. so the fix is to move the tunnel endpoints further into the network. instead of a full mesh of PE-PE tunnels, you only need a full mesh of P(n)-P(n) tunnels ✨
the physical topology doesn't change, so the PEs are still subtended to the P(n) nodes. the result is that there's no TE on the links between PEs and P(n) nodes 🦋
we already did the heavy lifting for computing LSP counts in the earlier sections, so we get to reuse it. the analysis from before works on this new model too, we just need to tweak the formulae 🙌 the easiest way is to remove a layer from the network. we introduce the term "tunnel end-point" (TEP) and swap the P(n) nodes for TEPs. so the flat snowflake network from Figure 3 turns into the one in Figure 7, and you can make the same changes to all of the sample topologies 🎀
PE PE PE PE PE PE
\ \/ \/ /
PE--TEP TEP TEP TEP--PE
\ | | /
\| |/
PE--TEP---P(1)------P(1)---TEP--PE
/ \ / \
PE \ / PE
\/
P(1)
/|\
/ | \
/ | \
PE--TEP TEP TEP--PE
/ /\ \
PE PE PE PEFigure 7 : An Example Snowflake Network with Tunnel End-Points
to do the scaling math, we only need to swap the PE counts in the formulae for TEP counts, and notice there's one fewer layer in the network 🤓 for example, in the flat snowflake network from Figure 7, the number of LSPs seen at a TEP is:
L(TEP) = 2*(S(TPE) - 1)in our sample networks, S(TPE) is usually around 50 or 100 (the original values of S(2)), so L(TEP) is under 200, which is totally manageable 💜
the number of LSPs handled by a P(1) node can be worked out from the original formula for LSPs seen at a P(2) node, since all we did was drop n in P(n) from 2 to 1. so our new formula is:
L(1) = M(1)*(2*S(TEP) - M(1) - 1)with M(1) = 10 and S(TEP) = 100, this gives us L(1) = 1890. also very manageable 🥰
10.1. Pros and Cons of the Alternate Solution ✨
Okay bestie, on the face of it this alternate solution is giving main character energy 💅. By contracting the edges of the tunnels into the network, we showed a dramatic reduction in the number of tunnels needed, and there's no requirement to apply any additional scaling techniques. Iconic 🦋
But what about the PE-P(n) links? 🙈 Earlier in this document we assumed there was some requirement for PE-PE LSPs with TE properties that extended to the PE-P(n) links at both ends of each LSP. So there was a requirement to provide reservation-based QoS on those links, to be able to discriminate traffic flows for priority-based treatment, and to be able to distinguish applications and sources that send data based on the LSPs that carry the data. Lowkey a lot to ask of the edges, ngl 😭
It might be argued that, since the PE-P(n) links don't offer any routing options (each such link provides the only access to the network for a PE), most of the benefits of tunnels are lost on these peripheral links. Fair point! 🧚 However, TE is not just about routing. Just as important are the abilities to make resource reservations, to prioritize traffic, and to discriminate between traffic from different applications, customers, or VPNs. Not me saying "it's just routing" 🙅♀️
Furthermore, in multihoming scenarios where each PE is connected to more than one P(n) or where a PE has multiple links to a single P(n), there may be a desire to pre-select the link to be used and to direct the traffic to that link using a PE-PE LSP. Note that multihoming has not been considered in this document. 🌸
Operationally, P(n)-P(n) LSPs come with the additional management overhead that we see for hierarchical LSPs described in Section 6. The LSPs have to be configured and established through additional configuration or management operations that aren't carried out at the PEs. As described in Section 6, automesh [RFC4972] could be used to ease this task. But it must be noted that, as mentioned above, some of the key uses of tunnels require that traffic is classified and placed in an appropriate tunnel according to its traffic class, end-points, originating application, and customer (such as client VPN). This information may not be readily available for each packet at the P(n) nodes, since it is PE-based information. Of course, it's possible to conceive of techniques to make this information available, such as assigning a different label for each class of traffic, but that gives rise to the original problem of larger numbers of LSPs. Circular drama 💀
So our conclusion is, therefore, that this alternate technique may be suitable for the general distribution of traffic based solely on the destination, or on a combination of the destination and key fields carried in the IP header. In this case it can provide a very satisfactory answer to the scaling issues in an MPLS-TE network. ✨ But if more sophisticated packet classification and discrimination is required, this technique will make the desired function hard to achieve, and the trade-off between scaling and feature-level will swing too far towards solving the scaling issue at the expense of delivery of function to the customer. Scaling is slay, but not if the customer loses the features 🥲
11. Management Considerations 🎀
The management issues of the models presented in this document have been discussed in-line. No one solution is without its management overhead. Honestly, it's the same story everywhere 🍓
Note, however, that scalability of management tools is one of the motivators for this work, and network scaling solutions that reduce the active management of LSPs at the cost of additional effort to manage the more static elements of the network represent a benefit. Basically, it's worth the extra effort to set up MP2P or FA LSPs if it means the network can be scaled to a larger size without being constrained by the management tools 👑
The MP2P technique may prove harder to debug through OAM methods than the FA LSP approach. Heads up, debugging gonna be a vibe check 🔍
12. Security Considerations 🔐
Okay, this section is serious, so bestie, pay attention 😤 The techniques described in this document use existing or yet-to-be-defined signaling protocol extensions and are subject to the security provided by those extensions. Note that we are talking about tunneling techniques used within the network, and that both approaches are vulnerable to the creation of bogus tunnels that deliver data to an egress or consume network resources. Yikes 💀
The fact that the MP2P technique may prove harder to debug through OAM methods than the FA LSP approach is a security concern, since it is important to be able to detect misconnections. Misconnections being undetected is literally a nightmare 😭
General issues of the relationship between scaling and security are covered in Section 1.1, but the details are beyond the scope of this document. Readers are referred to [MPLS-SEC] for details of MPLS security techniques. 🔒
13. Recommendations 🌈
The analysis in this document suggests that the ability to signal MP2P MPLS-TE LSPs is a desirable addition to the operator's MPLS-TE toolkit. Yay, toolkit upgrade unlocked 🧰
At this stage, no further recommendations are made, but it would be valuable to consult more widely to discover:
- The concerns of other service providers with respect to network scalability.
- More opinions on the realistic constraints to the network parameters listed in Section 4.
- Desirable values for the cost-effectiveness of the network (parameter K).
- The applicability, manageability, and support for the two techniques described.
- The feasibility of combining the two techniques, as discussed in Section 9.
- The level of concern over the loss of functionality that would occur if the alternate solution described in Section 10 was adopted.
So bestie, go ask around 🫶 We need more opinions before we call it.
14. Acknowledgements 💜
The authors are grateful to Jean-Louis Le Roux for discussions and review input. Thanks to Ben Niven-Jenkins, JP Vasseur, Loa Andersson, Anders Gavler, Ben Campbell, and Tim Polk for their comments. Thanks to Dave Allen for useful discussion of the math. The math squad really carried 🧮✨
15. Normative References 📚
[RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP)
Hierarchy with Generalized Multi-Protocol Label Switching
(GMPLS) Traffic Engineering (TE)", RFC 4206, October
2005.16. Informative References 🦋
[RFC2961] Berger, L., Gan, D., Swallow, G., Pan, P., Tommasi, F.,
and S. Molendini, "RSVP Refresh Overhead Reduction
Extensions", RFC 2961, April 2001.
[RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
Tunnels", RFC 3209, December 2001.
[RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen,
P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-
Protocol Label Switching (MPLS) Support of Differentiated
Services", RFC 3270, May 2002.
[RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label
Switching (GMPLS) Signaling Resource ReserVation
Protocol-Traffic Engineering (RSVP-TE) Extensions", RFC
3473, January 2003.
[RFC3985] Bryant, S., Ed., and P. Pate, Ed., "Pseudo Wire Emulation
Edge-to-Edge (PWE3) Architecture", RFC 3985, March 2005.
[RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast
Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090,
May 2005.
[RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3
Provider-Provisioned Virtual Private Networks (PPVPNs)",
RFC 4110, July 2005.
[RFC4972] Vasseur, JP., Ed., Leroux, JL., Ed., Yasukawa, S.,
Previdi, S., Psenak, P., and P. Mabbey, "Routing
Extensions for Discovery of Multiprotocol (MPLS) Label
Switch Router (LSR) Traffic Engineering (TE) Mesh
Membership", RFC 4972, July 2007.
[RFC5036] Andersson, L., Ed., Minei, I., Ed., and B. Thomas, Ed.,
"LDP Specification", RFC 5036, October 2007.
[MP2P-RSVP] Yasukawa, Y., "Supporting Multipoint-to-Point Label
Switched Paths in Multiprotocol Label Switching Traffic
Engineering", Work in Progress, October 2008.
[MPLS-SEC] Fang, L., Ed., "Security Framework for MPLS and GMPLS
Networks", Work in Progress, November 2008.Authors' Addresses 🧚
Seisho Yasukawa
NTT Corporation
9-11, Midori-Cho 3-Chome
Musashino-Shi, Tokyo 180-8585 Japan
Phone: +81 422 59 4769
EMail: s.yasukawa@hco.ntt.co.jp
Adrian Farrel
Old Dog Consulting
EMail: adrian@olddog.co.uk
Olufemi Komolafe
Cisco Systems
96 Commercial Street
Edinburgh
EH6 6LX
United Kingdom
EMail: femi@cisco.com📦 where this page comes from
- ietf: rfc5439 2009-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms