RFC 6894
Methodology for Benchmarking MPLS Traffic Engineered (MPLS-TE) Fast Reroute Protection
🎀 RFC 6894: Benchmarking MPLS-TE Fast Reroute Protection 💖
Part 1 of 6
Okay bestie, this is RFC 6894 📄, titled "Methodology for Benchmarking MPLS Traffic Engineered (MPLS-TE) Fast Reroute Protection." It was published in March 2013 and is Informational, which means it's published for information and isn't a standards track spec 🧠. It's an IETF document with ISSN 2070-1721. The authors are R. Papneja (Huawei Technologies), S. Vapiwala and J. Karthik (Cisco Systems), S. Poretsky (Allot Communications), S. Rao (Qwest Communications), and JL. Le Roux (France Telecom) 🫶. The header doesn't list anything it obsoletes or updates, so it's a standalone doc 🌸.
Abstract ✨
This doc is all about how to benchmark MPLS Fast Reroute (FRR) protection mechanisms for link and node protection 🔥. It gives you test methodologies and a testbed setup for measuring how fast Fast Reroute failovers happen. It also looks at the stuff that can change recovery times, like the underlying links, for real-time apps running on MPLS Traffic Engineered (MPLS-TE) tunnels 💅.
Status of This Memo
This document is not an Internet Standards Track specification. It's published for informational purposes 🦋.
It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community 💜. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you're curious 🤓.
For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc6894 🌈.
Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved ©️.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were on the publication date. Please read these carefully because they spell out your rights and restrictions for this doc 📜. Code Components pulled from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License.
This doc may include material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The people who control the copyright in some of that material may not have given the IETF Trust permission to allow modifications outside the IETF Standards Process. Unless you get an adequate license from those copyright holders, this doc may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process. The only exceptions are formatting it for publication as an RFC or translating it into languages other than English 🙈.
Table of Contents 🧚
1. Introduction ....................................................3
2. Document Scope ..................................................5
3. Existing Definitions and Requirements ...........................5
4. General Reference Topology ......................................6
5. Test Considerations .............................................7
5.1. Failover Events ............................................7
5.2. Failure Detection ..........................................8
5.3. Use of Data Traffic for MPLS Protection Benchmarking .......8
5.4. LSP and Route Scaling ......................................9
5.5. Selection of IGP ...........................................9
5.6. Restoration and Reversion ..................................9
5.7. Offered Load ...............................................9
5.8. Tester Capabilities .......................................10
5.9. Failover Time Measurement Methods .........................10
6. Reference Test Setup ...........................................11
6.1. Link Protection ...........................................12
6.1.1. Link Protection: 1-Hop Primary (from PLR)
and 1-Hop Backup Tail-End Tunnels ..................12
6.1.2. Link Protection: 1-Hop Primary (from PLR)
and 2-Hop Backup Tail-End Tunnels ..................13
6.1.3. Link Protection: 2-Hop (or More) Primary (from PLR)
and 1-Hop Backup Tail-End Tunnels ..................14
6.1.4. Link Protection: 2-Hop (or More) Primary (from PLR)
and 2-Hop Backup Tail-End Tunnels ..................15
6.2. Node Protection ...........................................16
6.2.1. Node Protection: 2-Hop Primary (from PLR)
and 1-Hop Backup Tail-End Tunnels ..................16
6.2.2. Node Protection: 2-Hop Primary (from PLR)
and 2-Hop Backup Tail-End Tunnels ..................17
6.2.3. Node Protection: 3-Hop (or More) Primary (from PLR)
and 1-Hop Backup Tail-End Tunnels ..................18
6.2.4. Node Protection: 3-Hop (or More) Primary (from PLR)
and 2-Hop Backup Tail-End Tunnels ..................19
7. Test Methodology ...............................................19
7.1. MPLS-FRR Forwarding Performance ...........................20
7.1.1. Head-End PLR Forwarding Performance ................20
7.1.2. Midpoint PLR Forwarding Performance ................21
7.2. Head-End PLR with Link Failure ............................22
7.3. Midpoint PLR with Link Failure ............................24
7.4. Head-End PLR with Node Failure ............................25
7.5. Midpoint PLR with Node Failure ............................26
8. Reporting Format ...............................................27
9. Security Considerations ........................................29
10. Acknowledgements ..............................................29
11. References ....................................................29
11.1. Normative References .....................................29
11.2. Informative References ...................................30
Appendix A. Fast Reroute Scalability Table ........................31
Appendix B. Abbreviations .........................................341. Introduction ✨
This doc describes how to benchmark MPLS Fast Reroute (FRR) protection mechanisms 💖. It uses a lot of the terminology defined in [RFC6414] 📚.
Protection mechanisms help recover client services from planned or unplanned link or node failures 🛟. MPLS-FRR protection mechanisms are usually deployed in network infrastructure where MPLS is used to set up point-to-point traffic engineered tunnels (tunnel). Their whole goal is to shrink the service disruption period by cutting down recovery time for the most common failures 🏃♀️.
Network elements from different manufacturers react differently to network failures, and that changes how well the network can recover and perform 👀. So service providers really need a common benchmark to understand how network elements behave 🔥.
Service availability comes down to two things: how often failures happen and how long they last ⏱️. Failures can be split into two types, correlated and uncorrelated, and either kind may be planned or unplanned 🧩.
Planned failures are usually predictable 📅. Network implementations should be able to handle both planned and unplanned failures and recover gracefully within a time frame to keep service assurance up 🫶. That's why failover recovery time is one of the biggest benchmarks a service provider looks at when picking building blocks for their network 👑.
A correlated failure happens when two or more failures occur together 💀. A classic example is a logical resource, like a Layer-2 (L2) link, failing because it depends on a common physical resource, like a common conduit, that fails. In MPLS protection, failures caused by Shared Risk Link Groups (SRLGs) [RFC4202] count as correlated failures 🌸.
MPLS-FRR [RFC4090] allows for the possibility that Label Switched Paths (LSPs) get reoptimized in the minutes after failover 🕰️. IP traffic would then be rerouted along the preferred path for the post-failure topology. So MPLS-FRR may include extra steps after the failure is detected and the failover event happens [RFC6414] ✨.
Here's the sequence, in order 🦋:
(1) Failover Event - Primary path (working path) fails
(2) Failure Detection - Failover event is detected
(3a) Failover - Working path switched to backup path
(3b) Reoptimization of working path (possible change from backup
path)
(4) Restoration (see Section 3.3.5 of [RFC6414])
(5) Reversion (see Section 3.3.6 of [RFC6414])2. Document Scope 🎯
This doc gives detailed test cases, plus different topologies and scenarios, that should be considered to effectively benchmark MPLS-FRR protection mechanisms and failover times on the data plane 💅. It also covers different failover events and scaling considerations 🌈.
All benchmarking test cases in this doc apply to facility backup [RFC4090] 🔧. The test cases cover a set of interesting failure scenarios, and the procedures attached to them benchmark how well the Device Under Test (DUT) recovers from failures 🧪. Data-plane traffic is used to benchmark failover times ⚡.
Testing MPLS-TE protection mechanisms as applied to MPLS Transport Profile, and IP fast reroute applied to MPLS networks, were not considered. Those are outside the scope of this doc 🙅♀️.
That said, the test setups for MPLS-based L3 and L2 services do consider LDP over MPLS RSVP-TE configurations 🫧.
Benchmarking correlated failures is outside the scope of this doc 🚫. Detection using Bidirectional Forwarding Detection (BFD) is also outside the scope, but it does get mentioned in the discussion sections 🙈.
The performance of the control plane is outside the scope of this doc too 🛑.
As described above, MPLS-FRR may include a reoptimization of the working path, and that can come with possible packet transfer impairments 💫. Characterizing reoptimization is beyond the scope of this memo, so we're leaving it alone for now 💜.
3. Existing Definitions and Requirements ✨
okay so first things first, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc are read the way BCP 14 [RFC2119] says 🎀 ngl that's the rulebook for them. [RFC2119] is mostly written for Standards Track docs, but this one is Informational, so it only borrows some of these words 🦋
you're expected to already know the common MPLS vocab, some of which lives in [RFC4090] 🧚
this doc also uses a bunch of terms from [RFC6414], plus some from other BMWG docs [RFC1242] [RFC2285] [RFC4689] 💅 Appendix B has the list of abbreviations if you get lost 🌸
4. General Reference Topology 🌈
okay bestie, Figure 1 is the basic reference testbed, and it works for every test case in this doc 😭 the Tester is a Traffic Generator (TG), a Traffic Analyzer (TA), and an Emulator all together 💜 the Tester hooks up to the test network, and the DUT can change depending on the test case. it sends and receives IP traffic to the tunnel ingress and does signaling protocol emulation so the lab can act like a real network 🔥 the Tester may also do MPLS-TE signaling so it can act as the ingress node of the MPLS tunnel ✨ and the lines in the figures are physical connections, just so we're clear 🫶
+---------------------------+
| +------------|---------------+
| | | |
| | | |
+--------+ +--------+ +--------+ +--------+ +--------+
TG--| R1 |-----| R2 |----| R3 | | R4 | | R5 |
| |-----| |----| |----| |---| |
+--------+ +--------+ +--------+ +--------+ +--------+
| | | | |
| | | | |
| +--------+ | | TA
+---------| R6 |---------+ |
| |----------------------+
+--------+
Figure 1the tester MUST record the number of lost, duplicate, and out-of-order packets 💀 it should also record arrival and departure times (lowercase here, it's just a suggestion bestie) so failover time, Additive Latency, and Reversion Time can be measured 📊 the tester may be one device or a whole test system pretending to be all the different roles along a primary or backup path 🙈
the label stack depends on three things:
- Type of protection (Link versus Node)
- Number of remaining hops of the primary tunnel from the Point of Local Repair (PLR) [RFC6414]
- Number of remaining hops of the backup tunnel from the PLR
because it depends on all of that, it is RECOMMENDED that you benchmark failover times on all the topologies in Section 6 🍓 no skipping, it's giving thoroughness 👑
5. Test Considerations 💖
this section is the fundamentals of MPLS Protection testing, so pay attention 💅 it covers:
- The types of network events that cause failover (Section 5.1)
- Indications for failover (Section 5.2)
- The use of data traffic (Section 5.3)
- Label Switched Path Scaling (Section 5.4)
- IGP Selection (Section 5.5)
- Reversion of LSP (Section 5.6)
- Traffic generation (Section 5.7)
5.1. Failover Events 🌸
okay so failover to the backup tunnel is mostly triggered by link or node failures seen downstream of the Point of Local Repair (PLR) 🦋 the failure events from [RFC6414] are below, and yes it's a lot, but it's giving completeness ✨
Link Failure Events
- Interface Shutdown on PLR side with physical/link alarm
- Interface Shutdown on remote side with physical/link alarm
- Interface Shutdown on PLR side with RSVP hello enabled
- Interface Shutdown on remote side with RSVP hello enabled
- Interface Shutdown on PLR side with BFD
- Interface Shutdown on remote side with BFD
- Fiber Pull on the PLR side (both Transmit (TX) and Receive (RX) or just the TX)
- Fiber Pull on the remote side (both TX and RX or just the RX)
- Online Insertion and Removal (OIR) on PLR side
- OIR on remote side
- Sub-interface failure on PLR side (e.g., shutting down of a VLAN)
- Sub-interface failure on remote side
- Parent interface shutdown on PLR side (an interface bearing multiple sub-interfaces)
- Parent interface shutdown on remote side
Node Failure Events
- A System reload initiated by either a graceful shutdown or a power failure
- A system crash due to a software failure or an assert
5.2. Failure Detection 🧚
link failure detection [RFC6414] time depends on the link type and whichever failure detection protocols are running 🫧 for SONET / SDH, the alarm type (like LOS, AIS, or RDI) can be used 🌈 other link types have L2 alarms, but they may not be fast enough for detection 😭 ethernet links with MPLS/IP don't have L2 failure indicators, so they rely on L3 signaling for detection 💜 however, for directly connected devices, remote fault indication in the ethernet auto-negotiation scheme could be considered as a kind of L2 link failure indicator 👀
MPLS has different failure detection techniques, like BFD or RSVP hellos 🔥 these can cover the L3 failure indicators ethernet links need, or help other non-ethernet links detect failures faster 💅 however, these fast failure detection mechanisms are out of scope, so we're not testing them here 🙅♀️
the test procedures in this doc can be used for local or remote failure scenarios for comprehensive benchmarking, and to evaluate failover performance independent of the failure detection techniques 🦋
5.3. Use of Data Traffic for MPLS Protection Benchmarking 🎀
currently, end customers use packet loss as a key metric for failover time [RFC6414] 💖 Failover Packet Loss [RFC6414] is externally observable and directly affects application performance 😩 MPLS protection is expected to minimize packet loss when a failure happens, so it's important to develop a standard router benchmarking methodology for MPLS protection that uses packet loss as the metric 🫶 at a known forwarding rate, packet loss can be measured and failover time determined ✨ measuring control-plane signaling to set up backup paths is not enough to verify failover 🙅♀️ failover is best determined when packets are actually traversing the backup path 🛤️
another perk of using packet loss for failover time is that it allows a black-box test environment 🖤 data traffic is offered to the DUT at line-rate, an emulated network failure is forced, and packet loss is measured externally to calculate convergence time 📈 this setup is independent of the DUT architecture, which is so iconic 👑
this methodology also considers packets in error and duplicate packets [RFC4689] that could be generated during the failover process 🌷 the methodologies treat lost, out-of-order [RFC4689], and duplicate packets as impaired packets that contribute to the failover time 💀
5.4. LSP and Route Scaling 🧁
failover time performance may vary with the number of established primary and backup tunnel LSPs and installed routes 🎢 however, this procedure should be used for any number of LSPs (L) and any number of routes protected by the PLR (R) 💅 the values of L and R must be recorded 📝
5.5. Selection of IGP 🌍
the underlying IGP could be ISIS-TE or OSPF-TE for the methodology proposed here 🧠 see [RFC6412] for IGP options to consider and report 📚
5.6. Restoration and Reversion 🔄
path restoration [RFC6414] gives you a way to restore an alternate primary LSP after a failure, and to switch traffic from the backup path back to the restored primary path (that's reversion) 🫧 in MPLS-FRR, reversion [RFC6414] can be implemented as Global Reversion or Local Reversion 🌈 it's important to include restoration and reversion as a step in each test case, so you can measure the packet loss, out-of-order packets, or duplicate packets that get produced 🔍
Note: In addition to restoration and reversion, reoptimization can take place while the failure is still not recovered, but it depends on the user configuration and reoptimization timers. ⏰
5.7. Offered Load 🛒
it's suggested that there be three or more traffic streams, as long as there is a steady and constant flow rate for all of them 🌊 to monitor DUT performance for recovery times, a set of route prefixes should be advertised before traffic is sent, and the traffic should be aimed at these routes 🎯
prefix-dependency behaviors are key in IP, and tests with route-specific flows spread across the routing table will reveal this dependency 🗺️ generating traffic to every prefix reachable by the protected tunnel (probably round-robin, where traffic goes to all the prefixes but one at a time in a cycle) is not recommended 🙅♀️ round-robin traffic generation to all prefixes is not recommended, because the time to hit all the prefixes may be higher than the failover time 😵 this reduces the granularity of the measured results, and the results observed may not be accurate 🥺
5.8. Tester Capabilities 🎀
it is RECOMMENDED that the Tester used to execute each test case have the following capabilities 💖
- Ability to establish MPLS-TE tunnels and push/pop labels.
- Ability to produce a failover event [RFC6414].
- Ability to insert a timestamp in each data packet's IP payload.
- An internal time clock to control timestamping, time measurements, and time calculations.
- Ability to disable or tune specific L2 and L3 protocol functions on any interface.
- Ability to react upon the receipt of path error from the PLR.
the Tester MAY be capable of making non-data-plane convergence observations and use those observations for measurements 🫶
5.9. Failover Time Measurement Methods 📏
failover time [RFC6414] is calculated using one of the following three methods 🧮
- Packet-Loss-Based Method (PLBM): (Number of packets dropped/ packets per second * 1000) milliseconds. This method could also be referred to as the Loss-Derived method. 📉
- Time-Based Loss Method (TBLM): This method relies on the ability of the traffic generators to provide statistics that reveal the duration of failure in milliseconds based on when the packet loss occurred (interval between non-zero packet loss and zero loss). ⏱️
- Timestamp-Based Method (TBM): This method of failover calculation is based on the timestamp that gets transmitted as payload in the packets originated by the generator. The traffic analyzer records the timestamp of the last packet received before the failover event and the first packet after the failover, and derives the time based on the difference between these two timestamps. Note: The payload could also contain sequence numbers for out-of-order packet calculation and duplicate packets. 🕰️
TBM would be able to detect reversion impairments beyond loss, so it is RECOMMENDED as the failover time method 🏆👑
6. Reference Test Setup ✨
okay so bestie, this is where we get to the fun part 🎀 Figure 1 gave us the big picture, and this section zooms in on the specific test setups you should consider for benchmarking failover time wherever the DUT sits along the primary tunnel 🦋
Every topology here maps back onto Figure 1 (no cap, it's all one family 🫶). These setups cover every local protection scenario, and they're aimed at the testbed you need when the DUT is the PLR, whether it's acting as head-end or midpoint 💅 Each topology comes with the label stack at the PLR. Penultimate Hop Popping (PHP) MAY be used, and if you use it, you must report that.
Figures 2 through 9 all follow the same little cast of characters, and they're just subsets of Figure 1:
- a) HE is Head-End
- b) T/E is Tail-End
- c) MID is Midpoint
- d) MP is Merge Point
- e) PLR is Point of Local Repair
- f) PRI is Primary Path
- g) BKP denotes Backup Path and Nodes
- h) UR is Upstream Router
6.1. Link Protection
6.1.1. Link Protection: 1-Hop Primary (from PLR) and 1-Hop Backup Tail-End Tunnels
+-------+ +--------+ +--------+
| R1 | | R2 | PRI| R3 |
| UR/HE |--| HE/MID |----| MP/T/E |
| | | PLR |----| |
+-------+ +--------+ BKP+--------+
Figure 2 Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 0 0
Layer3 VPN (PE-PE) 1 1
Layer3 VPN (PE-P) 2 2
Layer2 VC (PE-PE) 1 1
Layer2 VC (PE-P) 2 2
Midpoint LSPs 0 0Here's the rundown 💖
- a) For the P-P case, R2 and R3 act as P routers
- b) For the PE-PE cases, R2 acts as a PE and R3 acts as a remote PE
- c) For the PE-P cases, R2 acts as a PE router, R3 acts as a P router, and R5 acts as a remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, and R3 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.1.2. Link Protection: 1-Hop Primary (from PLR) and 2-Hop Backup Tail-End Tunnels
+-------+ +--------+ +--------+
| R1 | | R2 | | R3 |
| UR/HE | | HE/MID |PRI | MP/T/E |
| |----| PLR |----| |
+-------+ +--------+ +--------+
|BKP |
| +--------+ |
| | R6 | |
|----| BKP |----|
| MID |
+--------+
Figure 3 Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 0 1
Layer3 VPN (PE-PE) 1 2
Layer3 VPN (PE-P) 2 3
Layer2 VC (PE-PE) 1 2
Layer2 VC (PE-P) 2 3
Midpoint LSPs 0 1Please note the following 🌸
- a) For the P-P case, R2 and R3 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R3 acts as a remote PE
- c) For PE-P cases, R2 acts as a PE router, R3 acts as a P router, and R5 acts as a remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, and R3 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.1.3. Link Protection: 2-Hop (or More) Primary (from PLR) and 1-Hop Backup Tail-End Tunnels
+--------+ +--------+ +--------+ +--------+
| R1 | | R2 |PRI | R3 |PRI | R4 |
| UR/HE |----| HE/MID |----| MP/MID |------| T/E |
| | | PLR |----| | | |
+--------+ +--------+ BKP+--------+ +--------+
Figure 4 Traffic No. of Labels Num of labels
before failure after failure
IP TRAFFIC (P-P) 1 1
Layer3 VPN (PE-PE) 2 2
Layer3 VPN (PE-P) 3 3
Layer2 VC (PE-PE) 2 2
Layer2 VC (PE-P) 3 3
Midpoint LSPs 1 1Please note the following 🧚
- a) For the P-P case, R2, R3, and R4 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R4 acts as a remote PE c) For PE-P cases, R2 acts as a PE router, R3 acts as a P router, and R5 acts as remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, R3, and R4 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.1.4. Link Protection: 2-Hop (or More) Primary (from PLR) and 2-Hop Backup Tail-End Tunnels
+--------+ +--------+PRI +--------+ PRI +--------+
| R1 | | R2 | | R3 | | R4 |
| UR/HE |----| HE/MID |----| MP/MID|------| T/E |
| | | PLR | | | | |
+--------+ +--------+ +--------+ +--------+
BKP| |
| +--------+ |
| | R6 | |
+---| BKP |-
| MID |
+--------+
Figure 5 Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 1 2
Layer3 VPN (PE-PE) 2 3
Layer3 VPN (PE-P) 3 4
Layer2 VC (PE-PE) 2 3
Layer2 VC (PE-P) 3 4
Midpoint LSPs 1 2Please note the following 💜
- a) For the P-P case, R2, R3, and R4 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R4 acts as a remote PE
- c) For PE-P cases, R2 acts as a PE router, R3 acts as a P router, and R5 acts as remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, R3 and R4 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.2. Node Protection
6.2.1. Node Protection: 2-Hop Primary (from PLR) and 1-Hop Backup Tail-End Tunnels
+--------+ +--------+ +--------+ +--------+
| R1 | | R2 |PRI | R3 | PRI | R4 |
| UR/HE |----| HE/MID |----| MID |------| MP/T/E |
| | | PLR | | | | |
+--------+ +--------+ +--------+ +--------+
|BKP |
-----------------------------
Figure 6 Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 1 0
Layer3 VPN (PE-PE) 2 1
Layer3 VPN (PE-P) 3 2
Layer2 VC (PE-PE) 2 1
Layer2 VC (PE-P) 3 2
Midpoint LSPs 1 0Please note the following 🦋
- a) For the P-P case, R2, R3, and R4 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R4 acts as a remote PE
- c) For PE-P cases, R2 acts as a PE router, R4 acts as a P router, and R5 acts as remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, R3, and R4 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.2.2. Node Protection: 2-Hop Primary (from PLR) and 2-Hop Backup Tail-End Tunnels
+--------+ +--------+ +--------+ +--------+
| R1 | | R2 | | R3 | | R4 |
| UR/HE | | HE/MID |PRI | MID |PRI | MP/T/E |
| |----| PLR |----| |----| |
+--------+ +--------+ +--------+ +--------+
| |
BKP| +--------+ |
| | R6 | |
---------| BKP |---------
| MID |
+--------+
Figure 7 Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 1 1
Layer3 VPN (PE-PE) 2 2
Layer3 VPN (PE-P) 3 3
Layer2 VC (PE-PE) 2 2
Layer2 VC (PE-P) 3 3
Midpoint LSPs 1 1Please note the following 🌈
- a) For the P-P case, R2, R3, and R4 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R4 acts as a remote PE
- c) For PE-P cases, R2 acts as a PE router, R4 acts as a P router, and R5 acts as remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, R3, and R4 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.2.3. Node Protection: 3-Hop (or More) Primary (from PLR) and 1-Hop Backup Tail-End Tunnels
+--------+ +--------+PRI+--------+PRI+--------+PRI+--------+
| R1 | | R2 | | R3 | | R4 | | R5 |
| UR/HE |--| HE/MID |---| MID |---| MP |---| T/E |
| | | PLR | | | | | | |
+--------+ +--------+ +--------+ +--------+ +--------+
BKP| |
--------------------------
Figure 8 Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 1 1
Layer3 VPN (PE-PE) 2 2
Layer3 VPN (PE-P) 3 3
Layer2 VC (PE-PE) 2 2
Layer2 VC (PE-P) 3 3
Midpoint LSPs 1 1Please note the following ✨
- a) For the P-P case, R2, R3, R4, and R5 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R5 acts as a remote PE
- c) For PE-P cases, R2 acts as a PE router, R4 acts as a P router, and R5 acts as remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, R3, R4, and R5 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
6.2.4. Node Protection: 3-Hop (or More) Primary (from PLR) and 2-Hop Backup Tail-End Tunnels ✨
Okay bestie, here's the setup 🧚. R2 is the PLR, and the primary path runs through R3, R4, and R5. The backup goes through R6, which is a midpoint that bypasses the protected node. 🦋
+--------+ +--------+ +--------+ +--------+ +--------+
| R1 | | R2 | | R3 | | R4 | | R5 |
| UR/HE | | HE/MID |PRI| MID |PRI| MP |PRI| T/E |
| |-- | PLR |---| |---| |---| |
+--------+ +--------+ +--------+ +--------+ +--------+
BKP| |
| +--------+ |
| | R6 | |
---------| BKP |-------
| MID |
+--------+
Figure 9
Traffic No. of Labels No. of labels
before failure after failure
IP TRAFFIC (P-P) 1 2
Layer3 VPN (PE-PE) 2 3
Layer3 VPN (PE-P) 3 4
Layer2 VC (PE-PE) 2 3
Layer2 VC (PE-P) 3 4
Midpoint LSPs 1 2Please note the following: 💅
- a) For the P-P case, R2, R3, R4, and R5 act as P routers
- b) For PE-PE cases, R2 acts as a PE and R5 acts as a remote PE
- c) For PE-P cases, R2 acts as a PE router, R4 acts as a P router, and R5 acts as remote PE router (please refer to Figure 1 for complete setup)
- d) For the midpoint case, R1, R2, R3, R4, and R5 act as HE, midpoint/PLR, and tail-end, respectively (as shown in the figure above)
7. Test Methodology 🎀
Okay so this part is the actual playbook 📖. The procedure here works for all eight base test cases and their topologies. The backup tunnels and the primary tunnels are set up to use the same bandwidth, which is so fair 🥹. To test failover with every label stack depth you'd see in real deployments, it is RECOMMENDED to run all of the test cases in this section. Also, the forwarding performance tests in Section 7.1 MUST be done before the failover tests, no skipping 🙅♀️.
The considerations of Section 4 of [RFC2544] apply when you're reading the results too, so don't forget them 🌸.
7.1. MPLS-FRR Forwarding Performance
Before you can benchmark failover time [RFC6414] for MPLS protection, you need a baseline of how fast the test topology (including the DUT) forwards traffic 📊. Forwarding performance is measured as throughput, as defined in [RFC5695], in packets per second (pps). This section has two test cases for it, with the DUT set up as a head-end PLR, a midpoint PLR, and an egress PLR ✨.
7.1.1. Head-End PLR Forwarding Performance
Objective:
To benchmark the maximum rate (pps) on the PLR (as head-end) over the primary LSP and backup LSP. 🎯
Test Setup:
- A. Select any one topology out of the eight from Section 6.
- B. Select or enable IP, L3 VPN, or L2 VPN services with the DUT as head-end PLR.
- C. The DUT will also have two interfaces connected to the traffic generator/analyzer. (If the node downstream of the PLR is not a simulated node, then the ingress of the tunnel should have one link connected to the traffic generator, and the node downstream of the PLR or the egress of the tunnel should have a link connected to the traffic analyzer).
Procedure:
- Establish the primary LSP on R2 required by the topology selected.
- Establish the backup LSP on R2 required by the selected topology.
- Verify that primary and backup LSPs are up and that the primary is protected.
- Verify that Fast Reroute protection is enabled and ready.
- Set up traffic streams as described in Section 5.7.
- Send MPLS traffic over the primary LSP at the throughput supported by the DUT (Section 6 of [RFC2544]).
- Record the throughput over the primary LSP.
- Trigger a link failure as described in Section 5.1.
- Verify that the offered load gets mapped to the backup tunnel and measure the Additive Backup Delay [RFC6414].
- 30 seconds after failover, stop the offered load and measure the throughput, packet loss, out-of-order packets, and duplicate packets over the backup LSP.
- Adjust the offered load and repeat steps 6 through 10 until the throughput values for the primary and backup LSPs are equal.
- Record the final throughput, which corresponds to the offered load that will be used for the head-end PLR failover test cases.
7.1.2. Midpoint PLR Forwarding Performance
Objective:
To benchmark the maximum rate (pps) on the PLR (as midpoint) over the primary LSP and backup LSP. 🎯
Test Setup:
- A. Select any one topology out of the eight from Section 6.
- B. The DUT will also have two interfaces connected to the traffic generator.
Procedure:
- Establish the primary LSP on R1 required by the topology selected.
- Establish the backup LSP on R2 required by the selected topology.
- Verify that primary and backup LSPs are up and that the primary is protected.
- Verify that Fast Reroute protection is enabled and ready.
- Set up traffic streams as described in Section 5.7.
- Send MPLS traffic over the primary LSP at the throughput supported by the DUT (Section 6 of [RFC2544]).
- Record the throughput over the primary LSP.
- Trigger a link failure as described in Section 5.1.
- Verify that the offered load gets mapped to the backup tunnel and measure the Additive Backup Delay [RFC6414].
- 30 seconds after failover, stop the offered load and measure the throughput, packet loss, out-of-order packets, and duplicate packets over the backup LSP.
- Adjust the offered load and repeat steps 6 through 10 until the throughput values for the primary and backup LSPs are equal.
- Record the final throughput, which corresponds to the offered load that will be used for the midpoint PLR failover test cases.
7.2. Head-End PLR with Link Failure
Objective:
To benchmark the MPLS failover time due to link failure events described in Section 5.1 experienced by the DUT, which is the head-end PLR. ⏱️
Test Setup:
- A. Select any one topology out of the eight from Section 6.
- B. Select or enable IP, L3 VPN, or L2 VPN services with the DUT as head-end PLR.
- C. The DUT will also have two interfaces connected to the traffic generator/analyzer. (If the node downstream of the PLR is not a simulated node, then the ingress of the tunnel should have one link connected to the traffic generator, and the node downstream to the PLR or the egress of the tunnel should have a link connected to the traffic analyzer).
Test Configuration:
- Configure the number of primaries on R2 and the backups on R2 as required by the topology selected.
- Configure the test setup to support reversion.
- Advertise prefixes (as per the FRR Scalability Table in Appendix A) by the tail-end.
Procedure:
The test case in Section 7.1.1, "Head-End PLR Forwarding Performance", MUST be completed first to obtain the throughput to use as the offered load. 🚨
- Establish the primary LSP on R2 required by the topology selected.
- Establish the backup LSP on R2 required by the selected topology.
- Verify that primary and backup LSPs are up and that the primary is protected.
- Verify that Fast Reroute protection is enabled and ready.
- Set up traffic streams for the offered load as described in Section 5.7.
- Provide the offered load from the tester at the throughput [RFC1242] level obtained from the test case in Section 7.1.1.
- Verify that traffic is switched over the primary LSP without packet loss.
- Trigger a link failure as described in Section 5.1.
- Verify that the offered load gets mapped to the backup tunnel and measure the Additive Backup Delay [RFC6414].
- 30 seconds after failover, stop the offered load and measure the total failover packet loss [RFC6414].
- Calculate the failover time benchmark using the selected failover time calculation method (TBLM, PLBM, or TBM) [RFC6414].
- Restart the offered load and restore the primary LSP to verify that reversion occurs and measure the Reversion Packet Loss [RFC6414].
- Calculate the Reversion Time benchmark using the selected failover time calculation method (TBLM, PLBM, or TBM) [RFC6414].
- Verify that the head-end signals new LSP and protection should be in place again.
It is RECOMMENDED that this procedure be repeated for each of the link failure triggers defined in Section 5.1. 🔁
7.3. Midpoint PLR with Link Failure
Objective:
To benchmark the MPLS failover time due to link failure events described in Section 5.1 experienced by the DUT, which is the midpoint PLR. ⏱️
Test Setup:
- A. Select any one topology out of the eight from Section 6.
- B. The DUT will also have two interfaces connected to the traffic generator.
Test Configuration:
- Configure the number of primaries on R1 and the backups on R2 as required by the topology selected.
- Configure the test setup to support reversion.
- Advertise prefixes (as per the FRR Scalability Table in Appendix A) by the tail-end.
Procedure:
The test case in Section 7.1.2, "Midpoint PLR Forwarding Performance", MUST be completed first to obtain the throughput to use as the offered load. 🚨
- Establish the primary LSP on R1 as required by the topology selected.
- Establish the backup LSP on R2 as required by the selected topology.
- Perform steps 3 through 14 from Section 7.2, "Head-End PLR with Link Failure".
It is RECOMMENDED that this procedure be repeated for each of the link failure triggers defined in section 5.1. 💜
7.4. Head-End PLR with Node Failure 💥
okay so this one is the head-end PLR getting hit by a node failure 😭 we're checking how fast MPLS fails over when the DUT is the head-end PLR and the node failure events from Section 5.1 happen.
Test Setup:
- A. Pick any one of the eight topologies from Section 6. 🗺️
- B. Turn on IP, L3 VPN, or L2 VPN services, with the DUT acting as the head-end PLR. 💅
- C. The DUT also needs two interfaces hooked up to the traffic generator/analyzer. 🔌
Test Configuration:
- Set up the number of primaries on R2, plus the backups on R2, to match the topology you picked. 🎀
- Configure the test setup to support reversion. 🔁
- Have the tail-end advertise prefixes (use the FRR Scalability Table in Appendix A). 📣
Procedure:
The test case in Section 7.1.1, "Head-End PLR Forwarding Performance", MUST be completed first so you can grab the throughput to use as your offered load. This is non-negotiable, bestie. 🫶
- Bring up the primary LSP on R2 for your chosen topology. 🦋
- Bring up the backup LSP on R2 for your chosen topology. 🦋
- Check that the primary and backup LSPs are both up and that the primary is protected. ✅
- Check that Fast Reroute protection is enabled and ready to go. 🔥
- Set up traffic streams for the offered load, as described in Section 5.7. 🌊
- Send the offered load from the tester at the throughput [RFC1242] level you got from Section 7.1.1. 📈
- Make sure traffic is switching over to the primary LSP with zero packet loss. 💖
- Trigger a node failure, as described in Section 5.1. 💀
- Do steps 9 through 14 in Section 7.2, "Head-End PLR with Link Failure". 🧚
It is RECOMMENDED that you repeat this procedure for each of the node failure triggers defined in Section 5.1. 💜
7.5. Midpoint PLR with Node Failure 🌸
so same energy, but now the DUT is the midpoint PLR 👀 we're benchmarking the MPLS failover time when node failure events from Section 5.1 hit the DUT in the middle of the path.
Test Setup:
- A. Pick any one topology from Sections 6.1 to 6.2. 🗺️
- B. The DUT also needs two interfaces connected to the traffic generator. 🔌
Test Configuration:
- Set up the number of primaries on R1, plus the backups on R2, to match the topology you picked. 🎀
- Configure the test setup to support reversion. 🔁
- Have the tail-end advertise prefixes (use the FRR Scalability Table in Appendix A). 📣
Procedure:
The test case in Section 7.1.1, "Midpoint PLR Forwarding Performance", MUST be completed first so you can grab the throughput to use as your offered load. Do not skip this, periodt. 💅
- Bring up the primary LSP on R1 for your chosen topology. 🦋
- Bring up the backup LSP on R2 for your chosen topology. 🦋
- Check that the primary and backup LSPs are both up and that the primary is protected. ✅
- Check that Fast Reroute protection is enabled and ready. 🔥
- Set up traffic streams for the offered load, as described in Section 5.7. 🌊
- Send the offered load from the tester at the throughput [RFC1242] level you got from Section 7.1.1. 📈
- Make sure traffic is switching over to the primary LSP with zero packet loss. 💖
- Trigger a node failure, as described in Section 5.1. 💀
- Do steps 9 through 14 in Section 7.2, "Head-End PLR with Link Failure". 🧚
It is RECOMMENDED that you repeat this procedure for each of the node failure triggers defined in Section 5.1. 💜
8. Reporting Format 📝
so for every test, it is RECOMMENDED that you report your results in this format. it's giving organized. ✨
Parameter Units
IGP used for the test ISIS-TE / OSPF-TE
Interface types Gige,POS,ATM,VLAN, etc.
Packet Sizes offered to the DUT Bytes (at L3)
Offered Load (Throughput) Packets per second
IGP routes advertised Number of IGP routes
Penultimate Hop Popping Used/Not Used
RSVP hello timers Milliseconds
Number of Protected tunnels Number of tunnels
Number of VPN routes installed Number of VPN routes
on the head-end
Number of VC tunnels Number of VC tunnels
Number of midpoint tunnels Number of tunnels
Number of Prefixes protected by Number of LSPs
Primary
Topology being used Section number, and
figure reference
Failover event Event type
Reoptimization Yes/NoBenchmarks (record these for each test case, bestie 🎀):
Failover-
Failover Time seconds
Failover Packet Loss packets
Additive Backup Delay seconds
Out-of-Order Packets packets
Duplicate Packets packets
Failover Time Calculation Method Method Used
Reversion-
Reversion Time seconds
Reversion Packet Loss packets
Additive Backup Delay seconds
Out-of-Order Packets packets
Duplicate Packets packets
Failover Time Calculation Method Method Used9. Security Considerations 🔐
okay this section is serious, so listen up 🚨
The benchmarking described in this memo is limited to technology characterization using controlled stimuli in a laboratory environment. It uses dedicated address space and sticks to the constraints in the sections above. Keep it contained. 🧪
The benchmarking network topology is an independent test setup. It MUST NOT be connected to devices that might forward the test traffic into a production network, or misroute traffic to the test management network. This is the big one. Do not plug your lab into prod. 💀
Also, benchmarking is done on a "black-box" basis. It relies only on measurements you can observe from outside the DUT/SUT. 🖤
Special capabilities SHOULD NOT exist in the DUT/SUT just for benchmarking purposes. Any network security implications coming from the DUT/SUT SHOULD be identical in the lab and in production networks. No lab-only cheats, no lab-only vulnerabilities. Be honest, be consistent. 🫶
10. Acknowledgements 🌷
big thank yous to everyone who made this happen 💖
We'd like to thank Jean Philip Vasseur for his invaluable input to the document. Curtis Villamizar gets a shoutout for suggesting text on defining and needing to benchmark Correlated failures. And Bhavani Parise gets props for textual input and review. We also want to thank Al Morton, Arun Gandhi, Amrit Hanspal, Karu Ratnam, Raveesh Janardan, Andrey Kiselev, and Mohan Nanduri for their formal reviews of this document. The real MVPs. 👑
11. References 📚
11.1. Normative References
[RFC1242] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.
[RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast
Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090,
May 2005.
[RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding
Benchmarking Methodology for IP Flows", RFC 5695, November
2009.
[RFC6412] Poretsky, S., Imhoff, B., and K. Michielsen, "Terminology
for Benchmarking Link-State IGP Data-Plane Route
Convergence", RFC 6412, November 2011.
[RFC6414] Poretsky, S., Papneja, R., Karthik, J., and S. Vapiwala,
"Benchmarking Terminology for Protection Performance", RFC
6414, November 2011.11.2. Informative References
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998.
[RFC4202] Kompella, K., Ed., and Y. Rekhter, Ed., "Routing
Extensions in Support of Generalized Multi-Protocol Label
Switching (GMPLS)", RFC 4202, October 2005.
[RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
"Terminology for Benchmarking Network-layer Traffic
Control Mechanisms", RFC 4689, October 2006.Appendix A. Fast Reroute Scalability Table 📊
this part gives you the recommended numbers for checking how well fast reroute implementations scale 🦋 it also suggests typical numbers for IGP/VPNv4 Prefixes, LSP Tunnels, and VC entries. Based on what the DUT supports, you can pick appropriate scaling limits for your testbed. ✨
A.1. FRR IGP Table
No. of Head-End TE Tunnels IGP Prefixes
1 100
1 500
1 1000
1 2000
1 5000
2 (Load Balance) 100
2 (Load Balance) 500
2 (Load Balance) 1000
2 (Load Balance) 2000
2 (Load Balance) 5000
100 100
500 500
1000 1000
2000 2000A.2. FRR VPN Table
No. of Head-End TE Tunnels VPNv4 Prefixes
1 100
1 500
1 1000
1 2000
1 5000
1 10000
1 20000
1 Max
2 (Load Balance) 100
2 (Load Balance) 500
2 (Load Balance) 1000
2 (Load Balance) 2000
2 (Load Balance) 5000
2 (Load Balance) 10000
2 (Load Balance) 20000
2 (Load Balance) MaxA.3. FRR Midpoint LSP Table
The number of midpoint TE LSPs could be configured at recommended levels: 100, 500, 1000, 2000, or the max supported number. 🌈
A.4. FRR VC Table
No. of Head-End TE Tunnels VC entries
1 100
1 500
1 1000
1 2000
1 Max
100 100
500 500
1000 1000
2000 2000Appendix B. Abbreviations 🎀
okay bestie, this is the cheat sheet 💅 every acronym in the doc gets its own little entry, so you never have to guess what something means. skim it, screenshot it, and you're set 🫶
AIS - Alarm Indication Signal
BFD - Bidirectional Fault Detection
BGP - Border Gateway Protocol
BKP - Backup Path and Nodes
CE - Customer Edge
DUT - Device Under Test
FRR - Fast Reroute
HE - Head-End
IGP - Interior Gateway Protocol
IP - Internet Protocol
LOS - Loss of Signal
LSP - Label Switched Path
MID - Midpoint
MP - Merge Point
MPLS - Multiprotocol Label Switching
N-Nhop - Next - Next Hop
Nhop - Next Hop
OIR - Online Insertion and Removal
P - Provider
PE - Provider Edge
PHP - Penultimate Hop Popping
PLBM - Packet-Loss-Based Method
PLR - Point of Local Repair
PRI - Primary Path
RSVP - Resource reSerVation Protocol
RX - Receive
SRLG - Shared Risk Link Group
TA - Traffic Analyzer
TBM - Timestamp-Based Method
TE - Traffic Engineering
TG - Traffic Generator
TX - Transmit
UR - Upstream Router
VC - Virtual Circuit
VPN - Virtual Private NetworkAuthors' Addresses ✨
and here are the people who wrote this thing 👑 their names, orgs, and how to reach them, all in one place. the squad is giving serious research energy 🧚
Rajiv Papneja
Huawei Technologies
2330 Central Expressway
Santa Clara, CA 95050
USA
EMail: rajiv.papneja@huawei.com
Samir Vapiwala
Cisco Systems
300 Beaver Brook Road
Boxborough, MA 01719
USA
EMail: svapiwal@cisco.com
Jay Karthik
Cisco Systems
300 Beaver Brook Road
Boxborough, MA 01719
USA
EMail: jkarthik@cisco.com
Scott Poretsky
Allot Communications
300 TradeCenter
Woburn, MA 01801
USA
EMail: sporetsky@allot.com
Shankar Rao
Qwest Communications
950 17th Street
Suite 1900
Denver, CO 80210
USA
EMail: shankar.rao@du.edu
JL. Le Roux
France Telecom
2 av Pierre Marzin
22300 Lannion
France
EMail: jeanlouis.leroux@orange.comthat's the whole doc, bestie 🦋💖
📦 where this page comes from
- ietf: rfc6894 2013-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms