RFC 4377

Operations and Management (OAM) Requirements for Multi-Protocol Label Switched (MPLS) Networks

📅 published February 2006, from ietf

RFC 4377: Operations and Management (OAM) Requirements for Multi-Protocol Label Switched (MPLS) Networks 🎀

okay bestie, this one is an Informational RFC from the Network Working Group, dated February 2006 🌸. It was written by T. Nadeau, M. Morrow, and G. Swallow from Cisco Systems, Inc., plus D. Allan from Nortel Networks and S. Matsushima from Japan Telecom. The header doesn't list anything it obsoletes or updates, so it's a standalone doc 💅.

Status of This Memo ✨

This memo gives information to the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright (C) The Internet Society (2006).

Abstract 🦋

okay so this doc lays out Operations and Management (OAM) requirements for Multi-Protocol Label Switching (MPLS), plus the apps that run on top of MPLS, like pseudo-wire voice and virtual private network services. The requirements came from network operators who have seriously deployed MPLS networks, so this is real-world stuff, not theory 💜.

Table of Contents 🧚

1. Introduction ....................................................2
2. Document Conventions ............................................2
3. Motivations .....................................................4
4. Requirements ....................................................4
5. Security Considerations ........................................11
6. References .....................................................12
7. Acknowledgements ...............................................13

1. Introduction ✨

okay, so this section describes what user and data plane Operations and Management (OAM) needs to look like for Multi-Protocol Label Switching (MPLS) 🌈. Same deal as the abstract: the requirements were gathered from operators with lots of MPLS deployment experience, and they cover OAM for MPLS as well as for MPLS applications.

Here's the lowkey important bit 👀. There are no specific mechanisms proposed yet to address these requirements. The goal is to pin down a set of requirements that are commonly applicable to MPLS OAM right now. Specifically, the ones that fit the most common MPLS networks that service provider organizations were running when this doc was written 🍓. Then these requirements can be the base for building network management tools, guide how the tools that already exist evolve, and shape the OAM functions that are built right into the protocols MPLS networks use. iconic, honestly 💅.

2. Document Conventions 💖

2.1. Terminology 🫶

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

Next up, the glossary of terms. These definitions are the exact wording from the RFC, so keep them in a code block for accuracy 📖:

Queuing/buffering Latency: The delay caused by packet queuing (value
                           is variable since it is dependent on the
                           packet arrival rate, the packet length,
                           and the link throughput).

Probe-based-detection:     Active measurement tool that can measure
                           the consistency of an LSP [RFC4379].

Defect:                    Any error condition that prevents a Label
                           Switched Path (LSP) from functioning
                           correctly.  For example, loss of an
                           Interior Gateway Protocol (IGP) path will
                           most likely result in an LSP not being
                           able to deliver traffic to its
                           destination.  Another example is the
                           interruption of the path for a TE tunnel.
                           These may be due to physical circuit
                           failures or failure of switching nodes to
                           operate as expected.

                           Multi-vendor/multi-provider network
                           operation typically requires agreed upon
                           definitions of defects (when it is broken
                           and when it is not) such that both
                           recovery procedures and service level
                           specification impact can be specified.

Head-end Label Switching
Router (LSR):              The beginning of an LSP.  A head-end LSR
                           is also referred to as an ingress LSR.

Tail-end Label Switching
Router (LSR):              The end of an LSP.  A tail-end LSR is also
                           referred to as an egress LSR.

Propagation Latency:       The delay added by the propagation of the
                           packet through the link (fixed value that
                           depends on the distance of the link and
                           the propagation speed).

Transmission Latency:      The delay added by the transmission of the
                           packet over the link, i.e., the time it
                           takes to put the packet over the media
                           (value that depends on the link throughput
                           and packet length).

Processing Latency:        The delay added by all the operations
                           related to the switching of labeled
                           packets (value is node implementation
                           specific and may be considered fixed and
                           constant for a given type of equipment).

Node Latency:              The delay added by the network element
                           resulting from of the sum of the
                           transmission, processing, and
                           queuing/buffering latency.

One-hop Delay:             The fixed delay experienced by a packet to
                           reach the next hop resulting from the of
                           the propagation latency, the transmission
                           latency, and the processing latency.

Minimum Path Latency:      The sum of the one-hop delays experienced
                           by the packet when traveling from the
                           ingress to the egress LSR.

Variable Path Latency:     The variation in the sum of the delays
                           experienced by packets transiting the
                           path, otherwise know as jitter.

2.2. Acronyms 🌸

  • ASBR: Autonomous System Border Router
  • CE: Customer Edge
  • PE: Provider Edge
  • SP: Service Provider
  • ECMP: Equal-Cost Multi-path
  • LSP: Label Switched Path
  • LSP Ping: Label Switched Path Ping
  • LSR: Label Switching Router
  • OAM: Operations and Management
  • RSVP: Resource reSerVation Protocol
  • LDP: Label Distribution Protocol
  • DoS: Denial of Service

3. Motivations 🦋

This doc was made to provide requirements that could be used to build consistent and useful OAM functionality. The point is to meet the operational needs of the service providers (SPs) who have deployed MPLS or are deploying it now. It's giving practical purpose 🧚.

4. Requirements 👑

The following sections list the OAM requirements gathered from service providers who have deployed MPLS, along with services based on MPLS networks. Each requirement is spelled out in detail so its applicability is clear 🔥. Even though the requirements here are defined by the IETF, they were made consistent with requirements gathered by other standards bodies, such as the ITU [Y1710]. literally the best kind of teamwork 🫶✨.

4.1. Detection of Label Switched Path Defects

Okay so finding a broken LSP MUST not need someone manually hopping through every single LSR that's switching traffic for it 💅. Nobody wants to visit each LSR along the data plane path one by one 😭. Instead this function MUST be automated and able to run at an operator-specified frequency from the LSP's origination point 🦋. That means the solutions have to be interoperable so the automation actually works across the board ✨.

Automation matters even more when huge numbers of LSPs are being tested. Automated ingress LSR to egress LSR testing is wanted for some LSPs 🌸. The goal is to catch LSP path defects before customers do 👀, which means detection and correction have to be predictable and fit inside the SLA the service is sold under 💼. Basically, the time an OAM tool takes to detect a defect plus the time an operational support system takes to react (by fixing it or telling the customer) has to land within the SLA bounds 🎀.

Synchronizing detection time bounds across the tools that detect broken LSPs is required 🔄. If you don't specify defect detection time bounds, test results can get ambiguous 🙈. If you know how long detection takes, automated responses can be designed around resiliency and SLA reporting 📊. And if synchronization is possible, an operational framework can guide the design and specification of MPLS applications 👑.

An ICMP-based ping [RFC792] can be sent through an LSP as an IP payload, but using it to check that an LSP is defect-free can give erroneous results, both positive and negative 🙃. Because ICMP traffic uses legally addressable IP addressing, ICMP messages that start inside an LSP can "fall out of the LSP" somewhere along the path 😬. Since ICMP packets are routable, that can produce a falsely positive response. And when you need to test the data plane of a specific LSP, it's hard to guarantee that ICMP ping traffic gets parsed and hashed to the same equal-cost multi-paths (ECMP) as the data traffic 🧩.

Any detection mechanism that depends on getting status back over a return path SHOULD offer multiple return options, so that one of them is expected to be unaffected by the original defect 🛟. A false negative could happen with a mechanism that needs a functional MPLS return path. MPLS LSPs are unidirectional, so the forward LSP under test might be working fine while the response from the destination LSR gets lost. That makes the source LSR think the forward LSP is defective 😱. But if an alternate return path could be specified, like IP for example, the source could name it as the return path to the destination. Then it would receive a response saying the return LSP is defective 🔁.

The OAM packet MUST follow the customer data path exactly so it reflects the path liveliness that customer data actually uses 💖. The big cases to watch are forwarding mechanisms like ECMP inside the operator's network, where flows are load-shared across parallel paths (i.e., equal IGP cost) 🛣️. Where customer traffic may be spread over multiple paths, you need to detect failures on any of the path permutations 🌈. Where the spreading mechanism is payload specific, payloads need forwarding that's common with the traffic under test 🎯. Meeting all of this adds complexity, since ECMP connectivity permutations have to get exercised and defect detection has to happen in a reasonable amount of time ⏱️.

4.2. Diagnosis of a Broken Label Switched Path

Diagnosing a broken LSP and isolating the failed component (i.e., link or node) on the path is required 🔍. Recovery actions for mis-branching defects in an LDP network are a particularly difficult case 💀. The best way to diagnose defects and isolate the failed component is a path trace function that can return the entire list of LSRs and links used by a certain LSP (or at least the set of LSRs/links up to the location of the defect) 🗺️. The tracing capability SHOULD include the ability to trace recursive paths, like when nested LSPs are used 🪆. This path trace function MUST also be capable of diagnosing LSP mis-merging, by letting you compare expected vs. actual forwarding behavior at any LSR in the path 🔬. The path trace capability SHOULD be capable of being executed from the head-end Label Switching Router (LSR), and it may permit downstream path components to be traced from an intermediate mid-point LSR 📍. Last but not least, the path trace function MUST have the ability to support the ECMP scenarios described in Section 4.1 ✨.

4.3. Path Characterization

Path characterization is all about revealing the details of LSR forwarding operations 🧐. Those details can then be compared during later testing relevant to OAM functionality 📋. This includes but is not limited to:

  • consistent use of pipe or uniform time to live (TTL) models by an LSR [RFC3443].
  • sufficient details that allow the test origin to exercise all path permutations related to load spreading (e.g., ECMP).
  • stack operations performed by the LSR, such as pushes, pops, and TTL propagation at penultimate hop LSRs.

4.4. Service Level Agreement Measurement

Mechanisms are required to measure the diverse aspects of Service Level Agreements, which include:

  • latency - amount of time required for traffic to transit the network
  • packet loss
  • jitter - measurement of latency variation
  • defect free forwarding - the service is considered to be available, or the service is unavailable and other aspects of performance measurement do not have meaning.

Such measurements can be made independently of the user traffic, or via a hybrid of user traffic measurement and OAM probing 🎀.

At least one mechanism is required to measure the number of OAM packets 🔢. On top of that, the ability to measure the quantitative aspects of LSPs, such as jitter, delay, latency, and loss, MUST be available so you can determine whether traffic for a specific LSP is traveling within the operator-specified tolerances 📏.

Any method considered SHOULD be capable of measuring the latency of an LSP with minimal impact on network resources 🌿. See Section 2.1 for definitions of the various quantitative aspects of LSPs 📚.

4.5. Frequency of OAM Execution

The operator MUST have the flexibility to configure OAM parameters to meet their specific operational requirements ⚙️.

This includes the frequency of executing any OAM functions. Synchronizing OAM operations is required so SLA measurements stay consistent 🎯. Here's the tea: defect conditions like mis-branching or misdirection of traffic can cause flapping if probe-based detection mechanisms have significant mismatches in their detection frequency 🌀. You can fix that by synchronizing the rate, or by having the probes self-identify their probe rate 🤝. For example, when probing mechanisms are bootstrapping, they might negotiate and ultimately agree on a probing rate, which gives a consistent probing frequency and avoids those problems ✅.

One observation is that wide-spread deployment of MPLS, common implementation of monitoring tools, and the need for inter-carrier synchronization of defect and service level specification handling will push OAM parameters toward commonly agreed values 🌍. Those values have to be harmonized with the surrounding technologies (e.g., SONET/SDH, ATM) to be useful 🔗. This gets especially important as networks scale, since mis-configuration can result in churn, alarm flapping, etc. 📉

4.6. Alarm Suppression, Aggregation, and Layer Coordination

Network elements MUST provide alarm suppression functionality that stops superfluous alarms from being generated 🔕. They can do this by simply discarding them (or not generating them in the first place), or by aggregating them together, which greatly reduces the number of notifications emitted. Read together with the requirement in Section 4.7 below, this typically requires fault notification to the LSP egress, which may have specific time constraints if the application using the LSP independently implements path continuity testing (for example, ATM I.610 Continuity check (CC)[I610]). These constraints apply to LSPs that are monitored 👀.

MPLS applications can all try to respond to defects at the same time 🌪️. Layer-3 MPLS VPNs that use Traffic Engineered tunnels are a good example, because when a failure hits the LSP carrying the Traffic Engineered tunnel, it also affects the VPN traffic that uses that tunnel's LSP. Mechanisms are required to coordinate network responses to defects 🤹.

4.7. Support for OAM Inter-working for Fault Notification

An LSR supporting the inter-working of one or more networking technologies over MPLS MUST be able to translate an MPLS defect into the native technology's error condition 🔄. For example, errors occurring over an MPLS transport LSP that supports an emulated ATM VC MUST translate errors into native ATM OAM Alarm Indication Signal (AIS) cells at the termination points of the LSP 🦋. The mechanism SHOULD consider possible bounded detection time parameters, e.g., a "hold off" function before reacting, so it can synchronize with the OAM functions ⏳.

One goal is alarm suppression by the upper layer using the LSP. As observed in Section 4.5, this requires MPLS to perform detection in a bounded timeframe, so it can start alarm suppression before the upper layer independently detects the defect 🏃‍♀️.

4.8. Error Detection and Recovery

Recovery from a fault by a network element can be facilitated by MPLS OAM procedures 🛠️. These procedures detect a broader range of defects than simple link and node failures 🔭. Since MPLS LSPs may span multiple routing areas and service provider domains, fault recovery and error detection should be possible in these configurations as well as in the more simplified single-area/domain configurations 🌐.

Recovery from faults SHOULD be automatic 🤖. It is a requirement that faults SHOULD be detected (and possibly corrected) by the network operator before customers of the service in question detect them 👑.

4.9. Standard Management Interfaces 🎀

okay so MPLS is literally everywhere now, which means everyone needs to handle OAM the same way 💅 That's where the standard IETF MPLS-related MIB modules come in (think [RFC3813][RFC3812][RFC3814]) for fault, statistics, and configuration management. They give operators a common programmatic interface to reach Operations and Management functions and check their statuses 🦋 ngl there are still gaps where MIB modules don't cover OAM or other features yet, so MIB modules for new protocol functions or network tools are required 📚

4.10. Detection of Denial of Service Attacks 🛡️

The ability to detect denial of service (DoS) attacks against the data or control planes MUST be part of any security management related to MPLS OAM tools or techniques. This one is non-negotiable, bestie 🙅

4.11. Per-LSP Accounting Requirements 💸

In an MPLS network, service providers can measure traffic from an LSR to the egress of the network using some MPLS-related MIBs, for example. So it's totally reasonable to know how much traffic is traveling from location to location (aka a traffic matrix) by analyzing the flow of traffic 📊 Traffic accounting in an MPLS network boils down to these three items:

  1. Collecting information to design network

For the purpose of optimized network design, a service provider may offer the traffic information. Optimizing network design needs this info ✨

  1. Providing a Service Level Specification

Providers and their customers MAY need to verify high-level service level specifications, either to keep continuously optimizing their networks or to offer guaranteed bandwidth services. So traffic accounting to monitor MPLS applications is required 💖

  1. Inter-AS environment

Service providers that offer inter-AS services require accounting of those services 🌐

These three motivations need to satisfy the following:

  • In (1) and (2), collecting information on a per-LSP basis is the minimum level of granularity for collecting accounting information at both the ingress and egress of an LSP 🎯
  • In (3), the SP's ASBR carries out interconnection functions as an intermediate LSR. So identifying a pair of ingress and egress LSRs using each LSP is needed to determine the cost of the service a customer is using 💅

4.11.1. Requirements 📝

Accounting on a per-LSP basis covers the following set of functions:

  1. At an ingress LSR, accounting of traffic through LSPs that begin at each egress in question.
  2. At an intermediate LSR, accounting of traffic through LSPs for each pair of ingress to egress.
  3. At an egress LSR, accounting of traffic through LSPs for each ingress.
  4. All LSRs containing LSPs that are being measured need to have a common identifier to distinguish each LSP. The identifier MUST be unique to each LSP, and its mapping to the LSP SHOULD be provided, whether from manual or automatic configuration 🔑

In the case of non-merged LSPs, this can be achieved by simply reading traffic counters for the label stack associated with the LSP at any LSR along its path. However, to measure merged LSPs, an LSR MUST have a means to distinguish the source of each flow so the statistics don't get mixed up 🫶

4.11.2. Location of Accounting 📍

It's not realistic for LSRs to perform these operations on every LSP in a network 😭 At a minimum, per-LSP based accounting SHOULD be performed at the edges of the network, meaning at the edges of both LSPs and the MPLS domain 🌸

5. Security Considerations 🔐

Provisions to any of the network mechanisms designed to satisfy the requirements described here are required to prevent their unauthorized use. Likewise, these network mechanisms MUST provide a means by which an operator can prevent denial of service attacks if those mechanisms are used in such an attack 🚨

LSP mis-merging is a bigger deal than just a network defect, bestie. It can happen for a number of potential reasons, and some of them (because of MPLS label stacking) are brand new to MPLS 💀

The performance of diagnostic functions and path characterization involves extracting a significant amount of information about network construction that the network operator MAY consider private 🙈

6. References 📖

6.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.

6.2. Informative References

[RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol Label Switched (MPLS) Data Plane Failures", RFC 4379, February 2006.

[RFC3812] Srinivasan, C., Viswanathan, A., and T. Nadeau, "Multiprotocol Label Switching (MPLS) Traffic Engineering (TE) Management Information Base (MIB)", RFC 3812, June 2004.

[RFC3813] Srinivasan, C., Viswanathan, A., and T. Nadeau, "Multiprotocol Label Switching (MPLS) Label Switching Router (LSR) Management Information Base (MIB)", RFC 3813, June 2004.

[RFC3814] Nadeau, T., Srinivasan, C., and A. Viswanathan, "Multiprotocol Label Switching (MPLS) Forwarding Equivalence Class To Next Hop Label Forwarding Entry (FEC-To-NHLFE) Management Information Base (MIB)", RFC 3814, June 2004.

[Y1710] ITU-T Recommendation Y.1710, "Requirements for OAM Functionality In MPLS Networks"

[I610] ITU-T Recommendation I.610, "B-ISDN operations and maintenance principles and functions", February 1999

[RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998.

[RFC792] Postel, J., "Internet Control Message Protocol", STD 5, RFC 792, September 1981.

[RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing in Multi-Protocol Label Switching (MPLS) Networks", RFC 3443, January 2003.

7. Acknowledgements 💜

The authors wish to acknowledge and thank the following individuals for their valuable comments to this document: Adrian Smith, British Telecom; Chou Lan Pok, SBC; Mr. Ikejiri, NTT Communications; and Mr. Kumaki, KDDI. Hari Rakotoranto, Miya Kohno, Cisco Systems; Luyuan Fang, AT&T; Danny McPherson, TCB; Dr. Ken Nagami, Ikuo Nakagawa, Intec Netcore, and David Meyer. Thank you all, you're iconic 👑

Authors' Addresses 📮

Comments should be made directly to the MPLS mailing list at mpls@lists.ietf.org.

Thomas D. Nadeau Cisco Systems, Inc. 300 Beaver Brook Road Boxboro, MA 01719 Phone: +1-978-936-1470 EMail: tnadeau@cisco.com

Monique Jeanne Morrow Cisco Systems, Inc. Glatt-Com, 2nd Floor CH-8301 Switzerland Phone: (0)1 878-9412 EMail: mmorrow@cisco.com

George Swallow Cisco Systems, Inc. 300 Beaver Brook Road Boxboro, MA 01719 Phone: +1-978-936-1398 EMail: swallow@cisco.com

David Allan Nortel Networks 3500 Carling Ave. Ottawa, Ontario, CANADA Phone: 1-613-763-6362 EMail: dallan@nortel.com

Satoru Matsushima Japan Telecom 1-9-1, Higashi-Shinbashi, Minato-ku Tokyo, 105-7316 Japan Phone: +81-3-6889-1092 EMail: satoru@ft.solteria.net

Copyright (C) The Internet Society (2006).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 🧾

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or the extent to which any license under such rights might or might not be available. It also doesn't represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement 🌈

Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA).

📦 where this page comes from