RFC 9551

Framework of Operations, Administration, and Maintenance (OAM) for Deterministic Networking (DetNet)

📅 published March 2024, from ietf

RFC 9551: Framework of OAM for DetNet 💖

Quick intro, bestie! 🎀 This is RFC 9551, titled Framework of Operations, Administration, and Maintenance (OAM) for Deterministic Networking (DetNet). It's published by the Internet Engineering Task Force (IETF) and dated March 2024. The ISSN is 2070-1721, and it's Informational (so it's not a Standards Track spec 🦋). The authors are G. Mirsky (Ericsson), F. Theoleyre (CNRS), G. Papadopoulos (IMT Atlantique), CJ. Bernardos (UC3M), B. Varga (Ericsson), and J. Farkas (Ericsson). The title block doesn't list any RFCs that it obsoletes or updates, so it doesn't replace or change anything else 🌸

Abstract

Okay so, Deterministic Networking (DetNet), as defined in RFC 8655, is all about giving you bounded end-to-end latency on top of the network infrastructure. That covers both Layer 2 bridged and Layer 3 routed segments, and it's giving main character energy 👑 This document's main job is to spell out the specific requirements for Operations, Administration, and Maintenance (OAM) that you'd want for keeping a deterministic network running. It's also going to be used in future work that defines how OAM protocols apply to a deterministic network and how they get extended ✨

Once the OAM framework is in place in DetNet, an operator gets a real-time view of the network. That view shows whether the network is actually meeting the Service Level Objective (SLO) assigned to each DetNet flow, like packet delay, delay variation, and packet-loss ratio. Iconic, honestly 🔥

Status of This Memo

This document is not an Internet Standards Track specification. It's published just for informational purposes 🧚

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, and it went through public review before being approved for publication by the Internet Engineering Steering Group (IESG). Not every document the IESG approves is a candidate for any level of Internet Standard, so check out Section 2 of RFC 7841 for more on that 🫶

You can find information about the current status of this document, any errata, and how to send feedback at https://www.rfc-editor.org/info/rfc9551.

Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. 💅

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents, which you can read at https://trustee.ietf.org/license-info. These are the rules in effect on the date this document was published, so please read them carefully, because they explain your rights and restrictions for this document. Any Code Components pulled out of this document have to include the Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're also provided without warranty, as described in the Revised BSD License 🌈

Table of Contents

1.  Introduction
  1.1.  Definitions
  1.2.  Requirements Language
2.  Role of OAM in DetNet
3.  Operation
  3.1.  Information Collection
  3.2.  Continuity Check
  3.3.  Connectivity Verification
  3.4.  Route Tracing
  3.5.  Fault Verification/Detection
  3.6.  Fault Localization and Characterization
  3.7.  Use of Hybrid OAM in DetNet
4.  Administration
  4.1.  Collection of Metrics
  4.2.  Worst-Case Metrics
5.  Maintenance
  5.1.  Replication/Elimination
  5.2.  Resource Reservation
6.  Requirements
  6.1.  For the DetNet Forwarding Sub-layer
  6.2.  For the DetNet Service Sub-layer
7.  IANA Considerations
8.  Security Considerations
9.  Privacy Considerations
10. References
  10.1.  Normative References
  10.2.  Informative References
Acknowledgments
Authors' Addresses

1. Introduction ✨

Okay, so Deterministic Networking (DetNet) [RFC8655] is all about providing a bounded end-to-end latency on top of the network infrastructure. It covers both Layer 2 bridged and Layer 3 routed segments, and that work spans the data plane, OAM, time synchronization, management, control, and security 💜

Operations, Administration, and Maintenance (OAM) tools are super important for IP networks [RFC7276]. DetNet OAM should give you a toolset for fault detection, fault localization, and performance measurement 🍓

This document's main purpose is to spell out the specific requirements for the OAM features you'd want for keeping a deterministic/reliable network running. It specifically looks at what a deterministic network needs when it's carrying critical flows 🦋

In this document, the term "OAM" follows the definition in [RFC6291]. DetNet is expected to implement an OAM framework that keeps a real-time view of the network infrastructure, and of how well the network respects the Service Level Objectives (SLOs) assigned to each DetNet flow. Those SLOs include in-order packet delivery, packet delay, delay variation, and packet-loss ratio 🌸

This document lists the OAM functional requirements for a DetNet domain. That list can then be used for gap analysis of the OAM tools that already exist, so you can figure out:

  • possible enhancements of existing tools, or
  • whether new OAM tools are required to support proactive and on-demand path monitoring and service validation.

1.1. Definitions 🎀

This document uses definitions, especially the one for a DetNet flow, from Section 2.1 of [RFC8655]. The following terms are used throughout this document, as defined below:

  • DetNet OAM domain: a DetNet network used by the monitored DetNet flow. A DetNet OAM domain (also referred to in this document as "OAM domain") may have Maintenance End Points (MEPs) on its edge and Maintenance Intermediate Points (MIPs) within.
  • DetNet OAM instance: a function that monitors a DetNet flow for defects and/or measures its performance metrics. In this document, the shorter version "OAM instance" is used interchangeably.
  • Maintenance End Point (MEP): an OAM instance that is capable of generating OAM test packets in the particular sub-layer of the DetNet OAM domain.
  • Maintenance Intermediate Point (MIP): an OAM instance along the DetNet flow in the particular sub-layer of the DetNet OAM domain. An active MIP MUST respond to an OAM message generated by the MEP at its sub-layer of the same DetNet OAM domain. 🧚
  • Control and management plane: the control and management planes are used to configure and control the network. Relative to a DetNet flow, the control and/or management plane can be out of band.
  • Active measurement methods: (as defined in [RFC7799]) these methods modify a DetNet flow by injecting specially constructed test packets [RFC2544].
  • Passive measurement methods: (as defined in [RFC7799]) these methods infer information by observing unmodified existing flows.
  • Hybrid measurement methods: (as defined in [RFC7799]) these are a combination of elements from both active and passive measurement methods.
  • In-band OAM: an active OAM method that is in band within the monitored DetNet OAM domain when it traverses the same set of links and interfaces, receiving the same QoS and Packet Replication, Elimination, and Ordering Functions (PREOF) treatment as the monitored DetNet flow.
  • Out-of-band OAM: an active OAM method whose path through the DetNet domain may not be topologically identical to the path of the monitored DetNet flow. Its test packets may receive different QoS and/or PREOF treatment, or both.
  • On-path telemetry: on-path telemetry can be realized as a hybrid OAM method. The origination of the telemetry information is inherently in band, since packets in a DetNet flow are used as triggers. Collection of the on-path telemetry information can be performed using in-band or out-of-band OAM methods. 💖

1.2. Requirements Language 🌈

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. The requirements language is used in Sections 1.1 and 6, and it applies to the implementations of DetNet OAM. 🫶

2. Role of OAM in DetNet ✨

okay so DetNet networks are expected to give you communications with predictable low packet delay, packet loss, and packet misordering 🎀 Most critical apps will define a set of SLOs to be required for the DetNet flows they generate, so the stakes are high 😭

To keep those strict guarantees, DetNet can use an orchestrator that's able to monitor and maintain the network 👑 Usually a Software-Defined Network (SDN) controller places DetNet flows in the deployed network based on their SLOs. That means resources have to be provisioned a priori for regular operation of the network. it's giving planning ahead 🧚

Most of the existing OAM tools can be used in DetNet networks, but they can only cover some aspects of deterministic networking. Fulfilling strict guarantees is essential for DetNet flows, which leads to new DetNet-specific functionalities that must be covered with OAM. Filling these gaps is inevitable and needs accurate consideration of DetNet specifics. Just like DetNet flows, their OAM also needs careful end-to-end engineering 💖

For example, placing MEPs along the path of a DetNet flow is not always a trivial task. It may require proper design together with the design of the service component of a given DetNet flow 🦋

There are several DetNet-specific challenges for OAM. Bounded network characteristics (e.g., delay, loss) are inseparable service parameters, so Performance Monitoring (PM) OAM is a key topic for DetNet 📈 OAM tools are needed to monitor each SLO without impacting the DetNet flow characteristics. Another challenge is strict resource allocation. Resources used by OAM must be considered and allocated so they don't disturb DetNet flows 💅

The DetNet Working Group has defined two sub-layers:

  • The DetNet service sub-layer, at which a DetNet service (e.g., service protection) is provided.
  • The DetNet forwarding sub-layer, which optionally provides resource allocation for DetNet flows over paths provided by the underlying network.

OAM mechanisms already exist for the DetNet forwarding sub-layer, but the service sub-layer needs new OAM procedures 🌸 These new OAM functions must allow, for example, recognizing and discovering DetNet relay nodes, getting information about their configuration, and checking their operation or status.

DetNet service sub-layer functions use a sequence number for PREOF, which creates a challenge for inserting OAM packets in the DetNet flow 🙈

Fault tolerance also assumes that multiple paths could be provisioned to maintain an end-to-end circuit by adapting to existing conditions. The DetNet Controller Plane (e.g., a central controller/orchestrator) controls the PREOF on a node. OAM is expected to support monitoring and troubleshooting PREOF on a particular node and within the domain 🔥

Note that a distributed architecture of the DetNet Control Plane can also control PREOF in those scenarios where DetNet solutions involve more than one single central controller 🌈

The DetNet forwarding sub-layer is based on preexisting technologies and has much better OAM coverage. However, the forwarding sub-layer is terminated at DetNet relay nodes, so the end-to-end OAM state of forwarding may only be created based on the status of multiple forwarding sub-layer segments serving a given DetNet flow. (e.g., in the case of DetNet MPLS, there may be no end-to-end LSP below the DetNet pseudowire) ngl that's a lot of moving pieces 🫶

3. Operation 🌷

OAM features will enable DetNet with robust operation for both forwarding and routing purposes ⭐

It's worth noting that the test and data packets are expected to follow the same path. In other words, connectivity verification has to be conducted in band without impacting data traffic 💜 Test packets are expected to share fate with the monitored data traffic, without introducing congestion in normal network conditions.

3.1. Information Collection ✨

Information about the state of the network can be collected using several mechanisms. Some protocols, e.g., the Simple Network Management Protocol (SNMP), poll for updated data. Other protocols, such as YANG-Push [RFC8641], can be used to set up subscriptions for the data defined in the YANG data models, so it gets published periodically or when the underlying data changes. Either way, the information is collected and sent using the DetNet Controller Plane 🎀

We can also characterize methods of transporting OAM information relative to the path of data. For instance, OAM information may be transported in band or out of band relative to the DetNet flow. With the former, the telemetry information uses resources allocated for the monitored DetNet flow. If an in-band method of transporting telemetry is used, the amount of generated information needs to be carefully analyzed, and additional resources must be reserved 👀

[RFC9197] defines the in-band transport mechanism, where telemetry information is collected in the data packet on which the information is generated. Two tracing methods are described:

  • end-to-end, i.e., from the ingress and egress nodes, and
  • hop-by-hop, i.e., like end-to-end with additional information from transit nodes.

[RFC9326] and [HYBRID-TWO-STEP] are examples of out-of-band telemetry transport. In the former case, information is transported by each node traversed by the data packet of the monitored DetNet flow, in a specially constructed packet. In the latter, information is collected in a sequence of follow-up packets that traverse the same path as the data packet of the monitored DetNet flow. In both methods, transporting the telemetry can avoid using resources allocated for the DetNet domain 🦋

3.2. Continuity Check 🌸

A continuity check is used to monitor the continuity of a path, i.e., that there exists a way to deliver packets between MEP A and MEP B. The continuity check detects a network failure in one direction: from the MEP transmitting test packets to the remote egress MEP. The continuity check in a DetNet OAM domain monitors the DetNet forwarding sub-layer, so it is not affected by a PREOF that operates at the DetNet service sub-layer ([RFC8655]) 💅

3.3. Connectivity Verification 💖

In addition to the Continuity Check, DetNet solutions have to verify connectivity. This verification considers an additional constraint: the absence of misconnection. The misconnection error state is entered after several consecutive test packets from other DetNet flows are received. The definition of the conditions for entry and exit of a misconnection error state is outside the scope of this document 🙃 Connectivity verification in a DetNet OAM domain monitors the DetNet forwarding sub-layer, so it is not affected by PREOF that operates at the DetNet service sub-layer ([RFC8655]).

3.4. Route Tracing 🗺️

Ping and traceroute are two ubiquitous tools that help localize and characterize a failure in the network using an echo request/reply mechanism. They help identify a subset of the routers in the path. However, to be predictable, resources are reserved per flow in DetNet. So DetNet needs route tracing tools that are able to trace the route for a specific flow 🧭 Tracing can also be used for the discovery of the Path Maximum Transmission Unit (PMTU) or the location of elements of PREOF for the particular route in the DetNet domain.

DetNet is not expected to use Equal-Cost Multipath (ECMP) [RFC8939]. As a result, DetNet OAM in an ECMP environment is outside the scope of this document 🚫

3.5. Fault Verification/Detection 🔍

DetNet expects to operate fault-tolerant networks, so mechanisms that can detect faults before they impact network performance are needed 🛡️

The network has to detect when a fault has occurred, i.e., when the network has deviated from its expected behavior. Fault detection can be based on proactive OAM protocols like the continuity check, or on-demand methods like ping. While the network must report an alarm, the cause may not be identified precisely. Examples of such alarms are a significant degradation of the end-to-end reliability, or a buffer overflow 🚨

3.6. Fault Localization and Characterization 🕵️‍♀️

The ability to localize a network defect and provide its characterization are necessary elements of network operation 💜

  • Fault localization: a process of deducing the location of a network failure from a set of observed failure indications. For example, this might be achieved by tracing the route of the DetNet flow in which the network failure was detected. Another method of fault localization can correlate reports of failures from a set of interleaved sessions monitoring path continuity.
  • Fault characterization: a process of identifying the root cause of the problem. For instance, misconfiguration or malfunction of PREOF elements can be the cause of erroneous packet replication or extra packets being flooded in the DetNet domain 🍓

3.7. Use of Hybrid OAM in DetNet 🌈

Hybrid OAM methods are used in performance monitoring and are defined in [RFC7799] as follows:

Hybrid Methods are Methods of Measurement that use a combination of Active Methods and Passive Methods.

A hybrid measurement method can produce metrics as close to measured using a passive measurement method. Passive methods measure metrics closest to the network's actual conditions. A hybrid method is considered an approximation of a passive measurement method, even if it alters something in a data packet, even if that's as little as the value of a designated field in the packet encapsulation 🤏 One example of such a hybrid measurement method is the Alternate-Marking Method (AMM) described in [RFC9341]. As with all on-path telemetry methods, AMM in a DetNet domain with the IP data plane is, by design, in band with respect to the monitored DetNet flow. Because the marking is applied to a data flow, the measured metrics are directly applicable to the DetNet flow. AMM minimizes the additional load on the DetNet domain by using nodal collection and computation of performance metrics, optionally in combination with out-of-band telemetry collection for further network analysis 💫

4. Administration 👑

The ability to expose a collection of metrics to support an operator's decision-making is essential 📊 The following performance metrics are useful:

  • Queuing Delay: the time elapsed between enqueuing a packet and its transmission to the next hop.
  • Buffer occupancy: the number of packets present in the buffer for each of the existing flows.
  • Per DetNet flow: a metric reflecting end-to-end performance for a given flow. Each of the paths has to be isolated in a multipath routing environment.
  • Per-path: detection of a misbehaving path or paths when multiple paths are used for service protection.
  • Per-device: detection of a misbehaving device 🕵️

4.1. Collection of Metrics 🧁

It's important to optimize the volume and frequency of statistics/measurement collection, whether the mechanisms are distributed, centralized, or both 💖 Periodic and event-triggered collection of information characterizing the state of a network is an example of mechanisms that achieve this optimization ✨

4.2. Worst-Case Metrics 📉

okay so DetNet is all about real-time comms on a heterogeneous multi-hop architecture 🛣️, which means lots of different hops and lots of different gear. To make correct decisions, the DetNet Controller Plane [RFC8655] needs timely info about packet losses and delays for each flow and each hop of the paths. Just knowing the average end-to-end stats? Not enough, bestie 😭. The collected info has to be good enough that a system can predict the worst-case scenario 💀.

5. Maintenance 🔧

so the service protection (provided by the DetNet Service sub-layer) is built to fix simple network failures faster than the DetNet Controller Plane's expected response time 🏃‍♀️. When stuff happens that messes with network operation, like link up/down, a device crashing or rebooting, or flows starting and ending, the DetNet Controller Plane has to do repair and reoptimization actions so it can permanently keep the SLOs of all active flows safe, with minimal waste of resources 💅. The Controller Plane is expected to keep retrieving the state of the network, evaluate conditions and trends to see whether a reconfiguration is even worth it, and quantify the following:

  • the cost of the suboptimality: resources may not be used optimally (i.e., a better path exists).
  • the reconfiguration cost: the DetNet Controller Plane needs an ability to trigger some reconfigurations. For this transient period, resources may be twice reserved, and control packets have to be transmitted.

So, reconfiguration may only be triggered if the gain is significant 🎯. no notes.

5.1. Replication/Elimination 🔁

when multiple paths are reserved between two MEPs, packet replication may be used to add redundancy and cover for transmission errors and collisions 🫶. For instance, in Figure 1, the source device S sends a packet to devices A and B so it can reach the destination node R.

                          ===> (A) => (C) => (E) ===
                        //        \\//   \\//       \\
              source (S)          //\\   //\\         (R) (root)
                        \\       //  \\ //  \\      //
                          ===> (B) => (D) => (F) ===

Figure 1: Packet Replication and Elimination Functions 🌈

5.2. Resource Reservation 🎀

because the quality of service tied to a path may degrade, the network has to provision additional resources along the path 🌸.

6. Requirements 📋

okay so according to [RFC8655], DetNet functionality is split into forwarding and service sub-layers. The forwarding sub-layer has DetNet transit nodes and may allocate resources for a DetNet flow over paths provided by the underlay network. The service sub-layer has DetNet relay nodes and provides a DetNet service (e.g., service protection). This section lists the general requirements for DetNet OAM, plus the requirements for each DetNet sub-layer in a DetNet domain 💜.

  1. It MUST be possible to initiate a DetNet OAM session from a MEP located at a DetNet node towards a MEP or MEPs downstream from that DetNet node within the given domain at a particular DetNet sub-layer.
  2. It MUST be possible to initiate a DetNet OAM session using any of the DetNet Controller Plane solutions, e.g., a centralized controller.
  3. DetNet OAM MUST support proactive OAM monitoring and measurement methods.
  4. DetNet OAM MUST support on-demand OAM monitoring and measurement methods.
  5. DetNet OAM MUST support unidirectional OAM methods, continuity checks, connectivity verification, and performance measurements.
  6. DetNet OAM MUST support bidirectional DetNet flows, but it is not required to support bidirectional OAM methods for bidirectional DetNet flows. DetNet OAM test packets used for monitoring and measurements of a bidirectional DetNet flow MUST be in band in both directions.
  7. DetNet OAM MUST support proactive monitoring of a DetNet device's reachability for a given DetNet flow.
  8. DetNet OAM MUST support hybrid performance measurement methods.
  9. Calculated performance metrics MUST include, but are not limited to, throughput, packet-loss, out-of-order, delay, and delay-variation metrics. [RFC6374] gives detailed info on performance measurement and performance metrics 📊.

6.1. For the DetNet Forwarding Sub-layer 🚚

DetNet OAM MUST support:

  1. PMTU discovery.
  2. Remote Defect Indication (RDI) notification to the DetNet OAM instance performing continuity checking.
  3. the monitoring of levels of resources allocated for a particular DetNet flow. Such resources include, but are not limited to, buffer utilization and scheduler transmission calendar.
  4. the monitoring of any subset of paths traversed through the DetNet domain by a DetNet flow.

6.2. For the DetNet Service Sub-layer 🧚

the OAM functions for the DetNet service sub-layer let you do things like recognize and discover DetNet relay nodes, gather info about their configuration, and check how they're operating or what their status is 👀.

The requirements on OAM for a DetNet relay node are that DetNet OAM MUST:

  1. provide OAM functions for the DetNet service sub-layer.
  2. support the discovery of DetNet relay nodes in a DetNet network.
  3. support the discovery of PREOF locations in the domain.
  4. support the collection of information specific to the DetNet service sub-layer (configuration/operation/status) from DetNet relay nodes.
  5. support exercising functionality of PREOF in the domain.
  6. work for DetNet data planes: MPLS and IP.
  7. support a defect notification mechanism, like Alarm Indication Signal. Any DetNet relay node providing service for a given DetNet flow MAY originate a defect notification addressed to any subset of DetNet relay nodes along that flow.
  8. be able to measure metrics (e.g. delay) inside a collection of OAM sessions, specially for complex DetNet flows, with PREOF features.

7. IANA Considerations 🦋

this document has no IANA actions, so nothing to see here 🙌.

8. Security Considerations 🔐

okay, this part is serious, so listen up 🚨. This document lists the OAM requirements for a DetNet domain and does not raise any security concerns or issues beyond the ones common to networking and those specific to DetNet that are discussed in Section 9 of [RFC9055]. But the analysis of OAM security concerns in Section 6 of [RFC7276] also applies to DetNet OAM, including the use of OAM for network reconnaissance 👀💀. Someone can totally scope out your network with OAM if you're not careful, so don't sleep on it.

9. Privacy Considerations 🙈

privacy stuff from Section 13 of [RFC9055] also applies to DetNet OAM 🫣. If any privacy mechanism is used for the monitored DetNet flow, then the same privacy method MUST be applied to the active DetNet OAM used to monitor the flow. no exceptions, periodt 💅.

10. References 📚

10.1. Normative References

10.2. Informative References

Acknowledgments 🫶

the authors want to give a huge shoutout to Pascal Thubert for the review, the insightful questions, and the helpful comments 💖✨.

Authors' Addresses ✨

okay bestie, this is where the whole crew gets their shoutout 🫶 the people who wrote RFC 9551 are listed here with their contact info, so if you have questions (or you're just obsessed and want to say hi) this is where to find them 💌 I've kept every address, phone number, email, and URI exactly as it was, so nothing gets lost in the vibe shift 🎀

Greg Mirsky 🦋

Greg Mirsky
Ericsson
Email: gregimirsky@gmail.com

Fabrice Theoleyre 🌸

Fabrice Theoleyre
CNRS
ICube Lab, Pole API
300 boulevard Sebastien Brant - CS 10413
67400 Illkirch - Strasbourg
France
Phone: +33 368 85 45 33
Email: fabrice.theoleyre@cnrs.fr
URI:   https://fabrice.theoleyre.cnrs.fr/

Georgios Papadopoulos 🧚

Georgios Papadopoulos
IMT Atlantique
Office B00 - 102A
2 Rue de la Châtaigneraie
35510 Cesson-Sévigné - Rennes
France
Phone: +33 299 12 70 04
Email: georgios.papadopoulos@imt-atlantique.fr

Carlos J. Bernardos 💜

Carlos J. Bernardos
Universidad Carlos III de Madrid
Av. Universidad, 30
28911 Leganes, Madrid
Spain
Phone: +34 91624 6236
Email: cjbc@it.uc3m.es
URI:   http://www.it.uc3m.es/cjbc/

Balazs Varga 🍓

Balazs Varga
Ericsson
Budapest
Magyar Tudosok krt. 11.
1117
Hungary
Email: balazs.a.varga@ericsson.com

Janos Farkas 👑

Janos Farkas
Ericsson
Budapest
Magyar Tudosok krt. 11.
1117
Hungary
Email: janos.farkas@ericsson.com

that's the whole squad, bestie! 🌈✨ five of them are from Ericsson, CNRS, IMT Atlantique, and UC3M, all doing the hard protocol work so the rest of us can actually ship 💅 no cap, they're iconic 😭🫶

📦 where this page comes from