RFC 9322

In Situ Operations, Administration, and Maintenance (IOAM) Loopback and Active Flags

📅 published November 2022, from ietf

RFC 9322: In Situ Operations, Administration, and Maintenance (IOAM) Loopback and Active Flags 🦋

Okay bestie, so this is an Internet Standards Track doc from the IETF 💅 published in November 2022. The authors are T. Mizrahi (Huawei), F. Brockners (Cisco), S. Bhandari (Thoughtspot), B. Gafni (Nvidia), and M. Spiegel (Barefoot Networks). It's listed under ISSN 2070-1721, and the title block doesn't say it obsoletes or updates anything, so it's standing on its own as a main character 👑.

Abstract ✨

IOAM collects operational and telemetry info right inside packets while they travel a path between two points in the network 🛤️. This doc adds two new flags to the IOAM Trace Option headers: the Loopback flag and the Active flag.

Status of This Memo 📝

This is an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force (IETF) 🌈. It represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication. For more on Internet Standards, check out Section 2 of RFC 7841.

For the current status of this document, any errata, and how to send feedback, head to https://www.rfc-editor.org/info/rfc9322.

Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Read these carefully, because they spell out your rights and restrictions for this document. Code Components pulled out of this document have to include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Revised BSD License.

Table of Contents 📚

   1.  Introduction
   2.  Conventions
     2.1.  Requirements Language
     2.2.  Terminology
   3.  New IOAM Trace Option Flags
   4.  Loopback in IOAM
     4.1.  Loopback: Encapsulating Node Functionality
       4.1.1.  Loopback Packet Selection
     4.2.  Receiving and Processing Loopback
     4.3.  Loopback on the Return Path
     4.4.  Terminating a Looped-Back Packet
   5.  Active Measurement with IOAM
   6.  IANA Considerations
   7.  Performance Considerations
   8.  Security Considerations
   9.  References
     9.1.  Normative References
     9.2.  Informative References
   Acknowledgments
   Contributors
   Authors' Addresses

1. Introduction 🌸

IOAM [RFC9197] is how you monitor traffic in the network, by sticking IOAM data fields into in-flight data packets 📦.

IOAM data can show up in one of four IOAM options: Pre-allocated Trace, Incremental Trace, Proof of Transit (POT), and Edge-to-Edge. This document adds two new flags to the Pre-allocated and Incremental Trace options: the Loopback and Active flags.

The Loopback flag asks each transit device along the path to loop a truncated copy of the data packet back to the sender 🔁. The Active flag tells you that a packet is being used for active measurement. "Active measurement," as used in this document, is defined in [RFC7799].

2. Conventions 💅

2.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 🎀

2.2. Terminology

Abbreviations used in this document:

IOAM:   In situ Operations, Administration, and Maintenance
OAM:    Operations, Administration, and Maintenance [RFC6291]

3. New IOAM Trace Option Flags 🦋

This document defines two new flags in the Pre-allocated and Incremental Trace options:

  • Bit 1 "Loopback" (L-bit): When set, the Loopback flag triggers the sending of a copy of a packet back towards the source, as further described in Section 4. 🔙
  • Bit 2 "Active" (A-bit): When set, the Active flag indicates that a packet is an active measurement packet rather than a data packet, where "active" is used in the sense defined in [RFC7799]. The packet may be an IOAM probe packet or a replicated data packet (the second and third use cases of Section 5). 🧪

4. Loopback in IOAM 🔁

Okay, the Loopback flag is the star here ⭐. It asks each transit device on the path to loop a truncated copy of the data packet back to the sender. That lets an IOAM encapsulating node trace the path to a given destination and get per-hop data about both the forward and return paths 🗺️. It's meant to be a faster alternative to Traceroute 🏃‍♀️: the encapsulating node gets responses from several transit nodes in less than one round-trip time (RTT), and it only has to send a single packet.

As Figure 1 shows, an IOAM encapsulating node can push an IOAM encapsulation with the Loopback flag onto some or all of the packets it forwards, using one of the IOAM encapsulation types, like [IOAM-NSH] or [IOAM-IPV6-OPTIONS]. The transit node and the decapsulating node both make copies of the packet and loop them back to the encapsulating node. The decapsulating node also ends the IOAM encapsulation and forwards the packet on toward the destination. Both looped-back copies get terminated at the encapsulating node 🧚.

 +--------+     +--------+     +--------+     +--------+     +--------+
 |        |     |  IOAM  |.....|  IOAM  |.....|  IOAM  |     |        |
 +--------+     +--------+     +--------+     +--------+     +--------+
 | L2/L3  |<===>| L2/L3  |<===>| L2/L3  |<===>| L2/L3  |<===>| L2/L3  |
 +--------+     +--------+     +--------+     +--------+     +--------+
   Source      Encapsulating    Transit      Decapsulating   Destination
                    Node           Node           Node

                <------------  IOAM-Domain  ----------->

                     IOAM encap. with Loopback flag
 Data packet  ------->============================>----------->
                                   |             |
                  IOAM looped back |             |
                     <=============+             |
                                 IOAM looped back|
                     <===========================+

                       Figure 1: Loopback in IOAM

Loopback only works if there's a return path from the transit nodes and destination nodes back toward the source (the encapsulating node) 🛤️. More specifically, loopback only applies to encapsulations where the identity of the encapsulating node is available in the encapsulation header. And if an encapsulating node gets a looped-back packet that it didn't originate, that packet gets dropped 🗑️.

4.1. Loopback: Encapsulating Node Functionality ✨

The encapsulating node either generates synthetic packets with an IOAM trace option that has the Loopback flag set, or it sets the Loopback flag on a subset of the in-transit data packets. Loopback gets used either proactively or on-demand, meaning when a failure is detected 🚨. The encapsulating node also needs to make sure there's enough space in the IOAM header for loopback to work. That covers transit nodes adding trace data on the original path and then again on the return path.

An IOAM trace option that has the Loopback flag set MUST have the value '1' in the most significant bit of IOAM-Trace-Type and '0' in the rest of the bits of IOAM-Trace-Type. So every transit node that processes this trace option only adds a single data field, which is the Hop_Lim and node_id data field. A transit node that receives a packet with an IOAM trace option that has the Loopback flag set, where IOAM-Trace-Type isn't '1' in the most significant bit and '0' in the rest of the bits, MUST NOT loop back a copy of the packet. The reason for allowing only a single data field per hop is to minimize the impact of amplification attacks 🛡️.

IOAM encapsulating nodes MUST NOT push an IOAM encapsulation with the Loopback flag onto data packets that already include an IOAM encapsulation. This requirement exists to prevent IOAM Loopback nesting, where looped-back packets could get looped back again in a nested IOAM-Domain 🪆.

4.1.1. Loopback Packet Selection 🎀

If an IOAM encapsulating node puts the Loopback flag on all the traffic it forwards, it may lead to a ridiculous number of looped-back packets, which may overload the network and the encapsulating node 😵. So an IOAM encapsulating node that supports the Loopback flag MUST support the ability to put the Loopback flag selectively on a subset of the packets it forwards.

Various methods of packet selection and sampling have been defined before, like [RFC7014] and [RFC5475]. An IOAM encapsulating node can use similar techniques to apply loopback to a subset of the forwarded traffic 🎯.

The subset of traffic that is forwarded or transmitted with a Loopback flag SHOULD NOT exceed 1/N of the interface capacity on any of the IOAM encapsulating node's interfaces. This applies to the total traffic that carries a Loopback flag, including traffic forwarded by the IOAM encapsulating node and probe packets that the IOAM encapsulating node generates. Here, N is a parameter that network operators can configure. If there is an upper bound, M, on the number of IOAM transit nodes in any path in the network, then configuring N such that N >> M (i.e., N is much greater than M) is RECOMMENDED. The reasoning is that a packet with the Loopback flag triggers a looped-back packet from each IOAM transit node along the path, for a total of M looped-back packets. So if N >> M, the number of looped-back packets is way lower than the number of data packets forwarded by the IOAM encapsulating node 📉. It is RECOMMENDED that the default value of N satisfies N>100 and gets used in the absence of explicit operator configuration, or if there is no prior knowledge about the network topology or size.

An IOAM-Domain where the Loopback flag is used MUST be configured so that there is expected to be a return path from each of the IOAM transit and IOAM decapsulating nodes. If that expectation doesn't apply, or if the encapsulating node's identity isn't available in the encapsulation header, then configuration MUST NOT enable the Loopback flag to be set 🚫.

4.2. Receiving and Processing Loopback ✨

okay so when the Loopback flag is set, the transit nodes processing this option are like "bet, make a copy and send it back to the source of the packet" 💅. Here the source is the IOAM encapsulating node, and it's assumed the source address is available in the encapsulation header 🫶. So the original packet's source address becomes the destination address of the copy 🦋. If IOAM runs over an encapsulation that doesn't include the address of the encapsulating node, then the transit/decapsulating node does not loop back a copy of the original packet 🙈.

The copy's source address is the address of the node doing the copying 🌸. How that address gets assigned is encapsulation specific. For example, if an IPv6 encapsulation is used, the source address can be assigned as specified in [RFC6724] 💜.

The copy also gets truncated, which means any payload sitting after the IOAM option(s) is removed before the looped-back packet heads back toward the encapsulating node 💀. Making the copy, and especially the truncation, may need some encapsulation-specific updates in the encapsulation header 🎀. Meanwhile, the original packet keeps going to its destination ✨.

The L-bit MUST be cleared in the copy of the packet that a node sends back towards the source 🔥.

An IOAM node that supports reception and processing of the Loopback flag MUST support the ability to limit the rate of the looped-back packets 👑. The rate of looped-back packets SHOULD be limited so that the number of looped-back packets is significantly lower than the number of packets the device forwards 🌈. The looped-back data rate SHOULD NOT exceed 1/N of the interface capacity on any of the IOAM node's interfaces 😭. Using N>100 is RECOMMENDED ⭐. Depending on the IOAM node's architecture considerations, the loopback response rate may be limited to a lower number so the IOAM node doesn't get overloaded 🧚.

4.3. Loopback on the Return Path 🍓

On its way back toward the source, the copied packet is processed like any other packet with IOAM information 🦋. That includes adding requested data at each transit node, assuming there's still sufficient space 💖.

4.4. Terminating a Looped-Back Packet 🌸

Once the return packet reaches the IOAM-Domain boundary, IOAM decapsulation happens just like it does for any other packet containing IOAM information 🫶. Note that the looped-back packet does not have the L-bit set 💅.

The IOAM encapsulating node that started the original loopback packet recognizes a received packet as an IOAM looped-back packet by checking the Node ID in the Hop_Lim/node_id field that corresponds to the first hop 🔍. If the Node ID and IOAM-Namespace match the current IOAM node, it means this is a looped-back packet that the current IOAM node initiated, and it gets processed accordingly ✨. If there's no match in the Node ID, the packet is processed like a conventional IOAM-encapsulated packet 🙈.

An IOAM encapsulating node can be either an endpoint (like an IPv6 host) or a switch/router that pushes a tunnel encapsulation onto data packets 👑. In both cases, the functionality above helps avoid IOAM data leaks from the IOAM-Domain 🔒. Specifically, if an IOAM looped-back packet reaches an IOAM boundary node that is not the IOAM node that initiated the loopback, that node does not process the packet as a loopback 😭. The IOAM encapsulation is removed, which stops IOAM information from leaking out of the IOAM-Domain 🛑. Since the packet doesn't have any payload, it gets terminated 💀.

5. Active Measurement with IOAM 🧪

Active measurement methods [RFC7799] make use of synthetically generated packets to facilitate measurement 🦋. This section presents use cases of active measurement using the IOAM Active flag ✨.

The Active flag indicates that a packet is used for active measurement 💜. An IOAM decapsulating node that receives a packet with the Active flag set in one of its Trace options must terminate the packet 🛑. The Active flag is intended to simplify the implementation of decapsulating nodes by indicating that the packet should not be forwarded further 💅. It is not intended as a replacement for existing active OAM protocols, which may run in higher layers and make use of the Active flag 🌈.

An example of an IOAM deployment scenario is shown in Figure 2 👀. The figure depicts two endpoints: a source and a destination 🫶. Data traffic from the source to the destination is forwarded through a set of network devices, including an IOAM encapsulating node (which incorporates one or more IOAM options), a decapsulating node (which removes the IOAM options), and optionally one or more transit nodes 🎀. The IOAM options are encapsulated in one of the IOAM encapsulation types, e.g., [IOAM-NSH] or [IOAM-IPV6-OPTIONS] 🧚.

 +--------+     +--------+     +--------+     +--------+     +--------+
 |        |     |  IOAM  |.....|  IOAM  |.....|  IOAM  |     |        |
 +--------+     +--------+     +--------+     +--------+     +--------+
 | L2/L3  |<===>| L2/L3  |<===>| L2/L3  |<===>| L2/L3  |<===>| L2/L3  |
 +--------+     +--------+     +--------+     +--------+     +--------+
   Source      Encapsulating    Transit      Decapsulating   Destination
                   Node           Node           Node

                <------------  IOAM-Domain  ----------->

                      Figure 2: Network Using IOAM

This document focuses on three possible use cases of active measurement using IOAM 🌸. These use cases are described using the example of Figure 2 ✨.

Endpoint active measurement: synthetic probe packets are sent between the source and destination, traversing the IOAM-Domain 🫶. Since the probe packets are sent between the endpoints, they're treated as data packets by the IOAM-Domain and don't need special treatment at the IOAM layer 💅. Specifically, the Active flag is not used in this case, and the IOAM layer doesn't need to be aware that an active measurement mechanism is used at a higher layer 🙈.

IOAM active measurement using probe packets within the IOAM-Domain: probe packets are generated and transmitted by the IOAM encapsulating node and are expected to be terminated by the decapsulating node 🦋. IOAM data related to probe packets may be exported by one or more nodes along the path by an exporting protocol that is outside the scope of this document (e.g., [IOAM-RAWEXPORT]) 💜. Probe packets include a Trace Option that has its Active flag set, indicating that the decapsulating node must terminate them 🛑. The specification of these probe packets, and the processing of these packets by the encapsulating and decapsulating nodes, is outside the scope of this document 🌈.

IOAM active measurement using replicated data packets: probe packets are created by the encapsulating node by selecting some or all of the en route data packets and replicating them 🧚. A selected data packet and its (possibly truncated) copy are forwarded with one or more IOAM options, while the original packet is forwarded normally without IOAM options 🎀. To the extent possible, the original data packet and its replica are forwarded through the same path 🫶. The replica includes a Trace Option that has its Active flag set, indicating that the decapsulating node should terminate it ✨. The current document defines the role of the Active flag in allowing the decapsulating node to terminate the packet, but the replication functionality and the functionality of the decapsulating node in this context are outside the scope of this document 💅.

If the volume of traffic that incorporates the Active flag is large, it may overload the network and the IOAM node(s) that process the active measurement packet 😭. Thus, the rate of the traffic that includes the Active flag SHOULD NOT exceed 1/N of the interface capacity on any of the IOAM node's interfaces 🔥. Using N>100 is RECOMMENDED 💖. Depending on the IOAM node's architecture considerations, the rate of Active-enabled IOAM packets may be limited to a lower number in order to avoid overloading the IOAM node 🌸.

6. IANA Considerations 📋

IANA has allocated the following bits in the "IOAM Trace-Flags" registry as follows 🎀:

Bit 1  "Loopback" (L-bit)

Bit 2  "Active" (A-bit)

This document is specified as the "Reference" in the registry for both bits 💜.

Note that bit 0 is the most significant bit in the "IOAM Trace-Flags" registry 🤓. This bit was allocated by [RFC9197] as the 'Overflow' bit 🦋.

7. Performance Considerations 🚀

Each of the flags defined in this document may have performance implications 📈. When using the loopback mechanism, a copy of the data packet is sent back to the sender, which means it generates more traffic than the endpoints originally sent 😳. Using active measurement with the Active flag requires synthetic (overhead) traffic 🧪.

Each of the mechanisms that use the flags above has a cost in terms of network bandwidth and may potentially load the node that analyzes the data 💅. Therefore, it MUST be possible to use each of the mechanisms on a subset of the data traffic 🫶. An encapsulating node needs to be able to set the Loopback and Active flags selectively, in a way that considers the effect on network performance, as further discussed in Sections 4.1.1 and 5 ✨.

Transit and decapsulating nodes that support loopback need to be able to limit the looped-back packets (as discussed in Section 4.2) so that the mechanisms are used at a rate that doesn't significantly affect network bandwidth and doesn't overload the source node in the case of loopback 🌈💖

8. Security Considerations 🔐

Okay bestie, the security stuff for IOAM in general is covered in [RFC9197]. The main thing to know is that an attacker may try to use the functionality defined in this document to attack the network. 😬

IOAM is assumed to be deployed in a restricted administrative domain, which limits the scope of those threats and their effect. That's a fundamental assumption for IOAM security, and [RFC9197] goes into it further. Even with that limited scope, though, security threats should still be considered and mitigated. An attacker may try to overload network devices by injecting synthetic packets that include an IOAM Trace Option with one or more of the flags defined in this document. An on-path attacker may also maliciously set one or more of the flags on transit packets. 🚨

Loopback flag: an attacker that sets this flag, either in synthetic packets or transit packets, can potentially cause amplification. Each device along the path creates a copy of the data packet and sends it back to the source. The attacker can also potentially use the Loopback flag for a DDoS attack, since multiple devices send looped-back copies of a packet to a single victim. 💥

Active flag: the impact of synthetic packets with the Active flag is no worse than synthetic data packets where the Active flag is not set. 🙂 By setting the Active flag in en route packets, an attacker can stop those packets from reaching their destination, because the packet is terminated by the decapsulating device. Note that an on-path attacker can get the same result by changing the destination address of a packet. Amplification is another risk: if an attacker makes transit switches replicate more packets than they're meant to (either by setting the Active flag or by sending synthetic packets), traffic gets amplified and bandwidth degrades. As mentioned in Section 5, the replication mechanism isn't in scope for this document. A spec that defines replication functionality should also address the security aspects of that mechanism.

Some of the threats discussed here may be worse in a wide area network with nested IOAM-Domains. For example, if there are two nested IOAM-Domains that both use loopback, a looped-back copy in the outer IOAM-Domain may be forwarded through another (inner) IOAM-Domain, and it may then be subject to loopback in that inner domain. The amplification ends up worse than in the conventional case. 😱

To mitigate the performance-related attacks described in Section 7, it should be possible for IOAM-enabled devices to selectively apply the mechanisms that use the flags defined here to a subset of the traffic. It should also be possible to limit the performance of synthetically generated packets to a configurable rate. Specifically, IOAM nodes should be able to:

  • Limit the rate of IOAM packets with the Loopback flag (IOAM encapsulating nodes) as discussed in Section 4.1.1.
  • Limit the rate of looped back packets (IOAM transit and decapsulating nodes) as discussed in Section 4.2.
  • Limit the rate of IOAM packets with the Active flag (IOAM encapsulating nodes) as discussed in Section 5.

As defined in Section 4, transit nodes that process a packet with the Loopback flag only add a single data field and truncate any payload that follows the IOAM option(s). That significantly limits the possible impact of an amplification attack. 🛡️✨

9. References 📚

9.1. Normative References

9.2. Informative References

Acknowledgments 🫶

The authors thank Martin Duke, Tommy Pauly, Donald Eastlake, Paul Kyzivat, Bernard Aboba, Greg Mirsky, and other members of the IPPM working group for many helpful comments. 💜

Contributors 🌸

The Editors would like to recognize the contributions of the following individuals to this document.

Ramesh Sivakolundu Cisco Systems, Inc. 170 West Tasman Dr. San Jose, CA 95134 United States of America Email: sramesh@cisco.com

Carlos Pignataro Cisco Systems, Inc. 7200-11 Kit Creek Road Research Triangle Park, NC 27709 United States of America Email: cpignata@cisco.com

Aviv Kfir Nvidia Email: avivk@nvidia.com

Jennifer Lemon Broadcom 270 Innovation Drive San Jose, CA 95134 United States of America Email: jennifer.lemon@broadcom.com

Authors' Addresses 🦋

Tal Mizrahi Huawei Israel Email: tal.mizrahi.phd@gmail.com

Frank Brockners Cisco Systems, Inc. 3rd Floor Hansaallee 249 40549 Duesseldorf Germany Email: fbrockne@cisco.com

Shwetha Bhandari Thoughtspot 3rd Floor Indiqube Orion Garden Layout HSR Layout 24th Main Rd Bangalore 560 102 Karnataka India Email: shwetha.bhandari@thoughtspot.com

Barak Gafni Nvidia Suite 100 350 Oakmead Parkway Sunnyvale, CA 94085 United States of America Email: gbarak@nvidia.com

Mickey Spiegel Barefoot Networks, an Intel company 4750 Patrick Henry Drive Santa Clara, CA 95054 United States of America Email: mickey.spiegel@intel.com

📦 where this page comes from