RFC 9378
In Situ Operations, Administration, and Maintenance (IOAM) Deployment
In Situ Operations, Administration, and Maintenance (IOAM) Deployment 🦋
okay bestie, this is RFC 9378 ✨ It's an Informational doc from the IETF, published in April 2023, with ISSN 2070-1721. Its authors are F. Brockners (Ed., Cisco), S. Bhandari (Ed., Thoughtspot), D. Bernier (Bell Canada), and T. Mizrahi (Ed., Huawei). Editor credits are the "Ed." tags, btw 💅. The title block doesn't list any documents it obsoletes or updates, so this one is standing alone 🫶
Abstract
Okay so IOAM is about collecting operational and telemetry info inside the packet itself while it travels between two points in the network 📦✨ This doc gives you a framework for deploying IOAM, plus deployment considerations and guidance. Lowkey the whole vibe of the doc 💜
Status of This Memo
This doc is not an Internet Standards Track spec. It's published just for info 🌸
It's a product of the Internet Engineering Task Force (IETF), so it represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you want the deets 💅
You can find the current status, any errata, and how to send feedback at https://www.rfc-editor.org/info/rfc9378 🔥
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved ©️💖
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read them, because they describe your rights and restrictions 🙈 Code Components pulled out of this doc must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction
2. Conventions
3. IOAM Deployment: Domains and Nodes
4. Types of IOAM
4.1. Per-Hop Tracing IOAM
4.2. Proof of Transit IOAM
4.3. E2E IOAM
4.4. Direct Export IOAM
5. IOAM Encapsulations
5.1. IPv6
5.2. NSH
5.3. BIER
5.4. GRE
5.5. Geneve
5.6. Segment Routing
5.7. Segment Routing for IPv6
5.8. VXLAN-GPE
6. IOAM Data Export
7. IOAM Deployment Considerations
7.1. IOAM-Namespaces
7.2. IOAM Layering
7.3. IOAM Trace Option-Types
7.4. Traffic-Sets That IOAM Is Applied To
7.5. Loopback Flag
7.6. Active Flag
7.7. Brown Field Deployments: IOAM-Unaware Nodes
8. IOAM Manageability
9. IANA Considerations
10. Security Considerations
11. Informative References
Acknowledgements
Authors' Addresses1. Introduction ✨
IOAM collects OAM info within the packet while it traverses a particular network domain 📦 The "in situ" part just means the OAM data gets added to the data packets themselves, instead of being sent in packets that exist only for OAM 🧚
IOAM complements stuff like Ping, Traceroute, and other active probing mechanisms 💅 When you think about "active" vs "passive" OAM, IOAM is kind of a hybrid type. In situ mechanisms don't need extra packets sent. IOAM just adds info to data packets that are already flowing, so it can't be called passive either 🦋 In terms of the classification in [RFC7799], IOAM could be portrayed as Hybrid Type I.
IOAM can be used where mechanisms using, e.g., ICMP don't apply or don't give the results you want 🔥 Situations like these include:
- proving that a certain traffic flow takes a predefined path,
- verifying the Service Level Agreement (SLA) for live data traffic,
- providing detailed statistics on traffic distribution paths in networks that spread traffic across multiple paths, or
- providing scenarios where probe traffic might be handled differently from regular data traffic by the network devices 👀
2. Conventions 🎀
Here are the abbreviations used in this doc, bestie 💜
BIER: Bit Index Explicit Replication [RFC8279]
Geneve: Generic Network Virtualization Encapsulation [RFC8926]
GRE: Generic Routing Encapsulation [RFC2784]
IOAM: In situ Operations, Administration, and Maintenance
MTU: Maximum Transmission Unit
NSH: Network Service Header [RFC8300]
OAM: Operations, Administration, and Maintenance
POT: Proof of Transit
VXLAN-GPE: Virtual eXtensible Local Area Network - Generic Protocol
Extension [VXLAN-GPE]3. IOAM Deployment: Domains and Nodes 🌈
[RFC9197] defines the scope of IOAM and the different types of IOAM nodes 📚 For readability, this section gives a quick overview of where IOAM applies and what the main node roles are. For more details, go to [RFC9197] 🫶
IOAM is focused on "limited domains," as defined in [RFC8799] 🏡 It is not meant for deployment on the global Internet 🙅♀️ The part of the network that uses IOAM is called the "IOAM-Domain." For example, an IOAM-Domain could be an enterprise campus with physical connections between devices, or an overlay network that uses virtual connections or tunnels between those devices 🧩 An IOAM-Domain is defined by its perimeter, or edge.
The operator of an IOAM-Domain is expected to put provisions in place so that packets carrying IOAM data fields don't leak past the edge of the IOAM-Domain, e.g., by using packet filtering methods 🚧
The operator should also think about how IOAM could affect things like ECMP load-balancing schemes 🎢 For example, load-balancing schemes based on packet length could be impacted by the bigger packet size IOAM causes. It's also worth looking at path MTU, which means making sure the MTU of every link inside a domain is big enough to handle the increased packet size from IOAM 📏 And of course, think about ICMP message handling 💅
An IOAM-Domain is made up of "IOAM encapsulating nodes," "IOAM decapsulating nodes," and "IOAM transit nodes" 🍓 A node's role (encapsulating, transit, or decapsulating) is defined within an IOAM-Namespace (more on that below). A node can have different roles in different IOAM-Namespaces, which is very main-character of it 👑
An IOAM encapsulating node adds one or more IOAM Option-Types to packets that IOAM is enabled for 💖 If IOAM is only enabled for a selected subset of traffic, the IOAM encapsulating node is responsible for applying IOAM to that subset.
An IOAM transit node updates one or more of the IOAM-Data-Fields 🔄 If both the Pre-allocated and the Incremental Trace Option-Types are present in the packet, each IOAM transit node will update at most one of those Option-Types. If both Trace Option-Types show up in a packet, it's up to the IOAM data processing systems (see Section 6) to combine the data from the two Trace Option-Types into one view of the packet's whole journey 🗺️ A transit node does not add new IOAM Option-Types to a packet, and it does not change the IOAM-Data-Fields of an IOAM Edge-to-Edge (E2E) Option-Type 🙅♀️
An IOAM decapsulating node removes any IOAM Option-Types from packets 🧹
The role of an IOAM encapsulating, IOAM transit, or IOAM decapsulating node is always performed within a specific IOAM-Namespace 🎯 So an IOAM node that's a decapsulating node for IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove the IOAM Option-Types for IOAM-Namespace "A" from the packet. An IOAM decapsulating node sitting at the edge of an IOAM-Domain removes all IOAM Option-Types and their associated encapsulation headers for all IOAM-Namespaces from the packet 🧼
IOAM-Namespaces let each namespace define and interpret IOAM-Data-Fields in its own way 🌷 Check Section 7.1 for a discussion of IOAM-Namespaces.
Here's the roles picture, Figure 1 ✨
Export of Export of Export of Export of
IOAM data IOAM data IOAM data IOAM data
(optional) (optional) (optional) (optional)
^ ^ ^ ^
| | | |
| | | |
User +---+----+ +---+----+ +---+----+ +---+----+
packets |Encapsu-| | Transit| | Transit| |Decapsu-|
-------->|lating |====>| Node |====>| Node |====>|lating |-->
|Node | | A | | B | |Node |
+--------+ +--------+ +--------+ +--------+
Figure 1: Roles of IOAM NodesIOAM nodes that add or remove IOAM-Data-Fields can also update those fields at the same time 🔁 In other words, IOAM encapsulating or decapsulating nodes can also be IOAM transit nodes. And not every node in an IOAM-Domain has to be an IOAM transit node 🙌 For example, a deployment might require packets to traverse a set of firewalls that support IOAM. In that case, only the firewall nodes would be IOAM transit nodes, not every node 🔥
(Part 2 of 6 is next: Section 4, Types of IOAM 🦋)
4. Types of IOAM ✨
okay so IOAM has a few different modes it can run in 🫶 and they're told apart by a few things: what kind of IOAM data fields are riding along in the packet, what data is actually being collected, what type of node is collecting or updating it, and whether (and how) nodes export the IOAM info. the way all of these stack together is literally the whole vibe 💅
Here's the lineup:
- Per-hop tracing: OAM info about each IOAM node a packet traverses gets collected and stored inside the user data packet as it moves through the IOAM-Domain 🦋 Possible uses include:
- Understanding the different paths that different packets take between an IOAM encapsulating node and an IOAM decapsulating node in a network that uses load balancing, such as Equal Cost Multi-Path (ECMP) 🛣️ This info could be used to tune the ECMP algorithm for optimized network resource usage.
- For operations and troubleshooting, figuring out which path a particular packet (or set of packets) takes, plus how much jitter and delay each IOAM node on that path adds to the overall delay and jitter between the encapsulating and decapsulating nodes 🔍
- Proof of Transit: info that a verifier node can use to check whether a packet went through all the nodes it's supposed to go through gets stored inside the user data packet 🔐 For example, it could confirm that a packet really passed through every service in a service function chain (like checking it hit the firewalls it was meant to hit), or that it really took the expected path.
- Edge-to-Edge (E2E): OAM info that's specific to the edges of an IOAM-Domain gets collected and stored in the user data packet 🌐 E2E OAM could be used to gather operational info about a network domain, like the delay and jitter that domain adds, or its traffic matrix.
- Direct Export: OAM info about each IOAM node a packet traverses is collected and immediately exported to a collector 📤 Direct Export is for when you want per-hop tracing info but gathering it inside the packet isn't feasible. Unlike per-hop tracing, which automatically correlates the info, Direct Export means a collector has to correlate the info from the individual nodes itself. Also, all nodes enabled for Direct Export need to be capable of exporting the IOAM info to the collector.
4.1. Per-Hop Tracing IOAM
"IOAM tracing data" is expected to be collected at every IOAM transit node a packet traverses, so you get visibility into the entire path that packet takes within an IOAM-Domain 👀 In other words, in a typical deployment, all nodes in an IOAM-Domain would take part in IOAM, which means they'd be IOAM transit nodes, IOAM encapsulating nodes, or IOAM decapsulating nodes 🎀 If not all nodes in a domain are IOAM capable, IOAM tracing info (i.e., node data, see below) will only be collected on the nodes that are IOAM capable. Nodes that aren't IOAM capable will forward the packet without any changes to the IOAM-Data-Fields 🙈 The maximum number of hops and the minimum path MTU of the IOAM-Domain are assumed to be known.
IOAM offers two different Trace Option-Types: the "Incremental" Trace Option-Type and the "Pre-allocated" Trace Option-Type 💜 For a discussion about which of the two is the most suitable for an implementation and/or deployment, see Section 7.3.
Every node data entry holds info for a particular IOAM transit node that a packet traversed 🧚 The IOAM decapsulating node removes any IOAM Option-Types and then processes and/or exports the associated data. All IOAM-Data-Fields are defined in the context of an IOAM-Namespace.
IOAM tracing can collect the following types of info, for example:
- Identification of the IOAM node. An IOAM node identifier can match a device identifier or a particular control point or subsystem within a device.
- Identification of the interface that a packet was received on, i.e., ingress interface.
- Identification of the interface that a packet was sent out on, i.e., egress interface.
- Time of day when the packet was processed by the node, as well as the transit delay. Different definitions of processing time are feasible and expected, though it's important that all devices in an IOAM-Domain follow the same definition ⏰
- Generic data, which is format-free info where the syntax and semantics are defined by the operator in a specific deployment 🧩 For a specific IOAM-Namespace, all IOAM nodes should interpret the generic data the same way. Examples of generic IOAM data include geolocation info (where the node was at the time the packet was processed), buffer queue fill level or cache fill level at that time, or even a battery charge level 🔋
- Info to detect whether IOAM trace data was added at every hop, or whether certain hops in the domain weren't IOAM transit nodes 🕵️
- Data about how the packet traversed a node (transit delay, buffer occupancy in case the packet was buffered, and queue depth in case the packet was queued) 🚦
The Incremental Trace Option-Type and Pre-allocated Trace Option-Type are defined in [RFC9197].
4.2. Proof of Transit IOAM
The IOAM Proof of Transit Option-Type is there to support path or service function chain [RFC7665] verification use cases 🔗 Proof of transit could use methods like nested hashing or nested encryption of the IOAM data.
The IOAM Proof of Transit Option-Type consists of a fixed-size "IOAM Proof of Transit Option header" and "IOAM Proof of Transit Option data fields". For details, see [RFC9197].
4.3. E2E IOAM
The IOAM E2E Option-Type carries data that's added by the IOAM encapsulating node and interpreted by the IOAM decapsulating node 💌 IOAM transit nodes may process the data, but they must not modify it.
The IOAM E2E Option-Type consists of a fixed-size "IOAM Edge-to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data fields". For details, see [RFC9197].
4.4. Direct Export IOAM
Direct Export is an IOAM mode of operation where IOAM data is directly exported to a collector, rather than being collected within the data packets 📬 The IOAM Direct Export Option-Type consists of a fixed-size "IOAM direct export option header". Direct Export for IOAM is defined in [RFC9326].
5. IOAM Encapsulations 🎁
IOAM data fields and their associated data types are defined in [RFC9197] 📦 The IOAM data field can be carried by a bunch of transport protocols, including NSH, Segment Routing, Geneve, BIER, IPv6, and more.
5.1. IPv6
The IOAM encapsulation for IPv6 is defined in [IOAM-IPV6-OPTIONS], which also talks about IOAM deployment considerations for IPv6 networks 🌍
5.2. NSH
The IOAM encapsulation for NSH is defined in [IOAM-NSH].
5.3. BIER
The IOAM encapsulation for BIER is defined in [BIER-IOAM].
5.4. GRE
The IOAM encapsulation for GRE is outlined as part of the "EtherType Protocol Identification of In-situ OAM Data" in [IOAM-ETH] 🧵
5.5. Geneve
The IOAM encapsulation for Geneve is defined in [IOAM-GENEVE].
5.6. Segment Routing
The IOAM encapsulation for Segment Routing is defined in [MPLS-IOAM].
5.7. Segment Routing for IPv6
The IOAM encapsulation for Segment Routing over IPv6 is defined in [IOAM-SRV6] 🛤️
5.8. VXLAN-GPE
The IOAM encapsulation for VXLAN-GPE is defined in [IOAM-VXLAN-GPE].
6. IOAM Data Export 🚀
IOAM nodes collect info for packets traversing a domain that supports IOAM 📊 IOAM decapsulating nodes, as well as IOAM transit nodes, can choose to retrieve IOAM info from the packet, process it further, and export it using, e.g., IP Flow Information Export (IPFIX).
Raw data export of IOAM data using IPFIX is discussed in [IOAM-RAWEXPORT] 🧾 "Raw export of IOAM data" refers to a mode of operation where a node exports the IOAM data exactly as it was received in the packet. The exporting node doesn't interpret, aggregate, or reformat the IOAM data before exporting it. Raw export is there to support an operational model where processing and interpretation of IOAM data is decoupled from the encapsulating/updating/decapsulating work, which is also called "IOAM data-plane operation" ✂️ Figure 2 shows this separation of concerns for IOAM export. Exporting IOAM data is performed by the "IOAM node", which does the IOAM data-plane operation, whereas interpreting the IOAM data is done by one or several IOAM data processing systems 🧠
The whole point of the separation is to offload interpretation, aggregation, and formatting of IOAM data from the node doing the data-plane work. In other words, a node focused on data-plane operations (forwarding packets and handling IOAM data) isn't tasked with also interpreting the IOAM data. It can leave that to another system or a set of systems 🙌 For scalability reasons, a single IOAM node could export IOAM data to several systems that process it. Similarly, several monitoring or analytics systems can be used to further process the data received from the IOAM preprocessing systems. Figure 2 shows an overview of IOAM export, including IOAM data processing systems and monitoring and analytics systems.
+--------------+
+-------------+ |
| Monitoring/ | |
| Analytics | |
| system(s) |-+
+-------------+
^
| Processed/interpreted/
| aggregated IOAM data
|
+--------------+
+-------------+ |
| IOAM data | |
| processing | |
| system(s) |-+
+-------------+
^
| Raw export of
| IOAM data
|
+--------------+-------+------+--------------+
| | | |
| | | |
User +---+----+ +---+----+ +---+----+ +---+----+
packets |Encapsu-| | Transit| | Transit| |Decapsu-|
-------->|lating |====>| Node |====>| Node |====>|lating |-->
|Node | | A | | B | |Node |
+--------+ +--------+ +--------+ +--------+
Figure 2: IOAM Framework with Data Export7. IOAM Deployment Considerations ✨
okay so this section is the "how do we actually run this in a real network" part 🧚 It covers a few concepts you need before you deploy IOAM in a network domain: IOAM-Namespaces, IOAM Layering, the traffic-sets IOAM gets applied to, and IOAM Loopback. For the definitions of IOAM-Namespaces and IOAM Layering, go check [RFC9197]. IOAM Loopback is defined in [RFC9322]. bestie, you need both 💅
7.1. IOAM-Namespaces 🎀
IOAM-Namespaces add extra context to IOAM Option-Types and their IOAM-Data-Fields. They're defined in Section 4.3 of [RFC9197], and the Namespace-ID is part of the IOAM Option-Type definition. For IOAM Trace Option-Types, see Section 4.4 of [RFC9197]. For the IOAM E2E Option-Type, see Section 4.6 of [RFC9197]. IOAM-Namespaces are seriously versatile, so here are their main uses 🦋
- IOAM-Namespaces can be used by an operator to tell different operational domains apart 🏰 Devices at the domain edges can filter on Namespace-IDs to keep IOAM-Domain isolation proper.
- IOAM-Namespaces give IOAM-Data-Fields their context 💖 That's what makes them unique and lets management stations or network controllers actually interpret them. Like, the node identifier field doesn't have to be unique across a deployment. An operator might want different node identifiers for different IOAM layers, even inside the same device. Or node identifiers might just not be unique for organizational reasons, like after two formerly separate organizations merge 🙈 What does have to stay unique is the combination of node_id and Namespace-ID. Namespace-IDs can also define how certain IOAM-Data-Fields get interpreted. IOAM offers three different timestamp format options, and the Namespace-ID can be used to figure out which one is in play ⏰ IOAM-Data-Fields without an associated unit (buffer occupancy, for example) are interpreted within the context of an IOAM-Namespace 📦 The Namespace-ID could also tell different interface types apart. An interface-id might point to a physical interface, like finding out which member of an aggregated link handled a packet coming in or going out 🔌 In another case, an interface-id might refer to a logical interface, like a tunnel 🚇 So Namespace-IDs can be used to separate those interface types too.
- IOAM-Namespaces can also identify different sets of devices in a deployment (say, different device types) 🧩 If an operator wants to insert different IOAM-Data-Fields depending on the device, the devices can be grouped into multiple IOAM-Namespaces. The reasons could be that the IOAM feature set differs between device groups, or that the goal is to use packet header space more efficiently. It could also come from hardware or operational limits on how much trace data can be added and processed, which can stop a full trace from being collected for a flow 😭
- Give different Namespace-IDs to different sets of nodes or network partitions, then use the Namespace-ID as a selector at the IOAM encapsulating node. A full trace for a flow can be collected and rebuilt from partial traces spread across different packets of the same flow 🧵 For example, an operator could split a domain's devices into two IOAM-Namespaces so that, on average, only every second hop gets recorded by any device. To see the full picture, you have to correlate the captured IOAM-Data-Fields from the two IOAM-Namespaces 🔍
- Give different Namespace-IDs to different sets of nodes or network partitions, then use a separate instance of an IOAM Option-Type for each Namespace-ID. A full trace for a flow can be collected and rebuilt from partial traces coming from each IOAM Option-Type in each packet of the flow 🌈 For example, an operator could split a domain's devices into two IOAM-Namespaces, with each one represented by its own IOAM Option-Type in the packet. Each node only records data for the IOAM-Namespace it belongs to, and it ignores the other IOAM Option-Type whose IOAM-Namespace it isn't part of 🫶 To see the full deployment, you again need to correlate the captured IOAM-Data-Fields from the two IOAM-Namespaces 💜
7.2. IOAM Layering 🦋
When several encapsulation protocols get stacked on top of each other (tunneling is the classic case), IOAM-Data-Fields can show up in different protocol fields at different layers 🍓 Layering lets operators instrument whichever protocol layer they want to measure. It follows a ships-in-the-night model, which means the IOAM-Data-Fields in one layer are independent of the ones in another layer 🚢 Even though the word "layering" makes you think there's a hierarchy and some kind of relationship, in IOAM the layers are independent and don't assume any relationship between each other 🙃 Different layers can share the same IOAM encapsulation mechanisms, but they don't have to. The semantics of the IOAM-Data-Fields can be associated across different layers, but they don't have to be either 🤷♀️ For example, a node that inserts node-id information into two different layers could use node-id=10 for one layer and node-id=1000 for the other 🎀
Figure 3 shows an example of IOAM Layering 👑 It's a Geneve tunnel carried over IPv6, starting at node A and ending at node D. IOAM information gets encapsulated in IPv6 and in Geneve. At the IPv6 layer, node A is the IOAM encapsulating node (into IPv6), node D is the IOAM decapsulating node, and nodes B and C are IOAM transit nodes 🚦 At the Geneve layer, node A is the IOAM encapsulating node (into Geneve), and node D is the IOAM decapsulating node (from Geneve) 🌸 Using IOAM at both layers, like in this example, could reveal which nodes of an underlay (here, the IPv6 network) a tunneled packet crosses in an overlay (here, the Geneve network). That assumes the IOAM information node A and node D encapsulate into Geneve and into IPv6 is associated with each other 🔗 iykyk, it's giving detective energy 🕵️♀️✨
+---+----+ +---+----+
User |Geneve | |Geneve |
packets |Encapsu-| |Decapsu-|
-------->|lating |==================================>|lating |-->
| Node | | Node |
| A | | D |
+--------+ +--------+
^ ^
| |
v v
+--------+ +--------+ +--------+ +--------+
|IPv6 | | IPv6 | | IPv6 | |IPv6 |
|Encapsu-| | Transit| | Transit| |Decapsu-|
|lating |====>| Node |====>| Node |====>|lating |
| Node | | | | | | Node |
| A | | B | | C | | D |
+--------+ +--------+ +--------+ +--------+
Figure 3: IOAM Layering ExampleOkay bestie, picking up right where we left off 💖
7.3. IOAM Trace Option-Types ✨
IOAM has two Option-Types for tracing 🦋: the "Incremental" Trace Option-Type and the "Pre-allocated" Trace Option-Type. "Incremental" means the packet gets bigger at every IOAM node that adds its IOAM-Data-Fields 📈. "Pre-allocated" means the IOAM encapsulating node reserves room for all the IOAM-Data-Fields across the whole IOAM-Domain up front 🧁. Here's the breakdown:
Pre-allocated Trace Option: This one is a container of node data fields with space already reserved for each node to fill in. It's great when it's efficient to allocate the space once and then index into the array to write the data while the packet is in transit. Software forwarders often fit this vibe 💻.
Incremental Trace Option: This one is a container of node data fields where each node allocates and pushes its node data right after the option header 🚀.
Which IOAM Trace Option-Types you can support isn't just about what the operator configures 🎀. It can also be limited by protocol constraints that belong to the specific encapsulating protocol. If the encapsulating protocol supports both Option-Types, the operator picks, through configuration, which Trace Option-Type(s) get used for a particular domain 💅. In that case, a deployment can mix devices that use either the Incremental or the Pre-allocated Trace Option-Type 🌈. For example, say different kinds of packet forwarders, with different kinds of IOAM implementations, live in the same deployment, and the encapsulating protocol supports both Option-Types. Then the deployment can mix devices using either one, and as a result, both Option-Types can show up in the same packet 😭. IOAM decapsulating nodes remove both types of Trace Option-Types from the packet 🧹.
The two Option-Types serve different packet-forwarding infrastructures and let IOAM tracing be implemented in an optimized way:
Pre-allocated Trace Option: Some packet forwarder implementations find it efficient to allocate space for the maximum number of nodes that IOAM Trace Data-Fields should be collected from. They then insert or update information at flexible locations in the packet, based on a pointer read from a field in the packet 🎯. The IOAM encapsulating node allocates an array sized to the maximum number of nodes that IOAM Trace Data-Fields should be collected from. IOAM transit nodes index into that array to fill in data while the packet travels 🧭. Software forwarders often fall into this class of packet-forwarder implementations 💜. The operator configures the maximum number of nodes that IOAM information could be collected from on the IOAM encapsulating node 🛠️. The operator has to make sure that the packet carrying the pre-allocated array does not exceed the MTU of any link in the IOAM-Domain 🚨.
Incremental Trace Option: Looking up a pointer inside the packet and then inserting or updating information at a flexible location because of that lookup is costly for some forwarding infrastructures 😬. Hardware-based packet-forwarding infrastructures often fall into this category 🔌. So hardware-based packet forwarders could choose to support the IOAM Incremental Trace Option-Type 🧠. The Incremental Trace Option-Type means transit nodes don't have to read the full array in the Trace Option-Type ✨, and packets are allowed to grow up to the MTU of the IOAM-Domain 📏. IOAM transit nodes expand the packet and insert their IOAM-Data-Fields as long as there is space available, meaning as long as the packet stays within the bounds of the MTU of the links in the IOAM-Domain 🌸. The operator does not need to configure the IOAM encapsulating node with the maximum number of nodes that IOAM information could be collected from 🙌. The operator has to ensure that the minimum MTU of the links in the IOAM-Domain is known to all IOAM transit nodes 🫶.
7.4. Traffic-Sets That IOAM Is Applied To 🌈
IOAM can be deployed on all of the live user traffic or only on subsets of it 🎀. For example, you could scope it per interface, use an access control list, or use a flow specification that defines a specific set of traffic 🍓.
7.5. Loopback Flag 🔁
IOAM Loopback makes each transit device along a packet's path send a copy of the data packet back to the source 🪞. This lets an IOAM encapsulating node trace the path to a given destination and get per-hop data about both the forward and the return path 💫. Loopback gets turned on when the encapsulating node sets the Loopback flag 🚩. Looped-back packets use the source address of the original packet as their destination address, and the address of the node that performed the Loopback operation as their source address 🦋. Nodes that loop a packet back clear the Loopback flag before sending the copy toward the source 🧹. Loopback applies to IOAM deployments where the encapsulating node is either a host or the start of a tunnel 🏠. For details on IOAM Loopback, please refer to [RFC9322] 📚.
7.6. Active Flag 🧪
The Active flag tells you that a packet is an active OAM packet and not regular user data traffic 🔥. It's expected to be used for active measurement using IOAM 📊. For details on the Active flag, please refer to [RFC9322] 💅.
Example use cases for the Active flag include:
Endpoint detailed active measurement: Synthetic probe packets are sent between the source and destination 🎯. These probe packets include a Trace Option-Type (incremental or pre-allocated) 🧩. Since they travel between the endpoints, the IOAM-Domain treats them as data packets, and they don't need special handling at the IOAM layer 🙌. The source, which is also the IOAM encapsulating node, can choose to set the Active flag, which explicitly signals that these probe packets are meant for telemetry collection 📡.
IOAM active measurement using probe packets: Probe packets are generated and sent by an IOAM encapsulating node toward a destination that is also the IOAM decapsulating node 🎈. They include a Trace Option-Type (incremental or pre-allocated) that has its Active flag set 🩷.
IOAM active measurement using replicated data packets: Probe packets are created by an IOAM encapsulating node that picks some or all of the en route data packets and replicates them 👯. A selected data packet and its (possibly truncated) copy are forwarded with one or more IOAM options, while the original packet goes on normally without IOAM options 🧁. To the extent possible, the original data packet and its replica take the same path 🛤️. The replica includes a Trace Option-Type with its Active flag set, which tells the IOAM decapsulating node to terminate it 🛑. In this case, the IOAM Active flag makes sure the replicated traffic is not forwarded beyond the IOAM-Domain 🔒.
7.7. Brown Field Deployments: IOAM-Unaware Nodes 🏚️
A network can be a mix of IOAM-aware and IOAM-unaware nodes 🤝. The encapsulations of IOAM-Data-Fields into different protocols (see also Section 5) are defined so that data packets with IOAM-Data-Fields do not get dropped by IOAM-unaware nodes 🙏. For example, packets carrying the IOAM Trace Option-Types in IPv6 Hop-by-Hop extension headers have bits set to indicate "00 - skip over this option and continue processing the header" 🦸. That way, when an IOAM-unaware node gets a packet with IOAM-Data-Fields, it doesn't drop the packet 💖.
Deployments that use the IOAM Trace Option-Type(s) could benefit from being able to detect IOAM-unaware nodes 🕵️♀️. These are nodes that forward the packet but don't update or add IOAM-Data-Fields in the IOAM Trace Option-Types 😶. The node data defined as part of the IOAM Trace Option-Type(s) includes a Hop_Lim field associated with the node identifier, which helps detect missed nodes, i.e., "holes" in the trace 🕳️. Monitoring and analytics systems could use this information to account for IOAM-unaware nodes in the network 📈.
8. IOAM Manageability 🛠️
The YANG model for configuring IOAM in network nodes that support IOAM is defined in [IOAM-YANG] 🌷.
A deployment can use IOAM profiles to limit the scope of IOAM features 🎯. That makes implementation, verification, and interoperability testing simpler for specific use cases that don't need the full functionality of IOAM 🧸. An IOAM profile defines a use case (or a set of use cases) for IOAM, plus a set of rules that restrict the scope and features of the IOAM specification, limiting it to a subset of the full functionality 📝. IOAM profiles are defined in [IOAM-PROFILES] 🦄.
For deployments where the IOAM capabilities of a node are unknown, [RFC9359] could be used to discover the enabled IOAM capabilities of nodes 🔍.
9. IANA Considerations 🌟
This document has no IANA actions. ✨
10. Security Considerations 💅
Okay bestie, this one is serious, so let's go 🔒. As [RFC7276] explains, if an attacker successfully hits an OAM protocol in general (and IOAM specifically), they can stop failures or anomalies from being detected, or make it look like problems exist when they don't 😭.
The Proof of Transit Option-Type (Section 4.2) is what verifies the path that data packets take 🛤️. All the security considerations for POT are covered in [PROOF-OF-TRANSIT] ✨.
Security considerations for using IOAM flags, especially the Loopback flag, live in [RFC9322] 🚩.
IOAM data can be eavesdropped on 👀. Your user data's confidentiality isn't at risk in this situation, but the IOAM data elements can be used for network reconnaissance. That means attackers could gather info about network paths, performance, queue states, buffer occupancy, and other stuff 🕵️♀️. Recon is an unlikely threat when IOAM lives inside a tightly controlled physical domain 🏠. But if the deployment isn't limited to a single LAN and instead spans multiple interconnected sites (for example, over an overlay network), the inter-site links are expected to be secured (e.g., with IPsec) so nobody can eavesdrop from outside or sneak in malicious or false data 🔐. Another possible fix is "Direct Exporting" [RFC9326] 📤. With this approach, the IOAM-related trace information isn't carried in the customer data packets. Instead, it triggers the export of (secured) packet-related IOAM information at every node 🦋. IOAM data export, and securing that export, is outside the scope of this document 🙅♀️.
IOAM can also be used to implement or amplify Denial-of-Service (DoS) attacks 🚨. For example, a malicious attacker could add an IOAM header to packets, or change an IOAM header in packets already en route, to eat up the resources of network devices taking part in IOAM or of the collectors analyzing the IOAM data 💀. Another example is a packet-length attack, where an attacker pushes headers associated with IOAM Option-Types into data packets. That makes the packets bigger than the MTU, which leads to fragmentation or drops 📦. These DoS attacks can be mitigated by deploying IOAM in confined administrative domains, and by limiting the rate and/or percentage of packets that an IOAM encapsulating node adds IOAM information to. It also helps to limit the rate and/or percentage of packets that an IOAM transit or IOAM decapsulating node creates to export IOAM information extracted from data packets carrying IOAM information 🛡️.
Even though IOAM focuses on limited domains [RFC8799], there might be deployments where it matters that IOAM transit nodes and IOAM decapsulating nodes know the data they received hasn't been tampered with 😱. In those cases, the IOAM data should be integrity protected. Integrity protection of IOAM data fields is described in [IOAM-DATA-INTEGRITY] 🔏.
Also, IOAM options may include timestamps 🕰️. So if network devices use synchronization protocols, any attack on the time protocol [RFC7384] can compromise the integrity of the timestamp-related data fields. Synchronization attacks can be mitigated by combining a secured time distribution scheme (e.g., [RFC8915]) with redundant clock sources [RFC5905] and/or redundant network paths for the time distribution protocol [RFC8039] ⏱️🔐.
At the management plane, attacks can happen when IOAM-enabled nodes are misconfigured or maliciously configured in a way that enables other attacks 🧑💻. So IOAM configuration should be secured in a way that authenticates authorized users and verifies the integrity of configuration procedures 🔑.
Notably, IOAM is expected to be deployed in limited network domains [RFC8799], which confines the potential attack vectors within that limited domain 🧱. To keep threats inside the current network domain, the network operator is expected to enforce policies that stop IOAM traffic from leaking outside the IOAM-Domain, and that stop an attacker from introducing malicious or false IOAM data to be processed and used within the IOAM-Domain 🚧. IOAM data leakage could lead to privacy issues 🙈. Picture an IOAM encapsulating node that's a home gateway in an operator's network. A home gateway is often tied to an individual 🏡. Revealing IOAM data, such as "IOAM node identifier" or geolocation information, outside the limited domain could be harmful for that user 💔. Note that Direct Exporting [RFC9326] can mitigate the potential threat of IOAM data leaking through data packets 🌸.
11. Informative References ✨
okay bestie, here's the full reading list 📚 Every reference the spec leans on is here. Labels stay exactly as written so you can ctrl+F them 💅
[BIER-IOAM] Min, X., Zhang, Z., Liu, Y., Nainar, N., and C. Pignataro, "BIER Encapsulation for IOAM Data", Work in Progress, Internet-Draft, draft-xzlnp-bier-ioam-05, 27 January 2023, <https://datatracker.ietf.org/doc/html/draft-xzlnp-bier-ioam-05>.
[IOAM-DATA-INTEGRITY] Brockners, F., Bhandari, S., Mizrahi, T., and J. Iurman, "Integrity of In-situ OAM Data Fields", Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-data-integrity-03, 24 November 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-data-integrity-03>.
[IOAM-ETH] Weis, B., Ed., Brockners, F., Ed., Hill, C., Bhandari, S., Govindan, V., Pignataro, C., Ed., Nainar, N., Ed., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M. Spiegel, "EtherType Protocol Identification of In-situ OAM Data", Work in Progress, Internet-Draft, draft-weis-ippm-ioam-eth-05, 21 February 2022, <https://datatracker.ietf.org/doc/html/draft-weis-ippm-ioam-eth-05>.
[IOAM-GENEVE] Brockners, F., Ed., Bhandari, S., Govindan, V., Pignataro, C., Ed., Nainar, N., Ed., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Lapukhov, P., Gafni, B., Kfir, A., and M. Spiegel, "Geneve encapsulation for In-situ OAM Data", Work in Progress, Internet-Draft, draft-brockners-ippm-ioam-geneve-05, 19 November 2020, <https://datatracker.ietf.org/doc/html/draft-brockners-ippm-ioam-geneve-05>.
[IOAM-IPV6-OPTIONS] Bhandari, S., Ed. and F. Brockners, Ed., "In-situ OAM IPv6 Options", Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-ipv6-options-10, 28 February 2023, <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-ipv6-options-10>.
[IOAM-NSH] Brockners, F., Ed. and S. Bhandari, Ed., "Network Service Header (NSH) Encapsulation for In-situ OAM (IOAM) Data", Work in Progress, Internet-Draft, draft-ietf-sfc-ioam-nsh-11, 30 September 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-sfc-ioam-nsh-11>.
[IOAM-PROFILES] Mizrahi, T., Brockners, F., Bhandari, S., Ed., Sivakolundu, R., Pignataro, C., Kfir, A., Gafni, B., Spiegel, M., and T. Zhou, "In Situ OAM Profiles", Work in Progress, Internet-Draft, draft-mizrahi-ippm-ioam-profile-06, 17 February 2022, <https://datatracker.ietf.org/doc/html/draft-mizrahi-ippm-ioam-profile-06>.
[IOAM-RAWEXPORT] Spiegel, M., Brockners, F., Bhandari, S., and R. Sivakolundu, "In-situ OAM raw data export with IPFIX", Work in Progress, Internet-Draft, draft-spiegel-ippm-ioam-rawexport-06, 21 February 2022, <https://datatracker.ietf.org/doc/html/draft-spiegel-ippm-ioam-rawexport-06>.
[IOAM-SRV6] Ali, Z., Gandhi, R., Filsfils, C., Brockners, F., Nainar, N., Pignataro, C., Li, C., Chen, M., and G. Dawra, "Segment Routing Header encapsulation for In-situ OAM Data", Work in Progress, Internet-Draft, draft-ali-spring-ioam-srv6-06, 10 July 2022, <https://datatracker.ietf.org/doc/html/draft-ali-spring-ioam-srv6-06>.
[IOAM-VXLAN-GPE] Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE Encapsulation for In-situ OAM Data", Work in Progress, Internet-Draft, draft-brockners-ipxpm-ioam-vxlan-gpe-03, 4 November 2019, <https://datatracker.ietf.org/doc/html/draft-brockners-ippm-ioam-vxlan-gpe-03>.
[IOAM-YANG] Zhou, T., Ed., Guichard, J., Brockners, F., and S. Raghavan, "A YANG Data Model for In-Situ OAM", Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-yang-06, 27 February 2023, <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-yang-06>.
[MPLS-IOAM] Gandhi, R., Ed., Brockners, F., Wen, B., Decraene, B., and H. Song, "MPLS Data Plane Encapsulation for In Situ OAM Data", Work in Progress, Internet-Draft, draft-gandhi-mpls-ioam-10, 10 March 2023, <https://datatracker.ietf.org/doc/html/draft-gandhi-mpls-ioam-10>.
[PROOF-OF-TRANSIT] Brockners, F., Ed., Bhandari, S., Ed., Mizrahi, T., Ed., Dara, S., and S. Youell, "Proof of Transit", Work in Progress, Internet-Draft, draft-ietf-sfc-proof-of-transit-08, 31 October 2020, <https://datatracker.ietf.org/doc/html/draft-ietf-sfc-proof-of-transit-08>.
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, DOI 10.17487/RFC2784, March 2000, <https://www.rfc-editor.org/info/rfc2784>.
[RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, <https://www.rfc-editor.org/info/rfc5905>.
[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. Weingarten, "An Overview of Operations, Administration, and Maintenance (OAM) Tools", RFC 7276, DOI 10.17487/RFC7276, June 2014, <https://www.rfc-editor.org/info/rfc7276>.
[RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, October 2014, <https://www.rfc-editor.org/info/rfc7384>.
[RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function Chaining (SFC) Architecture", RFC 7665, DOI 10.17487/RFC7665, October 2015, <https://www.rfc-editor.org/info/rfc7665>.
[RFC7799] Morton, A., "Active and Passive Metrics and Methods (with Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, May 2016, <https://www.rfc-editor.org/info/rfc7799>.
[RFC8039] Shpiner, A., Tse, R., Schelp, C., and T. Mizrahi, "Multipath Time Synchronization", RFC 8039, DOI 10.17487/RFC8039, December 2016, <https://www.rfc-editor.org/info/rfc8039>.
[RFC8279] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., Przygienda, T., and S. Aldrin, "Multicast Using Bit Index Explicit Replication (BIER)", RFC 8279, DOI 10.17487/RFC8279, November 2017, <https://www.rfc-editor.org/info/rfc8279>.
[RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., "Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300, January 2018, <https://www.rfc-editor.org/info/rfc8300>.
[RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, <https://www.rfc-editor.org/info/rfc8799>.
[RFC8915] Franke, D., Sibold, D., Teichel, K., Dansarie, M., and R. Sundblad, "Network Time Security for the Network Time Protocol", RFC 8915, DOI 10.17487/RFC8915, September 2020, <https://www.rfc-editor.org/info/rfc8915>.
[RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., "Geneve: Generic Network Virtualization Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November 2020, <https://www.rfc-editor.org/info/rfc8926>.
[RFC9197] Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi, Ed., "Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022, <https://www.rfc-editor.org/info/rfc9197>.
[RFC9322] Mizrahi, T., Brockners, F., Bhandari, S., Gafni, B., and M. Spiegel, "In Situ Operations, Administration, and Maintenance (IOAM) Loopback and Active Flags", RFC 9322, DOI 10.17487/RFC9322, November 2022, <https://www.rfc-editor.org/info/rfc9322>.
[RFC9326] Song, H., Gafni, B., Brockners, F., Bhandari, S., and T. Mizrahi, "In Situ Operations, Administration, and Maintenance (IOAM) Direct Exporting", RFC 9326, DOI 10.17487/RFC9326, November 2022, <https://www.rfc-editor.org/info/rfc9326>.
[RFC9359] Min, X., Mirsky, G., and L. Bo, "Echo Request/Reply for Enabled In Situ OAM (IOAM) Capabilities", RFC 9359, DOI 10.17487/RFC9359, April 2023, <https://www.rfc-editor.org/info/rfc9359>.
[VXLAN-GPE] Maino, F., Ed., Kreeger, L., Ed., and U. Elzur, Ed., "Generic Protocol Extension for VXLAN (VXLAN-GPE)", Work in Progress, Internet-Draft, draft-ietf-nvo3-vxlan-gpe-12, 22 September 2021, <https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-vxlan-gpe-12>.
Acknowledgements 🫶
the authors wanna give a huge shoutout 📣 to Tal Mizrahi, Eric Vyncke, Nalini Elkins, Srihari Raghavan, Ranganathan T S, Barak Gafni, Karthik Babu Harichandra Babu, Akshaya Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew Yourtchenko, Aviv Kfir, Tianran Zhou, Zhenbin (Robin), Joe Clarke, Al Morton, Tom Herbet, Haoyu Song, and Mickey Spiegel for the comments and advice on IOAM 💖 the whole squad showed up 👑
Authors' Addresses 🎀
Frank Brockners (editor) 🌸
Cisco Systems, Inc.
Hansaallee 249, 3rd Floor
40549 DUESSELDORF
Germany
Email: fbrockne@cisco.comShwetha Bhandari (editor) 🦋
Thoughtspot
3rd Floor, Indiqube Orion
Garden Layout, HSR Layout
24th Main Rd
Bangalore 560 102
KARNATAKA
India
Email: shwetha.bhandari@thoughtspot.comDaniel Bernier 🧚
Bell Canada
Canada
Email: daniel.bernier@bell.caTal Mizrahi (editor) 💜
Huawei
8-2 Matam
Haifa 3190501
Israel
Email: tal.mizrahi.phd@gmail.com📦 where this page comes from
- ietf: rfc9378 2023-04
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms