RFC 9197
Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)
RFC 9197: Data Fields for In Situ Operations, Administration, and Maintenance (IOAM) 🦋
okay bestie, this one is the IOAM data-fields spec, and it's a Standards Track document from the IETF (Internet Engineering Task Force) 💅 It was published in May 2022 under ISSN 2070-1721. The editors are F. Brockners (Cisco), S. Bhandari (Thoughtspot), and T. Mizrahi (Huawei), and all three are listed as "Ed." (editors). The header doesn't list anything this one obsoletes or updates, so it's standing on its own 🌸
Abstract
okay so IOAM (In situ Operations, Administration, and Maintenance) collects operational and telemetry info inside the packet while that packet travels along a path between two points in the network 🧚 this doc talks about the data fields and their data types. IOAM-Data-Fields can ride along in a bunch of different protocols: Network Service Header (NSH), Segment Routing, Generic Network Virtualization Encapsulation (Geneve), or IPv6. And IOAM can complement OAM mechanisms that are based on things like ICMP or other kinds of probe packets 💖
Status of This Memo
this is an Internet Standards Track document ✨
it's a product of the IETF, so it represents the consensus of the IETF community. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). if you want more on Internet Standards, check Section 2 of RFC 7841 🎀
for the current status, any errata, and how to give feedback, go to https://www.rfc-editor.org/info/rfc9197 🌈
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. 🔒
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. please read these carefully, because they spell out your rights and restrictions for this document 📜 Code Components pulled out of this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as the Revised BSD License describes 💜
Table of Contents 🗂️
1. Introduction
2. Conventions
3. Scope, Applicability, and Assumptions
4. IOAM Data-Fields, Types, and Nodes
4.1. IOAM Data-Fields and Option-Types
4.2. IOAM-Domains and Types of IOAM Nodes
4.3. IOAM-Namespaces
4.4. IOAM Trace Option-Types
4.4.1. Pre-allocated and Incremental Trace Option-Types
4.4.2. IOAM Node Data Fields and Associated Formats
4.4.2.1. Hop_Lim and node_id Short
4.4.2.2. ingress_if_id and egress_if_id Short
4.4.2.3. Timestamp Seconds
4.4.2.4. Timestamp Fraction
4.4.2.5. Transit Delay
4.4.2.6. Namespace-Specific Data
4.4.2.7. Queue Depth
4.4.2.8. Checksum Complement
4.4.2.9. Hop_Lim and node_id Wide
4.4.2.10. ingress_if_id and egress_if_id Wide
4.4.2.11. Namespace-Specific Data Wide
4.4.2.12. Buffer Occupancy
4.4.2.13. Opaque State Snapshot
4.4.3. Examples of IOAM Node Data
4.5. IOAM Proof of Transit Option-Type
4.5.1. IOAM Proof of Transit Type 0
4.6. IOAM Edge-to-Edge Option-Type
5. Timestamp Formats
5.1. PTP Truncated Timestamp Format
5.2. NTP 64-Bit Timestamp Format
5.3. POSIX-Based Timestamp Format
6. IOAM Data Export
7. IANA Considerations
7.1. IOAM Option-Type Registry
7.2. IOAM Trace-Type Registry
7.3. IOAM Trace-Flags Registry
7.4. IOAM POT-Type Registry
7.5. IOAM POT-Flags Registry
7.6. IOAM E2E-Type Registry
7.7. IOAM Namespace-ID Registry
8. Management and Deployment Considerations
9. Security Considerations
10. References
10.1. Normative References
10.2. Informative References
Acknowledgements
Contributors
Authors' Addresses1. Introduction ✨
this doc defines the data fields for In situ Operations, Administration, and Maintenance (IOAM) 🎀 IOAM records OAM info within the packet while that packet traverses a particular network domain. "In situ" just means the OAM data gets added to the regular data packets, instead of being sent inside packets that exist only for OAM 🦋 IOAM is used to complement tools like Ping or Traceroute.
in terms of "active" or "passive" OAM, IOAM counts as a hybrid OAM type 💅 "In situ" mechanisms don't need any extra packets to be sent. IOAM just adds info to data packets that are already flying around, so it can't be called passive either. using the classification from [RFC7799], IOAM could be portrayed as Hybrid Type I 🌸
IOAM is the answer where mechanisms using, e.g., ICMP don't apply or don't give the results you want. examples: proving that a certain traffic flow takes a predefined path, Service Level Agreement (SLA) verification for data traffic, detailed statistics on traffic distribution in networks that spread traffic across multiple paths, or scenarios where probe traffic gets handled differently from regular data traffic by network devices 😭
the term "in situ OAM" started out because of OAM mechanisms that add info into a packet. this document uses IOAM as the name for the whole IOAM technology 💜 IOAM includes the "in situ" mechanisms, but it also includes mechanisms that could trigger the creation of extra packets dedicated to OAM 🧚
2. Conventions 🎀
okay the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document get interpreted as described in BCP 14 [RFC2119] [RFC8174], but only when they show up in all capitals, exactly like shown here 💅 lowercase versions are just regular words 🙈
here's the abbreviation and definition list 🌈
E2E: Edge to Edge
Geneve: Generic Network Virtualization Encapsulation [RFC8926]
IOAM: In situ Operations, Administration, and Maintenance
MTU: Maximum Transmission Unit
NSH: Network Service Header [RFC8300]
OAM: Operations, Administration, and Maintenance
PMTU: Path MTU
POT: Proof of Transit
Short format: refers to an IOAM-Data-Field that comprises 4 octets
SID: Segment Identifier
SR: Segment Routing
VXLAN-GPE: Virtual eXtensible Local Area Network, Generic
Protocol Extension [NVO3-VXLAN-GPE]
Wide format: refers to an IOAM-Data-Field that comprises 8 octets3. Scope, Applicability, and Assumptions 🧚
IOAM comes with a set of constraints, plus guiding principles and concepts that go hand in hand with defining the IOAM-Data-Fields 💖 this section describes all of those. how IOAM-Data-Fields and their concepts get applied in an actual IOAM deployment is out of scope for this doc, so check [IPPM-IOAM-DEPLOYMENT] for deployment considerations 🦋
Scope: This doc defines the data fields and associated data types for IOAM 🎀 IOAM-Data-Fields can be encapsulated in a bunch of protocols, including NSH, Segment Routing, Geneve, and IPv6. the specifics of those protocols are outside this doc's scope. each encapsulation is expected to get its own RFC, designed jointly by the working group that develops or maintains the encapsulation protocol and the IETF IP Performance Measurement (IPPM) Working Group 🌸
Domain (or scope) of in situ OAM deployment: IOAM is focused on "limited domains", as defined in [RFC8799] ✨ for IOAM, a limited domain could be, for example, an enterprise campus with physical connections between devices, or an overlay network that uses virtual connections or tunnels between those devices 💜 a limited domain that uses IOAM may be made up of one or several "IOAM-Domains", each told apart by its own namespace identifiers. an IOAM-Domain is bounded by its perimeter or edge, and IOAM-Domains may overlap inside the limited domain 🌈 designers of protocol encapsulations for IOAM specify mechanisms that keep IOAM data inside an IOAM-Domain 🔒 the operator of such a domain is also expected to set up provisions so IOAM data doesn't leak past the edge of an IOAM-Domain, for example with packet filtering methods 😭
the operator SHOULD think about the potential operational impact of IOAM on a few things 💅 first, ECMP processing: load-balancing schemes based on packet length could be affected by the bigger packet size IOAM causes. second, PMTU: make sure the MTU of all links within a domain is big enough to support the increased packet size from IOAM. third, ICMP message handling: for IPv6, IOAM support for ICMPv6 echo request/reply is desired, which would mean ICMPv6 extensions that let IOAM-Data-Fields get copied from an echo request message into the echo reply message 🎀
IOAM control points: IOAM-Data-Fields get added to or removed from user traffic by the devices that form the edge of a domain 🧚 devices in an IOAM-Domain can add, update, or remove IOAM-Data-Fields. edge devices of an IOAM-Domain can be hosts or network devices 🫶
Traffic sets that IOAM is applied to: IOAM can be deployed on all of the user traffic, or only on subsets of it 💖 using IOAM on a selected set of traffic (for example per interface, or based on an access control list or a flow specification that defines a specific set of traffic) could be useful in deployments where the cost of processing IOAM-Data-Fields, whether by encapsulating, transit, or decapsulating nodes, is a concern for performance or operations 🦋 so limiting the amount of traffic IOAM is applied to could be beneficial in some deployments 🌸
Encapsulation independence: the definition of IOAM-Data-Fields is independent of the protocols they get encapsulated into ✨ IOAM-Data-Fields can be encapsulated into several encapsulating protocols 💅
Layering: if several encapsulation protocols are stacked on top of each other (like in tunneling), IOAM-Data-Fields could be present at multiple layers 🌈 the behavior follows the "ships-in-the-night" model, meaning the IOAM-Data-Fields in one layer are independent from the ones in another layer 🧚 layering lets operators instrument whichever protocol layer they want to measure 💜 the different layers could, but don't have to, share the same IOAM encapsulation mechanisms 🎀
IOAM implementation: the definition of the IOAM-Data-Fields takes into account the specifics of devices with hardware data planes and devices with software data planes 🔥
4. IOAM Data-Fields, Types, and Nodes 🌸
this section breaks down IOAM-related nomenclature and describes data types like IOAM-Data-Fields, IOAM-Types, and IOAM-Namespaces, plus the different types of IOAM nodes 💖✨
4.1. IOAM Data-Fields and Option-Types
okay so an IOAM-Data-Field is basically a set of bits with a defined format and meaning 💖 and it can be stored at a certain place in a packet for the purpose of IOAM ✨
IOAM gets used for lots of different things, so IOAM-Data-Fields are sorted into categories 🎀 IOAM calls these categories "IOAM-Option-Types". There's a common registry maintained for IOAM-Option-Types (see Section 7.1 for details), and different IOAM-Data-Fields are defined to match each of them 🧚
This document defines four IOAM-Option-Types 👑:
- Pre-allocated Trace Option-Type
- Incremental Trace Option-Type
- POT Option-Type
- E2E Option-Type
Future IOAM-Option-Types can be allocated by IANA, as described in Section 7.1 🦋
4.2. IOAM-Domains and Types of IOAM Nodes
Section 3 already mentioned that IOAM is expected to be deployed in a limited domain [RFC8799] 🌸 One or more IOAM-Option-Types get added to a packet when it enters an IOAM-Domain, and they get removed when it exits the domain 🚪 Inside the IOAM-Domain, the IOAM-Data-Fields MAY be updated by the network nodes the packet goes through 🔥 An IOAM-Domain is made up of "IOAM encapsulating nodes", "IOAM decapsulating nodes", and "IOAM transit nodes" 💅 A node's role (encapsulating, transit, or decapsulating) is defined within an IOAM-Namespace (see below), and a node can have different roles in different IOAM-Namespaces, which is so iconic 👑
A device that adds at least one IOAM-Option-Type to the packet is called an "IOAM encapsulating node" 🫶 A device that removes an IOAM-Option-Type is called an "IOAM decapsulating node" 🧹 Nodes inside the domain that know about IOAM data and read, write, and/or process it are called "IOAM transit nodes" 🎀 IOAM nodes that add or remove the IOAM-Data-Fields can also update the IOAM-Data-Fields at the same time 🙈 In other words, IOAM encapsulating or decapsulating nodes can also be IOAM transit nodes at the same time, periodt 💜 Not every node in an IOAM-Domain needs to be an IOAM transit node, though 🌈 For example, a deployment might require that packets go through a set of firewalls that support IOAM. In that case, only the firewall nodes would be IOAM transit nodes, rather than all nodes 🧱
An IOAM encapsulating node adds one or more IOAM-Option-Types (from the list of IOAM-Types, see Section 7.1) into packets that IOAM is enabled for 💖 If IOAM is only enabled for a selected subset of the traffic, the IOAM encapsulating node is responsible for applying the IOAM functionality to that subset ✨
An IOAM transit node reads, writes, and/or processes one or more of the IOAM-Data-Fields 🦋 If both the Pre-allocated and the Incremental Trace Option-Types are present in the packet, each IOAM transit node, based on its configuration and the available implementation of IOAM, MAY populate IOAM trace data in either a Pre-allocated or an Incremental Trace Option-Type, but not both 🌸 Not populating any of the Trace Option-Types is also valid behavior for an IOAM transit node 🫶 A transit node MUST ignore IOAM-Option-Types that it does not understand 🙅♀️ A transit node MUST NOT add new IOAM-Option-Types to a packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type. Hard rules, bestie 🔒
An IOAM decapsulating node removes IOAM-Option-Type(s) from packets 🧹
The role of an IOAM encapsulating, IOAM transit, or IOAM decapsulating node is always performed within a specific IOAM-Namespace 💜 So an IOAM node that is, e.g., an IOAM decapsulating node for IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove the IOAM-Option-Types for IOAM-Namespace "A" from the packet 🎯 This applies even to IOAM-Option-Types the node does not understand, like an IOAM-Option-Type other than the four described above that gets added in a future revision 🔮
IOAM-Namespaces allow for namespace-specific definitions and interpretations of IOAM-Data-Fields 🌈 An interface identifier could, for example, point to a physical interface (e.g., to figure out which physical interface of an aggregated link is used when receiving or transmitting a packet) 🔌 In another case, it could refer to a logical interface (e.g., in the case of tunnels) 🚇 Please refer to Section 4.3 for details on IOAM-Namespaces 📚
4.3. IOAM-Namespaces
IOAM-Namespaces add further context to IOAM-Option-Types and their associated IOAM-Data-Fields ✨ The IOAM-Option-Types and associated IOAM-Data-Fields are interpreted as defined in this document, no matter what the value of the IOAM-Namespace is 💅 IOAM-Namespaces still give you a way to group nodes to support different deployment approaches of IOAM (see a few example use cases below) 🧩 They also help with issues that can pop up because IOAM-Data-Fields aren't globally unique (e.g., IOAM node identifiers do not have to be globally unique) 🙈 The significance of IOAM-Data-Fields is always within a particular IOAM-Namespace, so the Namespace-ID field always needs to be carried along with the IOAM data-fields themselves 🎀
An IOAM-Namespace is identified by a 16-bit namespace identifier (Namespace-ID) 🔢 The IOAM-Namespace field is included in all the IOAM-Option-Types defined in this document and MUST be included in all future IOAM-Option-Types 🔒 The Namespace-ID value is split into two subranges 🍰:
- an operator-assigned range from
0x0001to0x7FFFand - an IANA-assigned range from
0x8000to0xFFFF.
The IANA-assigned range is meant to let future extensions bring new, interoperable IOAM functionality 🌟 The operator-assigned range is meant to be domain specific and managed by the network operator 🛠️ The Namespace-ID value of 0x0000 is the "Default-Namespace-ID" 🌸 The Default-Namespace-ID means no specific namespace is associated with the IOAM-Data-Fields in the packet 🤍 The Default-Namespace-ID MUST be supported by all nodes implementing IOAM, no exceptions 👑 One use case for the Default-Namespace-ID is deployments that don't use specific namespaces for some or all of their packets that carry IOAM-Data-Fields 🧁
Namespace identifiers let IOAM-capable devices figure out:
- whether one or more IOAM-Option-Types need to be processed by a device. If the
Namespace-IDin a packet doesn't match anyNamespace-IDthe node is configured to operate on, then the node MUST NOT change the contents of the IOAM-Data-Fields 🚫 - which IOAM-Option-Type needs to be processed or updated when there are multiple IOAM-Option-Types in the packet. Multiple IOAM-Option-Types can show up in a packet when IOAM-Domains overlap or in a layered IOAM deployment 🥞
- whether one or more IOAM-Option-Types have to be removed from the packet, e.g., at a domain edge or domain boundary 🚪
IOAM-Namespaces support several different uses 💖:
- IOAM-Namespaces can be used by an operator to tell different IOAM-Domains apart. Devices at the edges of an IOAM-Domain can filter on Namespace-IDs to keep the IOAM-Domain properly isolated 🛡️
- IOAM-Namespaces give extra context to IOAM-Data-Fields, so they can be used to make sure IOAM-Data-Fields are unique and get interpreted properly by management stations or network controllers 🧠 The node identifier field (
node_id, see below) doesn't need to be unique in a deployment. That could be the case if an operator wants different node identifiers for different IOAM layers, even within the same device, or if node identifiers aren't unique for other organizational reasons, like after a merger of two formerly separate organizations 🤝 TheNamespace-IDcan be used as a context identifier, so the combination ofnode_idandNamespace-IDwill always be unique 🔑 - IOAM-Namespaces can also define how certain IOAM-Data-Fields are interpreted. IOAM offers three different timestamp format options, and the
Namespace-IDcan be used to determine the timestamp format ⏰ IOAM-Data-Fields that don't have a unit associated with them (e.g., buffer occupancy) are to be interpreted within the context of an IOAM-Namespace 📏 - IOAM-Namespaces can identify different sets of devices (e.g., different types of devices) in a deployment 🧸 If an operator wants to insert different IOAM-Data-Fields based on the device, the devices can be grouped into multiple IOAM-Namespaces 🗂️ This could be because the IOAM feature set differs between sets of devices, or for reasons of optimized space usage in the packet header 📦 It could also come from hardware or operational limits on the size of the trace data that can be added and processed, which can stop a full trace for a flow from being collected 🚧
- By assigning different IOAM Namespace-IDs to different sets of nodes or network partitions, and using a separate instance of an IOAM-Option-Type for each Namespace-ID, a full trace for a flow can be collected and built from partial traces, one from each IOAM-Option-Type in each of the packets in the flow 🧩 For example, an operator could group the devices of a domain into two IOAM-Namespaces, so each IOAM-Namespace is represented by one of two IOAM-Option-Types in the packet 👯 Each node records data only for the IOAM-Namespace it belongs to, and ignores the other IOAM-Option-Type whose IOAM-Namespace it doesn't belong to 🙃 To get the full view of the deployment, the captured IOAM-Data-Fields of the two IOAM-Namespaces need to be correlated 🧵✨
4.4. IOAM Trace Option-Types ✨
Okay bestie, so in a normal deployment, every node in an IOAM-Domain is joining the IOAM party 🎉. That means they're IOAM transit nodes, IOAM encapsulating nodes, or IOAM decapsulating nodes 💖. If not every node in the domain supports IOAM the way this document defines it, then IOAM tracing info (node data, more on that below) can only be collected from the nodes that do 😭. The non-IOAM nodes just forward the packet and leave the IOAM-Data-Fields totally untouched, no cap 🫶. The maximum number of hops and the minimum PMTU of the IOAM-Domain are assumed to be known 📏. There's also an overflow indicator (O-bit) 🚩, which is one way to handle the situation where the PMTU was underestimated, aka the number of IOAM-capable hops is bigger than the space available in the packet 💀.
To keep hardware and software both happy, IOAM tracing is defined as two separate options 🧚. A deployment can choose to configure and support one or both of them 💅.
Pre-allocated Trace-Option: This trace option is a container of node data fields (see below) with pre-allocated space for each node to fill in its info 📦. It's super useful when it's efficient to allocate the space once and index into the array to populate data in transit (software forwarders often fall into this class 💻). The IOAM encapsulating node allocates space for the Pre-allocated Trace Option-Type in the packet and sets the corresponding fields in this IOAM-Option-Type 🎀. It also allocates an array to store operational data retrieved from every node while the packet traverses the domain 🗂️. IOAM transit nodes update the content of the array and possibly update the checksums of outer headers ✍️. A pointer that's part of the IOAM trace data points to the next empty slot in the array 👉. An IOAM transit node that updates the Pre-allocated Trace-Option also updates the value of that pointer, which specifies where the next IOAM transit node fills in its data 🦋. The node data list array in the packet gets populated iteratively as the packet traverses the network, starting with the last entry. So node data list [n] is the first entry to be populated, node data list [n-1] is the second one, and so on 🌈.
Incremental Trace-Option: This trace option is a container of node data fields, where each node allocates and pushes its node data immediately after the option header 🚀. This is useful for some hardware implementations, because it means the transit network elements don't have to read the full array in the option, and it allows for as arbitrarily long packets as the MTU allows 🌸. The IOAM encapsulating node allocates space for the Incremental Trace Option-Type 📮. Based on the operational state and configuration, the IOAM encapsulating node sets the fields in the Option-Type that control which IOAM-Data-Fields have to be collected and how large the node data list can grow 🎯. IOAM transit nodes push their node data to the node data list, subject to any protocol constraints of the encapsulating layer ✨. They then decrease the remaining length available to subsequent nodes and adjust the lengths and possibly checksums in outer headers 📏.
IOAM encapsulating nodes and IOAM decapsulating nodes that support tracing MUST support both Trace Option-Types 🙌. For IOAM transit nodes, it is sufficient to support one of the Trace Option-Types 💜. If both options are utilized in a deployment at the same time, the Incremental Trace-Option MUST be placed before the Pre-allocated Trace-Option 🔥. Deployments that mix devices with either the Incremental Trace-Option or the Pre-allocated Trace-Option could end up with both Option-Types present in a packet 🍓. Given that the operator knows which equipment is deployed in a particular IOAM-Domain, the operator decides by configuration which type(s) of trace options will be used for that domain 🎀.
Every node data entry holds information for a particular IOAM transit node that is traversed by a packet 📝. The IOAM decapsulating node removes the IOAM-Option-Types and processes and/or exports the associated data 📤. Like all IOAM-Data-Fields, the IOAM-Data-Fields of the IOAM Trace Option-Types are defined in the context of an IOAM-Namespace 👑.
IOAM tracing can collect the following types of information:
- Identification of the IOAM node. An IOAM node identifier can match to a device identifier or a particular control point or subsystem within a device 🖥️.
- Identification of the interface that a packet was received on, i.e., ingress interface 📥.
- Identification of the interface that a packet was sent out on, i.e., egress interface 📤.
- Time of day when the packet was processed by the node, as well as the transit delay ⏰. Different definitions of processing time are feasible and expected, though it is important that all devices of an IOAM-Domain follow the same definition 🤝.
- Generic data, i.e., format-free information where the syntax and semantics of the information are defined by the operator in a specific deployment 🧩. For a specific IOAM-Namespace, all IOAM nodes have to interpret the generic data the same way. Examples for generic IOAM data include geolocation information (location of the node at the time the packet was processed), buffer queue fill level or cache fill level at the time the packet was processed, or even a battery-charge level 🔋.
- Information to detect whether IOAM trace data was added at every hop or whether certain hops in the domain weren't IOAM transit nodes 🕵️♀️.
It should be noted that the semantics of some of the node data fields that are defined below, such as the queue depth and buffer occupancy, are implementation specific. This approach is intended to allow IOAM nodes with various different architectures 🌈.
4.4.1. Pre-allocated and Incremental Trace Option-Types ✨
okay bestie, so the IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace-Option are basically twins 👯♀️ 🦋. Unless it says otherwise, their insides are identical. Both start with a fixed-size "trace option header" and then have a variable data space for the gathered info, which is the "node data list" 💅. One big rule: an IOAM transit node (so not an IOAM encapsulating node or IOAM decapsulating node) MUST NOT modify any fields in that fixed-size header except "Flags" and "RemainingLen". So it MUST NOT touch Namespace-ID, NodeLen, IOAM Trace-Type, or Reserved. Hands off 🙅♀️
Here's what the headers look like 🎀:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Namespace-ID |NodeLen | Flags | RemainingLen|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IOAM Trace-Type | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The trace option data MUST be aligned by 4 octets 📐. Picture the node data list stacked up like this:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
| | |
| node data list [0] | |
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D
| | a
| node data list [1] | t
| | a
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ ... ~ S
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p
| | a
| node data list [n-1] | c
| | e
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| | |
| node data list [n] | |
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+Now let's go field by field, because the details matter 🤓
Namespace-ID: this is a 16-bit identifier of an IOAM-Namespace 🏷️. The value 0x0000 is the "Default-Namespace-ID" (see Section 4.3), and it MUST be known to all the nodes implementing IOAM. For any other Namespace-ID that doesn't match a namespace the node is configured to operate on, the node MUST NOT change the contents of the IOAM-Data-Fields. Respect the boundaries, babe 🙅♀️
NodeLen: this is a 5-bit unsigned integer 🔢. It tells you how much data each node adds, counted in multiples of 4 octets. The Opaque State Snapshot field doesn't count toward this one.
If IOAM Trace-Type Bit 22 is not set, NodeLen is the actual length each node adds. If Bit 22 is set, then the real length a node adds is (NodeLen + length of the "Opaque State Snapshot" field), still in 4-octet units 🧠
Quick examples because I know you need them 💕: if 3 IOAM Trace-Type bits are set and none are wide format, NodeLen is 3. If 3 bits are set and 2 of them are wide, NodeLen is 5.
An IOAM encapsulating node MUST set NodeLen 📦. And a node receiving an IOAM Pre-allocated or Incremental Trace-Option relies on the NodeLen value, so it's kind of a big deal 😭
Flags: this is a 4-bit field, and IANA allocates the flags as specified in Section 7.3 🏛️. This document only allocates one flag:
- Bit 0: "Overflow" (O-bit), the most significant bit. Okay, this one is the drama queen 👑. If a network element is supposed to add node data but realizes there aren't enough octets left to record it, the network element MUST NOT add any fields, and it MUST set the overflow "O-bit" to "1" in the IOAM Trace-Option header. This is useful for transit nodes, since it lets them skip any further processing of the option 💀
RemainingLen: this is a 7-bit unsigned integer ⏳. It shows how much data space, in multiples of 4 octets, is still left for recording node data before the node data list counts as overflowed. The sender MUST assign the initial value of RemainingLen 🎯. The sender MAY calculate that value by working out how many node data bytes fit before going past the PMTU, assuming the sender knows the PMTU. After that, later nodes can just compare RemainingLen against NodeLen, plus the length of the "Opaque State Snapshot" if it applies, to see whether they can add their data 🧮. Whenever a node adds node data, it MUST decrease RemainingLen by however much it added ➖
In the Pre-allocated Trace-Option, RemainingLen also tells you where to write the node data element. The recording starts at RemainingLen - NodeLen - size of (opaque snapshot), measured in 4-octet units 🗺️. And if RemainingLen in a Pre-allocated Trace-Option is bigger than the length of the option (as the lower-layer header says, which is out of scope for this document), the node MUST NOT add any fields 🚫
IOAM Trace-Type: this is a 24-bit identifier that says which data types are in this node data list 🌈. It's a bit field, and the bits defined in this document are explained in Section 4.4.2. The data fields in each node data element are packed in the same bit order as the IOAM Trace-Type field:
- Bit 0 (most significant bit): when set, Hop_Lim and node_id (short format) are in the node data.
- Bit 1: when set, ingress_if_id and egress_if_id (short format) are in the node data.
- Bit 2: when set, timestamp seconds are in the node data.
- Bit 3: when set, timestamp fraction is in the node data.
- Bit 4: when set, transit delay is in the node data.
- Bit 5: when set, IOAM-Namespace-specific data in short format is in the node data.
- Bit 6: when set, queue depth is in the node data.
- Bit 7: when set, the Checksum Complement node data is in the node data.
- Bit 8: when set, Hop_Lim and node_id in wide format are in the node data.
- Bit 9: when set, ingress_if_id and egress_if_id in wide format are in the node data.
- Bit 10: when set, IOAM-Namespace-specific data in wide format is in the node data.
- Bit 11: when set, buffer occupancy is in the node data.
- Bits 12-21: undefined 🔮. These are open for future assignment in the IOAM Trace-Type Registry (Section 7.2). Every future node data field that matches one of these bits MUST be 4 octets long. An IOAM encapsulating node MUST set each undefined bit to 0. If an IOAM transit node gets a packet with one or more of these bits set to 1, it MUST do one of two things:
- add corresponding node data filled with the reserved value 0xFFFFFFFF after the node data fields for the IOAM Trace-Type bits defined above, so that the total node data this node adds, in units of 4 octets, equals NodeLen, or
- not add any node data fields to the packet, even for the IOAM Trace-Type bits defined above.
- Bit 22: when set, the variable-length Opaque State Snapshot field is present 🫧.
- Bit 23: reserved. It MUST be set to zero upon transmission and be ignored upon receipt. It's kept around so the IOAM Trace-Type bit field can be extended later 🧸
Section 4.4.2 covers the IOAM-Data-Types and their formats. Inside an IOAM-Domain, configuration knobs can restrict which combinations of these bits make up the IOAM Trace-Type ⚙️
Reserved: this is 8 bits. An IOAM encapsulating node MUST set it to zero upon transmission, and IOAM transit nodes MUST ignore the value they receive 🤫
Node data List [n]: this is a variable-length field, a list of node data elements where what's in each element depends on the IOAM Trace-Type 📋. The data fields are packed in the bit order of the IOAM Trace-Type field. Each node MUST prepend its node data element in front of the ones it received, so the transmitted list starts with that node's element as the first populated entry. The last element in the list is the node data of the first IOAM-capable node on the path 🏁. Building the list this way keeps the order the same for Incremental and Pre-allocated Trace-Options. In the Pre-allocated Trace-Option, the index stored in RemainingLen tells you the offset for the current active node data to fill in 🎯
4.4.2. IOAM Node Data Fields and Associated Formats 💖
okay bestie, this is where the actual node data fields live 🫶 All the IOAM-Data-Fields MUST be aligned by 4 octets 📏. If a node that's supposed to update an IOAM-Data-Field can't fill in a field set in the IOAM Trace-Type, the field value MUST be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 8-octet fields. That means "not populated," unless the field description below says otherwise 🙈
Some IOAM-Data-Fields, like interface identifiers or IOAM-Namespace-specific data, come in both a "short format" and a "wide format" ✨ Using one doesn't rule out the other, so a deployment can totally use both. For example, ingress_if_id_(short format) could identify the physical interface, while ingress_if_id_(wide format) could identify a logical sub-interface sitting on top of that physical one 🦋
The data fields and their data types are spelled out in the sections below. IOAM sticks to the syntax of each field and tries hard to skip the semantics 💅 That's why there are no units defined, like "buffer occupancy" or "queue depth." Nodes can report info in whatever format they already have, no conversions needed. Systems that do more processing with IOAM info, like a network management system, are expected to handle unit conversions themselves. The combo of a data field and its Namespace-ID is what gives the data its context 🧚
4.4.2.1. Hop_Lim and node_id Short
The "Hop_Lim and node_id short" field is a 4-octet field, and it's laid out like this 🍓
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Hop_Lim is a 1-octet unsigned integer, and it gets set to the Hop Limit value in the packet as it leaves the node that records this data 🏃♀️ Hop Limit info is how you figure out where a node sits along the communication path. It's copied from the lower layer, like the TTL in an IPv4 header or the Hop Limit field in an IPv6 header, once the packet is ready to go out. The meaning of Hop_Lim depends on whichever lower-layer protocol IOAM is encapsulated in, so its exact semantics are outside the scope of this memo 🤷♀️ If the lower level doesn't have a field equivalent to TTL / Hop Limit, this field MUST be set to 0xff.
node_id is a 3-octet unsigned integer, and its whole job is to uniquely identify a node within the IOAM-Namespace and its associated IOAM-Domain 🆔 How you allocate, manage, and map node_ids is beyond this document's scope. If you want the deployment side of node_id, check out [IPPM-IOAM-DEPLOYMENT] 📚
4.4.2.2. ingress_if_id and egress_if_id Short
The "ingress_if_id and egress_if_id" field is a 4-octet field, and it looks like this 👀
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingress_if_id | egress_if_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
ingress_if_id is a 2-octet unsigned integer that records the ingress interface the packet came in on 📥
egress_if_id is a 2-octet unsigned integer that records the egress interface the packet is forwarded out of 📤
Here's the tea 🍵 IOAM has its own IOAM-Namespaces for its data fields, so interface identifiers can be used pretty flexibly to represent whatever system resources are tied to packets coming in or going out. ingress_if_id could be a physical interface, a virtual or logical interface, or even a queue 🌈
4.4.2.3. Timestamp Seconds
The "timestamp seconds" field is a 4-octet unsigned integer field ⏰ It holds the absolute timestamp, in seconds, for when the node received the packet. This field has three possible formats, based on either the Precision Time Protocol (PTP) (see e.g., [RFC8877]), NTP [RFC5905], or POSIX [POSIX]. The three timestamp formats are specified in Section 5. In all three cases, the timestamp seconds field holds the 32 most significant bits of the timestamp format from Section 5 💅
If a node can't fill this field in, it assigns the value 0xFFFFFFFF. Plot twist 😭 this is a legitimate value too, and it's valid for 1 second in approximately 136 years. So the analyzer has to correlate several packets, or compare the value to its own time of day, to tell that it's actually an error indication 🕰️
4.4.2.4. Timestamp Fraction
The "timestamp fraction" field is a 4-octet unsigned integer field 🌸 Fraction is the fractional part of the number of seconds since the NTP epoch [RFC8877], and it tells you the time when the node received the packet. Like timestamp seconds, it has three possible formats, based on PTP (see e.g., [RFC8877]), NTP [RFC5905], or POSIX [POSIX]. These three formats are specified in Section 5. In all three cases, the timestamp fraction field holds the 32 least significant bits of the timestamp format from Section 5 ✨
If a node can't populate this field, it assigns 0xFFFFFFFF. This one's a fun little trap too 🙈 In the NTP format, that value is legitimate and valid for approximately 233 picoseconds in every second. So if the NTP format is in use, the analyzer has to correlate several packets to detect the error indication 🔍
4.4.2.5. Transit Delay
The "transit delay" field is a 4-octet unsigned integer in the range 0 to 2^31-1 ⏳ It's the time, in nanoseconds, that the packet spent in the transit node. That can be a nice hint about the queuing delay at the node 🚦 If the transit delay goes past 2^31-1 nanoseconds, the top bit 'O' gets set to flag overflow, and the value is set to 0x80000000 🚨
When this field is part of the data but the node filling it can't, the field position MUST be filled with the value 0xFFFFFFFF, which means not populated 🫶
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| O | transit delay |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4.4.2.6. Namespace-Specific Data
The "namespace-specific data" field is a 4-octet field that a node can use to add IOAM-Namespace-specific data 🎀 It's a "free-format" 4-octet bit field, and its meaning is only defined in the context of a specific IOAM-Namespace. So the namespace tells you what the bits mean 🧠
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| namespace-specific data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4.4.2.7. Queue Depth
The "queue depth" field is a 4-octet unsigned integer field 📦 It shows the current length of the egress interface queue, on the interface the packet is forwarded out of. The queue depth is counted as the current number of memory buffers the queue is using. A packet can eat up one or more memory buffers, depending on how big it is 🍰
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| queue depth |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4.4.2.8. Checksum Complement
The "Checksum Complement" field is a 4-octet node data field that holds the Checksum Complement value 🧮 It's useful when IOAM rides over encapsulations that use UDP transport, like VXLAN-GPE or Geneve. Without the Checksum Complement, nodes that add IOAM node data just update the UDP Checksum field, following what the encapsulation protocols recommend 💌
When the Checksum Complement is present, an IOAM encapsulating node or IOAM transit node that adds node data MUST pick one of these two options to keep the UDP Checksum value correct 🔒
- recompute the UDP Checksum field or
- use the Checksum Complement to make a checksum-neutral update in the UDP payload. The Checksum Complement is assigned a value that complements the rest of the node data fields added by the current node, so the existing UDP Checksum field stays correct ✨
IOAM decapsulating nodes MUST recompute the UDP Checksum field, since they can't tell whether previous hops changed the UDP Checksum field or the Checksum Complement field 🫠
Checksum Complement fields work in a similar way in [RFC7820] and [RFC7821] 🦋
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum Complement |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4.4.2.9. Hop_Lim and node_id Wide
The "Hop_Lim and node_id wide" field is an 8-octet field, laid out like this 👑
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id ~ |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ node_id (contd) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Hop_Lim is a 1-octet unsigned integer, and it's defined exactly as in Section 4.4.2.1 🔁
node_id is a 7-octet unsigned integer 🆔 It's a node identifier that uniquely identifies a node within the IOAM-Namespace and its associated IOAM-Domain. How node_ids get allocated, managed, and mapped is beyond the scope of this document 🙌
4.4.2.10. ingress_if_id and egress_if_id Wide
The "ingress_if_id and egress_if_id wide" field is an 8-octet field, and it looks like this 🌈
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingress_if_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| egress_if_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
ingress_if_id is a 4-octet unsigned integer that identifies the ingress interface the packet was received on 📥
egress_if_id is a 4-octet unsigned integer that identifies the egress interface the packet is forwarded out of 📤
4.4.2.11. Namespace-Specific Data Wide 🌈
okay so this one is a free-format 8-octet bit field 🫶 The node can stuff it with whatever IOAM-Namespace-specific data it wants. The meaning of those bits only makes sense inside a specific IOAM-Namespace, so it's giving "choose your own adventure" 📖
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| namespace-specific data ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ namespace-specific data (contd) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+4.4.2.12. Buffer Occupancy 🍓
The "buffer occupancy" field is a 4-octet unsigned integer, and it tells you how full the common buffer pool is for a set of queues 🪣 The units aren't fixed, since they're up to the implementation. So you read them in the context of an IOAM-Namespace and/or node identifier, if you're using one. The authors know that sometimes you need the units to match across the whole packet path, so they recommend using standard units like bytes. Iykyk 💅
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| buffer occupancy |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+4.4.2.13. Opaque State Snapshot 🔮
The "Opaque State Snapshot" field is variable-length, and it comes right after the fixed-length IOAM-Data-Fields defined above. It lets the network element stash arbitrary state in the node data field, with no predefined schema 🤯 The schema gets defined within an IOAM-Namespace, and the analyzer has to learn about it through some out-of-band mechanism. How that mechanism works is beyond the scope of this document, so don't ask me 🙈 A 24-bit "Schema ID" field, read within the IOAM-Namespace, says which schema is in use. The operator has to configure that schema on the network element.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length | Schema ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Opaque data |
~ ~
. .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Length: a 1-octet unsigned integer. It's the length, in multiples of 4 octets, of the Opaque data field that comes after Schema ID.
Schema ID: a 3-octet unsigned integer that identifies the schema of the Opaque data.
Opaque data: a variable-length field. It's read the way the schema identified by the Schema ID says to read it.
When this field is part of the data field, but the node filling it has no opaque state data to report, the Length MUST be set to 0 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 🫠 That's the "nothing to see here" combo, so please don't skip it.
4.4.3. Examples of IOAM Node Data 🧚
Okay, so the format of the entries in a packet's "node data list" array can change from packet to packet and deployment to deployment 🦋 Some deployments only care about recording node identifiers, while others want node identifiers and timestamps too. This section gives example entries for the "node data list" array, so you can see the vibes for each one ✨
0xD40000: If the IOAM Trace-Type is 0xD40000 (0b110101000000000000000000), then the format of node data is:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingress_if_id | egress_if_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp fraction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| namespace-specific data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+0xC00000: If the IOAM Trace-Type is 0xC00000 (0b110000000000000000000000), then the format is:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ingress_if_id | egress_if_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+0x900000: If the IOAM Trace-Type is 0x900000 (0b100100000000000000000000), then the format is:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp fraction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+0x840000: If the IOAM Trace-Type is 0x840000 (0b100001000000000000000000), then the format is:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| namespace-specific data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+0x940000: If the IOAM Trace-Type is 0x940000 (0b100101000000000000000000), then the format is:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp fraction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| namespace-specific data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+0x308002: If the IOAM Trace-Type is 0x308002 (0b001100001000000000000010), then the format is:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| timestamp fraction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Hop_Lim | node_id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| node_id(contd) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length | Schema ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Opaque data |
~ ~
. .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+4.5. IOAM Proof of Transit Option-Type 🔐
The IOAM Proof of Transit Option-Type is for path or service function chain [RFC7665] verification, which is the "prove that traffic actually went this way" use case 🛤️ How IOAM data for this Option-Type gets processed at encapsulating, decapsulating, and transit nodes is outside the scope of the document. Still, Proof of Transit approaches all need to uniquely identify a packet, and they all need to keep working on a bit of information that gets handed from node to node. So two pieces of information get added to the packet as IOAM-Data-Fields:
PktID: a unique identifier for the packet.
Cumulative: information that gets handed from node to node and updated by every node according to a verification algorithm.
The IOAM Proof of Transit Option-Type is made of a fixed-size "IOAM Proof of Transit Option header" plus "IOAM Proof of Transit Option data fields" 🧩
IOAM Proof of Transit Option header:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Namespace-ID |IOAM POT-Type | IOAM POT flags|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+IOAM Proof of Transit Option-Type IOAM-Data-Fields MUST be aligned by 4 octets. 📏 Yes, this one is strict, bestie 💀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| POT Option data field determined by IOAM POT-Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Namespace-ID: a 16-bit identifier of an IOAM-Namespace. The Namespace-ID value of 0x0000 is defined as the "Default-Namespace-ID" (see Section 4.3), and every node implementing IOAM MUST know it. 👑 For any other Namespace-ID value that doesn't match a Namespace-ID the node is set up to operate on, the node MUST NOT change the contents of the IOAM-Data-Fields.
IOAM POT-Type: an 8-bit identifier of a particular POT variant, which says what POT data is included. This document defines IOAM POT-Type 0:
0: POT data is a 16-octet field that carries data for POT procedures.
If a node receives an IOAM POT-Type value it doesn't understand, the node MUST NOT change, add to, or remove the contents of the IOAM-Data-Fields. 🙅♀️
IOAM POT flags: 8 bits. This document doesn't define any flags. Bits 0-7 are available for assignment (see Section 7.5). Bits that haven't been assigned MUST be set to zero upon transmission and be ignored upon receipt. 🌸
POT Option data: a variable-length field. Its type is determined by the IOAM POT-Type. 🫶
4.5.1. IOAM Proof of Transit Type 0 🦋
Okay bestie, this is the IOAM Proof of Transit Option for POT-Type 0 💅 Here's the layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Namespace-ID |IOAM POT-Type=0|R R R R R R R R|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
| PktID | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P
| PktID (contd) | O
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T
| Cumulative | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| Cumulative (contd) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+Namespace-ID: a 16-bit identifier of an IOAM-Namespace (see Section 4.3 above).
IOAM POT-Type: an 8-bit identifier of a particular POT variant that says which POT data is included (see Section 4.5 above). For this case, IOAM POT-Type is set to the value 0.
Bit 0-7: Undefined (see Section 4.5 above).
PktID: a 64-bit packet identifier.
Cumulative: a 64-bit Cumulative that gets updated at specific nodes by processing the per-packet PktID field and configured parameters.
Note: Larger or smaller sizes of "PktID" and "Cumulative" data are feasible and could be required for certain deployments, e.g., in case of space constraints in the encapsulation protocols used. Future documents could introduce different sizes of data for "Proof of Transit".
4.6. IOAM Edge-to-Edge Option-Type 🌈
The IOAM Edge-to-Edge Option-Type carries data that the IOAM encapsulating node adds and the IOAM decapsulating node interprets, so it's basically a message from one edge to the other 📬 IOAM transit nodes MAY process that data, but they MUST NOT modify it 🫶
The E2E Option-Type is made of two pieces: a fixed-size "IOAM Edge-to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data fields" 🎀 Here's the header:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Namespace-ID | IOAM E2E-Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Heads up: the IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST be aligned by 4 octets 🙈 Here's how the data field looks:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| E2E Option data field determined by IOAM-E2E-Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Namespace-ID: a 16-bit identifier of an IOAM-Namespace. The Namespace-ID value 0x0000 is defined as the "Default-Namespace-ID" (see Section 4.3) and MUST be known to all the nodes implementing IOAM. For any other Namespace-ID value that doesn't match any Namespace-ID the node is configured to operate on, the node MUST NOT change the contents of the IOAM-Data-Fields.
IOAM-E2E-Type: a 16-bit identifier that specifies which data types are used in the E2E Option data. The IOAM-E2E-Type value is a bit field, and the E2E Option data field elements get packed in the same order as the bits of the IOAM E2E-Type field, like this:
Bit 0: the most significant bit. When set, it indicates the presence of a 64-bit sequence number added to a specific "packet group", used to detect packet loss, packet reordering, or packet duplication within that group. The "packet group" is deployment dependent and defined at the IOAM encapsulating node, e.g., by n-tuple-based classification of packets. When this bit is set, "Bit 1" (for a 32-bit sequence number, see below) MUST be zero.
Bit 1: When set, it indicates the presence of a 32-bit sequence number added to a specific "packet group", used to detect packet loss, packet reordering, or packet duplication within that group. The "packet group" is deployment dependent and defined at the IOAM encapsulating node, e.g., by n-tuple-based classification of packets. When this bit is set, "Bit 0" (for a 64-bit sequence number, see above) MUST be zero.
Bit 2: When set, it indicates the presence of timestamp seconds, representing the time at which the packet entered the IOAM-Domain. Inside the IOAM encapsulating node, the time the timestamp gets retrieved can depend on the implementation. Some possibilities are 1) the time at which the packet was received by the node, 2) the time at which the packet was transmitted by the node, or 3) when a tunnel encapsulation is used, the point at which the packet is encapsulated into the tunnel. Each implementation has to document when the E2E timestamp that's going to be put in the packet gets retrieved. This 4-octet field has three possible formats, based on either PTP (see e.g., [RFC8877]), NTP [RFC5905], or POSIX [POSIX]. The three timestamp formats are specified in Section 5. In all three cases, the timestamp seconds field contains the 32 most significant bits of the timestamp format that is specified in Section 5. If a node isn't capable of populating this field, it assigns the value 0xFFFFFFFF. Note that this is a legitimate value that is valid for 1 second in approximately 136 years, so the analyzer has to correlate several packets or compare the timestamp value to its own time of day in order to detect the error indication.
Bit 3: When set, it indicates the presence of timestamp fraction, representing the time at which the packet entered the IOAM-Domain. This 4-octet field has three possible formats, based on either PTP (see e.g., [RFC8877]), NTP [RFC5905], or POSIX [POSIX]. The three timestamp formats are specified in Section 5. In all three cases, the timestamp fraction field contains the 32 least significant bits of the timestamp format that is specified in Section 5. If a node isn't capable of populating this field, it assigns the value 0xFFFFFFFF. Note that this is a legitimate value in the NTP format, valid for approximately 233 picoseconds in every second. If the NTP format is used, the analyzer has to correlate several packets in order to detect the error indication.
Bit 4-15: Undefined. An IOAM encapsulating node MUST set the value of these bits to zero upon transmission and ignore them upon receipt.
E2E Option data: a variable-length field. Its type is determined by the IOAM E2E-Type.
5. Timestamp Formats ✨
The IOAM-Data-Fields include a timestamp field that can be represented in one of three possible timestamp formats 🕰️ It's assumed that the management plane is responsible for determining which timestamp format gets used.
5.1. PTP Truncated Timestamp Format ⏱️
The Precision Time Protocol (PTP) uses an 80-bit timestamp format. The truncated timestamp format is a 64-bit field, made of the 64 least significant bits of the 80-bit PTP timestamp. The PTP truncated format is specified in Section 4.3 of [RFC8877], and the details are repeated below for the sake of completeness 💜
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Nanoseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Timestamp field format:
- Seconds: specifies the integer portion of the number of seconds since the PTP epoch.
- Size: 32 bits
- Units: seconds
- Nanoseconds: specifies the fractional portion of the number of seconds since the PTP epoch.
- Size: 32 bits
- Units: nanoseconds. The value of this field is in the range
0to(10^9)-1.
Epoch: the PTP epoch. For details, see e.g., [RFC8877].
Resolution: 1 nanosecond.
Wraparound: this time format wraps around every 2^32 seconds, which is roughly 136 years. The next wraparound will occur in the year 2106. 🔮
Synchronization Aspects: it's assumed that the nodes that run this protocol are synchronized among themselves. Nodes MAY be synchronized to a global reference time. Note that if PTP is used for synchronization, the timestamp MAY be derived from the PTP-synchronized clock, which allows the timestamp to be measured with respect to the clock of a PTP Grandmaster clock 🌸
5.2. NTP 64-Bit Timestamp Format 🧚
The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits long. This specification uses the NTP timestamp format that is specified in Section 4.2.1 of [RFC8877], and the details are repeated below for the sake of completeness 🎀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Fraction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Timestamp field format:
- Seconds: specifies the integer portion of the number of seconds since the NTP epoch.
- Size: 32 bits
- Units: seconds
- Fraction: specifies the fractional portion of the number of seconds since the NTP epoch.
- Size: 32 bits
- Units: the unit is
2^(-32)seconds, which is roughly equal to 233 picoseconds.
Epoch: the NTP epoch. For details, see [RFC5905].
Resolution: 2^(-32) seconds.
Wraparound: this time format wraps around every 2^32 seconds, which is roughly 136 years. The next wraparound will occur in the year 2036. 🌈
Synchronization Aspects: nodes that use this timestamp format will typically be synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY be derived from the NTP-synchronized clock, which allows the timestamp to be measured with respect to the clock of an NTP server ⭐
5.3. POSIX-Based Timestamp Format
okay so this one is the POSIX time format 💅🌸, and the spec for the timestamp format we use here is right below, bestie ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Timestamp field format 🎀
- Seconds 🕰️ specifies the integer portion of the number of seconds since the POSIX epoch.
- Size: 32 bits
- Units: seconds
- Microseconds ⏱️ specifies the fractional portion of the number of seconds since the POSIX epoch.
- Size: 32 bits
- Units: the unit is microseconds. The value of this field is in the range 0 to (10^6)-1.
Epoch 🌈: the POSIX epoch. For details, see [POSIX], Appendix A.4.16.
Resolution ✨: the resolution is 1 microsecond.
Wraparound 💀: this time format wraps around every 2^32 seconds, which is roughly 136 years. The next wraparound will occur in the year 2106.
Synchronization Aspects 🧚: it is assumed that nodes that use this timestamp format run the Linux operating system and hence use the POSIX time. In some cases, nodes MAY be synchronized to UTC using a synchronization mechanism that is outside the scope of this document, such as NTP [RFC5905]. Thus, the timestamp MAY be derived from the NTP-synchronized clock, allowing the timestamp to be measured with respect to the clock of an NTP server. 🫶
6. IOAM Data Export
IOAM nodes collect information for packets traversing a domain that supports IOAM. IOAM decapsulating nodes, as well as IOAM transit nodes, can choose to retrieve IOAM information from the packet, process the information further, and export the information using e.g., IP Flow Information Export (IPFIX). The mechanisms and associated data formats for exporting IOAM data are outside the scope of this document. 📤
A way to perform raw data export of IOAM data using IPFIX is discussed in [IPPM-IOAM-RAWEXPORT]. 🦋
7. IANA Considerations 👑
IANA has defined a registry group named "In Situ OAM (IOAM)". 🌸
This group includes the following registries:
- IOAM Option-Type
- IOAM Trace-Type
- IOAM Trace-Flags
- IOAM POT-Type
- IOAM POT-Flags
- IOAM E2E-Type
- IOAM Namespace-ID
The subsequent subsections detail the registries therein contained. 💖
7.1. IOAM Option-Type Registry
This registry defines 128 code points for the IOAM Option-Type field for identifying IOAM-Option-Types, as explained in Section 4. The following code points are defined in this document:
0: IOAM Pre-allocated Trace Option-Type
1: IOAM Incremental Trace Option-Type
2: IOAM POT Option-Type
3: IOAM E2E Option-TypeCode points 4-127 are available for assignment via the "IETF Review" process, as per [RFC8126]. 🎀
New registration requests MUST use the following template:
Name: name of the newly registered Option-Type
Code point: desired value of the requested code point
Description: brief description of the newly registered Option-Type
Reference: reference to the document that defines the new Option-Type🚨 The evaluation of a new registration request MUST also include checking whether the new IOAM-Option-Type includes an IOAM-Namespace field and that the IOAM-Namespace field is the first field in the newly defined header of the new Option-Type.
7.2. IOAM Trace-Type Registry
This registry defines code points for each bit in the 24-bit IOAM Trace-Type field for the Pre-allocated Trace Option-Type and Incremental Trace Option-Type defined in Section 4.4. Bits 0-11 are defined in this document in Paragraph 5 of Section 4.4.1:
Bit 0: hop_Lim and node_id in short format
Bit 1: ingress_if_id and egress_if_id in short format
Bit 2: timestamp seconds
Bit 3: timestamp fraction
Bit 4: transit delay
Bit 5: namespace-specific data in short format
Bit 6: queue depth
Bit 7: checksum complement
Bit 8: hop_Lim and node_id in wide format
Bit 9: ingress_if_id and egress_if_id in wide format
Bit 10: namespace-specific data in wide format
Bit 11: buffer occupancy
Bit 22: variable-length Opaque State Snapshot
Bit 23: reservedBits 12-21 are available for assignment via the "IETF Review" process, as per [RFC8126]. ✨
New registration requests MUST use the following template:
Bit: desired bit to be allocated in the 24-bit IOAM Trace Option-Type field for the Pre-allocated Trace Option-Type and Incremental Trace Option-Type
Description: brief description of the newly registered bit
Reference: reference to the document that defines the new bit7.3. IOAM Trace-Flags Registry
This registry defines code points for each bit in the 4-bit flags for the Pre-allocated Trace-Option and Incremental Trace-Option defined in Section 4.4. The meaning of Bit 0 (the most significant bit) for trace flags is defined in this document in Paragraph 3 of Section 4.4.1:
Bit 0: "Overflow" (O-bit)Bits 1-3 are available for assignment via the "IETF Review" process, as per [RFC8126]. 🌈
New registration requests MUST use the following template:
Bit: desired bit to be allocated in the 8-bit flags field of the Pre-allocated Trace Option-Type and Incremental Trace Option-Type
Description: brief description of the newly registered bit
Reference: reference to the document that defines the new bit7.4. IOAM POT-Type Registry
This registry defines 256 code points to define the IOAM POT-Type for the IOAM Proof of Transit Option (Section 4.5). The code point value 0 is defined in this document:
0: 16-Octet POT dataCode points 1-255 are available for assignment via the "IETF Review" process, as per [RFC8126]. 🧁
New registration requests MUST use the following template:
Name: name of the newly registered POT-Type
Code point: desired value of the requested code point
Description: brief description of the newly registered POT-Type
Reference: reference to the document that defines the new POT-Type7.5. IOAM POT-Flags Registry
This registry defines code points for each bit in the 8-bit flags for the IOAM POT Option-Type defined in Section 4.5.
Bits 0-7 are available for assignment via the "IETF Review" process, as per [RFC8126]. 💜
New registration requests MUST use the following template:
Bit: desired bit to be allocated in the 8-bit flags field of the IOAM POT Option-Type
Description: brief description of the newly registered bit
Reference: reference to the document that defines the new bit7.6. IOAM E2E-Type Registry
This registry defines code points for each bit in the 16-bit IOAM E2E-Type field for the IOAM E2E Option (Section 4.6). Bits 0-3 are defined in this document:
Bit 0: 64-bit sequence number
Bit 1: 32-bit sequence number
Bit 2: timestamp seconds
Bit 3: timestamp fractionBits 4-15 are available for assignment via the "IETF Review" process, as per [RFC8126]. 🦋
New registration requests MUST use the following template:
Bit: desired bit to be allocated in the 16-bit IOAM E2E-Type field
Description: brief description of the newly registered bit
Reference: reference to the document that defines the new bit7.7. IOAM Namespace-ID Registry
IANA has set up the "IOAM Namespace-ID" registry that contains 16-bit values and follows the template for requests shown below. The meaning of 0x0000 is defined in this document. IANA has reserved the values 0x0001 to 0x7FFF for private use (managed by operators), as specified in Section 4.3 of this document. Registry entries for the values 0x8000 to 0xFFFF are to be assigned via the "Expert Review" policy, as per [RFC8126]. 🔥
Upon receiving a new allocation request, a designated expert will perform the following:
- Review whether the request is complete, i.e., the registration template has been filled in. The expert will send incomplete requests back to the requester.
- Check whether the request is neither a duplicate of nor conflicting with either an already existing allocation or a pending allocation. In case of duplicates or conflicts, the expert will ask the requester to update the allocation request accordingly.
- Solicit feedback from relevant working groups and communities to ensure that the new allocation request has been properly reviewed and that rough consensus on the request exists. At a minimum, the expert will solicit feedback from the IPPM Working Group by posting the request to the ippm@ietf.org mailing list. The expert will allow for a 3-week review period on the mailing lists. If the feedback received from the relevant working groups and communities within the review period indicates rough consensus on the request, the expert will approve the request and ask IANA to allocate the new Namespace-ID. In case the expert senses a lack of consensus from the feedback received, the expert will ask the requester to engage with the corresponding working groups and communities to further review and refine the request.
It is intended that any allocation will be accompanied by a published RFC. In order to allow for the allocation of code points prior to the RFC being approved for publication, the designated expert can approve allocations once it seems clear that an RFC will be published. 🌸
0x0000: default namespace (known to all IOAM nodes)
0x0001 - 0x7FFF: reserved for private use
0x8000 - 0xFFFF: unassignedNew registration requests MUST use the following template:
Name: name of the newly registered Namespace-ID
Code point: desired value of the requested Namespace-ID
Description: brief description of the newly registered Namespace-ID
Reference: reference to the document that defines the new Namespace-ID
Status of the registration: Status can be either "permanent" or "provisional". Namespace-ID registrations following a successful expert review will have the status "provisional". Once the RFC that defines the new Namespace-ID is published, the status is changed to "permanent".8. Management and Deployment Considerations 🍓
This document defines the structure and use of IOAM-Data-Fields. This document does not define the encapsulation of IOAM-Data-Fields into different protocols. Management and deployment aspects for IOAM have to be considered within the context of the protocol IOAM-Data-Fields are encapsulated into and, as such, are out of scope for this document. For a discussion of IOAM deployment, please also refer to [IPPM-IOAM-DEPLOYMENT], which outlines a framework for IOAM deployment and provides best current practices. 🧚💜
9. Security Considerations 🔒✨
Okay bestie, this is the section where things get dramatic 😭 As [RFC7276] discusses, a successful attack on an OAM protocol in general, and on IOAM specifically, can stop failures or anomalies from being detected, or make it look like problems exist when they don't 💀. The main threats are compromising the integrity of IOAM data by maliciously changing IOAM options in transit, or injecting packets with maliciously generated IOAM options. Literally every node on the path of an IOAM-carrying packet can pull off this kind of attack 🙈
The Proof of Transit Option-Type (see Section 4.5) is used to verify the path of data packets, so you can prove that packets went through a defined set of nodes 🧚
Now the scary part 👀 If an attacker gets into several nodes in a network and can change the system software on those nodes, IOAM-Data-Fields could be misused and repurposed for something other than what this document specifies. One type of misuse is setting up a covert channel between network nodes 🫣
Confidentiality-wise, IOAM options aren't expected to contain user data, but they can still be used for network reconnaissance 🕵️♀️ Attackers could collect info about network paths, performance, queue states, buffer occupancy, and more. And if IOAM data leaks out of the IOAM-Domain, it could enable reconnaissance beyond that domain. One example of why an attacker would want this is to check how well an ongoing attack is working, like whether buffers and queues are overflowing 😬
IOAM can also be used to carry out Denial-of-Service (DoS) attacks or to amplify them 📉 For example, a malicious attacker can add an IOAM header to packets to eat up the resources of network devices taking part in IOAM, or of entities that receive, collect, or analyze the IOAM data. There's also the packet length attack, where an attacker pushes headers for IOAM-Option-Types into data packets, making them grow beyond the MTU size, which results in fragmentation or packet drops. And if POT is in use, an attacker could corrupt the POT data fields in a packet, causing a verification failure even if the packet actually followed the correct path 💔
IOAM options can include timestamps, so if network devices use synchronization protocols, any attack on the time protocol [RFC7384] can compromise the integrity of the timestamp-related data fields ⏰
At the management plane, attackers can set things up by misconfiguring, or by maliciously configuring, IOAM-enabled nodes in a way that enables other attacks. IOAM configuration should only be managed by authorized processes or users 🔐
IETF protocols need features that keep them secure 🛡️ IOAM-Data-Fields don't represent a protocol by themselves, but they add to the protocol they're encapsulated into. So any specification that defines how IOAM-Data-Fields are carried in an encapsulating protocol MUST provide a mechanism for cryptographic integrity protection of the IOAM-Data-Fields. That protection could come from a mechanism of the encapsulating protocol, or it could use the mechanisms specified in [IPPM-IOAM-DATA-INTEGRITY] ✨
This document doesn't define a specific IOAM encapsulation. Some IOAM encapsulation types can introduce their own specific security considerations, and a specification that defines an IOAM encapsulation is expected to address those encapsulation-specific considerations 🎀
IOAM is expected to be deployed in limited domains, which keeps the potential attack vectors inside that limited domain 🏰 A limited administrative domain gives the operator the means to select, monitor, and control access to all the network devices, which makes those devices trusted by the operator. To keep the threats above inside the current limited domain, the network operator is expected to enforce policies that stop IOAM traffic from leaking outside the IOAM-Domain, and stop IOAM data from outside the domain from being processed and used inside it 🧱
This document doesn't define the data contents of custom fields, like the "Opaque State Snapshot" and "namespace-specific data" IOAM-Data-Fields. Those custom data fields will have security considerations that match their defined data contents, and those need to be described wherever those formats are defined 📝
IOAM deployments that use both IOAM Trace Option-Types, meaning the Pre-allocated Trace Option-Type and the Incremental Trace Option-Type, can suffer from incomplete visibility if the information gathered through the two Trace Option-Types isn't correlated and aggregated properly 👀 If IOAM transit nodes use the IOAM-Data-Fields in the packet for further actions or insights, then transit nodes that only support one IOAM Trace Option-Type in a deployment using both have limited visibility. That means they can draw inappropriate conclusions or take wrong actions 🙃
The security considerations of a system that deploys IOAM, like any system, have to be reviewed for each deployment scenario, based on a system-specific threat analysis. That analysis can lead to specific security solutions that go beyond the scope of this document 🔍 In particular, if an IOAM deployment isn't confined to a single LAN but spans multiple interconnected sites (for example, using an overlay network), the inter-site links can be secured (for example, by IPsec) to avoid external threats 🌐
IOAM deployment considerations, including approaches to mitigate the threads and potential attacks discussed above, are outside the scope of this document. They're covered in [IPPM-IOAM-DEPLOYMENT] 💜
10. References 📚
10.1. Normative References
[POSIX] IEEE, "IEEE/Open Group 1003.1-2017 - IEEE Standard for Information Technology--Portable Operating System Interface (POSIX(TM)) Base Specifications, Issue 7", IEEE Std 1003.1-2017, January 2018, <https://standards.ieee.org/ieee/1003.1/7101/>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
[RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, "Network Time Protocol Version 4: Protocol and Algorithms Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, <https://www.rfc-editor.org/info/rfc5905>.
[RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/info/rfc8126>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
10.2. Informative References
[IPPM-IOAM-DATA-INTEGRITY] Brockners, F., Bhandari, S., Mizrahi, T., and J. Iurman, "Integrity of In-situ OAM Data Fields", Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-data-integrity-01, 2 March 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-data-integrity-01>.
[IPPM-IOAM-DEPLOYMENT] Brockners, F., Bhandari, S., Bernier, D., and T. Mizrahi, "In-situ OAM Deployment", Work in Progress, Internet-Draft, draft-ietf-ippm-ioam-deployment-01, 11 April 2022, <https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-deployment-01>.
[IPPM-IOAM-RAWEXPORT] Spiegel, M., Brockners, F., Bhandari, S., and R. Sivakolundu, "In-situ OAM raw data export with IPFIX", Work in Progress, Internet-Draft, draft-spiegel-ippm-ioam-rawexport-06, 21 February 2022, <https://datatracker.ietf.org/doc/html/draft-spiegel-ippm-ioam-rawexport-06>.
[IPV6-RECORD-ROUTE] Kitamura, H., "Record Route for IPv6 (RR6) Hop-by-Hop Option Extension", Work in Progress, Internet-Draft, draft-kitamura-ipv6-record-route-00, 17 November 2000, <https://datatracker.ietf.org/doc/html/draft-kitamura-ipv6-record-route-00>.
[NVO3-VXLAN-GPE] Maino, F., Ed., Kreeger, L., Ed., and U. Elzur, Ed., "Generic Protocol Extension for VXLAN (VXLAN-GPE)", Work in Progress, Internet-Draft, draft-ietf-nvo3-vxlan-gpe-12, 22 September 2021, <https://datatracker.ietf.org/doc/html/draft-ietf-nvo3-vxlan-gpe-12>.
[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. Weingarten, "An Overview of Operations, Administration, and Maintenance (OAM) Tools", RFC 7276, DOI 10.17487/RFC7276, June 2014, <https://www.rfc-editor.org/info/rfc7276>.
[RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, October 2014, <https://www.rfc-editor.org/info/rfc7384>.
[RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function Chaining (SFC) Architecture", RFC 7665, DOI 10.17487/RFC7665, October 2015, <https://www.rfc-editor.org/info/rfc7665>.
[RFC7799] Morton, A., "Active and Passive Metrics and Methods (with Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, May 2016, <https://www.rfc-editor.org/info/rfc7799>.
[RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way Active Measurement Protocol (OWAMP) and Two-Way Active Measurement Protocol (TWAMP)", RFC 7820, DOI 10.17487/RFC7820, March 2016, <https://www.rfc-editor.org/info/rfc7820>.
[RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 2016, <https://www.rfc-editor.org/info/rfc7821>.
[RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., "Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300, January 2018, <https://www.rfc-editor.org/info/rfc8300>.
[RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, <https://www.rfc-editor.org/info/rfc8799>.
[RFC8877] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for Defining Packet Timestamps", RFC 8877, DOI 10.17487/RFC8877, September 2020, <https://www.rfc-editor.org/info/rfc8877>.
[RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., "Geneve: Generic Network Virtualization Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November 2020, <https://www.rfc-editor.org/info/rfc8926>.
Okay so bestie, this is the final stretch of RFC 9197 and it's giving acknowledgements, contributors, and author addresses 🫶 Let's gooo 💅
Acknowledgements ✨
The authors wanted to thank Éric Vyncke, Nalini Elkins, Srihari Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew Yourtchenko, Aviv Kfir, Tianran Zhou, Zhenbin (Robin), and Greg Mirsky for their comments and advice 💖
This document leverages and builds on top of several concepts described in [IPV6-RECORD-ROUTE] 🦋 The authors would like to acknowledge the work done by the author Hiroshi Kitamura and everyone involved in writing it 🌸
The authors also want to gracefully acknowledge the useful review and insightful comments they got from Joe Clarke, Al Morton, Tom Herbert, Carlos J. Bernardos, Haoyu Song, Mickey Spiegel, Roman Danyliw, Benjamin Kaduk, Murray S. Kucherawy, Ian Swett, Martin Duke, Francesca Palombini, Lars Eggert, Alvaro Retana, Erik Kline, Robert Wilton, Zaheduzzaman Sarker, Dan Romascanu, and Barak Gafni 🎀✨
Contributors 🌈
This document was the collective effort of several authors, bestie 👑 The text and content were contributed by the editors and the coauthors listed below 💜
Carlos Pignataro Cisco Systems, Inc. Research Triangle Park 7200-11 Kit Creek Road NC 27709 United States of America Email: cpignata@cisco.com
Mickey Spiegel Barefoot Networks, an Intel company 101 Innovation Drive San Jose, CA 95134-1941 United States of America Email: mickey.spiegel@intel.com
Barak Gafni Nvidia Suite 100 350 Oakmead Parkway Sunnyvale, CA 94085 United States of America Email: gbarak@nvidia.com
Jennifer Lemon Broadcom 270 Innovation Drive San Jose, CA 95134 United States of America Email: jennifer.lemon@broadcom.com
Hannes Gredler RtBrick Inc. Email: hannes@rtbrick.com
John Leddy United States of America Email: john@leddy.net
Stephen Youell JP Morgan Chase 25 Bank Street London E14 5JP United Kingdom Email: stephen.youell@jpmorgan.com
David Mozes Email: mosesster@gmail.com
Petr Lapukhov Facebook 1 Hacker Way Menlo Park, CA 94025 United States of America Email: petr@fb.com
Remy Chang Barefoot Networks, an Intel company 101 Innovation Drive San Jose, CA 95134-1941 United States of America Email: remy.chang@intel.com
Daniel Bernier Bell Canada Canada Email: daniel.bernier@bell.ca
Authors' Addresses 🧚
Frank Brockners (editor) Cisco Systems, Inc. 3rd Floor Nordhein-Westfalen Hansaallee 249 40549 Duesseldorf Germany Email: fbrockne@cisco.com
Shwetha Bhandari (editor) Thoughtspot 3rd Floor Indiqube Orion Garden Layout HSR Layout 24th Main Rd Bangalore 560 102 Karnataka India Email: shwetha.bhandari@thoughtspot.com
Tal Mizrahi (editor) Huawei 8-2 Matam Haifa 3190501 Israel Email: tal.mizrahi.phd@gmail.com
That's the whole RFC, bestie! 🌈 IOAM got its whole glow-up, and you literally read the entire spec 😭💖
📦 where this page comes from
- ietf: rfc9197 2022-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms