RFC 9232
Network Telemetry Framework
RFC 9232: Network Telemetry Framework 🦋
Okay bestie, so this is RFC 9232 and it's called Network Telemetry Framework ✨ It's Informational (published for info, not a Standards Track spec 💅), with ISSN 2070-1721, dated May 2022. It was written by five people: H. Song at Futurewei, F. Qin at China Mobile, P. Martinez-Julia at NICT, L. Ciavaglia at Rakuten Mobile, and A. Wang at China Telecom 👑 The title block doesn't list anything it obsoletes or updates 🌸
Abstract 🌸
Network telemetry is how you get real insight into a network so management can be efficient and automated 🤖 It covers a bunch of techniques for generating remote data, collecting it, correlating it, and actually using it 💜 This doc lays out an architectural framework for network telemetry. It's motivated by the challenges that pop up when you run networks and by the requirements those challenges create 🔥 It also clarifies the terminology and sorts the modules and components of a network telemetry system from different angles. The framework and taxonomy give everyone a common ground for collecting related work and guide future technique and standard developments ✨
Status of This Memo 📜
Not gonna lie, this part is a bit formal, but here's the tea 🍓 This document is not an Internet Standards Track specification. It's just published for informational purposes.
It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community 🫶 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so see Section 2 of RFC 7841 for the details 🙈
For the current status, any errata, and how to give feedback, check https://www.rfc-editor.org/info/rfc9232 🦋
Copyright Notice ©️
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved 💖
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they're in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document 📖 Code Components pulled out of this document must include Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions. They're provided without warranty, as described in the Revised BSD License ⚖️
Table of Contents 📚
1. Introduction
1.1. Applicability Statement
1.2. Glossary
2. Background
2.1. Telemetry Data Coverage
2.2. Use Cases
2.3. Challenges
2.4. Network Telemetry
2.5. The Necessity of a Network Telemetry Framework
3. Network Telemetry Framework
3.1. Top-Level Modules
3.1.1. Management Plane Telemetry
3.1.2. Control Plane Telemetry
3.1.3. Forwarding Plane Telemetry
3.1.4. External Data Telemetry
3.2. Second-Level Function Components
3.3. Data Acquisition Mechanism and Type Abstraction
3.4. Mapping Existing Mechanisms into the Framework
4. Evolution of Network Telemetry Applications
5. Security Considerations
6. IANA Considerations
7. Informative References
Appendix A. A Survey on Existing Network Telemetry Techniques
A.1. Management Plane Telemetry
A.1.1. Push Extensions for NETCONF
A.1.2. gRPC Network Management Interface
A.2. Control Plane Telemetry
A.2.1. BGP Monitoring Protocol
A.3. Data Plane Telemetry
A.3.1. Alternate-Marking (AM) Technology
A.3.2. Dynamic Network Probe
A.3.3. IP Flow Information Export (IPFIX) Protocol
A.3.4. In Situ OAM
A.3.5. Postcard-Based Telemetry
A.3.6. Existing OAM for Specific Data Planes
A.4. External Data and Event Telemetry
A.4.1. Sources of External Events
A.4.2. Connectors and Interfaces
Acknowledgments
Contributors
Authors' Addresses1. Introduction ✨
Okay so network visibility is the ability of management tools to see the state and behavior of a network, and it's essential for running one successfully 👀 Network telemetry is all about network data that can do three things. First, it helps give insight into the current state of the network, including the devices and the forwarding, control, and management planes 🧚 Second, it can be generated and obtained through lots of techniques, including but not limited to network instrumentation and measurements 📊 Third, it can be processed for everything from service assurance to network security, using a wide variety of data analytics 🔐
In this document, "network telemetry" refers to both the data itself (that's "Network Telemetry Data") and the techniques and processes used to generate, export, collect, and consume that data for potentially automated management applications 💅 It goes beyond the classic network Operations, Administration, and Management (OAM) techniques and is expected to bring better flexibility, scalability, accuracy, coverage, and performance 🚀
Here's the lowkey problem though: the term "network telemetry" doesn't have an unambiguous definition 😭 Its scope and coverage cause confusion and misunderstandings. So it's really helpful to clarify the concept and give a clear architectural framework. That way we can articulate the field and better align the related techniques and standard works 🎀
To make that happen, we first talk about some key characteristics that clearly set network telemetry apart from conventional network OAM. We also show that some conventional OAM technologies can be treated as a subset of network telemetry technologies 🦋 Then we lay out an architectural framework with four modules. Each module is tied to a different category of telemetry data and its own procedures 🌈 All of the modules are structured the same way inside. They include components that let the operator configure data sources (what data to generate and how to make it available to client applications), components that instrument the underlying data sources, and components that do the actual rendering, encoding, and exporting of the generated data 💖 We show how the framework can benefit current and future network operations. Based on the distinction between modules and function components, we can map existing and emerging techniques and protocols into the framework. It can also make designing, maintaining, and understanding a network telemetry system way easier 🧚 Finally, we outline the evolution stages of a network telemetry system and discuss the potential security concerns 🔥
The purpose of the framework and taxonomy is to give everyone a common ground for collecting related work and to guide future technique and standard developments ✨ To the best of our knowledge, this is the first such effort for network telemetry in industry standards organizations 👑 Also, this document does not define specific technologies 🌸
1.1. Applicability Statement 🔒
Large-scale network data collection is a major threat to user privacy, and it may be indistinguishable from pervasive monitoring [RFC7258] 😭 The network telemetry framework in this document must not be applied to generating, exporting, collecting, analyzing, or retaining individual user data. That includes any data that can identify end users or characterize their behavior without consent 🙅♀️ Based on that principle, the framework isn't applicable to networks whose endpoints represent individual users, such as general-purpose access networks 💜
1.2. Glossary
okay bestie, before we go any further we need to get the vocab straight 📚🎀 there's an intended difference between network telemetry and OAM, but ngl it's not a hard line. network telemetry is basically an extension of OAM 🦋 it covers all the existing OAM protocols but puts way more focus on the newer, emerging stuff about all aspects of network data, from acquisition all the way to consumption 💅
- AI: Artificial Intelligence. in the network world, AI means machine-learning-based tech for automated network operation and other tasks 🤖
- AM: Alternate Marking. a flow performance measurement method, as specified in [RFC8321] 📈
- BMP: BGP Monitoring Protocol. specified in [RFC7854] 👀
- DPI: Deep Packet Inspection. the techniques that look at packets beyond the L3/L4 headers 🔍
- gNMI: gRPC Network Management Interface. a network management protocol from the OpenConfig Operator Working Group, mainly contributed by Google. see [gnmi] for details 🌈
- GPB: Google Protocol Buffer. an extensible mechanism for serializing structured data. see [gpb] for details 🧚
- gRPC: gRPC Remote Procedure Call. an open-source, high-performance RPC framework that gNMI is based on. see [grpc] for details ⭐
- IPFIX: IP Flow Information Export Protocol. specified in [RFC7011] 🌸
- IOAM: In situ OAM [RFC9197]. a data plane on-path telemetry technique 🛰️
- JSON: JavaScript Object Notation. an open standard file format and data interchange format that uses human-readable text to store and transmit data objects, as specified in [RFC8259] 💜
- MIB: Management Information Base. a database used for managing the entities in a network 🗂️
- NETCONF: Network Configuration Protocol. specified in [RFC6241] 🛠️
- NetFlow: a Cisco protocol used for flow record collecting, as described in [RFC3954] 🍓
- Network Telemetry: the process and instrumentation for acquiring and using network data remotely, for network monitoring and operation. it's a general term for a big set of network visibility techniques and protocols, covering things like data generation, collection, correlation, and consumption. network telemetry deals with today's network operation issues and helps us smoothly evolve toward future intent-driven autonomous networks 🦋✨
- NMS: Network Management System. refers to applications that let network administrators manage a network 👩💻
- OAM: Operations, Administration, and Maintenance. a group of network management functions that provide network fault indication, fault localization, performance information, and data and diagnosis functions. most conventional network monitoring techniques and protocols belong to network OAM 🩺
- PBT: Postcard-Based Telemetry. a data plane on-path telemetry technique. a representative technique is described in [IPPM-IOAM-DIRECT-EXPORT] 📮
- RESTCONF: an HTTP-based protocol that gives you a programmatic interface for accessing data defined in YANG, using the datastore concepts defined in NETCONF, as specified in [RFC8040] 🌐
- SMIv2: Structure of Management Information Version 2. defines MIB objects, as specified in [RFC2578] 📐
- SNMP: Simple Network Management Protocol. versions 1, 2, and 3 are specified in [RFC1157], [RFC3416], and [RFC3411], respectively 🕰️
- XML: Extensible Markup Language. a markup language for data encoding that is both human readable and machine readable, as specified by W3C [W3C.REC-xml-20081126] 💅
- YANG: a data modeling language for defining data sent over network management protocols such as NETCONF and RESTCONF. YANG is defined in [RFC6020] and [RFC7950] 🎀
- YANG ECA: a YANG model for Event-Condition-Action policies, as defined in [NETMOD-ECA-POLICY] 🫶
- YANG-Push: a mechanism that lets subscriber applications request a stream of updates from a YANG datastore on a network device. details are specified in [RFC8639] and [RFC8641] 🔥
2. Background
okay so "big data" is the term for those super huge data sets that can be analyzed computationally to reveal patterns, trends, and associations 📊💖 and networks are undoubtedly a source of big data, because of their scale and the sheer volume of traffic they forward 🌊 when a network's endpoints don't represent individual users (think industrial, data-center, and infrastructure contexts), network operations can often benefit from large-scale data collection without breaching user privacy 🔐
today you can access advanced big data analytics through a plethora of commercial and open-source platforms (e.g., Apache Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine learning) 🧚✨ thanks to the advance of computing and storage tech, network big data analytics give operators a chance to gain network insights and move toward network autonomy 🦋 some operators are starting to explore applying Artificial Intelligence (AI) to make sense of network data 🤖 software tools can use that data to detect and react to network faults, anomalies, and policy violations, and to predict future events 👀 in turn, network policy updates for planning, intrusion prevention, optimization, and self-healing may be applied 🩹
it's conceivable that an autonomic network [RFC7575] is the logical next step after Software-Defined Networking (SDN) 🌈 the goal is to reduce (or even eliminate) human labor, make more efficient use of network resources, and provide better services that line up more closely with customer requirements 💅 the IETF ANIMA Working Group is dedicated to developing and maintaining protocols and procedures for automated network management and control of professionally managed networks 🎀 the related technique of Intent-Based Networking (IBN) [NMRG-IBN-CONCEPTS-DEFINITIONS] requires network visibility and telemetry data so the network can be confirmed to be behaving as intended 🫶
but here's the tea 🍵 data processing has gotten way better and apps need more data to work well, yet networks are lagging behind at turning raw network data into useful, actionable information in efficient ways 😭 the bottleneck is shifting from data consumption to data supply 🚧 both the number of network nodes and the traffic bandwidth keep climbing fast 📈 network configuration and policy changes now happen in smaller time slots than before ⏱️ more subtle events and fine-grained data across all network planes need to be captured and exported in real time ⚡ in a nutshell, getting enough high-quality data out of the network in a way that's efficient, timely, and flexible is a challenge 💀 so we need to survey the existing technologies and protocols and find any potential gaps 🔍
in the rest of this section, we first clarify the scope of network data (i.e., telemetry data) relevant to this document 💖 then we talk about several key use cases for network operations today and in the future 🌸 next, we show why current network OAM techniques and protocols aren't enough for these use cases 🙈 the discussion underlines the need for new methods, techniques, and protocols, plus extensions of existing ones, which we group under the umbrella term "Network Telemetry" 👑
2.1. Telemetry Data Coverage
any information that can be extracted from networks (including the data plane, control plane, and management plane) and used to gain visibility or as a basis for actions counts as telemetry data 🦋 that includes statistics, event records and logs, snapshots of state, configuration data, and so on ✨ it also covers the outputs of any active and passive measurements [RFC7799] 📏 in some cases, raw data gets processed in the network before it's sent to a data consumer, and that processed data also counts as telemetry data 🧠 the value of telemetry data varies 💅 in some cases, if the cost is acceptable, less but higher-quality data is preferred over a ton of low-quality data 💎 a classification of telemetry data is provided in Section 3 📚
to preserve the privacy of end users, no user packet content should be collected 🙈 specifically, the data objects generated, exported, and collected by a network telemetry application should not include any packet payload from traffic associated with end-user systems 🔐 this one is non-negotiable, bestie 💜
2.2. Use Cases 💅✨
okay so this list is the essential stuff network ops is literally built on 😭 it's not exhaustive (there's more, babe), but it's enough to show why networks need the four big-data vibes: velocity, variety, volume, and veracity 🔥. these are the attributes of big data, and they show up in networks everywhere 🌐
- Security: Network intrusion detection and prevention systems need to watch network traffic and activities and act when something looks off 🚨. attacks keep getting sneakier and breaches keep getting more expensive, so new tools and techniques have to be built that rely on wider and deeper visibility into networks 👀. the ultimate goal is security with no, or only minimal, human intervention, and without messing up legitimate traffic flows 💖
- Policy and Intent Compliance: Network policies are the rules that limit who gets network access, give some services priority, or force specific treatment on traffic 📜. for example, a service function chain is a policy that makes selected flows pass through a set of ordered network functions 🧚. intent, as defined in [NMRG-IBN-CONCEPTS-DEFINITIONS], is a set of operational goals that a network should meet and outcomes it's supposed to deliver. it's declarative, meaning it says what without saying how to achieve or implement it 🎀. an intent needs a complex translation and mapping process before it can be applied to networks, so it's not a one-click thing 💀. while a policy or intent is enforced, its compliance needs to be verified and monitored continuously, using visibility provided through network telemetry data 📡. any violation must be reported immediately. this alerts the network administrator to the policy or intent violation, and it can lead to updates in how the policy or intent is applied so it stays in force 🫶
- SLA Compliance: A Service Level Agreement (SLA) is a service contract between a service provider and a client. it includes the metrics used to measure the service, plus the remedy or penalty procedures for when the service level misses the agreement 📝. users need to check that they're actually getting the service they were promised 🙈, and network operators need to figure out how to deliver services that meet the SLA, based on real-time network telemetry data, including data from network measurements 📊
- Root Cause Analysis: lots of network failures are the result of a chain of events 🔗. troubleshooting and recovery need a quick way to find the root cause of any observable issue 🔍. but the root cause isn't always easy to spot, especially when the failure is sporadic and there's a flood of event messages, some related to the same cause and some not 😵💫. machine learning can help with root cause analysis, but it's still up to the network to sense and provide the relevant diagnostic data. that data gets actively fed into, or passively retrieved by, the root cause analysis applications 🧠
- Network Optimization: this covers all short-term and long-term network optimization techniques, like load balancing, Traffic Engineering (TE), and network planning 🌸. operators want to optimize how much of their network gets used and differentiate services for a better Return on Investment (ROI) or lower Capital Expenditure (CAPEX) 💰. the first step is knowing real-time network conditions before applying policies for traffic manipulation 👑. sometimes microbursts need to be detected in a very short time frame, so fine-grained traffic control can be applied to avoid network congestion ⚡. long-term planning of network capacity and topology needs analysis of real-world network telemetry data collected over long periods of time 🗓️
- Event Tracking and Prediction: visibility into traffic paths and performance is critical for services and applications that rely on a healthy network 🌈. lots of related network events are of interest to operators. for example, operators want to learn where and why packets are dropped for an application flow 🕵️♀️. they also want a heads-up about issues in advance, so proactive actions can be taken to avoid catastrophic consequences 🚀
2.3. Challenges 🥺💜
for a long time, network operators have relied on SNMP [RFC3416], Command-Line Interface (CLI), or Syslog [RFC5424] to monitor the network 🖥️. some other OAM techniques described in [RFC7276] are also used to make troubleshooting easier 🛠️. but these conventional techniques just aren't enough for the use cases above, and here's why 👇
- Most use cases need to monitor the network continuously and refine data collection dynamically, in real time ⏱️. poll-based, low-frequency data collection is a bad fit for these apps 😬. subscription-based streaming data, pushed directly from the data source (like the forwarding chip), is preferred because it gives enough data quantity and precision at scale 🦋
- Comprehensive data is needed, from packet processing engines to traffic managers, line cards to main control boards, user flows to control protocol packets, device configurations to operations, and physical layers to application layers 🌍. conventional OAM only covers a narrow slice of this (SNMP, for example, only handles data from the Management Information Base (MIB)) 🥱. classical network devices can't provide all the necessary probes, so more open and programmable network devices are needed 🔧
- lots of application scenarios need to correlate network-wide data from multiple sources. that means distributed network devices, different components of one network device, or different network planes 🧩. a piecemeal solution often lacks the ability to consolidate data from multiple sources 😩. the composition of a complete solution, partly proposed by the Autonomic Resource Control Architecture (ARCA) [NMRG-ANTICIPATED-ADAPTATION], will be empowered and guided by a comprehensive framework 👑
- some conventional OAM techniques (like CLI and Syslog) lack a formal data model 📄. unstructured data makes tool automation and application extensibility harder 🙃. standardized data models are essential for programmable networks 💅
- although some conventional OAM techniques support data push (like SNMP Trap [RFC2981][RFC3877], Syslog, and sFlow [RFC3176]), the pushed data is limited to predefined management plane warnings (like SNMP Trap) or sampled user packets (like sFlow) 🫠. network operators need data with arbitrary source, granularity, and precision, and that's beyond what existing techniques can do 🚫
- conventional passive measurement techniques can either eat up excessive network resources and produce excessive redundant data, or lead to inaccurate results 😤. on the other hand, conventional active measurement techniques can interfere with user traffic, and their results are indirect 🙅♀️. techniques that can collect direct and on-demand data from user traffic are more favorable 💖
these challenges were addressed by newer standards and techniques, like IPFIX/Netflow, Packet Sampling (PSAMP), IOAM, and YANG-Push 🌟, and more are emerging 🌱. these standards and techniques need to be recognized and accommodated in a new framework 🎀
2.4. Network Telemetry ✨
Okay so, network telemetry has basically become the mainstream term for the whole vibe of collecting and consuming network data 📊💖. Techniques and protocols like IPFIX [RFC7011] and gRPC [grpc] are already widely deployed and literally everywhere 🎀. Network telemetry lets separate entities pull data from network devices, so that data can be visualized and analyzed for network monitoring and operation 👀. It covers conventional network OAM and has a much wider scope 🌸. It's expected that network telemetry can give autonomous networks the insight they need and fix the shortcomings of conventional OAM techniques 🦋.
Network telemetry usually assumes the data consumers are machines, not human operators 🤖. This means network telemetry can directly trigger automated network operation 💅. Some conventional OAM tools were designed to help human operators monitor and diagnose networks and guide manual operations instead, so the two approaches are really different 🙈. That difference leads to very different techniques, bestie ✨.
Telemetry techniques are still evolving and keep changing, but a few characteristics are widely accepted 🌈. Network telemetry is meant to be an umbrella term over a wide spectrum of techniques, so none of these characteristics is expected to be held by every specific technique 💜.
- Push and Streaming: Instead of polling data from network devices, telemetry collectors subscribe to streaming data that's pushed from data sources in network devices 🌊.
- Volume and Velocity: Telemetry data is meant to be consumed by machines rather than human beings, so the data volume can be huge 😭. Processing is optimized for the needs of real-time automation ⚡.
- Normalization and Unification: Telemetry aims to address overall network automation needs 🧠. Efforts are made to normalize the data representation and unify the protocols, which simplifies data analysis and allows integrated analysis across heterogeneous devices and data sources across a network 🫶.
- Model-Based: Telemetry data is modeled in advance, which lets applications configure and consume data with ease 🎀.
- Data Fusion: The data for a single application can come from multiple data sources (for example, cross-domain, cross-device, and cross-layer) that are based on a common name/ID, and they need to be correlated to take effect 🧚.
- Dynamic and Interactive: Network telemetry is meant to be used in a closed control loop for network automation, so it needs to run continuously and adapt to dynamic and interactive queries from the network operation controller 💅✨.
In addition, an ideal network telemetry solution may also have the following features or properties 👑:
- In-Network Customization: The data that gets generated can be customized in the network at runtime to cater to the specific needs of applications 💖. This needs support from a programmable data plane, which allows probes with custom functions to be deployed at flexible locations 🦋.
- In-Network Data Aggregation and Correlation: Network devices and aggregation points can figure out which events and what data need to be stored, reported, or discarded 🍓. This reduces the load on central collection and processing points while still making sure the right information is ready to be processed in a timely way ⭐.
- In-Network Processing: Sometimes it isn't necessary or feasible to gather all the information to a central point to be processed and acted upon 🤷♀️. Data processing can be done in the network instead, so reactive actions can be taken locally 🔥.
- Direct Data Plane Export: Data originating from data plane forwarding chips can be directly exported to the data consumer for efficiency 🚀. This matters most when the data bandwidth is large and real-time processing is required 😤.
- In-Band Data Collection: In addition to passive and active data collection approaches, this new hybrid approach allows data to be collected directly for any target flow on its entire forwarding path [OPSAWG-IFIT-FRAMEWORK] 🌸✨.
It's worth noting that a network telemetry system should not be intrusive to normal network operations 🙈. It should avoid the pitfall of the "observer effect", meaning it should not change the network behavior and affect forwarding performance 💀. Also, high-volume telemetry traffic may cause network congestion unless proper isolation or traffic engineering techniques are in place 😬. Otherwise, congestion control mechanisms can make sure telemetry traffic backs off if it exceeds the network capacity 🛑. [RFC8084] and [RFC8085] are relevant Best Current Practices (BCPs) in this space 📚💜.
Although a system for network telemetry often involves a remote data collecting and consuming entity, there are no inherent assumptions about how a system should be architected 🧩. A network architecture with a centralized controller (for example, SDN) seems like a natural fit for network telemetry, but telemetry can work in distributed fashions as well 🌈. For example, telemetry data producers and consumers can have a peer-to-peer relationship, in which a network node can be the direct consumer of telemetry data from other nodes 🫶.
2.5. The Necessity of a Network Telemetry Framework 🧠
Network data analytics (for example, machine learning) is applied to network operation automation, and it relies on abundant and coherent data from networks 🤖. Data acquisition that's limited to a single source and static in nature won't in many cases be enough to meet an application's telemetry data needs 😩. So multiple data sources, involving a variety of techniques and standards, will need to be integrated 🧩. It's desirable to have a framework that classifies and organizes different telemetry data sources and types, defines the components of a network telemetry system and how they interact, and helps coordinate and integrate multiple telemetry approaches across layers 💅. This allows flexible combinations of data for different applications while normalizing and simplifying interfaces ✨. Such a framework would benefit the development of network operation applications for the following reasons 👑:
- Future networks, autonomous or otherwise, depend on holistic and comprehensive network visibility 🔭. Use cases and applications work better when they're supported uniformly and coherently, using an integrated, converged mechanism and common telemetry data representations wherever feasible 💖. So the protocols and mechanisms should be consolidated into a minimum yet comprehensive set 🎀. A telemetry framework can help normalize technique developments 🦋.
- Network visibility has multiple viewpoints 👀. For example, the device viewpoint takes the network infrastructure as the monitoring object, from which the network topology and device status can be acquired 🖥️. The traffic viewpoint takes the flows or packets as the monitoring object, from which the traffic quality and path can be acquired 🌊. An application may need to switch its viewpoint during operation 🔄. It may also need to correlate a service and its impact on user experience (UE) to get the comprehensive picture 💜.
- Applications require network telemetry to be elastic, so they can make efficient use of network resources and reduce the impact of telemetry-related processing on network performance 🌸. For example, routine network monitoring should cover the entire network with a low data sampling rate 😌. Only when issues arise or critical trends emerge should telemetry data sources be modified and telemetry data rates be boosted as needed 🚨.
- Efficient data aggregation is critical for applications, since it reduces the overall quantity of data and improves the accuracy of analysis 📉✨.
A telemetry framework collects all the telemetry-related work from different sources and working groups within the IETF 🫶. This makes it possible to assemble a comprehensive network telemetry system and avoid repetitious or redundant work 🙌. The framework should cover the concepts and components from the standardization perspective 📜. This document describes the modules that make up a network telemetry framework and breaks the telemetry system down into a set of distinct components that existing and future work can easily map to 🧚💖.
3. Network Telemetry Framework 🌈
The top-level network telemetry framework splits network telemetry into four modules, based on the telemetry data object source, and shows how they relate to each other 🗺️. Once network operation applications acquire data from these modules, they can apply data analytics and take actions 🔥. At the next level, the framework breaks each module down into separate components 🧩. Each of these modules follows the same underlying structure: one component is dedicated to configuring data subscriptions and data sources, a second is dedicated to encoding and exporting data, and a third instruments the generation of telemetry related to the underlying resources 💅. Throughout the framework, the same set of abstract data-acquiring mechanisms and data types (Section 3.3) is applied 🎀. This two-level architecture with its uniform data abstraction helps pinpoint exactly where a protocol or technique sits in a network telemetry system, or breaks a network telemetry system into manageable parts 💜✨.
3.1. Top-Level Modules 🎀
okay so telemetry can show up on the forwarding plane, the control plane, and the management plane, and it can also come from stuff outside the network entirely 👀 so we sort network telemetry into four modules: management plane, control plane, forwarding plane, and external data and event telemetry. each one has its own interface for talking to network operation applications. it's giving organized 💅
+------------------------------+
| |
| Network Operation |<-------+
| Applications | |
| | |
+------------------------------+ |
^ ^ ^ |
| | | |
V V | V
+--------------+-----------|---+ +-----------+
| | Control | | | |
| | Plane | | | External |
| <---> | | | Data and |
| | Telemetry | | | Event |
| Management | ^ V | | Telemetry |
| Plane +-------|-------+ | |
| Telemetry | V | +-----------+
| | Forwarding |
| | Plane |
| <---> |
| | Telemetry |
| | |
+--------------+---------------+Figure 1: Modules in Layer Category of the Network Telemetry Framework
the reason we split it up like this is that each module has different telemetry data objects, which means different data sources and different export locations 🧠 those differences totally change what in-network data programming and processing can do, how the data gets encoded and transported, and how much bandwidth and latency it needs. data can be sent straight out, or it can be proxied through the control and management planes. both options have pros and cons, bestie ⚖️
ngl the network controller can also be the source of its own telemetry data 🎮 this could be data that's unique to it, or data it derives from what the network elements collected. some of the principles and taxonomy for control plane and management plane telemetry could also apply to the controller when it has to give telemetry data to network operation applications running outside. but this document is focused on network elements telemetry, so the controller details are out of scope 🙈
we sum up the big differences between the four modules in Table 1. they get compared from six angles:
- Data Object
- Data Export Location
- Data Model
- Data Encoding
- Telemetry Application Protocol
- Data Transport Method
Data Object is the target and source of each module ✨ since the data source changes, the best place to export the data also changes. for example, forwarding plane data mostly comes out of the forwarding Application-Specific Integrated Circuits (ASICs), while control plane data mostly comes from the protocol daemons running on the control CPU(s). for convenience and efficiency, it's preferred to export data off the device from somewhere near the source. and since the export locations have different capabilities, you end up picking different data models, encodings, and transport methods to balance performance and cost. the forwarding chip has high throughput but can't handle complex data or keep much state, while the main control CPU can do complex data and state processing but has limited bandwidth for high-throughput data. so the right telemetry protocol for each module can be different 🦋 the techniques in the table blocks are just some representative examples to show how diverse these modules are. they reflect the de facto state of the art and are by no means exhaustive (e.g., IPFIX can also be implemented over TCP and SCTP, but that is not recommended for the forwarding plane). the big takeaway is that you can't expect one universal protocol to cover all the network telemetry requirements 💜
+=============+===============+==========+==========+===============+
|Module |Management |Control |Forwarding|External Data |
| |Plane |Plane |Plane | |
+=============+===============+==========+==========+===============+
|Object |configuration |control |flow and |terminal, |
| |and operation |protocol |packet |social, and |
| |state |and |QoS, |environmental |
| | |signaling,|traffic | |
| | |RIB |stat., | |
| | | |buffer and| |
| | | |queue | |
| | | |stat., | |
| | | |FIB, | |
| | | |Access | |
| | | |Control | |
| | | |List (ACL)| |
+-------------+---------------+----------+----------+---------------+
|Export |main control |main |forwarding|various |
|Location |CPU |control |chip or | |
| | |CPU, |linecard | |
| | |linecard |CPU; main | |
| | |CPU, or |control | |
| | |forwarding|CPU | |
| | |chip |unlikely | |
+-------------+---------------+----------+----------+---------------+
|Data Model |YANG, MIB, |YANG, |YANG, |YANG, custom |
| |syslog |custom |custom | |
+-------------+---------------+----------+----------+---------------+
|Data Encoding|GPB, JSON, XML |GPB, JSON,|plain text|GPB, JSON, XML,|
| | |XML, plain| |plain text |
| | |text | | |
+-------------+---------------+----------+----------+---------------+
|Application |gRPC, NETCONF, |gRPC, |IPFIX, |gRPC |
|Protocol |RESTCONF |NETCONF, |traffic | |
| | |IPFIX, |mirroring,| |
| | |traffic |gRPC, | |
| | |mirroring |NETFLOW | |
+-------------+---------------+----------+----------+---------------+
|Data |HTTP(S), TCP |HTTP(S), |UDP |HTTP(S), TCP, |
|Transport | |TCP, UDP | |UDP |
+-------------+---------------+----------+----------+---------------+Table 1: Comparison of Data Object Modules
the interaction with the apps that consume telemetry data can be indirect, and some in-device data transfer is possible 🔄 for example, in management plane telemetry, the management plane needs to grab data from the data plane. some operational states can only be figured out from data plane sources, like interface status and statistics. another example: getting control plane telemetry data might mean you need to access the Forwarding Information Base (FIB) of the data plane. the plot thickens 🕵️♀️
and an app can involve more than one plane and talk to several planes at the same time 🤹♀️ like, an SLA compliance app might need both data plane telemetry and control plane telemetry. iconic multitasking
the requirements and challenges for each module are summarized below. note that requirements can apply across all telemetry modules, but we're highlighting the ones that really stand out for each particular plane 🌸
3.1.1. Management Plane Telemetry 🧚
the management plane of network elements talks to the Network Management System (NMS) and provides things like performance data, network logging data, network warning and defects data, and network statistics and state data. it includes lots of protocols, including the classic SNMP and syslog. regardless of the protocol, management plane telemetry has to deal with these requirements:
- Convenient Data Subscription: An application should have the freedom to choose which data is exported (see Section 3.3) and the means and frequency of how that data is exported (e.g., on-change or periodic subscription).
- Structured Data: For automatic network operation, machines will replace humans for network data comprehension. Data modeling languages, such as YANG, can efficiently describe structured data and normalize data encoding and transformation.
- High-Speed Data Transport: In order to keep up with the velocity of information, a data source needs to be able to send large amounts of data at high frequency. Compact encoding formats or data compression schemes are needed to reduce the quantity of data and improve the data transport efficiency. The subscription mode, by replacing the query mode, reduces the interactions between clients and servers and helps to improve the data source's efficiency.
- Network Congestion Avoidance: The application must protect the network from congestion with congestion control mechanisms or, at minimum, with circuit breakers. [RFC8084] and [RFC8085] provide some solutions in this space. 🚦
Okay bestie, let's get into it 🎀
3.1.2. Control Plane Telemetry ✨
Control plane telemetry is all about keeping an eye on the health of the different network control protocols at every layer of the protocol stack 👀 Knowing how these protocols are actually doing is super useful for detecting, localizing, and even predicting network issues, plus for network optimization, all in real time and with fine granularity 💅 It's giving main character energy. Some particular challenges and issues it faces are:
- How to correlate the End-to-End (E2E) Key Performance Indicators (KPIs) to a specific layer's KPIs. For example, IPTV users may describe their UE by the video smoothness and definition 📺 So when there's an unusually poor UE KPI or a service disconnection, it is non-trivial to delimit and pinpoint the issue in the responsible protocol layer (e.g., the transport layer or the network layer), the responsible protocol (e.g., IS-IS or BGP at the network layer), and finally the responsible device(s) with specific reasons. Lowkey a whole mystery novel 🕵️♀️
- Conventional OAM-based approaches for control plane KPI measurement, which include Ping (L3), Traceroute (L3), Y.1731 [y1731] (L2), and so on. One common issue behind these methods is that they only measure the KPIs instead of reflecting the actual running status of these protocols, making them less effective or efficient for control plane troubleshooting and network optimization 🙈 Not them only checking the numbers and missing the vibe!
- How more research is needed for the BGP monitoring protocol (BMP). BMP is an example of the control plane telemetry; it is currently used for monitoring BGP routes and enables rich applications, such as BGP peer analysis, Autonomous System (AS) analysis, prefix analysis, and security analysis 🌈 However, the monitoring of other layers, protocols, and the cross-layer, cross-protocol KPI correlations are still in their infancy (e.g., IGP monitoring is not as extensive as BMP), which requires further research 🧚 So yes, more research is needed, bestie.
Note that the requirement and solutions for network congestion avoidance are also applicable to the control plane telemetry 💖
3.1.3. Forwarding Plane Telemetry ✨
An effective forwarding plane telemetry system relies on the data that the network device can expose 🍓 The quality, quantity, and timeliness of that data must meet some stringent requirements, and this raises some challenges for the network data plane devices where the first-hand data originates.
- A data plane device's main function is user traffic processing and forwarding. While supporting network visibility is important, the telemetry is just an auxiliary function, and it should strive to not impede normal traffic processing and forwarding (i.e., the forwarding behavior should not be altered, and the trade-off between forwarding performance and telemetry should be well-balanced) 🫶
- Network operation applications require end-to-end visibility across various sources, which can result in a huge volume of data. However, the sheer quantity of data must not exhaust the network bandwidth, regardless of the data delivery approach (i.e., whether through in-band or out-of-band channels) 😭
- The data plane devices must provide timely data with the minimum possible delay. Long processing, transport, storage, and analysis delay can impact the effectiveness of the control loop and even render the data useless 💀
- The data should be structured, labeled, and easy for applications to parse and consume. At the same time, the data types needed by applications can vary significantly. The data plane devices need to provide enough flexibility and programmability to support the precise data provision for applications ✨
- The data plane telemetry should support incremental deployment and work even though some devices are unaware of the system 🦋
- The requirement and solutions for network congestion avoidance are also applicable to the forwarding plane telemetry 👑
Although not specific to the forwarding plane, these challenges are more difficult for the forwarding plane because of the limited resources and flexibility. Data plane programmability is essential to support network telemetry, and newer data plane forwarding chips are equipped with advanced telemetry features and provide flexibility to support customized telemetry functions 🔥
Technique Taxonomy: This pertains to how one instruments the telemetry; there can be multiple possible dimensions to classify the forwarding plane telemetry techniques 🌸
- Active, Passive, and Hybrid: This dimension pertains to the end-to-end measurement. Active and passive methods (as well as the hybrid types) are well documented in [RFC7799]. Passive methods include TCPDUMP, IPFIX [RFC7011], sFlow, and traffic mirroring. These methods usually have low data coverage. The bandwidth cost is very high in order to improve the data coverage. On the other hand, active methods include Ping, the One-Way Active Measurement Protocol (OWAMP) [RFC4656], the Two-Way Active Measurement Protocol (TWAMP) [RFC5357], the Simple Two-way Active Measurement Protocol (STAMP) [RFC8762], and Cisco's SLA Protocol [RFC6812]. These methods are intrusive and only provide indirect network measurements. Hybrid methods, including IOAM [RFC9197], Alternate Marking (AM) [RFC8321], and Multipoint Alternate Marking [RFC8889], provide a well-balanced and more flexible approach. However, these methods are also more complex to implement 🧠
- In-Band and Out-of-Band: Telemetry data carried in user packets before being exported to a data collector is considered in-band (e.g., IOAM [RFC9197]). Telemetry data that is directly exported to a data collector without modifying user packets is considered out-of-band (e.g., the postcard-based approach described in Appendix A.3.5). It is also possible to have hybrid methods, where only the telemetry instruction or partial data is carried by user packets (e.g., AM [RFC8321]) 🎀
- End-to-End and In-Network: End-to-end methods start from, and end at, the network end hosts (e.g., Ping). In-network methods work in networks and are transparent to end hosts. However, if needed, in-network methods can be easily extended into end hosts 🌈
- Data Subject: Depending on the telemetry objective, the methods can be flow based (e.g., IOAM [RFC9197]), path based (e.g., Traceroute), and node based (e.g., IPFIX [RFC7011]). The various data objects can be packet, flow record, measurement, states, and signal 🧚
3.1.4. External Data Telemetry ✨
Events that occur outside the boundaries of the network system are another important source of network telemetry 🌍 Correlating both internal telemetry data and external events with the requirements of network systems, as presented in [NMRG-ANTICIPATED-ADAPTATION], gives management operations a strategic and functional advantage 👑
As with other sources of telemetry information, the data and events must meet strict requirements, especially in terms of timeliness, which is essential to properly incorporate external event information into network management applications. The specific challenges are described as follows:
- The role of the external event detector can be played by multiple elements, including hardware (e.g., physical sensors, such as seismometers) and software (e.g., big data sources that can analyze streams of information, such as Twitter messages). Thus, the transmitted data must support different shapes but, at the same time, follow a common but extensible schema 🦋
- Since the main function of the external event detectors is to perform the notifications, their timeliness is assumed. However, once messages have been dispatched, they must be quickly collected and inserted into the control plane with variable priority, which is higher for important sources and events and lower for secondary ones 💜
- The schema used by external detectors must be easily adopted by current and future devices and applications. Therefore, it must be easily mapped to current data models, such as in terms of YANG 🍓
- As the communication with external entities outside the boundary of a provider network may be realized over the Internet, the risk of congestion is even more relevant in this context and proper countermeasures must be taken. Solutions such as network transport circuit breakers are needed as well 🔥
Organizing both internal and external telemetry information together will be key for the general exploitation of the management possibilities of current and future network systems, as reflected in the incorporation of cognitive capabilities to new hardware and software (virtual) elements 🧚✨
3.2. Second-Level Function Components ✨
okay so each plane's telemetry module can be split up even further into five distinct conceptual components 🧩 it's giving organized bestie energy
- Data Query, Analysis, and Storage: this one lives at the network operation application block in Figure 1 🎀 it's normally part of the network management system on the receiver side. On one hand, it's in charge of issuing data requirements. The data of interest can be modeled data through configuration or custom data through programming. The requirements can be queries for one-shot data or subscriptions for events or streaming data. On the other hand, it receives, stores, and processes the data that comes back from network devices. Data analysis can be interactive, so it can kick off further data queries 🔍 This component can reside in either network devices or remote controllers, and it can be centralized or distributed, with one or more instances. so flexible 💅
- Data Configuration and Subscription: this component manages data queries on devices 🦋 It decides the protocol and channel that applications use to get the data they want. It's also responsible for configuring the desired data that might not be directly available from data sources. The subscription data can be described by models, templates, or programs.
- Data Encoding and Export: this one decides how telemetry data gets delivered to the data analysis and storage component, with access control 🔐 The data encoding and the transport protocol may vary depending on the export location.
- Data Generation and Processing: the requested data has to be captured, filtered, processed, and formatted inside network devices, starting from the raw data sources 🌸 This may involve in-network computing and processing on either the fast path or the slow path in network devices. literally so much going on in here ✨
- Data Object and Source: this component decides which monitoring objects and original data sources are provisioned in the device 🧚 A data source usually just provides raw data that still needs further processing. Each data source can be considered a probe 👀 Some data sources can be dynamically installed, while others are more static.
+----------------------------------------+ +----------------------------------------+ |
| Data Query, Analysis, & Storage | |
|---|---|
| + |
+-------+++ -----------------------------+
| ^^^ | |||
|---|---|---|---|
| V |
+--+V--------------------+++------------+ +-----V---------------------+------------+ | +---------------------+-------+----------+ | |
| Data Configuration | |||
|---|---|---|---|
| & Subscription | Data Encoding | ||
| (model, template, | & Export | ||
| & program) |
+---------------------+------------------| | |
| Data Generation | ||
|---|---|---|
| & Processing |
+----------------------------------------| | |
| Data Object and Source | -+ | |
|---|---|---|
| -+ |
+----------------------------------------+
Figure 2: Components in the Network Telemetry Framework
3.3. Data Acquisition Mechanism and Type Abstraction 🌈
broadly speaking, network data can be acquired through subscription (push) or query (poll) 💜 A subscription is basically a contract between publisher and subscriber. After the initial setup, the subscribed data gets automatically delivered to registered subscribers until the subscription expires. There are two flavors of subscription: the subscriptions can be predefined, or the subscribers can be allowed to configure and tailor the published data to their specific needs 🎀
queries, on the other hand, are for when a client wants immediate, one-off feedback from network devices ⚡ The queried data may be pulled straight from some specific data source, or it may be synthesized and processed from raw data. Queries work really well for interactive network telemetry applications 🫶
in general, data can be pulled (i.e., queried) whenever it's needed, but in many cases pushing the data (i.e., subscription) is more efficient, and it can reduce the latency of a client detecting a change ⏱️ From the data consumer point of view, there are four types of data from network devices that a telemetry data consumer can subscribe to or query:
- Simple Data: data that is steadily available from some datastore or static probes in network devices 📦
- Derived Data: data that needs to be synthesized or processed in the network from raw data from one or more network devices 🧪 The data processing function can be statically or dynamically loaded into network devices.
- Event-triggered Data: data that gets conditionally acquired based on the occurrence of some events ⚠️ An example could be an interface changing operational state between up and down. Such data can be actively pushed through subscription or passively polled through query. There are many ways to model events, including using Finite State Machine (FSM) or Event Condition Action (ECA) [NETMOD-ECA-POLICY].
- Streaming Data: data that is continuously generated 🌊 It can be a time series or the dump of databases. For example, an interface packet counter is exported every second. Streaming data reflects real-time network states and metrics, and it requires large bandwidth and processing power. Streaming data is always actively pushed to the subscribers 📡
the above telemetry data types are not mutually exclusive! They're often composite 🫠 Derived data is composed of simple data, event-triggered data can be simple or derived, and streaming data can be based on some recurring event. The relationships between these data types are illustrated in Figure 3 👇
+----------------------+ +-----------------+
| Event-Triggered Data | <----+ Streaming Data |
+-------+---+----------+ +-----+---+-------+
| +--------------+ | ||
|---|---|---|
| +--> | Derived Data | <--+ |
| +------+------ + | ||
| V | ||
| +--------------+ |
+------>| Simple Data |<------+ +--------------+
Figure 3: Data Type Relationship
subscription usually deals with event-triggered data and streaming data, and query usually deals with simple data and derived data. But the other ways are also possible 🙃 Advanced network telemetry techniques are designed mainly for event-triggered or streaming data subscription and derived data query 💖
3.4. Mapping Existing Mechanisms into the Framework 🗺️
the table below shows how the existing mechanisms (mainly published in IETF, with the emphasis on the latest new technologies) are positioned in the framework 🍓 Given the vast body of existing work, we can't provide an exhaustive list, so the mechanisms in the tables should be considered as just examples. Also, some comprehensive protocols and techniques may cover multiple aspects or modules of the framework, so a name in a block only emphasizes one particular characteristic of it. More details about some of the listed mechanisms can be found in Appendix A 📖
+===============+=================+================+============+
| Management | Control Plane | Forwarding | |
|---|---|---|---|
| Plane | Plane |
+===============+=================+================+============+
| data | gNMI, NETCONF, | gNMI, NETCONF, | NETCONF, |
|---|---|---|---|
| configuration | RESTCONF, SNMP, | RESTCONF, | RESTCONF, |
| and subscribe | YANG-Push | YANG-Push | YANG-Push |
+---------------+-----------------+----------------+------------+
| data | MIB, YANG | YANG | IOAM, |
|---|---|---|---|
| generation | PSAMP, | ||
| and process | PBT, AM |
+---------------+-----------------+----------------+------------+
| data encoding | gRPC, HTTP, TCP | BMP, TCP | IPFIX, UDP |
|---|---|---|---|
| and export |
+---------------+-----------------+----------------+------------+
Table 2: Existing Work Mapping
although the framework is generally suitable for any network environment, multi-domain telemetry has some unique challenges that deserve further architectural consideration. That's out of the scope of this document 🌍
4. Evolution of Network Telemetry Applications 🚀
network telemetry is an evolving technical area ✨ As the network moves towards automated operation, network telemetry applications go through several stages of evolution, and each stage adds a new layer of requirements to the underlying network telemetry techniques. Each stage is built upon the techniques adopted by the previous stages, plus some new requirements 🧠
- Stage 0 - Static Telemetry: the telemetry data source and type are determined at design time. The network operator can only configure how to use it, and only with limited flexibility 🪨
- Stage 1 - Dynamic Telemetry: custom telemetry data can be dynamically programmed or configured at runtime without interrupting network operation, which allows a trade-off among resource, performance, flexibility, and coverage ⚖️
- Stage 2 - Interactive Telemetry: the network operator can continuously customize and fine-tune the telemetry data in real time to reflect the network operation's visibility requirements 🎛️ Compared with Stage 1, the changes are frequent, based on real-time feedback. At this stage some tasks can be automated, but human operators still need to sit in the middle and make decisions 🙋
- Stage 3 - Closed-Loop Telemetry: the telemetry is free from the interference of human operators, except for generating the reports 🤖 The intelligent network operation engine automatically issues the telemetry data requests, analyzes the data, and updates the network operations in closed control loops 🔁
existing technologies are ready for Stages 0 and 1, and individual applications for Stages 2 and 3 are also possible right now 💫 However, future autonomic networks may need a comprehensive operation management system that works at Stages 2 and 3 to cover all the network operation tasks. A well-defined network telemetry framework is the first step towards that direction 👑
5. Security Considerations 🔐✨
Okay bestie, this section is where things get SERIOUS 😭🫶 Network telemetry is complicated, and complicated means there's a lot that can go wrong. Telemetry data can be manipulated to exhaust network resources at each plane and at the data consumer 💀 Falsified or tampered data can mislead the decision-making process and paralyze networks 😱 Wrong configuration and programming for telemetry is just as harmful, no cap.
The telemetry data is highly sensitive 🙈 It exposes a lot of information about the network and how it's configured. Some of that info can make attacks way easier to design, like exact details of what software and patches have been installed, and it lets an attacker figure out whether a device may be exposed to unprotected security vulnerabilities. Not me being scared, but also yes, me being a little scared 😭
Since this document proposes a framework for network telemetry, and the telemetry mechanisms we talked about are way more extensive (both in how often messages get sent and how much traffic they make) than conventional network OAM concepts, we must also anticipate that new security considerations may come up too 👀 There are already plenty of techniques for securing the forwarding plane, the control plane, and the management plane. Still, it's important to check whether any new threat vectors are being enabled by using network telemetry procedures and mechanisms. Main character energy: always double-check the vibes 💅
This document proposes a conceptual architecture for collecting, transporting, and analyzing a wide variety of data sources to support network applications. The protocols, data formats, and configurations chosen to implement this framework will decide the specific security considerations. These considerations may include:
- Telemetry framework trust and policy models 🦋
- Role management and access control for enabling and disabling telemetry capabilities 🔑
- Protocol transport used for telemetry data and its inherent security capabilities 🚚
- Telemetry data stores, storage encryption, methods of access, and retention practices 🗄️
- Tracking telemetry events and any abnormalities that might identify malicious attacks using telemetry interfaces. 🕵️♀️
- Authentication and integrity protection of telemetry data to make data more trustworthy 🫶
- Segregating the telemetry data traffic from the data traffic carried over the network. For example, historically, management access and management data may be carried via an independent management network 🌸
Some of the security considerations above may be minimized or negated with policy management of network telemetry 🎀 In a network telemetry deployment, it would be advantageous to separate telemetry capabilities into different classes of policies, namely Role-Based Access Control and Event-Condition-Action policies. Also, potential conflicts between network telemetry mechanisms must be detected accurately and resolved quickly. Otherwise, unnecessary network telemetry traffic could spiral into an unintended, or even intended, denial-of-service attack 🚨
Further study of the security issues is still required. It's expected that security mechanisms and protocols will be developed and deployed along with a network telemetry system. Don't skip this part, babe 💀
6. IANA Considerations 🌷
Good news, no drama here 💖 This document has no IANA actions. Nothing to register, nothing to update, just vibes 🧚
7. Informative References 📚
[gnmi] Shakir, R., Shaikh, A., Borman, P., Hines, M., Lebsack,
C., and C. Marrow, "gRPC Network Management Interface",
IETF 98, March 2017,
<https://datatracker.ietf.org/meeting/98/materials/slides-
98-rtgwg-gnmi-intro-draft-openconfig-rtgwg-gnmi-spec-00>.
[gpb] Google Developers, "Protocol Buffers",
<https://developers.google.com/protocol-buffers>.
[grpc] gRPC, "gPPC: A high performance, open source universal RPC
framework", <https://grpc.io>.
[IPPM-IOAM-DIRECT-EXPORT]
Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F.,
Bhandari, S., Ed., Sivakolundu, R., and T. Mizrahi, Ed.,
"In-situ OAM Direct Exporting", Work in Progress,
Internet-Draft, draft-ietf-ippm-ioam-direct-export-07, 13
October 2021, <https://datatracker.ietf.org/doc/html/
draft-ietf-ippm-ioam-direct-export-07>.
[IPPM-POSTCARD-BASED-TELEMETRY]
Song, H., Mirsky, G., Filsfils, C., Abdelsalam, A., Zhou,
T., Li, Z., Mishra, G., Shin, J., and K. Lee, "In-Situ OAM
Marking-based Direct Export", Work in Progress, Internet-
Draft, draft-song-ippm-postcard-based-telemetry-12, 12 May
2022, <https://datatracker.ietf.org/doc/html/draft-song-
ippm-postcard-based-telemetry-12>.
[NETCONF-DISTRIB-NOTIF]
Zhou, T., Zheng, G., Voit, E., Graf, T., and P. Francois,
"Subscription to Distributed Notifications", Work in
Progress, Internet-Draft, draft-ietf-netconf-distributed-
notif-03, 10 January 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-netconf-
distributed-notif-03>.
[NETCONF-UDP-NOTIF]
Zheng, G., Zhou, T., Graf, T., Francois, P., Feng, A. H.,
and P. Lucente, "UDP-based Transport for Configured
Subscriptions", Work in Progress, Internet-Draft, draft-
ietf-netconf-udp-notif-05, 4 March 2022,
<https://datatracker.ietf.org/doc/html/draft-ietf-netconf-
udp-notif-05>.
[NETMOD-ECA-POLICY]
Wu, Q., Bryskin, I., Birkholz, H., Liu, X., and B. Claise,
"A YANG Data model for ECA Policy Management", Work in
Progress, Internet-Draft, draft-ietf-netmod-eca-policy-01,
19 February 2021, <https://datatracker.ietf.org/doc/html/
draft-ietf-netmod-eca-policy-01>.
[NMRG-ANTICIPATED-ADAPTATION]
Martinez-Julia, P., Ed., "Exploiting External Event
Detectors to Anticipate Resource Requirements for the
Elastic Adaptation of SDN/NFV Systems", Work in Progress,
Internet-Draft, draft-pedro-nmrg-anticipated-adaptation-
02, 29 June 2018, <https://datatracker.ietf.org/doc/html/
draft-pedro-nmrg-anticipated-adaptation-02>.
[NMRG-IBN-CONCEPTS-DEFINITIONS]
Clemm, A., Ciavaglia, L., Granville, L. Z., and J.
Tantsura, "Intent-Based Networking - Concepts and
Definitions", Work in Progress, Internet-Draft, draft-
irtf-nmrg-ibn-concepts-definitions-09, 24 March 2022,
<https://datatracker.ietf.org/doc/html/draft-irtf-nmrg-
ibn-concepts-definitions-09>.
[OPSAWG-DNP4IQ]
Song, H., Ed. and J. Gong, "Requirements for Interactive
Query with Dynamic Network Probes", Work in Progress,
Internet-Draft, draft-song-opsawg-dnp4iq-01, 19 June 2017,
<https://datatracker.ietf.org/doc/html/draft-song-opsawg-
dnp4iq-01>.
[OPSAWG-IFIT-FRAMEWORK]
Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "A
Framework for In-situ Flow Information Telemetry", Work in
Progress, Internet-Draft, draft-song-opsawg-ifit-
framework-17, 22 February 2022,
<https://datatracker.ietf.org/doc/html/draft-song-opsawg-
ifit-framework-17>.
[RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin,
"Simple Network Management Protocol (SNMP)", RFC 1157,
DOI 10.17487/RFC1157, May 1990,
<https://www.rfc-editor.org/info/rfc1157>.
[RFC2578] McCloghrie, K., Ed., Perkins, D., Ed., and J.
Schoenwaelder, Ed., "Structure of Management Information
Version 2 (SMIv2)", STD 58, RFC 2578,
DOI 10.17487/RFC2578, April 1999,
<https://www.rfc-editor.org/info/rfc2578>.
[RFC2981] Kavasseri, R., Ed., "Event MIB", RFC 2981,
DOI 10.17487/RFC2981, October 2000,
<https://www.rfc-editor.org/info/rfc2981>.
[RFC3176] Phaal, P., Panchen, S., and N. McKee, "InMon Corporation's
sFlow: A Method for Monitoring Traffic in Switched and
Routed Networks", RFC 3176, DOI 10.17487/RFC3176,
September 2001, <https://www.rfc-editor.org/info/rfc3176>.
[RFC3411] Harrington, D., Presuhn, R., and B. Wijnen, "An
Architecture for Describing Simple Network Management
Protocol (SNMP) Management Frameworks", STD 62, RFC 3411,
DOI 10.17487/RFC3411, December 2002,
<https://www.rfc-editor.org/info/rfc3411>.
[RFC3416] Presuhn, R., Ed., "Version 2 of the Protocol Operations
for the Simple Network Management Protocol (SNMP)",
STD 62, RFC 3416, DOI 10.17487/RFC3416, December 2002,
<https://www.rfc-editor.org/info/rfc3416>.
[RFC3877] Chisholm, S. and D. Romascanu, "Alarm Management
Information Base (MIB)", RFC 3877, DOI 10.17487/RFC3877,
September 2004, <https://www.rfc-editor.org/info/rfc3877>.
[RFC3954] Claise, B., Ed., "Cisco Systems NetFlow Services Export
Version 9", RFC 3954, DOI 10.17487/RFC3954, October 2004,
<https://www.rfc-editor.org/info/rfc3954>.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, DOI 10.17487/RFC4656, September 2006,
<https://www.rfc-editor.org/info/rfc4656>.
[RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual
Circuit Connectivity Verification (VCCV): A Control
Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085,
December 2007, <https://www.rfc-editor.org/info/rfc5085>.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
RFC 5357, DOI 10.17487/RFC5357, October 2008,
<https://www.rfc-editor.org/info/rfc5357>.
[RFC5424] Gerhards, R., "The Syslog Protocol", RFC 5424,
DOI 10.17487/RFC5424, March 2009,
<https://www.rfc-editor.org/info/rfc5424>.
[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for
the Network Configuration Protocol (NETCONF)", RFC 6020,
DOI 10.17487/RFC6020, October 2010,
<https://www.rfc-editor.org/info/rfc6020>.
[RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed.,
and A. Bierman, Ed., "Network Configuration Protocol
(NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011,
<https://www.rfc-editor.org/info/rfc6241>.
[RFC6812] Chiba, M., Clemm, A., Medley, S., Salowey, J., Thombare,
S., and E. Yedavalli, "Cisco Service-Level Assurance
Protocol", RFC 6812, DOI 10.17487/RFC6812, January 2013,
<https://www.rfc-editor.org/info/rfc6812>.
[RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
"Specification of the IP Flow Information Export (IPFIX)
Protocol for the Exchange of Flow Information", STD 77,
RFC 7011, DOI 10.17487/RFC7011, September 2013,
<https://www.rfc-editor.org/info/rfc7011>.
[RFC7258] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an
Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May
2014, <https://www.rfc-editor.org/info/rfc7258>.[RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. Weingarten, "An Overview of Operations, Administration, and Maintenance (OAM) Tools", RFC 7276, DOI 10.17487/RFC7276, June 2014, <https://www.rfc-editor.org/info/rfc7276>.
[RFC7540] Belshe, M., Peon, R., and M. Thomson, Ed., "Hypertext Transfer Protocol Version 2 (HTTP/2)", RFC 7540, DOI 10.17487/RFC7540, May 2015, <https://www.rfc-editor.org/info/rfc7540>.
[RFC7575] Behringer, M., Pritikin, M., Bjarnason, S., Clemm, A., Carpenter, B., Jiang, S., and L. Ciavaglia, "Autonomic Networking: Definitions and Design Goals", RFC 7575, DOI 10.17487/RFC7575, June 2015, <https://www.rfc-editor.org/info/rfc7575>.
[RFC7799] Morton, A., "Active and Passive Metrics and Methods (with Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, May 2016, <https://www.rfc-editor.org/info/rfc7799>.
[RFC7854] Scudder, J., Ed., Fernando, R., and S. Stuart, "BGP Monitoring Protocol (BMP)", RFC 7854, DOI 10.17487/RFC7854, June 2016, <https://www.rfc-editor.org/info/rfc7854>.
[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", RFC 7950, DOI 10.17487/RFC7950, August 2016, <https://www.rfc-editor.org/info/rfc7950>.
[RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, <https://www.rfc-editor.org/info/rfc8040>.
[RFC8084] Fairhurst, G., "Network Transport Circuit Breakers", BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017, <https://www.rfc-editor.org/info/rfc8084>.
[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, March 2017, <https://www.rfc-editor.org/info/rfc8085>.
[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017, <https://www.rfc-editor.org/info/rfc8259>.
[RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli, L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi, "Alternate-Marking Method for Passive and Hybrid Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321, January 2018, <https://www.rfc-editor.org/info/rfc8321>.
[RFC8639] Voit, E., Clemm, A., Gonzalez Prieto, A., Nilsen-Nygaard, E., and A. Tripathy, "Subscription to YANG Notifications", RFC 8639, DOI 10.17487/RFC8639, September 2019, <https://www.rfc-editor.org/info/rfc8639>.
[RFC8641] Clemm, A. and E. Voit, "Subscription to YANG Notifications for Datastore Updates", RFC 8641, DOI 10.17487/RFC8641, September 2019, <https://www.rfc-editor.org/info/rfc8641>.
[RFC8671] Evens, T., Bayraktar, S., Lucente, P., Mi, P., and S. Zhuang, "Support for Adj-RIB-Out in the BGP Monitoring Protocol (BMP)", RFC 8671, DOI 10.17487/RFC8671, November 2019, <https://www.rfc-editor.org/info/rfc8671>.
[RFC8762] Mirsky, G., Jun, G., Nydell, H., and R. Foote, "Simple Two-Way Active Measurement Protocol", RFC 8762, DOI 10.17487/RFC8762, March 2020, <https://www.rfc-editor.org/info/rfc8762>.
[RFC8889] Fioccola, G., Ed., Cociglio, M., Sapio, A., and R. Sisto, "Multipoint Alternate-Marking Method for Passive and Hybrid Performance Monitoring", RFC 8889, DOI 10.17487/RFC8889, August 2020, <https://www.rfc-editor.org/info/rfc8889>.
[RFC8924] Aldrin, S., Pignataro, C., Ed., Kumar, N., Ed., Krishnan, R., and A. Ghanwani, "Service Function Chaining (SFC) Operations, Administration, and Maintenance (OAM) Framework", RFC 8924, DOI 10.17487/RFC8924, October 2020, <https://www.rfc-editor.org/info/rfc8924>.
[RFC9069] Evens, T., Bayraktar, S., Bhardwaj, M., and P. Lucente, "Support for Local RIB in the BGP Monitoring Protocol (BMP)", RFC 9069, DOI 10.17487/RFC9069, February 2022, <https://www.rfc-editor.org/info/rfc9069>.
[RFC9197] Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi, Ed., "Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022, <https://www.rfc-editor.org/info/rfc9197>.
[W3C.REC-xml-20081126] Bray, T., Paoli, J., Sperberg-McQueen, M., Maler, E., and F. Yergeau, "Extensible Markup Language (XML) 1.0 (Fifth Edition)", World Wide Web Consortium Recommendation REC-xml-20081126, November 2008, <https://www.w3.org/TR/2008/REC-xml-20081126>.
[y1731] ITU-T, "Operations, administration and maintenance (OAM) functions and mechanisms for Ethernet-based networks", ITU-T Recommendation G.8013/Y.1731, August 2015, <https://www.itu.int/rec/T-REC-Y.1731/en>.
Appendix A. A Survey on Existing Network Telemetry Techniques ✨
This appendix is non-normative, so think of it as a cute little tour 🦋 It gives an overview of some existing techniques and standard proposals for each network telemetry module 💜
A.1. Management Plane Telemetry
A.1.1. Push Extensions for NETCONF
NETCONF [RFC6241] is a popular network management protocol recommended by IETF 🎀 Its core strength is managing configuration, but it can also be used for data collection, which is so versatile ngl 💅
YANG-Push [RFC8639] [RFC8641] extends NETCONF and lets subscriber applications request a continuous, customized stream of updates from a YANG datastore 🌊 Being able to see every change made to YANG configuration and operational objects unlocks new capabilities based on remote mirroring of configuration and operational state. It's giving main character energy ✨
Then there's a distributed data collection mechanism [NETCONF-DISTRIB-NOTIF] that runs over a UDP-based publication channel [NETCONF-UDP-NOTIF]. It gives enhanced efficiency for NETCONF-based telemetry, which is iconic 🫶
A.1.2. gRPC Network Management Interface
gRPC Network Management Interface (gNMI) [gnmi] is a network management protocol based on the gRPC [grpc] Remote Procedure Call (RPC) framework. One gRPC service definition can cover both configuration and telemetry, so it's a two-for-one 🍓
gRPC is an open-source micro-service communication framework built on HTTP/2 [RFC7540]. It has a bunch of capabilities that are well-suited for network telemetry, including:
- A full-duplex streaming transport model. When you pair it with a binary encoding mechanism, you get good telemetry efficiency 🔥
- Higher-level feature consistency across platforms, which common HTTP/2 libraries typically don't give you. This is especially valuable because telemetry data collectors usually live on a huge variety of platforms 🌈
- A built-in load-balancing and failover mechanism 🧚
A.2. Control Plane Telemetry
A.2.1. BGP Monitoring Protocol
BMP [RFC7854] is used to monitor BGP sessions, and it's intended to be a convenient interface for getting route views 👀
BGP routing information is collected from the monitored device(s) to the BMP monitoring station by setting up the BMP TCP session. The BGP peers are watched through the BMP Peer Up and Peer Down notifications. The BGP routes, including Adj_RIB_In [RFC7854], Adj_RIB_out [RFC8671], and local RIB [RFC9069], get encapsulated in the BMP Route Monitoring Message and the BMP Route Mirroring Message. That gives you both an initial table dump and real-time route updates, so no cap, it's a full picture 📸
BGP statistics are reported through the BMP Stats Report Message, which could be either timer triggered or event-driven ⏰ Future BMP extensions could further enrich BGP monitoring applications, and I'm honestly obsessed 💖
A.3. Data Plane Telemetry
A.3.1. Alternate-Marking (AM) Technology
The Alternate-Marking method enables efficient measurements of packet loss, delay, and jitter in both IP and Overlay Networks, as presented in [RFC8321] and [RFC8889] 🦋
This technique works for point-to-point and multipoint-to-multipoint flows. Alternate Marking creates batches of packets by alternating the value of 1 bit (or a label) in the packet header. These batches are unambiguously recognized over the network, and comparing packet counters for each batch lets you calculate packet loss. The same idea can be applied to delay measurement by selecting ad hoc packets with a marking bit dedicated to delay measurements ⏱️
The Alternate-Marking method needs two counters each marking period for each flow under monitor. For example, if you consider n measurement points and m monitored flows, the order of magnitude of the packet counters for each time interval is n*m*2 (1 per color) 🧮
Networks offer rich sets of performance measurement data, like packet counters, but conventional approaches run into limitations. The bottleneck is the generation and export of the data, plus how much data can reasonably be collected from the network 😭 On top of that, management tasks like deciding and configuring which data to generate lead to significant deployment challenges 💀
The Multipoint Alternate-Marking approach, described in [RFC8889], aims to fix this and make performance monitoring more flexible when a detailed analysis isn't needed 🌸
An application orchestrates network performance measurement tasks across the network to allow for optimized monitoring. It can choose how roughly or precisely to configure measurement points, depending on what the application needs 🎀
You can monitor a Multipoint Network without in-depth examination by using Network Clustering. Clusters are subnetworks that are portions of the entire network and preserve the same property of the entire network. So if there's packet loss or the delay is too high, specific filtering criteria can be applied to gather a more detailed analysis, using a different combination of clusters, all the way down to per-flow measurement, as described in the Alternate-Marking document [RFC8321] 🔍
In summary, an application can configure end-to-end network monitoring. If the network isn't having issues, this approximate monitoring is good enough and is very cheap in terms of network resources 💜 If there are problems, the application notices them from this approximate monitoring. Then, to localize the part of the network that has issues, it configures the measurement points more extensively, allowing more detailed monitoring. After the problem is detected and resolved, the initial approximate monitoring can be used again. Iykyk, it's the glow-up cycle 🌈✨
A.3.2. Dynamic Network Probe 🌸
okay so the Dynamic Network Probe (DNP) [OPSAWG-DNP4IQ] is a hardware-based thing that gives you a programmable way to customize what an application collects from the data plane 🔧✨ the big payoff is that you export way less data, which is so iconic. a full DNP solution covers three parts: data source, data subscription, and data generation 💅 the data subscription has to define the derived data that can be composed and derived from the raw data sources, and the data generation uses moderate in-network computing to produce the data you actually want 🧚
DNP can give the data plane telemetry some seriously unforeseeable flexibility, but it comes with its own drama 😭 the data plane has to be flexible enough to be dynamically reprogrammed at runtime, and the programming Application Programming Interface (API) is not defined yet 🙈
A.3.3. IP Flow Information Export (IPFIX) Protocol 🦋
bestie, think of traffic on a network as a bunch of flows moving through network elements 🌈 IPFIX [RFC7011] is how you transmit that traffic flow information for administrative or other purposes 💜 a typical IPFIX-enabled system has a pool of Metering Processes. These collect data packets at one or more Observation Points, optionally filter them, and aggregate information about those packets. Then an Exporter gathers each of the Observation Points together into an Observation Domain and sends the info over the IPFIX protocol to a Collector 🎀
A.3.4. In Situ OAM 🍓
the classic passive and active monitoring techniques are either inaccurate or resource consuming, and honestly that's not it 🙄 it's way better to directly grab data tied to a flow's packets as they pass through the network. IOAM [RFC9197] is a data generation technique. It embeds a new instruction header in user packets, and that instruction tells the network nodes to add the requested data to the packets 🔥 so at the end of the path, you can collect the packet's experience from the entire forwarding path. That firsthand data is literally invaluable for lots of network OAM applications 👑
ngl, IOAM has its own challenges too 💀 the issues around performance impact, security, scalability and overhead limits, encapsulation difficulties in some protocols, and cross-domain deployment all still need to be addressed.
A.3.5. Postcard-Based Telemetry 💌
postcard-based telemetry is the vibe as embodied in IOAM Direct Export (DEX) [IPPM-IOAM-DIRECT-EXPORT] and IOAM Marking [IPPM-POSTCARD-BASED-TELEMETRY]. It's a complementary technique to the passport-based IOAM [RFC9197] 🧚♀️ PBT directly exports data at each node through an independent packet. The tradeoff is higher bandwidth overhead and the need for data correlation, but PBT has several unique advantages anyway 💖 it can also help identify where a packet got dropped in case it's lost on its forwarding path 🕵️♀️
A.3.6. Existing OAM for Specific Data Planes 🌷
different data planes come with their own unique OAM requirements, so it's giving variety 🎨 the IETF has published OAM technique and framework documents (e.g., [RFC8924] and [RFC5085]) that target a bunch of data planes: Multiprotocol Label Switching (MPLS), L2 Virtual Private Network (VPN), Network Virtualization over Layer 3 (NVO3), Virtual Extensible LAN (VXLAN), Bit Index Explicit Replication (BIER), Service Function Chaining (SFC), Segment Routing (SR), and Deterministic Networking (DETNET) 🌐 and the data plane telemetry techniques we just talked about can be used to level up the OAM capability on all of those ✨
A.4. External Data and Event Telemetry 🌍
A.4.1. Sources of External Events 📰
okay so the network telemetry framework has to make sure that external event detectors (sources) are easy to connect to the management solutions (sinks). That's the only way the info from external detectors stays meaningful for management purposes 🫶 it means you need a list of potential external data sources that could matter for network management, and you have to match each one to the connectors and/or interfaces needed to hook it up 🔌
These are the categories of external event sources that might be of interest to network management:
- Smart objects and sensors. With the Internet of Things (IoT) booming, any network system will have lots of smart objects attached to its physical surroundings and logical operation environments 🌡️ most of these are sensors of many kinds (e.g., temperature, humidity, and presence), and the info they provide can be super useful for managing the network, even when they weren't deployed for that purpose 💅 elements of this source type usually come with a specific protocol for interaction, especially one of the IoT protocols, like the Constrained Application Protocol (CoAP) 🧸
- Online news reporters. Lots of online news services can share an enormous amount of info about events happening around the world 📱 some of those events can impact the network system managed by a given framework, so that info may be of interest to the management solution. For instance, security reports like Common Vulnerabilities and Exposures (CVEs) can be issued by the corresponding authority and used by the management solution to update the managed system, if needed 🛡️ instead of a specific protocol and data format, these sources usually follow a relaxed but structured format. That format will be part of both the ontology and information model of the telemetry framework 🧠
- Global event analyzers. Big data analyzers provide a huge amount of info and, even more interestingly, can identify events detected by analyzing many data streams from different origins 🤯 unlike the other source types, which focus on specific events, the detectors of this type detect generic events. For example, during a sports event, some unexpected movement makes it fascinating, and tons of people connect to sites reporting on it 🏟️ the underlying networks supporting the services covering the event can be affected by that, so their management solutions should be aware of it 👀 unlike the other source types, a new information model, format, and reporting protocol is required to integrate detectors of this type with the management solution 🎀
Additional detector types can be added to the system, but they'll generally be the result of composing the properties offered by these main classes 🌸
A.4.2. Connectors and Interfaces 🔗
so for external event detectors to be properly integrated with other management solutions, both elements have to expose interfaces and protocols that match their particular objective 💖 external event detectors are mainly focused on giving their info to their main consumers, and those consumers generally won't be limited to network management solutions. That means the framework must define the required connectors so the link between detectors (sources) and their consumers within the management systems (sinks) actually works 🧚
In some situations, the external event detector connects to the management system via the management plane. For those situations, there will be a special connector that provides the typical interfaces found in most other elements connected to the management plane 🌈 for instance, those interfaces could do this with a specific data model (YANG) and a specific telemetry protocol, such as NETCONF, YANG-Push, or gRPC 💜
Acknowledgments 🫶
we'd love to thank Rob Wilton, Greg Mirsky, Randy Presuhn, Joe Clarke, Victor Liu, James Guichard, Uri Blumenthal, Giuseppe Fioccola, Yunan Gu, Parviz Yegani, Young Lee, Qin Wu, Gyan Mishra, Ben Schwartz, Alexey Melnikov, Michael Scharf, Dhruv Dhody, Martin Duke, Roman Danyliw, Warren Kumari, Sheng Jiang, Lars Eggert, Éric Vyncke, Jean-Michel Combes, Erik Kline, Benjamin Kaduk, and so many others who gave helpful comments and suggestions to improve this document 👑✨
Contributors 💅
the other contributors of this document are Tianran Zhou, Zhenbin Li, Zhenqiang Li, Daniel King, Adrian Farrel, and Alexander Clemm 🌷
Authors' Addresses 💌
Haoyu Song
Futurewei
United States of America
Email: haoyu.song@futurewei.com
Fengwei Qin
China Mobile
China
Email: qinfengwei@chinamobile.com
Pedro Martinez-Julia
NICT
Japan
Email: pedro@nict.go.jp
Laurent Ciavaglia
Rakuten Mobile
France
Email: laurent.ciavaglia@rakuten.com
Aijun Wang
China Telecom
China
Email: wangaj3@chinatelecom.cn📦 where this page comes from
- ietf: rfc9232 2022-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms