RFC 9318

IAB Workshop Report: Measuring Network Quality for End-Users

📅 published October 2022, from ietf

Part 1 of 9 🎀

IAB Workshop Report: Measuring Network Quality for End-Users 🌸

Okay so this is RFC 9318, bestie 🧚. Written by W. Hardaker and O. Shapira, it's an Informational doc (not a standard!) published in October 2022 under ISSN 2070-1721. It comes from the Internet Architecture Board (IAB), the long-term planning crew for the Internet. The title block doesn't list anything it obsoletes or updates, so it isn't replacing or patching any older RFC 🙈

Abstract ✨

The Measuring Network Quality for End-Users workshop was held virtually by the IAB on September 14-16, 2021 💻. This report recaps the workshop, the topics that came up, and some preliminary conclusions drawn at the end of it.

This document is a report on what happened at the workshop. The views and positions in it belong to the workshop participants, so they don't necessarily reflect IAB views and positions 💅

Status of This Memo 📜

This document is not an Internet Standards Track specification; it's published for informational purposes 💖

This document is a product of the Internet Architecture Board (IAB) and holds information the IAB decided was worth keeping on the permanent record. It represents the consensus of the IAB. Documents the IAB approves for publication are not candidates for any level of Internet Standard, see Section 2 of RFC 7841 🫶

Info about the current status of this document, any errata, and how to give feedback on it can be found at https://www.rfc-editor.org/info/rfc9318 🔥

Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions regarding this document 👑

Table of Contents 🗂️

1.  Introduction
  1.1.  Problem Space
2.  Workshop Agenda
3.  Position Papers
4.  Workshop Topics and Discussion
  4.1.  Introduction and Overviews
    4.1.1.  Key Points from the Keynote by Vint Cerf
    4.1.2.  Introductory Talks
    4.1.3.  Introductory Talks - Key Points
  4.2.  Metrics Considerations
    4.2.1.  Common Performance Metrics
    4.2.2.  Availability Metrics
    4.2.3.  Capacity Metrics
    4.2.4.  Latency Metrics
    4.2.5.  Measurement Case Studies
    4.2.6.  Metrics Key Points
  4.3.  Cross-Layer Considerations
    4.3.1.  Separation of Concerns
    4.3.2.  Security and Privacy Considerations
    4.3.3.  Metric Measurement Considerations
    4.3.4.  Towards Improving Future Cross-Layer Observability
    4.3.5.  Efficient Collaboration between Hardware and Transport
            Protocols
    4.3.6.  Cross-Layer Key Points
  4.4.  Synthesis
    4.4.1.  Measurement and Metrics Considerations
    4.4.2.  End-User Metrics Presentation
    4.4.3.  Synthesis Key Points
5.  Conclusions
  5.1.  General Statements
  5.2.  Specific Statements about Detailed Protocols/Techniques
  5.3.  Problem Statements and Concerns
  5.4.  No-Consensus-Reached Statements
6.  Follow-On Work
7.  IANA Considerations
8.  Security Considerations
9.  Informative References
Appendix A.  Program Committee
Appendix B.  Workshop Chairs
Appendix C.  Workshop Participants
IAB Members at the Time of Approval
Acknowledgments
Contributors
Authors' Addresses

1. Introduction 🦋

The IAB holds occasional workshops to think about long-term issues and strategies for the Internet and to suggest future directions for the Internet architecture. This long-term planning function is complementary to the ongoing engineering work done by working groups of the Internet Engineering Task Force (IETF) 🧠

The Measuring Network Quality for End-Users workshop [WORKSHOP] was held virtually by the Internet Architecture Board (IAB) on September 14-16, 2021. This report recaps the workshop, the topics discussed, and some preliminary conclusions drawn at the end of it ✨

1.1. Problem Space 🔍

The Internet in 2021 is lowkey a totally different place from 10 years ago 😭 It's now a crucial part of everyone's daily life. People use it for their social lives, their jobs, routine shopping, and keeping up with major events. More and more people can access a gigabit connection, which would've been hard to imagine a decade ago 🚀 Thanks to security improvements, people also trust it for banking, buying stuff, and paying bills every month 💸

At the same time, some parts of the end-user experience haven't glowed up as much 🥲 Lots of users still have typical connection latencies stuck at decade-old levels. Even though data centers have gotten way more reliable, end users still often see service interruptions. Despite algorithmic advances in control theory, queuing delays in last-mile equipment still exceed the accumulated transit delays. Transport improvements like QUIC, Multipath TCP, and TCP Fast Open still aren't fully supported in some networks. Privacy and security upgrades are also not widely supported, like encrypted DNS to the local resolver 🙈

Part of why progress has stalled is the popular idea that throughput is often the only measure of how good an Internet connection is. Because the focus is so narrow, the Measuring Network Quality for End-Users workshop set out to discuss several topics:

  • What is user latency under typical working conditions?
  • How reliable is connectivity across longer time periods?
  • Do networks allow the use of a broad range of protocols?
  • What services can be run by network clients?
  • What kind of IPv4, NAT, or IPv6 connectivity is offered, and are there firewalls?
  • What security mechanisms are available for local services, such as DNS?
  • To what degree are the privacy, confidentiality, integrity, and authenticity of user communications guarded?
  • Improving these aspects of network quality will likely depend on measuring and exposing metrics in a meaningful way to all involved parties, including end users. Measuring and exposing the right metrics will let service providers and network operators focus on their users' experience, and will also empower users to choose the Internet Service Providers (ISPs) that can deliver the best experience for their needs 💅
  • What are the fundamental properties of a network that contribute to a good user experience?
  • What metrics quantify these properties, and how can we collect such metrics in a practical way?
  • What are the best practices for interpreting those metrics and incorporating them into a decision-making process?
  • What are the best ways to communicate these properties to service providers and network operators?
  • How can these metrics be displayed to users in a meaningful way?

2. Workshop Agenda 📋

The Measuring Network Quality for End-Users workshop was divided into the following main topic areas. See further discussion in Sections 4 and 5 🌟

  • Introduction overviews and a keynote by Vint Cerf
  • Metrics considerations
  • Cross-layer considerations
  • Synthesis
  • Group conclusions

3. Position Papers ✨

okay so the workshop got a whole pile of position papers from attendees 🫶 the workshop's web page [WORKSHOP] has the archives of the papers, presentations, and recorded videos, so go binge them if you're into this stuff 🎀

  • Ahmed Aldabbagh. "Regulatory perspective on measuring network quality for end users" [Aldabbagh2021]
  • Al Morton. "Dream-Pipe or Pipe-Dream: What Do Users Want (and how can we assure it)?" [Morton2021]
  • Alexander Kozlov. "The 2021 National Internet Segment Reliability Research"
  • Anna Brunstrom. "Measuring network quality - the MONROE experience"
  • Bob Briscoe, Greg White, Vidhi Goel, and Koen De Schepper. "A Single Common Metric to Characterize Varying Packet Delay" [Briscoe2021]
  • Brandon Schlinker. "Internet Performance from Facebook's Edge" [Schlinker2019]
  • Christoph Paasch, Kristen McIntyre, Randall Meyer, Stuart Cheshire, and Omer Shapira. "An end-user approach to the Internet Score" [McIntyre2021]
  • Christoph Paasch, Randall Meyer, Stuart Cheshire, and Omer Shapira. "Responsiveness under Working Conditions" [Paasch2021]
  • Dave Reed and Levi Perigo. "Measuring ISP Performance in Broadband America: A Study of Latency Under Load" [Reed2021]
  • Eve M. Schooler and Rick Taylor. "Non-traditional Network Metrics"
  • Gino Dion. "Focusing on latency, not throughput, to provide better internet experience and network quality" [Dion2021]
  • Gregory Mirsky, Xiao Min, Gyan Mishra, and Liuyan Han. "The error performance metric in a packet-switched network" [Mirsky2021]
  • Jana Iyengar. "The Internet Exists In Its Use" [Iyengar2021]
  • Jari Arkko and Mirja Kuehlewind. "Observability is needed to improve network quality" [Arkko2021]
  • Joachim Fabini. "Network Quality from an End User Perspective" [Fabini2021]
  • Jonathan Foulkes. "Metrics helpful in assessing Internet Quality" [Foulkes2021]
  • Kalevi Kilkki and Benajamin Finley. "In Search of Lost QoS" [Kilkki2021]
  • Karthik Sundaresan, Greg White, and Steve Glennon. "Latency Measurement: What is latency and how do we measure it?"
  • Keith Winstein. "Five Observations on Measuring Network Quality for Users of Real-Time Media Applications"
  • Ken Kerpez, Jinous Shafiei, John Cioffi, Pete Chow, and Djamel Bousaber. "Wi-Fi and Broadband Data" [Kerpez2021]
  • Kenjiro Cho. "Access Network Quality as Fitness for Purpose"
  • Koen De Schepper, Olivier Tilmans, and Gino Dion. "Challenges and opportunities of hardware support for Low Queuing Latency without Packet Loss" [DeSchepper2021]
  • Kyle MacMillian and Nick Feamster. "Beyond Speed Test: Measuring Latency Under Load Across Different Speed Tiers" [MacMillian2021]
  • Lucas Pardue and Sreeni Tellakula. "Lower-layer performance not indicative of upper-layer success" [Pardue2021]
  • Matt Mathis. "Preliminary Longitudinal Study of Internet Responsiveness" [Mathis2021]
  • Michael Welzl. "A Case for Long-Term Statistics" [Welzl2021]
  • Mikhail Liubogoshchev. "Cross-layer Cooperation for Better Network Service" [Liubogoshchev2021]
  • Mingrui Zhang, Vidhi Goel, and Lisong Xu. "User-Perceived Latency to Measure CCAs" [Zhang2021]
  • Neil Davies and Peter Thompson. "Measuring Network Impact on Application Outcomes Using Quality Attenuation" [Davies2021]
  • Olivier Bonaventure and Francois Michel. "Packet delivery time as a tie-breaker for assessing Wi-Fi access points" [Michel2021]
  • Pedro Casas. "10 Years of Internet-QoE Measurements. Video, Cloud, Conferencing, Web and Apps. What do we Need from the Network Side?" [Casas2021]
  • Praveen Balasubramanian. "Transport Layer Statistics for Network Quality" [Balasubramanian2021]
  • Rajat Ghai. "Using TCP Connect Latency for measuring CX and Network Optimization" [Ghai2021]
  • Robin Marx and Joris Herbots. "Merge Those Metrics: Towards Holistic (Protocol) Logging" [Marx2021]
  • Sandor Laki, Szilveszter Nadas, Balazs Varga, and Luis M. Contreras. "Incentive-Based Traffic Management and QoS Measurements" [Laki2021]
  • Satadal Sengupta, Hyojoon Kim, and Jennifer Rexford. "Fine-Grained RTT Monitoring Inside the Network" [Sengupta2021]
  • Stuart Cheshire. "The Internet is a Shared Network" [Cheshire2021]
  • Toerless Eckert and Alex Clemm. "network-quality-eckert-clemm-00.4"
  • Vijay Sivaraman, Sharat Madanapalli, and Himal Kumar. "Measuring Network Experience Meaningfully, Accurately, and Scalably" [Sivaraman2021]
  • Yaakov (J) Stein. "The Futility of QoS" [Stein2021]

4. Workshop Topics and Discussion 💅

the three-day workshop had a four-part agenda, and each part helped frame the chats 🌸 it kicked off with intro and problem space presentations (Section 4.1), then moved on to metrics considerations (Section 4.2), cross-layer considerations (Section 4.3), and a synthesis discussion (Section 4.4). once those four subsections wrapped up, there was a follow-on discussion to draw conclusions the participants could agree on (Section 5). it's giving very organised 🧚

4.1. Introduction and Overviews 🦋

the workshop opened with a broad look at the state of user Quality of Service (QoS) and Quality of Experience (QoE) on the Internet today 👑 the goal of the intro talks was to set the scene by describing the problem space, the solutions that exist right now, and where those solutions fall short 💜

the intro presentations looked at existing QoS and QoE measurements and how well they actually work 🔥 they also talked about how multiple users interact inside the same network, and how multiple layers of the OSI stack interact with each other 🙈 Vint Cerf gave a keynote on the history and importance of the topic, and he is literally a legend ✨

4.1.1. Key Points from the Keynote by Vint Cerf 🌈

okay bestie, this part is so good 😭 Vint says we might be working in a networking space with super different parameters compared to 30 years ago, and that means we should rethink not just which metric matters more than the other, but the whole metaphor too 💅

it's time for the experts to look at more than just tweaking TCP 🎀 they should also explore other protocols, like what QUIC has been doing lately. we should totally feel free to consider alternatives to TCP. TCP is not a teddy bear 🧸 nobody should be scared to swap it out for a transport layer with better properties that serve its users better 🫶

he also suggested running exercises to figure out the desirable properties, as opposed to the fundamental properties, like low latency 🍓 as we look at the parametric spaces, we can pick out those desirable properties. his example comes from ARPA: you want to know where the missile is now, not where it was 🚀 understanding that drives which parameters get created and picked in the design space, iconic 👑

when the parameter values get pushed to the extreme, like connectivity, totally new designs pop up 🦋 one famous case study is the interplanetary protocol, where "ping" stops meaning anything useful 🌌 so while we look at responsiveness, we shouldn't ignore connectivity either 💖

sadly, keeping backward compatibility is painful 💀 the work on designing IPv6 so it could transition from IPv4 might have gone better if backward compatibility had been considered. it's too late for IPv6, but it's not too late to think about this for potential future problems ✨

IPv6 still isn't fully implemented everywhere 🙈 it's been a long road to deployment since work started in 1996, and we aren't there yet. back in 1996, people thought it would be pretty easy to implement IPv6, and that turned out not to be true 😭 also in 1996 the dot-com boom began, where tons of money got spent really fast, and the moment got missed while the market grew exponentially. this should be a cautionary tale 🚨

one last point: think about performance across multiple hops on the Internet 🌸 we haven't seen many end-to-end metrics, because building end-to-end measurements across different network and business boundaries is genuinely hard. a great question to ask when designing a new protocol is "will the new protocol work across multiple network hops?" 🧚

multi-hop networks are gradually being replaced by huge, flat networks with enough connectivity between operators that systems end up 1 hop, or 2 hops at most, away from each other (e.g., Google, Facebook, and Amazon) 🌐 the fundamental architecture of the Internet is changing, and that's giving main character energy ⭐

4.1.2. Introductory Talks ✨

okay so the internet is this shared network built on IP protocols, using packet switching to link up a bunch of autonomous networks together 🌐. Ditching circuit switching is literally why it could scale further than any other network design we knew about 📈. The catch? There's no in-network regulation, so it's really hard to make sure every single user gets the best experience 🥲.

As internet use cases keep expanding, it's getting harder to predict which network characteristics actually line up with good user experiences 🔮. Different application classes, like video streaming and teleconferencing, can affect experience in ways that are complicated and hard to measure. And internet usage shifts super fast across the day, week, and year, which makes it even trickier to find the key metrics that predict a good experience 🙈.

QoS initiatives tried to fix this by strictly prioritizing different types of traffic 🚦. But QoS metrics don't always match up with user experience, and the usefulness of a QoS metric is limited even more by how hard it is to build solutions with the QoS characteristics you want 💀.

QoE initiatives tried to bring in the psychology of how quality is perceived, and built statistical models to optimize user experience 🧠. Despite all that modeling, QoE worked well for certain application classes. Sadly, generalizing the models was really hard, and how different applications affect each other when they share the same network is still an open problem 🫠.

The industry put its focus on giving end users more throughput and bandwidth, and it led to remarkable advances 🚀. In lots of places around the world, a home user gets gigabit speeds to their ISP, which is so wild that it would've been brushed off as science fiction a decade ago 🛸. But that focus on capacity came at the expense of neglecting another important core metric: latency ⏱️. So users whose experience suffered from high latency were told to upgrade their equipment to get more throughput instead. [MacMillian2021] showed that sometimes such an upgrade can lead to latency improvements, due to the economical reasons of overselling the "value-priced" data plans 💸.

As the industry kept giving end users more throughput while mostly ignoring latency, application designs started using all sorts of latency and short service disruption hiding techniques 🫣. For example, a user's web browser performance is closely tied to what's sitting in the browser's local cache 💻. These techniques can clearly improve the user experience when using stale data is possible, but this development decouples user experience even further from core metrics 🧵.

In the most recent 10 years, efforts by Dave Taht and the bufferbloat society have made significant progress in updating queuing algorithms to reduce latencies under load, compared to simpler FIFO queues 👑. Unfortunately, the home router industry hasn't implemented these algorithms yet, mostly due to marketing and cost concerns 😤. Most home router manufacturers depend on System on a Chip (SoC) acceleration to create products with a desired throughput. SoC manufacturers opt for simpler algorithms and aggressive aggregation, reasoning that a higher-throughput chip will have guaranteed demand 📦. Because consumers are mostly offered choices among different high-throughput devices, the perception that higher throughput means higher QoS keeps getting stronger 💅.

The home router isn't the only place that could benefit from clearer indications of acceptable performance for users 🏠. Since users see the Internet through the lens of applications, it's important that we call on application vendors to adopt solutions that stress lower latencies 🫶. Unfortunately, while bandwidth is straightforward to measure, responsiveness is trickier 🎯. Many applications have found a set of metrics that are helpful in their own realm, but those metrics don't generalize well and can't become universally applicable 🌀. And because the application space is so competitive, vendors may have economic reasons to keep their most useful metrics to themselves 🔒.

4.1.3. Introductory Talks - Key Points 🗝️

  1. Measuring bandwidth is necessary but is not alone sufficient. 💫
  2. In many cases, Internet users don't need more bandwidth but rather need "better bandwidth", i.e., they need other connectivity improvements. 🌟
  3. Users perceive the quality of their Internet connection based on the applications they use, which are affected by a combination of factors. There's little value in exposing a typical user to the entire spectrum of possible reasons for the poor performance perceived in their application-centric view. 🔍
  4. Many factors affecting user experience are outside the users' sphere of control. It's unclear whether exposing users to these other factors will help them understand the state of their network performance. In general, users prefer simple, categorical choices (e.g., "good", "better", and "best" options). 🎀
  5. The Internet content market is highly competitive, and many applications develop their own "secret sauce". 🤫

4.2. Metrics Considerations 📊

In the second agenda section, the workshop continued its discussion about metrics that can be used instead of or in addition to available bandwidth. Several workshop attendees presented deep-dive studies on measurement methodology 🧪.

4.2.1. Common Performance Metrics 💅

Okay bestie, losing Internet access entirely is the absolute worst user experience 💀. Unfortunately, unless rebooting the home router brings connectivity back, there's basically nothing a user can do except contact their service provider 😭. Still, collecting availability metrics on the client side is worth it ✨. These can help the user's ISP localize and resolve issues faster, and they let users choose between ISPs more wisely 🧚. You can measure availability directly by simply attempting connections from the client side to distant locations of interest. For example, Ookla's [Speedtest] uses a large number of Android devices to measure network and cellular availability around the globe. Ookla collects hundreds of millions of data points per day and uses them for accurate availability reporting 🔥. Another approach is to derive availability from the failure rates of other tests. For example, [FCC_MBA] and [FCC_MBA_methodology] use thousands of off-the-shelf routers, with measurement software developed by [SamKnows]. These routers run an array of network tests and report availability based on whether the test connections succeeded or not 🦋.

Measuring available capacity helps end users, but it's even more valuable for service providers and application developers 🌸. High-definition video streaming needs significantly more capacity than any other type of traffic. At the time of the workshop, video traffic made up 90% of overall Internet traffic and contributed to 95% of the revenues from monetization (via subscriptions, fees, or ads) 👑. So video streaming services, such as Netflix, have to keep coping with rapid changes in available capacity 🙈. Measuring capacity in real time lets them use the different adaptive bitrate (ABR) compression algorithms to ensure the best possible user experience. Measuring aggregated capacity demand lets ISPs get ready for traffic spikes 📈. For example, during the end-of-year holiday season, global demand for capacity has been shown to be 5-7 times higher than during other seasons 🎄. For end users, knowing their capacity needs can help them pick the best data plan for what they actually do online. In many cases, though, end users already have more than enough capacity, and adding more bandwidth won't improve their experience. After a point, capacity is no longer the limiting factor in user experience 💖. Finally, being able to tell the "throughput" apart from the "goodput" can help you spot when the network is saturated 🧐.

In measuring network quality, latency is defined as the time it takes a packet to traverse a network path from one end to the other ⏱️. At the time of this report, users in many places around the world can enjoy Internet access with high enough capacity and availability for their current needs 🌈. For these users, latency improvements, rather than bandwidth improvements, can lead to the biggest QoE gains 💜. The established latency metric is round-trip time (RTT), usually measured in milliseconds. But users often find RTT values unintuitive. Unlike other performance metrics, a high RTT means poor latency, and people usually assume higher scores are better 🙃. To fix that vibe, [Paasch2021] and [Mathis2021] present an inverse metric called "Round-trips Per Minute" (RPM) 🎀.

There's an important difference between "idle latency" and "latency under working conditions" 👀. The former is measured when the network is underused and reflects a best-case scenario. The latter is measured under a typical workload. Until recently, typical tools reported idle latency, which can be seriously misleading 😬. For example, data presented at the workshop shows that idle latencies can be up to 25 times lower than the latency under typical working loads. That's why it's essential to clearly separate the two when presenting latency to end users 🫶.

Data shows that rapid changes in capacity affect latency 📉. [Foulkes2021] attempts to quantify how often a rapid change in capacity can make network connectivity "unstable" (i.e., high latency with very little throughput) 😵. Such changes can be caused by infrastructure failures, but they're way more often caused by in-network phenomena, like changing traffic engineering policies or rapid changes in cross-traffic 🕸️.

Data presented at the workshop shows that 36% of measured lines have capacity metrics that vary by more than 10% throughout the day and across multiple days 🤯. Lots of variables cause this, including local connectivity methods (Wi-Fi vs. Ethernet), competing LAN traffic, device load/configuration, time of day, and local loop/backhaul capacity. These factor variations make it hard to measure capacity using only an end-user device or other end-network measurement 😩. A network router that sees aggregated traffic from multiple devices has a better vantage point for capacity measurements 🏠. Such a test can account for the totality of local traffic and run an independent capacity test. Still, various factors might limit the accuracy of such a test. Accurate capacity measurement requires multiple samples 📊.

Users perceive the Internet through the lens of applications, so it can be tricky to connect changes in capacity and latency to the quality of the end-user experience 🔍. For example, web browsers rely on cached page versions to shorten page load times and soften connectivity losses. Social networking apps often prefetch their "feed" items too 📱. These techniques make the core in-network metrics less indicative of what users actually experience, which means you have to collect data from the end-user applications themselves 💅.

It helps to tell apart applications that operate on a "fixed latency budget" from those that tolerate latency variance better 🎮. Cloud gaming is an example of an app that needs a "fixed latency budget", since one sudden latency spike can decide the "win/lose" ratio for a player 😤. Companies competing in the lucrative cloud gaming market make big infrastructure investments, like building entire data centers closer to their users. Those data centers show the economic benefit of having fewer latency spikes outweighs the deployment costs 💸. On the other hand, applications that handle latency spikes better can keep running reasonably well through short spikes 🌊. Even so, those apps can still benefit from consistently low latency, depending on how usage shifts. For example, Video-on-Demand (VOD) apps can work reasonably well when a video is watched linearly. But once the user tries to "switch a channel" or to "skip ahead", the experience suffers unless latency is sufficiently low 📺.

Finally, as applications keep evolving, in-application metrics are becoming more important 🚀. For example, VOD apps can assess QoE with application-specific metrics, like whether the video player is able to use the highest possible resolution, or whether the video is smooth or freezing 🎬. Developers can then use these metrics to prioritize future work ✅. All the popular video platforms (YouTube, Instagram, Netflix, and others) have built frameworks to collect and analyze VOD metrics at scale. One example is the Scuba framework used by Meta [Scuba] 🪩.

In-application metrics can be tough to use for comparative research, though 🥲. First, different applications often use different metrics to measure the same thing. For example, application A might measure video smoothness via "mean time to rebuffer", while application B relies on the "probability of rebuffering per second" for the same purpose 🤷‍♀️. Second, VOD is a major revenue source for companies like YouTube, Facebook, and Netflix, which gives them a proprietary incentive not to exchange in-application data 🔒. Last but not least, there's a privacy concern: in-application metrics can accurately describe an individual end user's activities and preferences 🙈🫣.

4.2.2. Availability Metrics 🛜

Availability is simply defined as whether or not a packet can be sent and then received by its intended recipient 📦. It's naively thought to be the simplest thing to measure, but it gets more complicated once you realize that continual, instantaneous measurements would be needed to catch the tiniest outages 😵‍💫. It's also hard to figure out the root cause of infallibility 🫠: was the user's line down, was something in the middle of the network, or was it the service the user was trying to reach? 🤔

4.2.3. Capacity Metrics 🧃

If network capacity doesn't meet user demands, network quality takes a hit 📉. Once capacity meets the demands, increasing capacity won't lead to further quality improvements 🙌.

The actual network connection capacity is determined by the equipment and the lines along the network path, and it varies throughout the day and across multiple days ⚡. Studies involving DSL lines in North America indicate that over 30% of DSL lines have capacity metrics that vary by more than 10% throughout the day and across multiple days 📊.

Some factors that affect actual capacity are:

  1. Competing traffic, either in the LAN or in the WAN environments. In the LAN setting, the competing traffic comes from the multiple devices sharing the Internet connection. In the WAN setting, it often comes from unrelated network flows that just happen to share the same network path 🏘️.
  2. Capabilities of the equipment along the path of the network connection, including the data transfer rate and the amount of memory used for buffering 🧠.
  3. Active traffic management measures, such as traffic shapers and policers that network providers often use 🚦.

There are other factors that can negatively affect actual line capacities too 🫤.

User demands follow the usage patterns and preferences of particular users 🥰. For example, large data transfers can soak up any available capacity, while media streaming apps need limited capacity to work correctly. Videoconferencing apps typically need less capacity than high-definition video streaming 🎥✨.

4.2.4. Latency Metrics 🦋

Okay so end-to-end latency is how long one packet takes to travel the network path from the user to their destination and then all the way back 🚀. It's not just one thing either. It's made up of several components, and they're all lowkey important 💅

  1. The propagation delay, which reflects the path distance and the individual link technologies (e.g., fiber vs. satellite). The propagation doesn't depend on the utilization of the network, to the extent that the network path remains constant.
  2. The buffering delay, which reflects the time segments spent in the memory of the network equipment that connect the individual network links, as well as in the memory of the transmitting endpoint. The buffering delay depends on the network utilization, as well as on the algorithms that govern the queued segments.
  3. The transport protocol delays, which reflect the time spent in retransmission and reassembly, as well as the time spent when the transport is "head-of-line blocked" 😭.
  4. Application delay, which some of the workshop submissions explicitly called out. It reflects the inefficiencies in the application layer 🙈.

Typically, end-to-end latency is measured when the network is idle. Results from those measurements mostly reflect the propagation delay but not the other kinds of delay 🌸. This report uses the term "idle latency" to refer to results achieved under idle network conditions.

Alternatively, if the latency is measured when the network is under its typical working conditions, the results reflect multiple types of delays ✨. This report uses the term "working latency" to refer to such results. Other sources use the term "latency under load" (LUL) as a synonym.

Data presented at the workshop revealed a substantial difference between idle latency and working latency 👑. Depending on the traffic direction and the technology type, the working latency is between 6 to 25 times higher than the idle latency. Iconic behavior, honestly 💀.

+============+============+========+=========+============+=========+
| Direction  | Technology |Working | Idle    | Working -  |Working /|
|            | Type       |Latency | Latency | Idle       |Idle     |
|            |            |        |         | Difference |Ratio    |
+============+============+========+=========+============+=========+
| Downstream | FTTH       |148     | 10      | 138        |15       |
+------------+------------+--------+---------+------------+---------+
| Downstream | Cable      |103     | 13      | 90         |8        |
+------------+------------+--------+---------+------------+---------+
| Downstream | DSL        |194     | 10      | 184        |19       |
+------------+------------+--------+---------+------------+---------+
| Upstream   | FTTH       |207     | 12      | 195        |17       |
+------------+------------+--------+---------+------------+---------+
| Upstream   | Cable      |176     | 27      | 149        |6        |
+------------+------------+--------+---------+------------+---------+
| Upstream   | DSL        |686     | 27      | 659        |25       |
+------------+------------+--------+---------+------------+---------+

                              Table 1

While historically the tooling for measuring latency focused on idle latency, the industry is now trending toward measuring working latency as well, e.g., Apple's [NetworkQuality] 📊.

4.2.5. Measurement Case Studies 🎀

The participants proposed several concrete methodologies for measuring network quality for end users, and some of them are really giving 🔥.

[Paasch2021] introduced a methodology for measuring working latency from the end-user vantage point. The suggested method incrementally adds network flows between the user device and a server endpoint until a bottleneck capacity is reached. From these measurements, a round-trip latency is measured and reported to the end user. The authors chose to report results with the RPM metric. The methodology had been implemented in Apple's macOS Monterey 🍓.

[Mathis2021] applied the RPM metric to the results of more than 4 billion download tests that M-Lab performed from 2010-2021. During this time frame, the M-Lab measurement platform underwent several upgrades that allowed the research team to compare the effect of different TCP congestion control algorithms (CCAs) on the measured end-to-end latency. The study showed that the use of cubic CCA leads to increased working latency, which is attributed to its use of larger queues 🧚.

[Schlinker2019] presented a large-scale study that aimed to establish a correlation between goodput and QoE on a large social network. The authors performed the measurements at multiple data centers from which video segments of set sizes were streamed to a large number of end users. The authors used the goodput and throughput metrics to determine whether particular paths were congested.

[Reed2021] presented the analysis of working latency measurements collected as part of the Measuring Broadband America (MBA) program by the Federal Communication Commission (FCC). The FCC does not include working latency in its yearly report, but it does offer it in the raw data files. The authors used a subset of the raw data to identify important differences in the working latencies across different ISPs 🔍.

[MacMillian2021] presented analysis of working latency across multiple service tiers. They found that, unsurprisingly, "premium" tier users experienced lower working latency compared to a "value" tier. The data demonstrated that working latency varies significantly within each tier. One possible explanation is the difference in equipment deployed in the homes 🏠.

These studies have stressed the importance of measuring working latency 🫶. At the time of this report, many home router manufacturers rely on hardware-accelerated routing that uses FIFO queues. Focusing on measuring working latency on these devices, and making the consumer aware of the effect of choosing one manufacturer vs. another, can help improve the home router situation. The ideal test would be able to identify the working latency and pinpoint the source of the delay (home router, ISP, server side, or some network node in between). Main character energy for the measurement world 👑.

Another source of high working latency comes from network routers exposed to cross-traffic. As [Schlinker2019] indicated, these can become saturated during the peak hours of the day 😩. Systematic testing of working latency in routers under load can help improve both our understanding of latency and the impact of deployed infrastructure.

4.2.6. Metrics Key Points 🌈

The metrics for network quality can be roughly grouped into the following:

  1. Availability metrics, which indicate whether the user can access the network at all.
  2. Capacity metrics, which indicate whether the actual line capacity is sufficient to meet the user's demands.
  3. Latency metrics, which indicate if the user gets the data in a timely fashion.
  4. Higher-order metrics, which include both the network metrics, such as inter-packet arrival time, and the application metrics, such as the mean time between rebuffering for video streaming.

The availability metrics can be seen as a derivative of either the capacity (zero capacity leading to zero availability) or the latency (infinite latency leading to zero availability). Math that makes total sense 🧠.

Key points from the presentations and discussions included the following:

  1. Availability and capacity are "hygienic factors" -- unless an application is capable of using extra capacity, end users will see little benefit from using over-provisioned lines 🧼.
  2. Working latency has a stronger correlation with the user experience than latency under an idle network load. Working latency can exceed the idle latency by order of magnitude 😭.
  3. The RPM metric is a stable metric, with positive values being better, that may be more effective when communicating latency to end users ✨.
  4. The relationship between throughput and goodput can be effective in finding the saturation points, both in client-side [Paasch2021] and server-side [Schlinker2019] settings 🎯.
  5. Working latency depends on the algorithm choice for addressing endpoint congestion control and router queuing 💜.

Finally, it was commonly agreed to that the best metrics are those that are actionable 💅.

4.3. Cross-Layer Considerations 🧩

In the cross-layer segment of the workshop, participants presented material on, and discussed, how to accurately measure exactly where problems occur. Discussion centered especially on the differences between physically wired and wireless connections, and on the difficulties of accurately determining problem spots when multiple different types of network segments are responsible for the quality. As an example, [Kerpez2021] showed that a limited bandwidth of 2.4 Ghz Wi-Fi bottlenecks the most frequently. In comparison, the wider bandwidth of the 5 Ghz Wi-Fi has only bottlenecked in 20% of observations 📶.

The participants agreed that no single component of a network connection has all the data required to measure the effects of network performance on the quality of the end-user experience 🥺.

  • Applications that are running on the end-user devices have the best insight into their respective performance, but they have limited visibility into the behavior of the network itself and are unable to act based on their limited perspective.
  • ISPs have good insight into QoS considerations, but they are not able to infer the effect of the QoS metrics on the quality of end-user experiences.
  • Content providers have good insight into the aggregated behavior of the end users, but they lack the insight on what aspects of network performance are leading indicators of user behavior.

The workshop had identified the need for a standard and extensible way to exchange network performance characteristics. Such an exchange standard should address (at least) the following:

  • A scalable way to capture the performance of multiple (potentially thousands of) endpoints.
  • The data exchange format should prevent data manipulation so that the different participants won't be able to game the mechanisms.
  • Preservation of end-user privacy. In particular, federated learning approaches should be preferred so that no centralized entity has access to the whole picture 🔐.
  • A transparent model for giving the different actors on a network connection an incentive to share the performance data they collect.
  • An accompanying set of tools to analyze the data 🛠️.

4.3.1. Separation of Concerns 💅

Okay so, collecting performance metrics, interpreting them, and acting on the interpretation are usually stuck together super tightly. Unfortunately, that setup is not the best for swapping cross-layer data, bestie 😭

  • actors that can collect particular performance metrics (e.g., the TCP RTT) don't necessarily have the context needed to interpret them properly 🙈
  • the actors that have the context and the computing/storage power to interpret metrics don't necessarily have the ability to control how the network or application behaves 💀
  • the actors that can control how networks and/or applications behave usually don't have access to complete measurement data 😤

The participants agreed that it's important to split those three things apart, so that:

  • the different actors that have the data, but can't interpret and/or act on it, should publish their measured data 📊 and
  • the actors with the expertise to interpret and synthesize performance data should publish the results of their interpretations 🧠✨

4.3.2. Security and Privacy Considerations 🔒

Protecting the privacy of Internet end users is honestly a hard requirement to meet in this problem space 🥺. There's an intrinsic trade-off here: the more data you collect about user activities, the more you risk infringing on their privacy while you do it. The participants agreed that seeing across multiple layers is necessary for an accurate measurement of network quality, but doing that in a way that minimizes privacy leakage is still an open question 🌸.

4.3.3. Metric Measurement Considerations 📏

  • The following TCP protocol metrics have been found to be effective and are available for passive measurement: 🎀
    • TCP connection latency measured using selective acknowledgment (SACK) or acknowledgment (ACK) timing, as well as the timing between TCP retransmission events, are good proxies for end-to-end RTT measurements 💖.
    • On the Linux platform, the tcp_info structure is the de facto standard for an application to inspect the performance of kernel-space networking. However, there is no equivalent de facto standard for user-space networking 🐧.
  • The QUIC and MASQUE protocols make passive performance measurements more challenging 😩.
    • An approach that uses federated measurement/hierarchical aggregation may be more valuable for these protocols 🧚.
    • The QLOG format seems to be the most mature candidate for such an exchange 👑.

4.3.4. Towards Improving Future Cross-Layer Observability 🔭

Okay, the ownership of the Internet is spread across multiple administrative domains, so measuring end-to-end performance data is genuinely difficult 😵‍💫. Plus, the immense scale of the Internet makes aggregating and analyzing all of that even harder. [Marx2021] presented a simple logging format that could potentially be used to collect and aggregate data from different layers 🦋.

Another thing hurting cross-layer measurement is that most current algorithms don't explicitly provide performance data that can be used in cross-layer analysis 🙃. The IETF community could be more diligent about identifying each protocol's key performance indicators and exposing them as part of the protocol specification 📝✨.

Despite all these challenges, it should still be possible to run limited-scope studies to better understand how user quality is affected by the interaction of the different components that make up the Internet 🌈. Also, recent development of federated learning algorithms suggests it might be possible to perform cross-layer performance measurements while preserving user privacy 🫶. Iconic, honestly 🥹.

4.3.5. Efficient Collaboration between Hardware and Transport Protocols 🛜

Ngl, with the arrival of low latency, low loss, and scalable throughput (L4S) congestion notification and control, there's an even bigger need for transport protocols and the underlying hardware to work in unison 🎶.

At the time of the workshop, the typical home router used a single FIFO queue that was large enough to amortize the lower-layer header overhead across multiple transport PDUs 📦. These designs worked well with the cubic congestion control algorithm, but the newer generation of algorithms can operate on much smaller queues 🏃‍♀️. To fully support latencies less than 1 ms, the home router needs to work efficiently on sequential transmissions of just a few segments, rather than being optimized for large packet bursts ⚡.

Another design trait common in home routers is packet aggregation, which further amortizes the overhead added by the lower-layer headers 🍓. Specifically, multiple IP datagrams are combined into a single, large transfer frame. However, this aggregation can add up to 10 ms to the packet sojourn delay 😬.

Following the famous "you can't improve what you don't measure" adage 💜, it's important to expose these aggregation delays in a way that lets people identify the source of the bottlenecks and make hardware more suitable for the next generation of transport protocols 🚀.

4.3.6. Cross-Layer Key Points 🗝️

  • There are significant differences in the characteristics of metrics to be measured, and in the optimizations required, between wireless and wired networks 📡.
  • Identifying an issue's root cause is hampered by the challenges of measuring multi-segment network paths 🕵️‍♀️.
  • No single component of a network connection has all the data required to measure the effects of the complete network performance on the quality of the end-user experience 🥲.
  • Actionable results require both proper collection and interpretation 🎯.
  • Coordination among network providers is important for successfully improving the measurement of end-user experiences 🤝.
  • Providing accurate measurements while preserving end-user privacy at the same time is challenging 🙈.
  • Passive measurements from protocol implementations may provide beneficial data 💅.

4.4. Synthesis 🎀

Finally, in the synthesis section of the workshop, the presentations and discussions focused on the next steps likely needed to make forward progress 🏁. Of particular concern is how to bring forward measurements that make sense to end users trying to choose between various networking subscription options 🛒.

4.4.1. Measurement and Metrics Considerations 📈

One important consideration is how decisions can be made, and what actions can be taken, based on collected metrics 🤔. Measurements must be integrated with applications in order to get true application views of congestion, because measurements over different infrastructure or via other applications may return incorrect results 🙅‍♀️. Congestion itself can be a temporary problem, and mitigation strategies may need to be different depending on whether it's expected to be a short-term or long-term phenomenon ⏳. A significant challenge exists in measuring short-term problems, which drives the need for continuous measurements to make sure critical moments and long-term trends are captured 📹. For short-term problems, workshop participants debated whether an issue that goes away is really a problem, or a sign that a network is properly adapting and self-recovering 🌱.

Important consideration must be taken when constructing metrics in order to understand the results 🔍. Measurements can also be affected by individual packet characteristics: differently sized packets typically have a linear relationship with their delay 📐. With this in mind, measurements can be divided into a delay based on geographical distances, a packet-size serialization delay, and a variable (noise) delay 🌍. Each of these three sub-component delays can be different and individually measured across each segment in a multi-hop path 🧩. Variable delay can also be significantly impacted by external factors, such as bufferbloat, routing changes, network load sharing, and other local or remote changes in performance 🌪️. Network measurements, especially load-specific tests, must also be run long enough to make sure any problems associated with buffering, queuing, etc. are captured ⏰. Measurement technologies should also distinguish between upstream and downstream measurements, as well as measure the difference between end-to-end paths and sub-path measurements 💫.

4.4.2. End-User Metrics Presentation 🎀

okay so figuring out what end users actually need takes real, informative measurements and metrics 📊 How do we give people the service they need or want? Can users even explain what they want clearly? 🤔 In end-user surveys, the answers are usually super high-level stuff like "reliability", "capacity", and "service bundling" 💅 Meanwhile, the technical requirements operators can work with, like "low-latency" and "congestion avoidance", aren't words end users know or use. Not the vibe for them, bestie 💀

Some metrics end users would actually care about are the number of users a service supports and the number of applications or streams a network can handle ✨ One possible fix for networking issues is incentive-based traffic management. For example, an application that asks for lower latency may also mean accepting lower bandwidth 🦋 Now, user-perceived latency must be considered, not just network latency 🙈 Measuring user experience from in-application to in-server latency, or network-to-network, may only be looking at the lowest-level latency, which is lowkey not the whole story. So picking the right protocol for a measurement is critical if you want to match what users actually experience. For instance, users do not transmit data over ICMP, even though it's a common measurement tool 😭

In-application measurements should think about how to handle different kinds of apps, like video streaming, file sharing, multi-user gaming, and real-time voice communications 🎮 It could be super helpful to ask users what trade-offs they're willing to accept. Would they rather have a network with low latency or one with higher bandwidth? Gamers might choose differently than home office users or content producers, for example 🌈

And then there's the question of how users can make these trade-offs fairly, without messing things up for other users 🫶 There's a real tension between solutions in this space and what they cost, plus which customers are actually willing to pay for these improvements.

Getting higher-priority traffic to users is tricky too. The challenge is getting networks to be willing to listen to client requests for higher incentives, even though commercial interests may not flow to them without a cost incentive 💸 Shared mediums are generally subject to oversubscribing. That means the number of users a network can support is either accurate on an underutilized network, or it assumes an average bandwidth or other usage metric that stops being accurate during utilization spikes 📉 Individual metrics are also affected by in-home devices, from cheap routers all the way to microwaves 🍓 and by (multi-)user behaviors during tests. So a single metric alone, or a single reading without context, may not help a user or operator figure out where the problem actually is. Iykyk 🧚

User comprehension of a network is still a really hard problem 🥺 Multiple workshop participants argued for a single number, possibly calculated with a weighted aggregation formula, or for a small number of measurements per expected usage (for example, a "gaming" score vs. a "content producer" score). Many agreed that some users may prefer simplified or color-coded ratings, like good/better/best, red/yellow/green, or bronze/gold/platinum 🥇

4.4.3. Synthesis Key Points ⭐

  • Some proposed metrics:
    • Round-trips Per Minute (RPM)
    • users per network
    • latency
    • 99% latency and bandwidth
  • Median and mean measurements are distractions from the real problems. 🙄
  • Shared network usage greatly affects quality. 📶
  • Long measurements are needed to capture all facets of potential network bottlenecks. ⏳
  • Better-funded research in all these areas is needed for progress. 💜
  • End users will best understand a simplified score or ranking system. 🏆

5. Conclusions 👑

During the final hour of the three-day workshop, the group collected the statements they considered summary statements. Later on, any statements that were still in contention got discarded (they're listed further below for completeness 🫣). For this document, the authors took the original list, sorted it into rough categories, applied some suggested edits from the mailing list, and then edited further for clarity and to add context ✨

5.1. General Statements 💖

  1. Bandwidth is necessary but not alone sufficient. 💅
  2. In many cases, Internet users don't need more bandwidth but rather need "better bandwidth", i.e., they need other improvements to their connectivity. 🫶
  3. We need both active and passive measurements -- passive measurements can provide historical debugging. 🔍
  4. We need passive measurements to be continuous, archivable, and queriable, including reliability/connectivity measurements. 📚
  5. A really meaningful metric for users is whether their application will work properly or fail because of a lack of a network with sufficient characteristics. 🥹
  6. A useful metric for goodness must actually incentivize goodness -- good metrics should be actionable to help drive industries towards improvement. 🚀
  7. A lower-latency Internet, however achieved, would benefit all end users. 🌸

5.2. Specific Statements about Detailed Protocols/Techniques 🧪

  1. Round-trips Per Minute (RPM) is a useful, consumable metric. 💜
  2. We need a usable tool that fills the current gap between network reachability, latency, and speed tests. 🛠️
  3. End users that want to be involved in QoS decisions should be able to voice their needs and desires. 🎀
  4. Applications are needed that can perform and report good quality measurements in order to identify insufficient points in network access. 📱
  5. Research done by regulators indicate that users/consumers prefer a simple metric per application, which frequently resolves to whether the application will work properly or not. 🙌
  6. New measurements and QoS or QoE techniques should not rely only or depend on reading TCP headers. 🙅‍♀️
  7. It is clear from developers of interactive applications and from network operators that lower latency is a strong factor in user QoE. However, metrics are lacking to support this statement directly. 🧐

5.3. Problem Statements and Concerns 😤

  1. Latency mean and medians are distractions from better measurements. 📉
  2. It is frustrating to only measure network services without simultaneously improving those services. 😩
  3. Stakeholder incentives aren't aligned for easy wins in this space. Incentives are needed to motivate improvements in public network access. Measurements may be one step towards driving competitive market incentives. 🏁
  4. For future-proof networking, it is important to measure the ecological impact of material and energy usage. 🌍
  5. We do not have incontrovertible evidence that any one metric (e.g., latency or speed) is more important than others to persuade device vendors to concentrate on any one optimization. 🤷‍♀️

5.4. No-Consensus-Reached Statements 🫠

Additional statements were discussed and recorded that did not have the group's consensus at the time, but they're listed here for completeness:

  1. We do not have incontrovertible evidence that bufferbloat is a prevalent problem. 🤔
  2. The measurement needs to support reporting localization in order to find problems. Specifically:
    • Detecting a problem is not sufficient if you can't find the location.
    • Need more than just English -- different localization concerns.
  3. Stakeholder incentives aren't aligned for easy wins in this space. 💀

6. Follow-On Work 🔭

There was discussion during the workshop about where future work should happen. The group agreed that some work could be done more immediately within existing IETF working groups (e.g., IPPM, DetNet, and RAW), while other longer-term research may be needed in IRTF groups 🌱

7. IANA Considerations 📝

This document has no IANA actions. ✅

8. Security Considerations 🔐

A few security-relevant topics came up at the workshop, including but not limited to:

  • what prioritization techniques can work without invading the privacy of the communicating parties and
  • how oversubscribed networks can essentially be viewed as a DDoS attack. 😱

9. Informative References ✨

okay bestie, the reference list is the big bibliography in the back, and it's giving a lot of IAB workshop papers 📚🦋 I've kept every label, author, title, date and link exactly as the original has them, typos included 🫶

Appendix A. Program Committee 💜

The program committee consisted of:

  • Jari Arkko
  • Olivier Bonaventure
  • Vint Cerf
  • Stuart Cheshire
  • Sam Crowford
  • Nick Feamster
  • Jim Gettys
  • Toke Hoiland-Jorgensen
  • Geoff Huston
  • Cullen Jennings
  • Katarzyna Kosek-Szott
  • Mirja Kühlewind
  • Jason Livingood
  • Matt Mathis
  • Randall Meyer
  • Kathleen Nichols
  • Christoph Paasch
  • Tommy Pauly
  • Greg White
  • Keith Winstein

Appendix B. Workshop Chairs 👑

The workshop chairs consisted of:

  • Wes Hardaker
  • Evgeny Khorov
  • Omer Shapira

Okay bestie, we're at the very end of RFC 9318 and it's time for the guest list! 🎀✨

Appendix C. Workshop Participants 🌸

The following is a list of participants who attended the workshop over a remote connection:

   Ahmed Aldabbagh
   Jari Arkko
   Praveen Balasubramanian
   Olivier Bonaventure
   Djamel Bousaber
   Bob Briscoe
   Rich Brown
   Anna Brunstrom
   Pedro Casas
   Vint Cerf
   Stuart Cheshire
   Kenjiro Cho
   Steve Christianson
   John Cioffi
   Alexander Clemm
   Luis M. Contreras
   Sam Crawford
   Neil Davies
   Gino Dion
   Toerless Eckert
   Lars Eggert
   Joachim Fabini
   Gorry Fairhurst
   Nick Feamster
   Mat Ford
   Jonathan Foulkes
   Jim Gettys
   Rajat Ghai
   Vidhi Goel
   Wes Hardaker
   Joris Herbots
   Geoff Huston
   Toke Høiland-Jørgensen
   Jana Iyengar
   Cullen Jennings
   Ken Kerpez
   Evgeny Khorov
   Kalevi Kilkki
   Joon Kim
   Zhenbin Li
   Mikhail Liubogoshchev
   Jason Livingood
   Kyle MacMillan
   Sharat Madanapalli
   Vesna Manojlovic
   Robin Marx
   Matt Mathis
   Jared Mauch
   Kristen McIntyre
   Randall Meyer
   François Michel
   Greg Mirsky
   Cindy Morgan
   Al Morton
   Szilveszter Nadas
   Kathleen Nichols
   Lai Yi Ohlsen
   Christoph Paasch
   Lucas Pardue
   Tommy Pauly
   Levi Perigo
   David Reed
   Alvaro Retana
   Roberto
   Koen De Schepper
   David Schinazi
   Brandon Schlinker
   Eve Schooler
   Satadal Sengupta
   Jinous Shafiei
   Shapelez
   Omer Shapira
   Dan Siemon
   Vijay Sivaraman
   Karthik Sundaresan
   Dave Taht
   Rick Taylor
   Bjørn Ivar Teigen
   Nicolas Tessares
   Peter Thompson
   Balazs Varga
   Bren Tully Walsh
   Michael Welzl
   Greg White
   Russ White
   Keith Winstein
   Lisong Xu
   Jiankang Yao
   Gavin Young
   Mingrui Zhang

IAB Members at the Time of Approval 💜

Internet Architecture Board members at the time this document was approved for publication were:

   Jari Arkko
   Deborah Brungard
   Lars Eggert
   Wes Hardaker
   Cullen Jennings
   Mallory Knodel
   Mirja Kühlewind
   Zhenbin Li
   Tommy Pauly
   David Schinazi
   Russ White
   Qin Wu
   Jiankang Yao

Acknowledgments 🫶

The authors would like to thank the workshop participants, the members of the IAB, and the program committee for creating and participating in many interesting discussions. So sweet of them, right? 🥹

Contributors 🍓

Thank you to the people that contributed edits to this document:

   Erik Auerswald
   Simon Leinen
   Brian Trammell

Authors' Addresses 🦋

   Wes Hardaker
   Email: ietf@hardakers.net


   Omer Shapira
   Email: omer_shapira@apple.com

That's the whole RFC, bestie!! 🌈💅 Every single workshop participant, IAB member, and contributor is written out above exactly as the original has them. Ily for reading all nine parts with me. 💖🧚

📦 where this page comes from