RFC 9506
Explicit Host-to-Network Flow Measurements Techniques
RFC 9506 🎀 Explicit Host-to-Network Flow Measurements Techniques
okay bestie, RFC 9506 is an Informational doc, so it's published for info and isn't a Standards Track spec 🧚. It came out of the IETF in October 2023, ISSN 2070-1721. The header doesn't list any document it obsoletes or updates, so it's not replacing or amending anything 🙌. It was written by a whole squad: M. Cociglio (Telecom Italia - TIM), A. Ferrieux (Orange Labs), G. Fioccola (Huawei Technologies), I. Lubashev (Akamai Technologies), F. Bulgarella (Telecom Italia - TIM), M. Nilo (Telecom Italia - TIM), I. Hamchaoui (Orange Labs), and R. Sisto (Politecnico di Torino) 💅✨
Abstract 💜
so this doc describes protocol-independent methods called Explicit Host-to-Network Flow Measurement Techniques 🦋, which can apply to transport-layer protocols between a client and a server. They use just a few marking bits in each packet's header for performance measurements, and the client and server have to work together 🫶. Both endpoints cooperate by marking packets and, possibly, mirroring the markings on the round-trip connection. These are especially iconic for protocols that encrypt transport headers, because they let passive, on-path network devices measure loss and delay 👀. The doc describes several methods you can use separately or jointly, depending on how many marking bits are available, what measurements you want, and the properties of the protocol they're applied to 🔥
Status of This Memo 📝
this doc is not an Internet Standards Track spec. It's published for informational purposes 🌸
it's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community 🫂. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so see Section 2 of RFC 7841 for that 🧐
for info on the current status of this doc, any errata, and how to give feedback, check https://www.rfc-editor.org/info/rfc9506 🔗
Copyright Notice ⚖️
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved 🎀
this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they describe your rights and restrictions with respect to this doc 📜. Code Components extracted from this doc must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Revised BSD License 💖
Table of Contents 🗺️
1. Introduction
2. Latency Bits
2.1. Spin Bit
2.2. Delay Bit
2.2.1. Generation Phase
2.2.2. Reflection Phase
2.2.3. T_Max Selection
2.2.4. Delay Measurement Using the Delay Bit
2.2.4.1. RTT Measurement
2.2.4.2. Half-RTT Measurement
2.2.4.3. Intra-domain RTT Measurement
2.2.5. Observer's Algorithm
2.2.6. Two Bits Delay Measurement: Spin Bit + Delay Bit
3. Loss Bits
3.1. T Bit -- Round-Trip Loss Bit
3.1.1. Round-Trip Loss
3.1.2. Setting the Round-Trip Loss Bit on Outgoing Packets
3.1.3. Observer's Logic for Round-Trip Loss Signal
3.1.4. Loss Coverage and Signal Timing
3.2. Q Bit -- sQuare Bit
3.2.1. Q Block Length Selection
3.2.2. Upstream Loss
3.2.3. Identifying Q Block Boundaries
3.2.3.1. Improved Resilience to Burst Losses
3.3. L Bit -- Loss Event Bit
3.3.1. End-To-End Loss
3.3.1.1. Loss Profile Characterization
3.3.2. L+Q Bits -- Loss Measurement Using L and Q Bits
3.3.2.1. Correlating End-to-End and Upstream Loss
3.3.2.2. Downstream Loss
3.3.2.3. Observer Loss
3.4. R Bit -- Reflection Square Bit
3.4.1. Enhancement of Reflection Block Length Computation
3.4.2. Improved Resilience to Packet Reordering
3.4.2.1. Improved Resilience to Burst Losses
3.4.3. R+Q Bits -- Loss Measurement Using R and Q Bits
3.4.3.1. Three-Quarters Connection Loss
3.4.3.2. End-To-End Loss in the Opposite Direction
3.4.3.3. Half Round-Trip Loss
3.4.3.4. Downstream Loss
3.5. E Bit -- ECN-Echo Event Bit
3.5.1. Setting the ECN-Echo Event Bit on Outgoing Packets
3.5.2. Using E Bit for Passive ECN-Reported Congestion
Measurement
3.5.3. Multiple E Bits
4. Summary of Delay and Loss Marking Methods
4.1. Implementation Considerations
5. Examples of Application
6. Protocol Ossification Considerations
7. Security Considerations
7.1. Optimistic ACK Attack
7.2. Delay Bit with RTT Obfuscation
8. Privacy Considerations
9. IANA Considerations
10. References
10.1. Normative References
10.2. Informative References
Acknowledgments
Contributors
Authors' Addresses1. Introduction ✨
packet loss and delay are hard, and they're everywhere in day-to-day network operation 😩. Proactively detecting, measuring, and locating them is crucial for keeping QoS high and fixing end-to-end throughput issues quickly 🚀
detecting and measuring loss and delay lets network operators confirm trouble reports on their own and, ideally, get a heads-up when problems are developing on the network 👀. Finding the cause of packet loss or excessive delay is the first step to fixing things and restoring QoS 💅
operators who want to measure loss and delay quantitatively have been relying heavily on info sitting in the clear in transport-layer headers (like TCP sequence and acknowledgment numbers) 📖. By passively watching a network path at multiple points inside their network, they could either quickly find the source of the problem within their own network or reliably blame an upstream or downstream network 🔍
with encrypted protocols, the transport-layer headers are encrypted, so passive loss and delay observations aren't possible, as also noted in [TRANSPORT-ENCRYPT] 🙈. Still, accurate measurement of the loss and delay experienced by encrypted transport-layer protocols is highly desired, especially by network operators who own or control the infrastructure between the client and server 🥺
you can't take the loss and delay measured on unencrypted connections between the same or similar endpoints and assume it applies to encrypted ones. Different protocols may use the network differently and be routed differently by the network, so it's necessary to directly measure the packet loss and delay experienced by users of encrypted protocols 🤷💖
the Alternate-Marking method [AltMark] defines a consolidated method to perform packet loss, delay, and jitter measurements on live traffic ✨. However, as mentioned in [IPv6AltMark], [AltMark] mainly applies to a network-layer-controlled domain managed with a Network Management System (NMS), where the Customer Premises Equipment (CPE) or the Provider Edge (PE) routers are the starting or ending nodes 🏢. [AltMark] provides measurement within a controlled domain in which the packets are marked 🏷️. So applying [AltMark] to end-to-end transport-layer connections isn't easy, because packet identification and marking by network nodes is prevented when encrypted transport-layer headers (e.g., QUIC, TCP with TLS) are being used 😭
this doc defines Explicit Host-to-Network Flow Measurement Techniques that are specifically designed for encrypted transport protocols 🔐. According to the definitions of [IPPM-METHODS], these measurement methods can be classified as Hybrid 🧩. They're embedded into a transport-layer protocol and are explicitly intended for exposing delay and loss rate info to on-path measurement devices 📡. Unlike [AltMark], most of these methods require collaborative endpoint nodes 🫶. Since these techniques make performance info directly visible to the path, they don't rely on an external NMS 🙌
these techniques are applicable to any transport-layer protocol connecting a client and a server 🌐. In this doc, the client and the server are also referred to as the endpoints of the transport-layer protocol 💜
the different methods described here can be used alone or in combination 🧃. Each technique uses few bits and exposes a specific measurement 🎯. It's assumed that the endpoints are collaborative in the sense of the measurements, because both the client and server need to cooperate 🤝
following the recommendation in [RFC8558] of making path signals explicit, this doc proposes adding some dedicated measurement bits to the clear portion of the transport protocol headers 🦋. These bits can be added to an unencrypted portion of a transport-layer header, e.g., UDP surplus space (see [UDP-OPTIONS] and [UDP-SURPLUS]) or reserved bits in a QUIC v1 header, as already done with the latency Spin bit (see Section 17.4 of [QUIC-TRANSPORT]) 🌈. Note that this doc does not recommend the use of any specific bits, as these would need to be chosen by the specific protocol implementations (see Section 5) 🙈
the Spin bit, Delay bit, and loss bits explained in this doc are inspired by [AltMark], [QUIC-MANAGEABILITY], [QUIC-SPIN], [TSVWG-SPIN], and [IPPM-SPIN] 💫
more details about performance measurements for QUIC are in the paper [ANRW19-PM-QUIC] 📄
2. Latency Bits 💖
this section introduces bits that can be used for round-trip latency measurements. Whenever this section refers to packets, it's only talking about packets with protocol headers that include the latency bits 🎀
in [QUIC-TRANSPORT], Section 17.4 introduces an explicit, per-flow transport-layer signal for hybrid measurement of RTT. This signal is a Spin bit that toggles once per RTT ⏱️. Section 4 of [QUIC-SPIN] discusses an additional two-bit Valid Edge Counter (VEC) that compensates for loss and reordering of the Spin bit and increases the fidelity of the signal in less than ideal network conditions 🔧
this doc introduces a standalone single-bit delay signal that passive observers can use to measure the RTT of a network flow 🦋, avoiding the Spin bit ambiguities that pop up as soon as network conditions deteriorate 💀✨
2.1. Spin Bit
okay so before we get into the good stuff, here's a quick recap of how the Spin bit works 💅 for the full tea, go read Section 3.8.2 of [QUIC-MANAGEABILITY] 🫶
The Spin bit is a signal generated by Alternate-Marking [AltMark], and the size of the alternation changes with the flight size each RTT 🌈
the latency Spin bit is a single-bit signal that toggles once per RTT ✨ which means it lets intermediate observation points measure latency for a connection-oriented communication 🦋
a "Spin bit period" is a set of packets that all have the same Spin bit value and are sent during one RTT time interval 🌸 and a "Spin bit period value" is that shared Spin bit value 💜
the client and the server each keep an internal per-connection spin value (so 0 or 1) that they use to set the Spin bit on outgoing packets 🎀 both endpoints start that spin value at 0 when a new connection starts, and then:
- when the client receives a packet with a packet number bigger than any it has seen so far, it flips the connection spin value to the opposite of the value in that packet 🙈
- when the server receives a packet with a packet number bigger than any it has seen so far, it sets the connection spin value to the same value as the one in that packet 💖
the endpoints then use the computed spin value to set the Spin bit on their outgoing packets 🧚 this makes a square wave, and a passive on-path observer can measure the time between pairs of consecutive edges seen in the same direction to work out the round-trip network delay of that flow 👑 the Spin bit lets you measure round-trip latency by only watching one direction of traffic, which is so iconic 🔥
ngl packet reordering can cause spurious edges, which means you need heuristics to fix them 💀 and the Spin bit performance falls apart as soon as network impairments show up, as explained in Section 2.2 🥲
2.2. Delay Bit
the Delay bit was designed to fix the accuracy problems the Spin bit has under rough network conditions 😭
- packet reordering creates spurious edges and messes up the delay estimate 🙃
- losing edges makes the Spin bit periods wrong, so the RTT measurements end up wrong too 💔
- application-limited senders make the Spin bit measure application delays instead of network delays 🍓
unlike the Spin bit, which gets set on every packet sent on the network, the Delay bit only gets set once per round trip ✨
when the Delay bit is in use, one single packet with a marked bit (the Delay bit) bounces between a client and a server for the entire connection lifetime 🦋 this lone packet is called the "delay sample" 💅
an observer sitting at an intermediate point can watch a single direction of traffic, track the delay sample and its relative timestamp, and measure the round-trip delay of the connection 🎀
the delay sample's lifetime has two phases: initialization and reflection 🌈 initialization is when the delay sample gets generated, and reflection is when that one packet bounces back and forth between the two endpoints 🫶
here's the elementary Delay bit mechanism, bestie 👇
+--------+ - - - - - +--------+
| | -----------> | |
| Client | | Server |
| | <----------- | |
+--------+ - - - - - +--------+
(a) No traffic at beginning.
+--------+ 0 0 1 - - +--------+
| | -----------> | |
| Client | | Server |
| | <----------- | |
+--------+ - - - - - +--------+
(b) The Client starts sending data and sets
the first packet as the delay sample.
+--------+ 0 0 0 0 0 +--------+
| | -----------> | |
| Client | | Server |
| | <----------- | |
+--------+ - - - 1 0 +--------+
(c) The Server starts sending data
and reflects the delay sample.
+--------+ 0 1 0 0 0 +--------+
| | -----------> | |
| Client | | Server |
| | <----------- | |
+--------+ 0 0 0 0 0 +--------+
(d) The Client reflects the delay sample.
+--------+ 0 0 0 0 0 +--------+
| | -----------> | |
| Client | | Server |
| | <----------- | |
+--------+ 0 0 0 1 0 +--------+
(e) The Server reflects the delay sample
and so on.
Figure 1: Delay Bit Mechanism2.2.1. Generation Phase
only the client is actively involved in the Generation Phase 🫶 it keeps an internal per-flow timestamp variable (ds_time) that gets updated every time a delay sample is sent 🕰️
when the connection starts, the client generates a new delay sample by setting the Delay bit of the first outgoing packet to 1 ✨ then it updates the ds_time variable with the timestamp of that transmission 💖
the server sets the Delay bit to 0 at the beginning of the connection, and its only job during the connection is described in Section 2.2.2 🎀
without any network impairments, the delay sample should keep bouncing between the client and server for the whole connection 🌸 but that's super unlikely, for two reasons:
- the packet carrying the Delay bit might get lost 😭
- an endpoint could stop or delay sending packets because the application is limiting how much traffic it sends 🙈
so to handle these problems, the client generates a new delay sample if more than a predetermined time (T_Max) has passed since the last delay sample was sent (reflections included) 🦋 note that T_Max should be greater than the max measurable RTT on the network 💅 see Section 2.2.3 for the details, bestie 🫶
2.2.2. Reflection Phase
reflection is how the delay sample gets bounced between a client and a server 🌈 the two endpoints behave almost the same way 🎀
- server-side reflection: when a delay sample arrives, the server marks the first packet in the opposite direction as the delay sample 🧚
- client-side reflection: when a delay sample arrives, the client marks the first packet in the opposite direction as the delay sample 👑 it also updates the
ds_timevariable when the outgoing delay sample is actually forwarded 💜
in both cases, if the outgoing delay sample is sent with a delay greater than a predetermined threshold after the incoming delay sample was received (1 ms by default), then the delay sample is not reflected and the outgoing Delay bit stays at 0 🙅♀️ this is so giving "reject what's not accurate" energy 💅
by doing this, the algorithm can throw out measurements that would overestimate the delay because the endpoints don't have enough traffic 🔥 so the maximum estimation error works out to twice the threshold (e.g., 2 ms) per measurement 🍓
2.2.3. T_Max Selection
the internal ds_time variable lets a client spot delay sample losses 🕵️♀️ since a lost delay sample is regenerated at the end of an explicit time (T_Max) counted from the last generation, an observer can use that same value to throw out a measurement and start a new one ✨
put another way, if the difference in time between two delay samples is greater than or equal to T_Max, then those two can't be used to produce a delay measure 🙈 which means the observers on the path also need to know the value of T_Max 💖
there are two ways to pick the T_Max value so that both the client and the observers know it 🌸 the first way is to have T_Max known a priori (T_Max_p), set inside the protocol specifications that implement the marking mechanism (for example, 1 second, which is usually bigger than the max expected RTT) 🎀 the second way needs a dynamic mechanism that adapts the duration of T_Max to the delay of the connection (T_Max_c) 🧚
for instance, the client and the observers could use the connection RTT as a basis for working out an effective T_Max 💅 they should start with a predetermined initial value so that T_Max = T_Max_p (e.g., 1 second), and then, once a valid RTT is measured, change T_Max so that T_Max = T_Max_c 🦋 either way, the selected T_Max should be large enough to soak up any possible variations in the connection delay 🌈 this also helps stop the mechanism from breaking when the observer can't recognize sudden RTT changes that go past T_Max 🙃
T_Max_c could be computed as two times the measured RTT plus a fixed amount of time (100 ms), so that you don't end up with tiny T_Max values when the RTTs are really small 🍓 the resulting formula is:
T_Max_c = 2RTT + 100 msif T_Max_c is greater than T_Max_p, then T_Max_c gets forced to the T_Max_p value 🫶 the value of 100 ms is only an example, though, and it can be chosen differently depending on the scenario 💜 for instance, an implementer might consider using existing protocol-specific values if that makes sense 🌷
the observer's T_Max should always be less than or equal to the client's T_Max, so that the observer doesn't count what's really the client's T_Max as a valid measurement 👑 to get that result, the client waits for two consecutive incoming samples and computes the two related RTTs ✨ then it takes the largest one as the basis for the T_Max_c formula 🔥 by this point, the observers have already measured a valid RTT, so they can compute their own T_Max_c 💖
2.2.4. Delay Measurement Using the Delay Bit
when the Delay bit is in use, a passive observer can use delay samples directly 🧚 that way it skips the inherent ambiguities in the RTT calculation that you can see in Spin bit analysis 🙈 no cap, this is the main character move 👑✨
2.2.4.1. RTT Measurement
okay so the delay sample generation process makes sure that only ONE packet with the Delay bit set to 1 bounces back and forth between two endpoints per round-trip time 🏓 no crowding allowed! ✨ to get the RTT measurement of a flow, an on-path passive observer does the math on the time gap between two delay samples seen going in a single direction 💅
to make it a valid measurement, the observer must verify that the distance in time between the two samples it's using is less than T_Max 🫶
=======================|======================>
= ********** -----Obs----> ********** =
= * Client * * Server * =
= ********** <------------ ********** =
<==============================================
(a) client-server RTT
==============================================>
= ********** ------------> ********** =
= * Client * * Server * =
= ********** <----Obs----- ********** =
<======================|=======================
(b) server-client RTTFigure 2: Round-Trip Time (Both Directions) 🌸
2.2.4.2. Half-RTT Measurement
okay this one is giving two-directions energy 🦋 an observer that can see both the forward and return traffic can use the delay samples to measure the "upstream" and "downstream" RTT components, aka the half-RTT measurements. it does this by measuring the time between a delay sample seen in one direction and the delay sample seen previously in the opposite direction 🔥
same as before, the observer must verify that the distance in time between the two samples it's using is less than T_Max 💜
ngl there's a little catch here 👀 the upstream and downstream sections of the paths between the endpoints and the observer (so observer-to-client vs. client-to-observer, and observer-to-server vs. server-to-observer) may have different delay characteristics, because of differences in network congestion and other factors 🙈
=======================>
= ********** ------|-----> **********
= * Client * Obs * Server *
= ********** <-----|------ **********
<=======================
(a) client-observer half-RTT
=======================>
********** ------|-----> ********** =
* Client * Obs * Server * =
********** <-----|------ ********** =
<=======================
(b) observer-server half-RTTFigure 3: Half Round-Trip Time (Both Directions) 🎀
2.2.4.3. Intra-domain RTT Measurement
intra-domain RTT is the portion of the whole RTT that a flow uses to traverse a provider's network 🧚 to measure it, two observers that can see traffic in both directions have to be used at the same time, one at the ingress and one at the egress of the network being measured. the intra-domain RTT is the difference between the two computed upstream (or downstream) RTT components. main character math 💅✨
=========================================>
= =====================>
= = ********** ---|--> ---|--> **********
= = * Client * Obs Obs * Server *
= = ********** <--|--- <--|--- **********
= <=====================
<=========================================
(a) client-observer RTT components (half-RTTs)
==================>
********** ---|--> ---|--> **********
* Client * Obs Obs * Server *
********** <--|--- <--|--- **********
<==================
(b) the intra-domain RTT resulting from the
subtraction of the above RTT componentsFigure 4: Intra-domain Round-Trip Time (Client-Observer: Upstream) 🌈
2.2.5. Observer's Algorithm
okay the observer keeps an internal per-flow variable to track when the last delay sample was seen 🕰️ and the flow characterization should be part of the protocol, bestie 🫶
if the observer is unidirectional, or if routing is asymmetric, then when it detects a delay sample:
- if a delay sample was also detected previously in the same direction, and the distance in time between them is less than T_Max - K, then the two delay samples can be used to calculate an RTT measurement. K is a protection threshold that absorbs differences in T_Max computation and delay variations between two consecutive delay samples (e.g., K = 10% T_Max). 🛡️
if the observer can see both forward and return traffic flows, and it can tell which direction holds the client and which holds the server (for example, by watching the connection handshake), then when it detects a delay sample:
- if a delay sample was also detected in the opposite direction, and the distance in time between them is less than T_Max - K, then the two delay samples can be used to measure the observer-client half-RTT or the observer-server half-RTT, depending on the direction of the last delay sample observed. 💖
ngl the accuracy depends on what the observer is able to see 👀 and the type of measurement is different, like we covered in the previous sections 📚
2.2.6. Two Bits Delay Measurement: Spin Bit + Delay Bit
the Spin and Delay bit algorithms work independently, no drama 🙅♀️ if both marking methods are used on the same connection, observers can pick the best measurement of the two that are available 👑
- when a precise measurement can be produced using the Delay bit, observers choose it; and
- when a Delay bit measurement is not available, observers choose the approximate Spin bit one. 🦋
3. Loss Bits 🍓
this section introduces bits that can be used for loss measurements 💅 whenever this section talks about packets, it means only packets whose protocol headers include the loss bits, since those are the only packets whose loss can be measured 🫶
- T: The "round-Trip loss" bit is used in combination with the Spin bit to measure round-trip loss. See Section 3.1. 🔁
- Q: The "sQuare" bit is used to measure upstream loss. See Section 3.2. 🟦
- L: The "Loss Event" bit is used to measure end-to-end loss. See Section 3.3. 🚨
- R: The "Reflection square" bit is used in combination with the Q bit to measure end-to-end loss. See Section 3.4. 🪞
loss measurements enabled by the T, Q, and L bits can be implemented by those loss bits alone (the T bit requires a working Spin bit). the two-bit combinations Q+L and Q+R unlock extra measurement opportunities, which we'll get into below ✨
each endpoint keeps its own counters, independently and separately, for each identifiable flow (or each sub-flow for multipath connections) 🧮
since loss is reported independently for each flow, all bits except for the L bit need a certain minimum number of packets to be exchanged per flow before any signal can be measured 🥺 so loss measurements work best for flows that transfer more than a minimal amount of data 💕
3.1. T Bit -- Round-Trip Loss Bit
the round-Trip loss bit marks a variable number of packets that get exchanged twice between the endpoints, in what's called a two round-trip reflection 🪞✨ a passive on-path observer, watching either direction, can count and compare the number of marked packets seen during the two reflections and estimate the loss rate the connection is experiencing 📉 here's the whole exchange:
- the client picks and sets the T bit to 1 to identify a first train of packets 🚂;
- when the server receives each packet in the first train, it sets the T bit to 1 and reflects back to the client a matching second train of packets, the same size as the first train it received;
- when the client receives each packet in the second train, it sets the T bit to 1 and reflects back to the server a matching third train of packets, the same size as the second train it received; and
- when the server receives each packet in the third train, it sets the T bit to 1 and finally reflects back to the client a matching fourth train of packets, the same size as the third train it received. 💖
packets in the first round trip (first and second trains) make up the Generation Phase, while packets in the second round trip (third and fourth trains) make up the Reflection Phase 🌙
a passive on-path observer can count and compare the number of marked packets seen during the two round trips (that's the first and third trains, or the second and fourth trains, depending on which direction is being observed) and estimate the loss rate the connection is experiencing 🕵️♀️ this keeps going over and over to get more measurements as long as the endpoints keep exchanging traffic 🔄 these measurements can be called round-trip losses 💜
since the packet rates in the two directions may be different, the number of marked packets in a train is set by the direction with the lowest packet rate 🐢 see Section 3.1.2 for details on packet generation 📖
3.1.1. Round-Trip Loss 🎀
Okay bestie, so the observer only watches a portion of the traffic going between the client and the server. Statistically, though, that sample lines up with the loss rate the connection is dealing with across the whole network path 🦋 That's the end-to-end Round-Trip Packet Loss (RTPL), and it's giving main character energy ✨ Here's how it looks in both directions:
=======================|======================>
= ********** -----Obs----> ********** =
= * Client * * Server * =
= ********** <------------ ********** =
<==============================================
(a) client-server RTPL
==============================================>
= ********** ------------> ********** =
= * Client * * Server * =
= ********** <----Obs----- ********** =
<======================|=======================
(b) server-client RTPL
Figure 5: Round-Trip Packet Loss (Both Directions)The same method also lets us get the half-RTPL measurement and the Intra-domain RTPL measurement, kind of like how we do RTT measurement 😭 Look at these slay diagrams:
=======================>
= ********** ------|-----> **********
= * Client * Obs * Server *
= ********** <-----|------ **********
<=======================
(a) client-observer half-RTPL
=======================>
********** ------|-----> ********** =
* Client * Obs * Server * =
********** <-----|------ ********** =
<=======================
(b) observer-server half-RTPL
Figure 6: Half Round-Trip Packet Loss (Both Directions)
=========================================>
=====================> =
********** ---|--> ---|--> ********** = =
* Client * Obs Obs * Server * = =
********** <--|--- <--|--- ********** = =
<===================== =
<=========================================
(a) observer-server RTPL components (half-RTPLs)
==================>
********** ---|--> ---|--> **********
* Client * Obs Obs * Server *
********** <--|--- <--|--- **********
<==================
(b) the intra-domain RTPL resulting from the
subtraction of the above RTPL components
Figure 7: Intra-domain Round-Trip Packet Loss (Observer-Server)3.1.2. Setting the Round-Trip Loss Bit on Outgoing Packets 💖
Real talk, the round-Trip loss signal needs a working Spin bit signal to keep its packet trains separated 🧚 A "pause" of at least one empty Spin bit period between each phase of the algorithm acts as that separator for the on-path observer. The T bit and the Spin bit are connected, and that connection is what helps the observer line up the packet trains 🫶
The client keeps a "generation token" count that starts at zero when the session begins and goes up every time a packet is received, marked or unmarked. The client also keeps a "reflection counter" that starts at zero at the beginning of the session.
The client is the one in charge of launching the trains of marked packets, and it runs this algorithm:
- Generation Phase. The client starts generating marked packets for two consecutive Spin bit periods. When the client sends a packet and a "generation token" is available, it marks the packet and uses up one "generation token". If no token is available, the outgoing packet goes out unmarked. At the end of the first Spin bit period spent in generation, the reflection counter gets unlocked so it can start counting incoming marked packets that will be reflected later 🌸
- Pause Phase. Once generation is done, the client pauses until it has watched one entire Spin bit period go by with no marked packets. The observer uses that period as a separator between generated and reflected packets. During this marking pause, every outgoing packet goes out with the T bit set to 0. The reflection counter keeps going up every time a marked packet shows up 🍓
- Reflection Phase. The client starts sending marked packets and decrements the reflection counter for each marked packet it sends, until the reflection counter hits zero. The "generation token" method from the Generation Phase is used here too. At the end of the first Spin bit period spent in reflection, the reflection counter gets locked so incoming reflected packets can't increment it 🦋
- Pause Phase 2. The Pause Phase gets repeated after the Reflection Phase. It works as the separator between the reflected packet train and a brand new packet train 👑
The generation token counter should be capped so that it doesn't cause trouble if the other endpoint's packet rate suddenly drops and that endpoint can't reflect the packets it collected. A cap value of 1 is recommended 💅
A server keeps a "marking counter" that starts at zero and goes up every time a marked packet arrives. When the server sends a packet and the "marking counter" is positive, it marks that packet and decrements the "marking counter". If the "marking counter" is zero, the outgoing packet goes out unmarked 🫶
Heads up, bestie: picking 2 RTT (two Spin bit periods) for the Generation Phase is a trade-off. You get a percentage of marked packets (aka how much traffic gets monitored) on one side, and measurement delay on the other 📏 With this value, the algorithm makes a measurement roughly every 6 RTT (2 generations, ~2 reflections, 2 pauses), and it marks ~1/3 of the packets in the slower direction (see Section 3.1.4). If you go with a Generation Phase of 1 RTT instead, you get a measurement every 4 RTT, and you monitor ~1/4 of the packets in the slower direction ✨
Also, problems can totally happen in some cases, especially if the rate suddenly changes 😭 The mechanism still worked well under normal traffic conditions in the implementation, so that's the good news 🌈
3.1.3. Observer's Logic for Round-Trip Loss Signal 🔥
The on-path observer counts the marked packets and tells the trains apart by spotting Spin bit periods (at least one) with no marked packets. The Round-Trip Packet Loss (RTPL) is the difference between the size of the Generation train and the size of the Reflection train 💜
Here's a cute example. Packets are shown as two bits, where the first one is the Spin bit and the second one is the round-Trip loss bit:
Generation Pause Reflection Pause
____________________ ______________ ____________________ ________
| | | | |
01 01 00 01 11 10 11 00 00 10 10 10 01 00 01 01 10 11 10 00 00 10
Figure 8: Round-Trip Loss Signal ExampleNotice that 5 marked packets were generated, and 4 of them got reflected 🙈
3.1.4. Loss Coverage and Signal Timing 🌸
One cycle of the round-Trip loss signaling algorithm has 2 RTTs of Generation phase, 2 RTTs of Reflection Phase, and 2 Pause Phases that are each at least 1 RTT long. So the loss signal shows up about 6 RTTs after the loss events happen ⏳
The observer can only detect the loss of marked packets that happens after it first sees the Generation Phase and before it sees the Reflection Phase afterwards 👀 So if the loss happens on the path that sends packets at a lower rate (typically ACKs in asymmetric scenarios like this), then 2/6 (1/3) of the packets get sampled for loss detection 🎀
If the loss happens on the path that sends packets at a higher rate, then lowPacketRate/(3*highPacketRate) of the packets get sampled for loss detection. For protocols that use ACKs, the share of packets sampled for loss in the higher-rate direction during unidirectional data transfer is 1/(3*packetsPerAck), and the value of packetsPerAck can change by protocol, by implementation, and by network conditions 💅
3.2. Q Bit -- sQuare Bit ⭐
The sQuare bit (Q bit) is named after the square wave its signal makes 🌈 This method is based on the Alternate-Marking method [AltMark], and the Q bit is the "packet color". It flips between 0 and 1 to mark back-to-back blocks of packets with different colors. The best part? This method doesn't need both endpoints to cooperate 🫶
[AltMark] describes two versions of the Alternate-Marking method, depending on whether the color switches on a fixed timer or after a fixed number of packets. Cooperating, synchronized observers on either end of a network segment can use the fixed-timer version to measure packet loss on that segment, by comparing packet counters for the same packet blocks. The length of the blocks can be picked based on how often you want to measure, but it must be long enough to keep things working properly with clock errors and network delay issues ⏱️
The Q bit method in this document goes with the color-switching-by-fixed-number-of-packets approach, and each block gets the same number of packets. That's iconic because it doesn't need cooperating or synchronized observers or network elements 🧚 Each probe can measure packet loss on its own without relying on an external NMS. For packet loss measurement, the only thing that matters is detecting that a loss event happened, not pinpointing the exact block where the losses occurred 💖
Following the fixed-number-of-packets method, the square wave signal is generated by switching the Q bit. Every outgoing packet carries the Q bit value, which starts at 0 and gets inverted after N packets have been sent (a sQuare Block, or just Q Block). That makes the Q Period 2*N 🦋
Observation points can estimate upstream losses by watching a single direction of the traffic flow and counting the number of packets in each observed Q Block, as described in Section 3.2.2 👑
3.2.1. Q Block Length Selection 🎀
okay so the observer has to know how long a Q Block is 👀 bestie, there are two ways to make sure that happens 💅 The first way is that the length is already known ahead of time and written right into the protocol specs that carry the marking mechanism ✨ The second way is that the sender picks it 🫶
if the sender is the one choosing, it should pick N (the Q Block length) based on how much loss and reordering it expects on the path 🦋 this is a total balancing act ngl 😭 If N is too small, the observation gets super unreliable when packets get reordered or lots of them go missing 💀 If N is too large, short flows won't give a useful upstream loss measurement at all 🙈 (see Section 3.2.2)
the value of N MUST be at least 64 and MUST be a power of 2 🔥 this is what lets an observer figure out the Q Block length just by watching one period of the square signal 👑 It also lets the observer spot flows that are setting the loss bits to random values (see Section 6) 🌈
if the sender doesn't have enough info to make a good call on Q Block length, it SHOULD use N=64 🍓 this value has been tested out in big field tests and it gave really good results 💖 Or the sender MAY pick a random power-of-2 N for each flow instead 🧚 that way some flows are more likely to land on a length that gives a great signal ✨
the sender MUST keep the value of N constant for a given flow 🔒 no switching it up mid-flow, bestie 💅
3.2.2. Upstream Loss 🌸
Blocks of N (the Q Block length) consecutive packets go out with the same Q bit value 💜 then the next block of N packets goes out with the Q bit flipped 🦋 so if you know N, an on-path observer can estimate the upstream loss after watching at least N packets 👀 The upstream loss rate (uloss) is one minus the average number of packets in a run of same-Q-value packets (p), divided by N: uloss=1-avg(p)/N 🧮
the observer has to be able to handle packet reordering, because reordering can blur the edges of the square signal 😭 (more on that in Section 3.2.3) 🫶
=====================>
********** -----Obs----> **********
* Client * * Server *
********** <------------ **********
(a) in client-server channel (uloss_up)
********** ------------> **********
* Client * * Server *
********** <----Obs----- **********
<=====================
(b) in server-client channel (uloss_down)
Figure 9: Upstream Loss3.2.3. Identifying Q Block Boundaries 🔍
packet reordering can make fake edges show up in the square signal 🙃 to fix that, the observer SHOULD look for packets with the current Q bit value up to X packets past the first packet that has the reversed Q bit value 💅 The value of X, a "Marking Block Threshold", SHOULD be less than N/2 ✨
choosing X is a trade-off, bestie 🎀 A really big Marking Block Threshold can rebuild Q Blocks even when there's a lot of reordering 🌈 but if loss goes over 50% for some Q Blocks, it might accidentally mash packets from multiple Q Blocks into fewer Q Blocks 😭
3.2.3.1. Improved Resilience to Burst Losses 💥
burst losses can mess with how accurate the Q measurements are 😵💫 Generally, burst losses can be absorbed and measured correctly if they're smaller than the established Q Block length 🦋 but if a whole Q Block's worth of packets gets lost in one burst, the observer might have zero idea it happened 💀
to get better at handling burst losses, an observer MAY treat a received Q Block that's bigger than the selected Q Block length as a sign that a burst loss happened 👀 The observer would then compute the loss as three times the Q Block length minus the measured block length 🧮 With that, the observer can detect burst losses of less than two blocks (for example, less than 128 packets for a Q Block length of 64 packets) 🎉 A burst loss spanning two or more consecutive periods would still go unnoticed by the observer (or get underestimated if a period longer than the Q Block length got formed) 😭
3.3. L Bit -- Loss Event Bit 💜
The Loss Event bit uses an Unreported Loss counter that the protocol implementing the marking mechanism keeps track of 🧠 To use the Loss Event bit, the protocol MUST let the sender identify lost packets 🔥 This is true for protocols like QUIC 👑 partially true for TCP and Stream Control Transmission Protocol (SCTP), since losses of pure ACKs aren't detected 🙈 and it is not true for protocols like UDP and IPv4/IPv6 💀
The Unreported Loss counter starts at 0 🍓 and the L bit of every outgoing packet tells you whether the Unreported Loss counter is positive (L=1 if the counter is positive, and L=0 otherwise) ✨
the value of the Unreported Loss counter goes down every time a packet with L=1 is sent 📉
the value of the Unreported Loss counter goes up for every packet the protocol declares lost, using whatever loss detection machinery the protocol has 🕵️♀️ If the protocol can take back a loss determination later, a positive Unreported Loss counter MAY go down because of that rescission 🔄 In general, it SHOULD NOT become negative due to the rescission, but it can happen in a few cases 😬
this loss signaling is similar to the loss signaling in [ConEx] 🫶 The difference is that the Loss Event bit reports the exact number of lost packets, whereas the signal mechanism in [ConEx] reports an approximate number of lost bytes 💖
for protocols such as TCP [TCP] that let network devices change data segmentation, it's possible that only part of a packet gets lost 🧩 In these cases, the sender MUST increment the Unreported Loss counter by the fraction of the packet data that was lost (so the Unreported Loss counter may become negative when a packet with L=1 is sent after a partial packet has been lost) 🙃
observation points can estimate the end-to-end loss, as determined by the upstream endpoint, by counting packets in this direction with the L bit equal to 1 📊 (see Section 3.3.1)
3.3.1. End-To-End Loss 🦋
The Loss Event bit lets an observer estimate the end-to-end loss rate by counting packets with L bit values of 0 and 1 for a given flow 🧮 The end-to-end loss ratio is the fraction of packets with L=1 💅
the assumption here is that upstream loss affects packets with L=0 and L=1 equally 🤞 If some loss is caused by tail-drop in a network device, that might be a bit of a simplification 🙈 If the sender's congestion controller cuts the packet send rate after loss, there may be enough of a delay before packets with L=1 get sent that they have a better chance of reaching the observer ✨
3.3.1.1. Loss Profile Characterization 🌈
The Loss Event bit lets an observer characterize the loss profile 🕵️♀️ since the distribution of observed packets with the L bit set to 1 roughly matches the distribution of packets lost between 1 RTT and 1 retransmission timeout (RTO) before (see Section 3.3.2.1) 🎯 So observing random single instances of the L bit set to 1 points to random single packet loss 💫 while observing blocks of packets with the L bit set to 1 points to loss hitting entire blocks of packets 😭
3.3.2. L+Q Bits -- Loss Measurement Using L and Q Bits 💖
Combining L and Q bits lets a passive observer watching a single direction of traffic accurately measure:
- upstream loss: sender-to-observer loss (see Section 3.2.2)
- downstream loss: observer-to-receiver loss (see Section 3.3.2.2)
- end-to-end loss: sender-to-receiver loss on the observed path (see Section 3.3.1) with loss profile characterization (see Section 3.3.1.1) ✨
3.3.2.1. Correlating End-to-End and Upstream Loss 🔗
Upstream loss is calculated by observing packets that did not suffer the upstream loss (Section 3.2.2) 🧐 End-to-end loss, however, is calculated by observing subsequent packets after the sender's protocol detected the loss 👀 So end-to-end loss is usually observed with a delay of between 1 RTT (loss declared due to multiple duplicate acknowledgments) and 1 RTO (loss declared due to a timeout) relative to the upstream loss ⏰
the flow RTT can sometimes be estimated by timing the protocol handshake messages 🤝 That RTT estimate can get a lot better by observing a dedicated protocol mechanism for conveying RTT information, like the Spin bit (see Section 2.1) or the Delay bit (see Section 2.2) 🫧
whenever the observer needs to run a computation that uses both upstream and end-to-end loss rate measurements, it SHOULD consider the upstream loss rate leading the end-to-end loss rate by approximately 1 RTT 💅 If the observer can't estimate the RTT of the flow, it SHOULD accumulate loss measurements over time periods of at least 4 times the typical RTT for the observed flows ⏳
if the calculated upstream loss rate is higher than the end-to-end loss rate calculated in Section 3.3.1, then either the Q Period is too short for the amount of packet reordering or there is observer loss (described in Section 3.3.2.3) 😵 If this happens, the observer SHOULD adjust the calculated upstream loss rate to match the end-to-end loss rate, unless the following applies 🙏
for a protocol such as TCP or SCTP, which doesn't track losses of pure ACK packets, observing a direction of traffic dominated by pure ACK packets could give a measured upstream loss that's higher than the measured end-to-end loss if those pure ACK packets are lost upstream 🌊 So if the measurement is applied to such protocols, and the observer can confirm that pure ACK packets dominate the observed traffic direction, the observer SHOULD adjust the calculated end-to-end loss rate to match the upstream loss rate 🎀
3.3.2.2. Downstream Loss 📉
Because downstream loss only affects packets that didn't suffer upstream loss, the end-to-end loss rate (eloss) relates to the upstream loss rate (uloss) and the downstream loss rate (dloss) like this: (1-uloss)(1-dloss)=1-eloss 🧮 So dloss=(eloss-uloss)/(1-uloss) 💖
3.3.2.3. Observer Loss 👀
A typical deployment of a passive observation system includes a network tap device that mirrors the network packets of interest to a device that does the analysis and measurement on those mirrored packets 🪞 The observer loss is the loss that happens on the mirror path 💫
observer loss affects the upstream loss rate measurement, since it makes the observer count fewer packets in a block of identical Q bit values (see Section 3.2.2) 😭 The end-to-end loss rate measurement, however, is unaffected by observer loss, since it measures the fraction of packets with the L bit value of 1, and observer loss would hit all packets equally (see Section 3.3.1) 🙌
if you have to adjust the upstream loss rate down to match the end-to-end loss rate as described in Section 3.3.2.1, that's a sign of observer loss 🚨 Its size is somewhere between the amount of that adjustment and the entire upstream loss measured in Section 3.2.2 📏 Alternatively, a high apparent upstream loss rate could mean there's significant packet reordering, possibly because packets belonging to a single flow are being spread across several upstream paths with different latency characteristics 🛤️✨
3.4. R Bit -- Reflection Square Bit 🪞✨
okay so the R bit literally needs the Q bit to be deployed alongside it 🫶 (no solo R bit era, sorry bestie). Unlike the Q signal, where packets go out in blocks of a fixed size, the Reflection square blocks (also an Alternate-Marking signal 🦋) change size based on these rules:
- when the transmission of a new block starts, its size is set equal to the size of the last Q Block whose reception has been completed; and
- if the reception of at least one further Q Block is completed before transmission of the block is terminated, the size of the block is updated to be the average size of the further received Q Blocks 📈
The Reflection square value starts at 0 and gets slapped onto the R bit of every outgoing packet 💖. It flips for the first time when a completed Q Block shows up in the incoming square signal (the one the other endpoint made with the Q bit) 🔄. The number of packets spotted in that first Q Block, called p, is used to make a reflection square signal that flips every M=p packets (at first). This new signal makes blocks of M packets marked with the R bit, and each of them is called "Reflection Block" (Reflection Block). yes, the name is said twice, iykyk 🎀
The M value gets updated every time a completed Q Block in the incoming square signal arrives, using M=round(avg(p)) 🧮.
The parameter avg(p), the average number of packets in a marking period, is computed based on all the Q Blocks received since the beginning of the current Reflection Block 📊.
The transmission of a Reflection Block counts as complete (and the signal flips) when the number of packets sent in that block is at least the latest computed M value ✅.
To get M computed properly, endpoints implementing the R bit have to identify where incoming Q Blocks start and end. The same approach described in Section 3.2.3 should be used 🧩.
By peeking at the R bit, unidirectional observation points can tell how much loss the entire unobserved channel has, plus the loss on the path from the sender to them 👀.
Since the Q Block travels in one direction and the matching reflected R Block travels the opposite way, the reflected R signal goes at the packet rate of the slowest direction 🐢. So if the observed direction is the slowest, there can be multiple Q Blocks sent in the unobserved direction before a complete Reflection Block is sent in the observed direction. And if the unobserved direction is the slowest, the observed direction can be sending R Blocks of the same size over and over before it can update the signal to account for a newly completed Q Block 🔁.
3.4.1. Enhancement of Reflection Block Length Computation
The rounding function used in the M computation adds errors 😤, but those can be minimized by storing the rounding applied each time M is computed and using it when computing M for the following Reflection Block 🧠.
This is done by adding the new r_avg parameter to the computation of M. The new formula is Mr=avg(p)+r_avg; M=round(Mr); r_avg=Mr-M, where the initial value of r_avg is equal to 0.
3.4.2. Improved Resilience to Packet Reordering
When a protocol implementing the marking mechanism is able to detect when packets are received out of order 👀, it can improve resilience to packet reordering beyond what is possible with the methods in Section 3.2.3 🙌.
This is achieved by updating the size of the current Reflection Block while it's being transmitted. The Reflection Block size gets updated every time an incoming reordered packet of the previous Q Block is detected. This can be done if and only if the transmission of the current Reflection Block is in progress and no packets of the following Q Block have been received 💜.
3.4.2.1. Improved Resilience to Burst Losses
Burst losses can affect the accuracy of R measurements the same way they affect accuracy of Q measurements 💥. Therefore, the recommendations in Section 3.2.3.1 apply equally to improving burst loss resilience for R measurements 🫶.
3.4.3. R+Q Bits -- Loss Measurement Using R and Q Bits
Since both sQuare and Reflection square bits are toggled at most every N packets (except for the first transition of the R bit, as explained before) 🔀, an on-path observer can count the number of packets of each marking block. Knowing the value of N, it can estimate the amount of loss the connection experiences 📉. An observer can calculate different measurements depending on whether it can observe a single direction of the traffic or both directions 👯.
Single directional observer:
- upstream loss in the observed direction: the loss between the sender and the observation point (see Section 3.2.2)
- "three-quarters" connection loss: the loss between the receiver and the sender in the unobserved direction plus the loss between the sender and the observation point in the observed direction
- end-to-end loss in the unobserved direction: the loss between the receiver and the sender in the opposite direction
Two directions observer (same metrics as above, applied to both directions, plus):
- client-observer half round-trip loss: the loss between the client and the observation point in both directions
- observer-server half round-trip loss: the loss between the observation point and the server in both directions
- downstream loss: the loss between the observation point and the receiver (applicable to both directions)
3.4.3.1. Three-Quarters Connection Loss
Except for the very first block, where there's nothing to reflect yet (a complete Q Block hasn't been received), packets are continuously R-bit marked into alternate blocks of size lower or equal to N 🧃. By knowing N, an on-path observer can estimate the amount of loss that occurred in the whole opposite channel, plus the loss from the sender up to it in the observation channel 🕵️. Same as the previous metric, the three-quarters connection loss rate (tqloss) is one minus the average number of packets in a block of packets with the same R value (t) divided by N: tqloss=1-avg(t)/N 💜.
=======================>
= ********** -----Obs----> **********
= * Client * * Server *
= ********** <------------ **********
<============================================
(a) in client-server channel (tqloss_up)
============================================>
********** ------------> ********** =
* Client * * Server * =
********** <----Obs----- ********** =
<=======================
(b) in server-client channel (tqloss_down)
Figure 10: Three-Quarters Connection LossThe following metrics derive from this last metric and the upstream loss produced by the Q bit 🌈.
3.4.3.2. End-To-End Loss in the Opposite Direction
End-to-end loss in the unobserved direction (eloss_unobserved) relates to the "three-quarters" connection loss (tqloss) and upstream loss in the observed direction (uloss) as (1-eloss_unobserved)(1-uloss)=1-tqloss. Hence, eloss_unobserved=(tqloss-uloss)/(1-uloss).
********** -----Obs----> **********
* Client * * Server *
********** <------------ **********
<==========================================
(a) in client-server channel (eloss_down)
==========================================>
********** ------------> **********
* Client * * Server *
********** <----Obs----- **********
(b) in server-client channel (eloss_up)
Figure 11: End-To-End Loss in the Opposite Direction3.4.3.3. Half Round-Trip Loss
If the observer is able to observe both directions of traffic, it can calculate two "half round-trip" loss measurements. These are loss from the observer to the receiver (in a given direction) and then back to the observer in the opposite direction 🔁. For both directions, "half round-trip" loss (hrtloss) relates to "three-quarters" connection loss (tqloss_opposite) measured in the opposite direction and the upstream loss (uloss) measured in the given direction as (1-uloss)(1-hrtloss)=1-tqloss_opposite. Hence, hrtloss=(tqloss_opposite-uloss)/(1-uloss).
=======================>
= ********** ------|-----> **********
= * Client * Obs * Server *
= ********** <-----|------ **********
<=======================
(a) client-observer half round-trip loss (hrtloss_co)
=======================>
********** ------|-----> ********** =
* Client * Obs * Server * =
********** <-----|------ ********** =
<=======================
(b) observer-server half round-trip loss (hrtloss_os)
Figure 12: Half Round-Trip Loss (Both Directions)3.4.3.4. Downstream Loss
If the observer can see both directions of traffic, it can calculate two downstream loss measurements. It can use either end-to-end loss and upstream loss (similar to the calculation in Section 3.3.2.2), or "half round-trip" loss and upstream loss in the opposite direction.
For the latter, dloss=(hrtloss-uloss_opposite)/(1-uloss_opposite).
=====================>
********** ------|-----> **********
* Client * Obs * Server *
********** <-----|------ **********
(a) in client-server channel (dloss_up)
********** ------|-----> **********
* Client * Obs * Server *
********** <-----|------ **********
<=====================
(b) in server-client channel (dloss_down)
Figure 13: Downstream Loss3.5. E Bit -- ECN-Echo Event Bit 🚦
While the primary focus of this document is on exposing packet loss and delay, modern networks can report congestion before they are forced to drop packets, as described in [ECN] 💫. When transport protocols keep ECN-Echo feedback under encryption, network operators can't observe this signal 🔒. When they're diagnosing network performance problems, knowing about congestion downstream of an observation point can be super instrumental 🔍.
If downstream congestion information is desired, this information can be signaled with an additional bit 🌸.
- E: The "ECN-Echo Event" bit is set to 0 or 1 according to the Unreported ECN-Echo counter, as explained below in Section 3.5.1. 🦋
3.5.1. Setting the ECN-Echo Event Bit on Outgoing Packets 🎀
okay so the Unreported ECN-Echo counter works exactly like the Unreported Loss counter from Section 3.3 💅, except it counts the packets the network delivered with Congestion Experienced (CE) markings, going by the ECN-Echo feedback from the receiver 🫶
this ECN-Echo signaling is giving cousin of the ECN signaling in [ConEx] ✨ the ECN-Echo mechanism in QUIC gives you the number of packets received with CE marks 🌸 for protocols like TCP, you can use the method in [ConEx-TCP] 🦋 and as [ConEx-TCP] says, that feedback can be leveled up even more with the method in [ACCURATE-ECN] 👑
3.5.2. Using E Bit for Passive ECN-Reported Congestion Measurement 🔍
a network observer can count packets with the CE codepoint and figure out the upstream CE-marking rate directly 🧚
observation points can also estimate ECN-reported end-to-end congestion by counting packets in this direction with an E bit equal to 1 💜
the upstream CE-marking rate and the end-to-end ECN-reported congestion can tell you something about the downstream CE-marking rate 🌈 but ngl, the presence of E bits along with L bits can kinda confuse precise estimates of upstream and downstream CE markings 😭 if the flow has packets that are not ECN capable 💀
3.5.3. Multiple E Bits 🍓
some protocols, like QUIC, support separate ECN-Echo counters 💖 for example, Section 13.4.1 of [QUIC-TRANSPORT] describes separate counters for ECT(0), ECT(1), and ECN-CE 🎀 so to better support protocols like that, multiple E bits can be used, one per corresponding ECN-Echo counter ✨
4. Summary of Delay and Loss Marking Methods 💖
okay bestie, this section is the recap 🫶 it sums up the marking methods from this document, which is basically a toolkit of techniques you can use separately, partly, or all together depending on what you need 🧰🌸
for delay measurement, you can use the Spin bit and/or the Delay bit 🦋 a unidirectional or bidirectional observer can be used 👀
+===============+======+=====================+=============+========+
| Method | # of | Available Delay | Impairments | # of |
| | bits | Metrics | Resiliency | meas. |
| | +==========+==========+ | |
| | | UniDir | BiDir | | |
| | | Observer | Observer | | |
+===============+======+==========+==========+=============+========+
| S: Spin Bit | 1 | RTT | x2, Half | low | very |
| | | | RTT | | high |
+---------------+------+----------+----------+-------------+--------+
| D: Delay | 1 | RTT | x2, Half | high | medium |
| Bit | | | RTT | | |
+---------------+------+----------+----------+-------------+--------+
| SD: Spin | 2 | RTT | x2, Half | high | very |
| Bit & Delay | | | RTT | | high |
| Bit * | | | | | |
+---------------+------+----------+----------+-------------+--------+
Table 1: Delay Comparison
x2 Same metric for both directions
* Both bits work independently; an observer could use less
accurate Spin bit measurements when Delay bit ones are
unavailable.for loss measurement, each row in Table 2 is a loss-marking method 🍭 for each method, the table tells you the number of bits needed in the header, the available metrics using a unidirectional or bidirectional observer, applicable protocols, measurement fidelity, and delay 📋
+============+====+==========================+====+=================+
| Method |Bits| Available Loss Metrics |Prto| Measurement |
| | | | | Aspects |
| | +============+=============+ +==========+======+
| | | UniDir | BiDir | | Fidelity |Delay |
| | | Observer | Observer | | | |
+============+====+============+=============+====+==========+======+
| T: Round- | $1 | RT | x2, Half RT | * |Rate by |~6 RTT|
| Trip Loss | | | | |sampling | |
| Bit | | | | |1/3 to | |
| | | | | |1/(3*ppa) | |
| | | | | |of pkts | |
| | | | | |over 2 | |
| | | | | |RTT | |
+------------+----+------------+-------------+----+----------+------+
| Q: sQuare | 1 | Upstream | x2 | * |Rate over |N pkts|
| Bit | | | | |N pkts |(e.g.,|
| | | | | |(e.g., |64) |
| | | | | |64) | |
+------------+----+------------+-------------+----+----------+------+
| L: Loss | 1 | E2E | x2 | # |Loss |Min: |
| Event Bit | | | | |shape |RTT, |
| | | | | |(and |Max: |
| | | | | |rate) |RTO |
+------------+----+------------+-------------+----+----------+------+
| QL: sQuare | 2 | Upstream | x2 | # |see Q |see Q |
| + Loss Ev. | +------------+-------------+----+----------+------+
| Bits | | Downstream | x2 | # |see Q|L |see L |
| | +------------+-------------+----+----------+------+
| | | E2E | x2 | # |see L |see L |
+------------+----+------------+-------------+----+----------+------+
| QR: sQuare | 2 | Upstream | x2 | * |Rate over |see Q |
| + Ref. Sq. | +------------+-------------+----+N*ppa +------+
| Bits | | 3/4 RT | x2 | * |pkts (see |N*ppa |
| | +------------+-------------+----+Q bit for |pkts |
| | | !E2E | E2E, | * |N) |(see Q|
| | | | Downstream, | | |bit |
| | | | Half RT | | |for N)|
+------------+----+------------+-------------+----+----------+------+
Table 2: Loss Comparison
* All protocols
# Protocols employing loss detection (with or without pure ACK
loss detection)
$ Require a working Spin bit
! Metric relative to the opposite channel
x2 Same metric for both directions
ppa Packets-Per-Ack
Q|L See Q if Upstream loss is significant; L otherwise
E2E End to end4.1. Implementation Considerations 🛠️
by combining the info from the two tables above, you can deduce that the solutions with 3 bits (i.e., QL or QR + S or D) or 4 bits (i.e., QL or QR + SD) allow for more complete and resilient measurements 💪✨
the methodologies from the previous sections are transport agnostic, so they can be applied in lots of situations 🌍 which method you pick also depends on the specific protocol 💅 for example, QL is a good combo 🍓 but if a protocol doesn't support the L bit, or can't set it, QR is the only viable solution 🥺
5. Examples of Application 🎬
this document describes several measurement methods, but it's not expected that all of them will be implemented together 🙈 for example, only some of the methods here (i.e., sQuare bit and Spin bit) are used in [CORE-COAP-PM] 🧚 also, the binding of a delay signal to QUIC is partly described in Section 17.4 of [QUIC-TRANSPORT], which adds only the Spin bit to the first byte of the short packet header, leaving two reserved bits for future use (see Section 17.2.2 of [QUIC-TRANSPORT]) 🦋
all the signals discussed in this document have been implemented in successful experiments for both QUIC and TCP 🏆 the scenarios considered let you monitor the interconnections inside a data center (Intra-DC), between data centers (Inter-DC), and end-to-end large-scale data transfers 🌐 for applying the methods here, it's assumed that the monitored flows follow stable paths and traverse the same measurement points 🛣️
the specific implementation details and the choice of bits used for the QUIC and TCP experiments are out of scope for this document 🙅♀️ a specification that defines the specific protocol application is expected to discuss implementation details depending on which bits will be implemented in the protocol, e.g., [CORE-COAP-PM] 💅 if bits used for specific measurements can also be used for other purposes by a protocol, that specification is expected to address ways for on-path observers to disambiguate the signals, or to discuss limitations on the conditions under which observers can expect a valid signal 🔍💖
6. Protocol Ossification Considerations 🧊
accurate loss and delay info is not required for the operation of any protocol 🙂 but having it on a sufficient number of flows is important for how networks run 🌸
the delay and loss bits are amenable to "greasing" described in [RFC8701] if protocol designers aren't ready to dedicate (and ossify) the bits used for loss reporting to this function 🎲 the greasing could be done similarly to the latency Spin bit greasing in Section 17.4 of [QUIC-TRANSPORT] 🧁 for example, protocol designers could decide that a fraction of flows should not encode loss and delay info, and instead the bits would be set to arbitrary values 🎭 setting any of the bits described here to arbitrary values would make the corresponding delay and loss info look like noise rather than the expected signal for the flow 📻 so observers would need to be ready to ignore those flows 🙈✨
7. Security Considerations 🔐✨
okay bestie, the methods here are transport agnostic 🚀, which means they could work with basically any transport-layer protocol. They're especially valuable for encrypted protocols 💖. They can be applied inside limited domains and across the Internet too, depending on the specific protocol application 🌐.
Passive loss and delay observations have been part of network operations for a long, long time 🕰️. So exposing loss and delay information to the network doesn't add new security concerns for protocols that are already observable 🙌.
Without packet loss, the Q and R bit signals don't reveal anything you couldn't already learn by counting packets crossing a network path 🔢. With packet loss, Q and R bits will disclose the loss. But that's information about the environment, not the endpoint state 🌦️. The L bit signal discloses internal state of the protocol's loss-detection machinery 🕵️, though that state can often be gleaned by timing packets and watching how the congestion controller responds 👀.
The measurements described here do not imply that new packets injected into the network can cause potential harm to the network itself or to data traffic. The measurements could be harmed by an attacker altering the marking of the packets or injecting artificial traffic 😤. Authentication techniques may be used where appropriate to guard against these traffic attacks 🛡️.
Hence, loss bits do not provide a viable new mechanism to attack data integrity and secrecy 💅.
The measurement fields introduced here are intended to be included in the packets. However, it's worth mentioning that it may be possible to use this information as a covert channel 🫣.
This document doesn't define a specific application, and the techniques can generally apply to different communication protocols operating in different security environments 🦋. A specification defining a specific protocol application is expected to address the respective security considerations and must consider specifics of the protocol and its expected operating environment. For example, security considerations for QUIC, discussed in Section 21 of [QUIC-TRANSPORT] and Section 9 of [QUIC-TLS], consider a possibility of active and passive attackers in the network as well as attacks on specific QUIC mechanisms 🧚.
7.1. Optimistic ACK Attack 🎭
A defense against an optimistic ACK attack, described in Section 21.4 of [QUIC-TRANSPORT], involves a sender randomly skipping packet numbers to detect a receiver acknowledging packet numbers that have never been received. The Q bit signal may inform the attacker which packet numbers were skipped on purpose and which had actually been lost (and are, therefore, safe for the attacker to acknowledge) 👀. To use the Q bit for this purpose, the attacker must first receive at least an entire Q Block of packets, which renders the attack ineffective against a delay-sensitive congestion controller 🙅♀️.
A protocol that is more susceptible to an optimistic ACK attack with the loss signal provided by the Q bit, and that uses a loss-based congestion controller, should shorten the current Q Block by the number of skipped packet numbers. For example, skipping a single packet number will invert the square signal one outgoing packet sooner 📉.
Similar considerations apply to the R bit. A shortened Reflection Block along with a matching skip in packet numbers does not necessarily imply a lost packet, since it could be due to a lost packet on the reverse path along with a deliberately skipped packet by the sender 🔄.
7.2. Delay Bit with RTT Obfuscation 🫥
Theoretically, delay measurements can be used to roughly evaluate the distance of the client from the server (using the RTT) or from any intermediate observer (using the client-observer half-RTT) 📍. As described in [RTT-PRIVACY], connection RTT measurements for geolocating endpoints are usually inferior to even the most basic IP geolocation databases. Honestly, the variability within RTT measurements (the jitter) is the most informative part 🤓. It can reveal insight into the operating environment of the endpoints, as well as the state of the networks (queuing delays) used by the connection 🌊.
Nevertheless, to further mask the actual RTT of the connection, the Delay bit algorithm can be slightly modified by, for example, delaying the client-side reflection of the delay sample by a fixed, randomly chosen time value 🎲. This would lead an intermediate observer to measure a delay greater than the real one 🎀.
This Additional Delay should be randomly selected by the client and kept constant for a certain amount of time across multiple connections. This ensures that the client-server jitter remains the same as if no Additional Delay had been inserted 🧘♀️. For example, a new Additional Delay value could be generated whenever the client's IP address changes 🔁.
Despite the Additional Delay, this Hidden Delay technique still allows an accurate measurement of the RTT components (observer-server) and all the intra-domain measurements used to distribute the delay in the network 📏. Furthermore, unlike the Delay bit, the Hidden Delay bit does not require the use of the client reflection threshold (1 ms by default). Removing this threshold may lead to an increase in the number of valid measurements produced by the algorithm 📈.
Note that the Hidden Delay bit does not affect an observer's ability to measure accurate RTT using other means, such as timing packets exchanged during connection establishment ⏱️.
8. Privacy Considerations 🌸
To minimize unintentional exposure of information, loss bits provide an explicit loss signal. That's a preferred way to share information, per [RFC8558] 🫶.
New protocols commonly have specific privacy goals, and loss reporting must ensure that loss information does not compromise those goals 🙈. For example, [QUIC-TRANSPORT] allows changing Connection IDs in the middle of a connection to reduce the likelihood of a passive observer linking old and new sub-flows to the same device (see Section 5.1 of [QUIC-TRANSPORT]) 🕶️. A QUIC implementation would need to reset all counters when it changes the destination (IP address or UDP port) or the Connection ID used for outgoing packets. It would also need to avoid incrementing the Unreported Loss counter for loss of packets sent to a different destination or with a different Connection ID 🧹.
It's also worth highlighting that, if these techniques are not widely deployed, an endpoint that uses them may be fingerprinted based on their usage . However, since there is no release of user data, the techniques seem unlikely to substantially increase the existing privacy risks 💜.
Furthermore, if there is experimental traffic with these bits set on the network, a network operator could potentially prioritize this marked traffic by placing it in a priority queue 🚦. This may result in better service being delivered, which could potentially mislead an experiment intended to benchmark the network 🧪.
9. IANA Considerations 📋
This document has no IANA actions. Nothing to register here, bestie 🍓.
10. References 📚
10.1. Normative References
[ECN] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<https://www.rfc-editor.org/info/rfc3168>.
[IPPM-METHODS]
Morton, A., "Active and Passive Metrics and Methods (with
Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799,
May 2016, <https://www.rfc-editor.org/info/rfc7799>.
[QUIC-TRANSPORT]
Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
Multiplexed and Secure Transport", RFC 9000,
DOI 10.17487/RFC9000, May 2021,
<https://www.rfc-editor.org/info/rfc9000>.
[RFC8558] Hardie, T., Ed., "Transport Protocol Path Signals",
RFC 8558, DOI 10.17487/RFC8558, April 2019,
<https://www.rfc-editor.org/info/rfc8558>.
[TCP] Eddy, W., Ed., "Transmission Control Protocol (TCP)",
STD 7, RFC 9293, DOI 10.17487/RFC9293, August 2022,
<https://www.rfc-editor.org/info/rfc9293>.10.2. Informative References ✨
okay bestie, these are the extra-credit reads 📚 they're not required to get the main idea, but they're where the rest of the story lives 🦋
- [ACCURATE-ECN] Briscoe, B., Kühlewind, M., and R. Scheffenegger, "More Accurate Explicit Congestion Notification (ECN) Feedback in TCP", Work in Progress, Internet-Draft, draft-ietf-tcpm-accurate-ecn-26, 24 July 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-26>. - [AltMark] Fioccola, G., Ed., Cociglio, M., Mirsky, G., Mizrahi, T., and T. Zhou, "Alternate-Marking Method", RFC 9341, DOI 10.17487/RFC9341, December 2022,
<https://www.rfc-editor.org/info/rfc9341>. - [ANRW19-PM-QUIC] Bulgarella, F., Cociglio, M., Fioccola, G., Marchetto, G., and R. Sisto, "Performance measurements of QUIC communications", Proceedings of the Applied Networking Research Workshop (ANRW '19), Association for Computing Machinery, DOI 10.1145/3340301.3341127, July 2019,
<https://doi.org/10.1145/3340301.3341127>. - [ConEx] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) Concepts, Abstract Mechanism, and Requirements", RFC 7713, DOI 10.17487/RFC7713, December 2015,
<https://www.rfc-editor.org/info/rfc7713>. - [ConEx-TCP] Kuehlewind, M., Ed. and R. Scheffenegger, "TCP Modifications for Congestion Exposure (ConEx)", RFC 7786, DOI 10.17487/RFC7786, May 2016,
<https://www.rfc-editor.org/info/rfc7786>. - [CORE-COAP-PM] Fioccola, G., Zhou, T., Nilo, M., Milan, F., and F. Bulgarella, "Constrained Application Protocol (CoAP) Performance Measurement Option", Work in Progress, Internet-Draft, draft-ietf-core-coap-pm-01, 19 October 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-core-coap-pm-01>. - [IPPM-SPIN] Trammell, B., Ed., "An Explicit Transport-Layer Signal for Hybrid RTT Measurement", Work in Progress, Internet-Draft, draft-trammell-ippm-spin-00, 9 January 2019,
<https://datatracker.ietf.org/doc/html/draft-trammell-ippm-spin-00>. - [IPv6AltMark] Fioccola, G., Zhou, T., Cociglio, M., Qin, F., and R. Pang, "IPv6 Application of the Alternate-Marking Method", RFC 9343, DOI 10.17487/RFC9343, December 2022,
<https://www.rfc-editor.org/info/rfc9343>. - [QUIC-MANAGEABILITY] Kühlewind, M. and B. Trammell, "Manageability of the QUIC Transport Protocol", RFC 9312, DOI 10.17487/RFC9312, September 2022,
<https://www.rfc-editor.org/info/rfc9312>. - [QUIC-SPIN] Trammell, B., Ed., De Vaere, P., Even, R., Fioccola, G., Fossati, T., Ihlar, M., Morton, A., and S. Emile, "Adding Explicit Passive Measurability of Two-Way Latency to the QUIC Transport Protocol", Work in Progress, Internet-Draft, draft-trammell-quic-spin-03, 14 May 2018,
<https://datatracker.ietf.org/doc/html/draft-trammell-quic-spin-03>. - [QUIC-TLS] Thomson, M., Ed. and S. Turner, Ed., "Using TLS to Secure QUIC", RFC 9001, DOI 10.17487/RFC9001, May 2021,
<https://www.rfc-editor.org/info/rfc9001>. - [RFC8701] Benjamin, D., "Applying Generate Random Extensions And Sustain Extensibility (GREASE) to TLS Extensibility", RFC 8701, DOI 10.17487/RFC8701, January 2020,
<https://www.rfc-editor.org/info/rfc8701>. - [RTT-PRIVACY] Trammell, B. and M. Kühlewind, "Revisiting the Privacy Implications of Two-Way Internet Latency Data", Passive and Active Measurement, pp. 73-84, Springer International Publishing, DOI 10.1007/978-3-319-76481-8_6, ISBN 9783319764801, March 2018,
<https://doi.org/10.1007/978-3-319-76481-8_6>. - [TRANSPORT-ENCRYPT] Fairhurst, G. and C. Perkins, "Considerations around Transport Header Confidentiality, Network Operations, and the Evolution of Internet Transport Protocols", RFC 9065, DOI 10.17487/RFC9065, July 2021,
<https://www.rfc-editor.org/info/rfc9065>. - [TSVWG-SPIN] Trammell, B., Ed., "A Transport-Independent Explicit Signal for Hybrid RTT Measurement", Work in Progress, Internet-Draft, draft-trammell-tsvwg-spin-00, 2 July 2018,
<https://datatracker.ietf.org/doc/html/draft-trammell-tsvwg-spin-00>. - [UDP-OPTIONS] Touch, J., "Transport Options for UDP", Work in Progress, Internet-Draft, draft-ietf-tsvwg-udp-options-23, 15 September 2023,
<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-udp-options-23>. - [UDP-SURPLUS] Herbert, T., "UDP Surplus Header", Work in Progress, Internet-Draft, draft-herbert-udp-space-hdr-01, 8 July 2019,
<https://datatracker.ietf.org/doc/html/draft-herbert-udp-space-hdr-01>.
Acknowledgments 🌸
The authors would like to thank the QUIC WG for their contributions, Christian Huitema for implementing Q and L bits in his picoquic stack, and Ike Kunze for providing constructive reviews and helpful suggestions. 🫶 we love a team that shows up with receipts 💅
Contributors 💜
The following people provided valuable contributions to this document:
Marcus Ihlar
Ericsson
Email: marcus.ihlar@ericsson.comJari Arkko
Ericsson
Email: jari.arkko@ericsson.comEmile Stephan
Orange
Email: emile.stephan@orange.comDmitri Tikhonov
LiteSpeed Technologies
Email: dtikhonov@litespeedtech.comAuthors' Addresses 🎀
Mauro Cociglio
Telecom Italia - TIM
Via Reiss Romoli, 274
10148 Torino
Italy
Email: mauro.cociglio@outlook.comAlexandre Ferrieux
Orange Labs
Email: alexandre.ferrieux@orange.comGiuseppe Fioccola
Huawei Technologies
Riesstrasse, 25
80992 Munich
Germany
Email: giuseppe.fioccola@huawei.comIgor Lubashev
Akamai Technologies
Email: ilubashe@akamai.comFabio Bulgarella
Telecom Italia - TIM
Via Reiss Romoli, 274
10148 Torino
Italy
Email: fabio.bulgarella@guest.telecomitalia.itMassimo Nilo
Telecom Italia - TIM
Via Reiss Romoli, 274
10148 Torino
Italy
Email: massimo.nilo@telecomitalia.itIsabelle Hamchaoui
Orange Labs
Email: isabelle.hamchaoui@orange.comRiccardo Sisto
Politecnico di Torino
Email: riccardo.sisto@polito.itThat's the whole spec, bestie! 🌈✨ Thanks for sticking with me through all 9 parts 🥹💖
📦 where this page comes from
- ietf: rfc9506 2023-10
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms