RFC 6703
Reporting IP Network Performance Metrics: Different Points of View
RFC 6703: Reporting IP Network Performance Metrics: Different Points of View 🎀
Okay bestie, let's start with the vibes. This doc is RFC 6703, titled Reporting IP Network Performance Metrics: Different Points of View 💖 It was written by A. Morton, G. Ramachandran, and G. Maguluri, all from AT&T Labs. It's an Informational doc (not a Standards Track spec 👀), with ISSN 2070-1721, and it's dated August 2012. The header doesn't list any RFCs it obsoletes or updates, so it's a standalone moment 🦋
Abstract ✨
Consumers of IP network performance metrics have tons of different uses in mind 🧚 This memo gives "long-term" reporting considerations (think hours, days, weeks, or months, NOT 10 seconds 💀) based on looking at the points of view of two key audiences. It explains how those audience categories change which metric parameters and options you pick when you're trying to get info that actually serves them. Main character energy 👑
Status of This Memo 📜
Real talk, this isn't an Internet Standards Track specification. It's published for informational purposes 🌸
It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so see Section 2 of RFC 5741 if you want the deets 🫶
Want the current status, any errata, or a way to send feedback? Check out http://www.rfc-editor.org/info/rfc6703 ✨
Copyright Notice ©️
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please review these documents carefully, because they describe your rights and restrictions with respect to this document 🙈 Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License.
This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of that material may not have granted the IETF Trust the right to allow modifications outside the IETF Standards Process. Without an adequate license from those copyright holders, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process. The one exceptions are formatting it for publication as an RFC or translating it into languages other than English 🌈
Table of Contents 🗂️
1. Introduction ....................................................4
2. Purpose and Scope ...............................................4
3. Reporting Results ...............................................5
3.1. Overview of Metric Statistics ..............................5
3.2. Long-Term Reporting Considerations .........................6
4. Effect of POV on the Loss Metric ................................8
4.1. Loss Threshold .............................................8
4.1.1. Network Characterization ............................8
4.1.2. Application Performance ............................11
4.2. Errored Packet Designation ................................11
4.3. Causes of Lost Packets ....................................11
4.4. Summary for Loss ..........................................12
5. Effect of POV on the Delay Metric ..............................12
5.1. Treatment of Lost Packets .................................12
5.1.1. Application Performance ............................13
5.1.2. Network Characterization ...........................13
5.1.3. Delay Variation ....................................14
5.1.4. Reordering .........................................15
5.2. Preferred Statistics ......................................15
5.3. Summary for Delay .........................................16
6. Reporting Raw Capacity Metrics .................................16
6.1. Type-P Parameter ..........................................17
6.2. A priori Factors ..........................................17
6.3. IP-Layer Capacity .........................................17
6.4. IP-Layer Utilization ......................................18
6.5. IP-Layer Available Capacity ...............................18
6.6. Variability in Utilization and Available Capacity .........19
6.6.1. General Summary of Variability .....................19
7. Reporting Restricted Capacity Metrics ..........................20
7.1. Type-P Parameter and Type-C Parameter .....................21
7.2. A Priori Factors ..........................................21
7.3. Measurement Interval ......................................22
7.4. Bulk Transfer Capacity Reporting ..........................22
7.5. Variability in Bulk Transfer Capacity .....................23
8. Reporting on Test Streams and Sample Size ......................23
8.1. Test Stream Characteristics ...............................23
8.2. Sample Size ...............................................24
9. Security Considerations ........................................25
10. Acknowledgements ..............................................25
11. References ....................................................25
11.1. Normative References .....................................25
11.2. Informative References ...................................261. Introduction ✨
When you design measurements of IP networks and present a result, knowing your audience is a key consideration 💅 To present a useful and relevant portrait of network conditions, you have to answer this question:
"How will the results be used?" 🤔
There are two main audience categories for the report of results:
- Network Characterization: describes conditions in an IP network for quality assurance, troubleshooting, modeling, Service Level Agreements (SLAs), etc. This point of view (POV) looks inward toward the network, where the report consumer intends to take their actions 🔧
- Application Performance Estimation: describes the network conditions in a way that helps figure out the effects on user applications, and ultimately the users themselves 🫶 This POV looks outward, toward the user(s), and accepts the network as is. This report consumer wants to estimate a network-dependent aspect of performance, or design some part of an application's accommodation of the network. (These are not application metrics; they're defined at the IP layer 🙈)
This memo looks at how these different POVs affect both the measurement design (the parameters and options of the metrics) and the statistics reported when serving the report consumer's needs 🦋
The IP Performance Metrics (IPPM) Framework [RFC2330] and other RFCs describing IPPM give background for this memo 🌸
2. Purpose and Scope 🎯
The purpose of this memo is to clearly delineate two POVs for using measurements, and to describe their effects on test design, including the selection of metric parameters and the reporting of results 💖
The scope mainly covers test design and reporting for the loss and delay metrics [RFC2680] [RFC2679]. It also discusses the delay variation [RFC3393] and reordering metrics [RFC4737] where applicable ✨
Capacity metrics are getting more and more relevant to the industry, so the memo also covers POV and reporting considerations for metrics that come from the Bulk Transfer Capacity Framework [RFC3148] and Network Capacity Definitions [RFC5136]. These memos describe two different categories of metrics 🌈
- Restricted [RFC3148]: includes restrictions on congestion control and the notion of unique data bits delivered, and
- Raw [RFC5136]: uses a definition of raw capacity without the restrictions of data uniqueness or congestion awareness.
At first glance, you might think each of these metrics has one obvious audience (raw = network characterization, restricted = application performance) 🤷♀️ But reality is messier, and that fits the overall topic of capacity measurement and reporting. For example, TCP is usually used in restricted capacity measurement methods, while UDP shows up in raw capacity measurement 🧚 The raw and restricted capacity metrics get their own sections, even though they share one common reporting issue: representing variability in capacity metric results as part of a long-term report 💜
Sampling (the design of the active packet stream that the measurements are based on) is discussed as well 🍓
3. Reporting Results 📊
This section gives an overview of recommendations, followed by additional considerations for reporting results in the "long term". Those considerations are based on the discussion and conclusions of the major sections that follow ⭐
3.1. Overview of Metric Statistics ✨
okay so this section is basically the cheat sheet for every metric in this memo 📝🎀
the minimal report needs BOTH loss and delay. no skipping either one, bestie 🫶
for packet loss, the loss ratio from [RFC2680] is a solid starting point 💅 especially the existing advice on setting the loss threshold waiting time. in Section 4.1.1 we worked out a waiting time of 51 seconds that should be enough to tell apart packets that are truly lost from packets with really long finite delays, under general measurement conditions. if you know more about your specific conditions you can shrink that threshold, and around 50 seconds is considered manageable in practice 🙈
we also note that a loss ratio calculated according to [Y.1540] would exclude errored packets from the numerator. in practice the difference between these two loss metrics is small, if there's any difference at all 🤷♀️ it depends on whether the last link before the Destination contributes errored packets
for packet delay, we recommend reporting BOTH the mean delay and the median delay, with lost packets marked as undefined (which [RFC2679] lets you do) 🍓 both stats are based on a conditional distribution. the condition is that the packet arrives before a waiting time dT, and dT is set to account for maximum packet lifetimes, same as we talked about for loss. using a long dT helps keep your delay distributions from getting chopped off ✂️
for Packet Delay Variation (PDV), the minimum delay of the conditional distribution should be used as the reference delay when computing PDV according to [Y.1540] or [RFC5481] and [RFC3393] 🦋 a useful thing to report is a "pseudo" range of delay variation, which is the difference between a high percentile of delay and the minimum delay. for example, the 99.9th percentile minus the minimum gives you a number you can compare against the objectives in [Y.1541] ⭐
for both raw capacity and restricted capacity, the main challenge is reporting the variability in a useful way 😭 the min, max, and range stats are suggested, along with a ratio of max to min and moving averages. honestly, a simple plot of the singleton results over time might work where summary metrics fail, or it can confirm that your summaries are valid 📈✨
3.2. Long-Term Reporting Considerations 🌸
[IPPM-RPT] describes how to conduct measurements and report results on a near-immediate time scale (10 seconds, which we consider to be "short-term") ⏱️
measurement intervals and reporting intervals don't have to be the same length 💖 sometimes the user only cares about the performance over a pretty long stretch of time (like days, weeks, or months, instead of 10 seconds). BUT running a measurement nonstop over a long period without recording intermediate results can be risky 😬 here's what can go wrong:
- a temporary power failure may cause loss of all results to date.
- measurement system timing synchronization signals may experience a temporary outage, causing subsets of measurements to be in error or invalid.
- maintenance on the measurement system or on its connectivity to the network under test may be necessary.
and for other reasons, like these 💅
- the constraint to collect measurements on intervals similar to user session length,
- the dual use of measurements in monitoring activities where results are needed on a period of a few minutes, or
- the ability to inspect results of a single measurement interval for deeper analysis,
there's real value in conducting measurements on intervals that are much shorter than the reporting interval 🧚
there are a few ways to aggregate a series of measurement results over time so you can make a statement about the longer reporting interval. one approach requires storing ALL the metric singletons collected throughout the reporting interval, even though the measurement interval stops and starts many times 📦
another approach is described in [RFC5835] as "temporal aggregation" 🎀 this one estimates the results for the reporting interval by combining lots of individual short-term measurement interval statistics into one long-term result. ideally, the result would look the same as if a continuous measurement had been conducted. a memo addressing the details of temporal aggregation is yet to be prepared 👀
yet another approach needs a numerical objective for the metric. the results of each measurement interval get compared with the objective, and every interval where the results meet the objective counts toward the fraction of time with performance as specified 💜 when the reporting interval contains many measurement intervals, you can present results like "metric A was less than or equal to objective X during Y% of time".
NOTE that numerical thresholds of acceptability are not set in IETF performance work and are therefore excluded from the scope of this memo.
in all measurements, it's important to avoid unintended synchronization with network events 🙅♀️ this topic is covered in [RFC2330] for Poisson-distributed inter-packet time streams and in [RFC3432] for Periodic streams. both avoid synchronization by using random start times 🎲
there are also network conditions where it's just more useful to report the connectivity status of the Source-Destination path. you want to tell apart the time intervals where connectivity can be demonstrated from the ones where connectivity doesn't appear to exist 🔌 [RFC2678] specifies a bunch of one-way and two-way connectivity metrics, getting more complex as you go. in this memo, we recommend that long-term reporting of loss, delay, and other metrics be limited to time intervals where connectivity can be demonstrated. the other intervals get summarized as the percent of time where connectivity does not appear to exist. this is the same approach ITU-T Recommendation [Y.1540] took 👑 there, performance parameters are only valid during periods of service "availability" (evaluated with a function based on packet loss), and sustained periods of loss ratio greater than a threshold are declared "unavailable" 📉
4. Effect of POV on the Loss Metric ✨
this section describes the ways the loss metric can be tuned to reflect the preferences of the two audience categories, or different POVs 🦋 the waiting time before declaring a packet lost (the loss threshold) is one area where there seems to be a difference, but being able to post-process the results may resolve it 💅
4.1. Loss Threshold 🌈
[RFC2680] defines the concept of a waiting time for packets to arrive, beyond which they're declared lost. the RFC declines to recommend a value and says instead that "good engineering, including an understanding of packet lifetimes, will be needed in practice". later, in the methodology, they give reasons for waiting "a reasonable period of time" and leave the definition of "reasonable" intentionally vague 🙃 so below, we estimate a practical bound on waiting time 🔥
4.1.1. Network Characterization 🔍
practical measurement experience has shown that unusual network circumstances can cause long delays 😳 one of them is when routing loops form during IGP re-convergence after a failure or a drastic link cost change. packets will loop between two routers until new routes are installed, or until the IPv4 Time-to-Live (TTL) field (or the IPv6 Hop Limit) decrements to zero. very long delays on the order of several seconds have been measured [Casner] [Cia03] 💀
so network characterization activities prefer a long waiting time, so they can tell these events apart from other causes of loss (like packet discard at a full queue, or tail drop) 🫶 this way, the metric design helps distinguish more reliably between packets that might still arrive and packets that are no longer traversing the network ✨
it's possible to calculate a worst-case waiting time, assuming a routing loop is the cause 🧮 we model the path between Source and Destination as a series of delays in links (t) and queues (q), because these are the dominant contributors to delay. (in active measurement, the Source and Destination hosts contribute minimal delay.) the normal path delay, D, across n queues (where TTL is decremented at a node with a queue) and n+1 links without encountering a loop is:
Path model with n=5
Source --- q1 --- q2 --- q3 --- q4 --- q5 --- Destination
t0 t1 t2 t3 t4 t5
n
---
\
D = t + > (t + q)
0 / i i
---
i = 1
Figure 1: Normal Path Delayand the time spent in the loop with L queues is:
Path model with n=5 and L=3
Time in one loop = (qx+tx + qy+ty + qz+tz)
qy -- qz
| ?/exit?
qx--/\
Src --- q1 --- q2 ---/ q3 --- q4 --- q5 --- Dst
t0 t1 t2 t3 t4 t5
j + L-1
---
\ (TTL - n)
R = C > (t + q) where C = ---------
/ i i max L
---
i=j
Figure 2: Delay Due to Rotations in a Loopwhere n is the total number of queues in the non-loop path (with n+1 links), j is the queue number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value at the Source (or Hop Count in IPv6) 🌀
if we take the delays of all links and queues as 100 ms each, TTL=255, the number of queues n=5, and the queues in the loop L=4, then using C_max:
D = 1.1 seconds and R ~= 50 seconds, and D + R ~= 51.1 secondswe note that link delays of 100 ms would span most continents 🌍 and a constant queue length of 100 ms is also very generous 🙈 when a loop occurs, it's almost certain to be resolved in 10 seconds or less. the value calculated above is an upper limit for almost any real-world circumstance 💖
a waiting time threshold parameter, dT, set consistent with this calculation, would not truncate the delay distribution (which could change its mathematical properties), because the packets that might arrive have been given enough time to traverse the network 🧚♀️
and it's worth noting that packets stored and deliberately forwarded at a much later time count as a replay attack on the measurement system 🚨 those are beyond the scope of normal performance reporting 🔒
4.1.2. Application Performance ✨
Okay so good news: app performance estimates aren't messed up by that long waiting time, bc most apps are totally fine with way shorter time limits 💅 The designer might be tempted to set the loss threshold to match one specific app's limit, but that threshold can be applied later, during post-processing 🎀 A shorter wait works like this: find packets with delays longer than the app's threshold and re-designate them as lost. So the measurement system can use one loss waiting time and support both the network and application POVs at the same time 👑
4.2. Errored Packet Designation 🦋
RFC 2680 designates packets that arrive containing errors as lost packets. A lot of packets that get corrupted by bit errors are discarded inside the network and never reach their intended destination 😭
That's consistent with apps that check payload integrity at higher layers and then discard the packet. BUT some apps prefer to deal with errored payloads on their own, and honestly, even a corrupted payload is better than no packet at all 🥺
So to cover that possibility, and to make network characterization more complete, it is recommended to tell apart packets that do not arrive (lost) from errored packets that do arrive (conditionally lost) 🧚
4.3. Causes of Lost Packets 💀
Many measurement systems use a waiting time to determine whether or not a packet is lost, but most of that waiting is in vain. The packets are no longer traversing the network and have not reached their destination 🙈
There are many causes of packet loss, including the following:
- Queue drop, or discard
- Corruption of the IP header, or other essential header information
- TTL expiration (or use of a TTL value that is too small)
- Link or router failure
- Layers below the Source-to-Destination IP layer can discard packets that fail error checking, and link-layer checksums often cover the entire packet
It is reasonable to consider a packet that has not arrived after a large amount of time to be lost (due to one of the causes above) because packets do not "live forever" in the network or have infinite delay ⏳
4.4. Summary for Loss 🍓
Given that measurement post-processing is possible (even encouraged in the definitions of IPPM), measurements of loss can easily serve both POVs:
- Use a long waiting time to serve network characterization and revise results for specific application delay thresholds as needed.
- Distinguish between errored packets and lost packets when possible to aid network characterization, and combine the results for application performance if appropriate.
5. Effect of POV on the Delay Metric 💜
This section describes the ways in which the delay metric can be tuned to reflect the preferences of the two consumer categories, or different POVs 🌈
5.1. Treatment of Lost Packets
The delay metric [RFC2679] specifies the treatment of packets that do not successfully traverse the network: their delay is undefined.
>>The Type-P-One-way-Delay from Src to Dst at T is undefined (informally, infinite)<< means that Src sent the first bit of a Type-P packet to Dst at wire-time T and that Dst did not receive that packet.
It is an accepted but informal practice to assign infinite delay to lost packets. Next we look at how these two different treatments line up with what measurement consumers need, whether they want to characterize networks or estimate application performance. We also look at the way lost packets have been treated in other metrics: delay variation and reordering 👀
5.1.1. Application Performance
Apps need to do different things depending on whether or not each packet arrives within some finite tolerance. In other words, a receiver's packet processing takes only one of two alternative directions (a "fork" in the road) 🛣️
- Packets that arrive within expected tolerance are handled by removing headers, restoring smooth delivery timing (as in a de-jitter buffer), restoring sending order, checking for errors in payloads, and many other operations.
- Packets that do not arrive when expected lead to attempted recovery from the apparent loss, such as retransmission requests, loss concealment, or forward error correction to replace the missing packet.
So it's important to keep a distinction between packets that actually arrive and those that do not. Therefore, it is preferable to leave the delay of lost packets undefined and to characterize the delay distribution as a conditional distribution (conditioned on arrival) 💖
5.1.2. Network Characterization
In this discussion, we assume that both loss and delay metrics will be reported for network characterization (at least) 📊
Assume that packets that do not arrive are reported as lost, usually as a fraction of all sent packets. If these lost packets are assigned an undefined delay, then the network's inability to deliver them (in a timely way) is only relegated to the loss metric when we report statistics on the delay distribution conditioned on the event of packet arrival (within the loss waiting time threshold). We can say that the delay and loss metrics are orthogonal in that they convey non-overlapping information about the network under test. This is a valuable property whose absence is discussed below ✨
However, if we assign infinite delay to all lost packets, then:
- The delay metric results are influenced both by packets that arrive and those that do not.
- The delay singleton and the loss singleton do not appear to be orthogonal (delay is finite when loss=0; delay is infinite when loss=1).
- The network is penalized in both the loss and delay metrics, effectively double-counting the lost packets.
As further evidence of overlap, consider the Cumulative Distribution Function (CDF) of delay when the value "positive infinity" is assigned to all lost packets. Figure 3 shows a CDF where a small fraction of packets are lost 📉
1 | - - - - - - - - - - - - - - - - - -+
| |
| _..----''''''''''''''''''''
| ,-''
| ,'
| / Mass at
| / +infinity
| / = fraction
|| lost
|/
0 |_____________________________________
0 Delay +o0
Figure 3: Cumulative Distribution Function for Delay
When Loss = +InfinityWe note that a delay CDF that is conditioned on packet arrival would not exhibit this apparent overlap with loss.
Infinity is a familiar mathematical concept, but it is kind of disconcerting to see any time-related metric reported as infinity 😵 Questions are bound to pop up and tend to distract from the goal of informing the consumer with a performance report.
5.1.3. Delay Variation
[RFC3393] excludes lost packets from samples, effectively assigning an undefined delay to packets that do not arrive in a reasonable time. Section 4.1 of [RFC3393] describes this specification and its rationale (ipdv = inter-packet delay variation in the quote below).
The treatment of lost packets as having "infinite" or "undefined" delay complicates the derivation of statistics for ipdv. Specifically, when packets in the measurement sequence are lost, simple statistics such as sample mean cannot be computed. One possible approach to handling this problem is to reduce the event space by conditioning. That is, we consider conditional statistics; namely we estimate the mean ipdv (or other derivative statistic) conditioned on the event that selected packet pairs arrive at the Destination (within the given timeout). While this itself is not without problems (what happens, for example, when every other packet is lost), it offers a way to make some (valid) statements about ipdv, at the same time avoiding events with undefined outcomes.
We note that the argument above applies to all forms of packet delay variation that can be constructed using the "selection function" concept of [RFC3393]. In recent work, the two main forms of delay variation metrics have been compared, and the results are summarized in [RFC5481] 🔍
5.1.4. Reordering
[RFC4737] defines metrics that are based on evaluation of packet arrival order and that include a waiting time before declaring that a packet is lost (to exclude the packet from further processing) 🔄
If packets are assigned a delay value, then the reordering metric would declare any packets with infinite delay to be reordered, because their sequence numbers will surely be less than the "Next Expected" threshold when (or if) they arrive. But this practice would fail to maintain orthogonality between the reordering metric and the loss metric. Confusion can be avoided by designating the delay of non-arriving packets as undefined and reserving delay values only for packets that arrive within a sufficiently long waiting time 🫶
5.2. Preferred Statistics 📈
Today in network characterization, the sample mean is one statistic that is almost ubiquitously reported. It is easily computed and understood by virtually everyone in this audience category. Also, the sample is usually filtered on packet arrival, so the mean is based on a conditional distribution 💅
The median is another statistic that summarizes a distribution, having somewhat different properties from the sample mean. The median is stable in distributions with a few outliers or without them. However, the median's stability prevents it from indicating when a large fraction of the distribution changes value. 50% or more values would need to change for the median to capture the change 🫠
Both the median and sample mean have difficulty with bimodal distributions. The median will reside in only one of the modes, and the mean may not lie in either mode range. For this and other reasons, additional statistics such as the minimum, maximum, and 95th percentile have value when summarizing a distribution 🌸
When both the sample mean and median are available, a comparison will sometimes be informative, because these two statistics are equal only under unusual circumstances, such as when the delay distribution is perfectly symmetrical 🪞
Also, these statistics are generally useful from the application performance POV, so there is a common set that should satisfy audiences 🎀
Plots of the delay distribution may also be useful when single-value statistics indicate that new conditions are present. An empirically derived probability distribution function will usually describe multiple modes more efficiently than any other form of result. Bestie, that's the whole vibe 🦋✨
5.3. Summary for Delay ✨
okay so bestie, here's the big takeaway for delay 💅 If you look at it from three angles, the answer is pretty clear:
- application/receiver analysis, where what happens next depends on whether the packet shows up or times out 📬
- straightforward network characterization that doesn't double-count defects 🧮
- staying consistent with the delay variation and reordering metric definitions 🔁
So the most efficient move is to split things into packets that are truly lost and packets that are just delayed, as long as their waiting time is long enough 🕰️ and to mark the delay of any packet that never arrives as undefined 🚫. No cap, it's the cleanest way to keep everything honest 🫶
6. Reporting Raw Capacity Metrics 🎀
Raw capacity is the stuff defined in [RFC5136], and it doesn't come with extra restrictions like data uniqueness or flow-control response to congestion 🌸
We're looking at IP-layer capacity, utilization (aka used capacity), and available capacity, for individual links and for complete paths 💖 These three form a triad, which is giving very "one affects the other two" energy: knowing one limits the other two (within their allowed range), and knowing two tells you the third ✨ The link metrics also share another vibe: they're all single-measurement-point metrics taken at the egress of a link 🦋 Path capacity and available capacity get derived by looking at the whole set of single-point link measurements and taking the minimum value 👑
6.1. Type-P Parameter 🧚
The idea of "packets of Type-P" comes from [RFC2330], and honestly the Type-P categorization is critical for every form of capacity measurement and reporting 😭 Sorting packets by header fields into different queues and schedulers is super common now 💅 When unused resources get shared across queues, the conditions in every packet category end up affecting capacity and related measurements 🌈 That's one source of variability, and every audience would love to see it reported in a useful, easy-to-understand way 🫶
Communication of Type-P inside the One-Way Active Measurement Protocol (OWAMP) and the Two-Way Active Measurement Protocol (TWAMP) is basically limited to the Diffserv Code Point (DSCP) [RFC4656]. DSCP is the most common qualifier for Type-P, so it's the main one you'll see 🍓
Each audience has its own set of Type-P qualifications and value combinations they care about 💜 Measurements and reports should be flexible enough to show both per-type and aggregate performance ✨
6.2. A priori Factors 🔍
The audience for network characterization might have detailed info about each link in a complete path (because they own it, for example), or info about some of the links but not others, or info about none of them 🤷♀️
Sometimes the measurement audience only knows about one link (the local access link) and still wants to measure one or more raw capacity metrics 🏠 This situation is super common and has inspired a ton of experimental measurement methods (e.g., http://www.caida.org/tools/taxonomy/) 🌟 Lots of these methods respect that users want an answer fast, in one trial, so the measurement interval stays short (a few seconds to a minute) ⏱️ For long-term reporting, you need to summarize a sample of those short-term results 📊
6.3. IP-Layer Capacity 🚀
For links, the theoretical maximum can be worked out from the physical-layer bit rate and the bit rate reduction caused by the layers between the physical layer and IP 🧠 When it's actually measured, though, it also takes extra stuff into account, like how well the sending device can process and forward traffic under different conditions 🙈 For example, routing updates showing up can kick off high-priority processes that temporarily drop the sending rate 📉
So yeah, the measured capacity of a link will vary, and the maximum capacity you observe only applies to a specific time, time interval, and other relevant circumstances ⏳
For paths made of a series of links, it's easy to see how the sources of variability pile up with each link 😵💫 We'll get into variability in more detail below 💅
6.4. IP-Layer Utilization 📈
The ideal metric definition of link utilization [RFC5136] is based on actual usage (bits successfully received during a time interval) and the maximum capacity for that same interval 🎯
In practice, you calculate link utilization by counting the IP-layer (or other layer) octets received over a time interval and dividing by the theoretical maximum number of octets that could have been delivered in that same interval 🧮 A commonly used interval is 5 minutes, and it's been good enough for network operations and design for a while now 🫡 Five minutes is kinda long compared to how long web pages take to download, but short when it comes to big file transfers and TV viewing 📺 It's fair to say a lot of variability gets hidden when you report a single (average) utilization value for each 5-minute interval 🙃 Some performance management systems have started offering 1-minute averages, which is iconic 👏
There's also a limit on how small a useful measurement interval can be 🔬 Intervals around the serialization time for a single Maximum Transmission Unit (MTU) packet will show on/off behavior and report 100% or 0% 💀 The smallest interval needs to be some multiple of MTU serialization time for averaging to actually work 🌸
6.5. IP-Layer Available Capacity 🌷
The available capacity of a link can be calculated using the capacity and utilization metrics 🧩
When available capacity of a link or path is estimated with some measurement technique, the following parameters should be reported:
- Name and reference to the exact method of measurement
- IP packet length, octets (including IP header)
- Maximum capacity that can be assessed in the measurement configuration
- Time duration of the measurement
- All other parameters specific to the measurement method
Lots of available capacity methods have a maximum capacity they can measure, and that maximum might be lower than the link's or path's actual available capacity 😬 So it's important to know the capacity value past which you won't see any more measured improvement 🛑
The application performance estimation audience might have a target capacity value in mind and just want to know if there's enough available capacity 🎀 This case is a lot simpler, as long as the measurable maximum is bigger than the target capacity 💖
6.6. Variability in Utilization and Available Capacity 🌈
Like with most metrics and measurements, checking how consistent or variable the results are gives you an intuitive feel for how confident you can be that a single value represents the other results, or for the spread of the underlying distribution of the singleton measurements 🫶
So how do you measure and summarize utilization so it describes the potential variability in a useful way? 🤔
And how can the variability in available capacity estimates be reported so the confidence in the results comes across too? 🤔
We've got some methods to suggest below 💅
6.6.1. General Summary of Variability 📊
With a set of singleton utilization or available capacity estimates, each one representing the time interval needed to figure out the estimate, we want to describe the variation across the set, kinda like summary statistics of a distribution 🧁 Three useful summary statistics are:
- Minimum,
- Maximum, and
- Range
You can also show the range as a ratio of maximum to minimum value, which makes it super easy to understand 🦋 For example, when maximum = 3*minimum, the max/min ratio is 3, and users might see variability of that order 👀 On the flip side, capacity estimates with a max/min ratio near 1 are pretty consistent and sit close to the central measure or statistic reported ✨
For an ongoing series of singleton estimates, a moving average of n estimates can give you a single value estimate that makes it easier to spot big changes in performance over time 📈 For example, within a window of n singletons observed in time interval t, a percentage change of x% counts as a substantial change and gets reported as an exception 🚨
Honestly, the most informative summary is often a two-axis plot instead of a table of stats, with time on the x-axis and the singleton value on the y-axis 🗓️ The time-series plot can show sudden changes in an otherwise stable range, spot bi-modality easily, and help you quickly check correlation with other time series 🔥 Plots of how often each singleton value shows up are useful for visualizing the variation too 💜
7. Reporting Restricted Capacity Metrics 👑
Restricted capacity is the stuff defined in [RFC3148], and these metrics do include criteria like data uniqueness or flow-control response to congestion 🔒
One main metric we care about is Bulk Transfer Capacity (BTC) for complete paths 🚛 [RFC3148] defines BTC as:
BTC = data_sent / elapsed_timefor a connection with congestion-aware flow control, where data_sent is the total number of unique payload bits (no headers) 📦
Here's a key thing: this definition differs from the raw capacity definition in Section 2.3.1 of [RFC5136], where IP-layer capacity includes all the bits in the IP header and payload 🫢 That means restricted capacity BTC starts out at a disadvantage compared to raw capacity at layers below TCP 😤 And it gets worse when one IP layer is encapsulated in another IP layer or some other tunneling protocol, because more and more of the fundamental transport capacity gets eaten up by header bits that are pure overhead to the BTC measurement 💀
Also, raw and restricted capacity metrics aren't orthogonal in the sense defined in Section 5.1.2 above 🧠 Their info about the network under test definitely overlaps, but they reveal two different and important sides of performance 💅
When you're thinking about the triad of raw capacity metrics, BTC is most akin to the "IP-Type-P Available Path Capacity" 🎯 At least, that's how a network user who wants to know what transmission performance a path might support would see it 🫶
7.1. Type-P Parameter and Type-C Parameter 🧚♀️
The idea of "packets of Type-P" comes from [RFC2330]. The considerations for restricted capacity are identical to the raw capacity section on this topic, with the addition that the various fields and options in the TCP header must be included in the description 🎀
The huge array of TCP flow-control options isn't really captured by Type-P, because they don't live in the TCP header bits 😩 So we're introducing a new notion here: TCP Configuration of "Type-C" 🌟 The elements of Type-C describe all the settings for TCP options and congestion control algorithm variables, including the main form of congestion control in use 🔥 Readers should consider the parameters and variables of [RFC3148] and [RFC6349] when constructing Type-C 🫡
Type-C is giving main character energy 🌈✨
7.2. A Priori Factors ✨
okay so the audience for network characterization might know a LOT about each link on a full path (maybe because they own it 👑), or they might know about some links but not others, or maybe they know nothing at all 🙈
there are cases where the audience only has info about one link, the local access link 🏠, and still wants to measure one or more BTC metrics. the stuff in Section 6.2 applies here too, so go check that out 💅
7.3. Measurement Interval 🦋
bestie, there are limits on what counts as a useful measurement interval for BTC. three things change how long the interval should be:
- measurements can include or skip the 3-way handshake of TCP connection establishment. that handshake takes at least
1.5 * RTT(round-trip time) and includes both the path delay and the host processing time for responses. the catch is that user experience includes the 3-way handshake for every new TCP connection 😭 - measurements can include or skip Slow-Start and focus on a chunk of the transfer that represents "equilibrium" (which needs to be defined for specific situations if you use it). but again, user experience includes Slow-Start for every new TCP connection 💀
- measurements can use a fixed block of data to transfer, where the block size relates to the file size of the application being measured. this gives you variable-size measurement intervals, so a path with faster BTC gets measured for less time than a path with slower BTC. that matters when path impairments are time-varying or transient ⏱️ people also tend to look away when a super huge file is transferring, so they don't really feel that long transfer. they just see the result (success or failure) and maybe an objective transfer time, which usually includes the 3-way handshake, Slow-Start, application file management processing time, and the BTC itself 🌸
individual measurement intervals can be short or long, but you still need to report results over the long term so you can capture the BTC variability between each interval. consistent BTC is a valuable commodity, right up there with the value you actually get 💜
7.4. Bulk Transfer Capacity Reporting 📝
when BTC of a link or path is estimated with some measurement technique, you should report the following parameters:
- Name and reference to the exact method of measurement
- Maximum Transmission Unit (MTU)
- Maximum BTC that can be assessed in the measurement configuration
- Time and duration of the measurement
- Number of BTC connections used simultaneously
- All other parameters specific to the measurement method, especially the congestion control algorithm in use
see also [RFC6349] 🎀
lots of BTC measurement methods have a maximum capacity they can measure, and that maximum might be lower than the available capacity of the link or path. so it's important to specify the measured BTC value past which there will be no measured improvement 🚧
the application performance estimation audience might have a target capacity value and just want to know whether there is enough BTC. this makes measuring link and path capacity a bit simpler, as long as the measurable maximum is higher than the target capacity 🔥
7.5. Variability in Bulk Transfer Capacity 🌈
like with most metrics and measurements, checking how consistent or variable the results are gives the user an intuitive feel for how much confidence they can have that any one value represents the other results, or the underlying distribution those singleton measurements came from 🥹
two big questions are looming:
- what ways can BTC be measured and summarized to describe the potential variability in a useful way?
- how can the variability in BTC estimates be reported, so that the confidence in the results is also conveyed?
-- we suggest the methods listed in Section 6.6.1 above, plus the additional results presentations given in [RFC6349] 💅✨
8. Reporting on Test Streams and Sample Size 🧚
this section covers two key things that get left out of reports all the time: describing the test stream the measurements are based on, and the sample size 👀
8.1. Test Stream Characteristics 🍓
network characterization has traditionally used Poisson-distributed inter-packet spacing, because it gives an unbiased sample 🧠 the average inter-packet spacing can be picked so you can observe specific network phenomena. other test streams are designed to sample some property of the network, like the presence of congestion, link bandwidth, or packet reordering.
if you're measuring a network to make inferences about applications or receiver performance, there are usually efficiencies from using a test stream that looks similar to the sender. sometimes it's essential to synthesize the sender stream, like with BTC estimates. other times it's fine to sample with a "known bias", for example a Periodic stream to estimate real-time application performance 🎵
8.2. Sample Size 📊
sample size is directly tied to the accuracy of your results and plays a critical role in the report. even if you only give the sample size (in number of packets) for each value or summary statistic, it still tells people how confident they can be in the result 💖
in practice, the sample size gets picked by weighing both statistical and practical factors. these include:
- the estimated variability of the quantity being measured.
- the desired confidence in the result (although this may depend on assumptions about the underlying distribution of the measured quantity).
- the effects of active measurement traffic on user traffic.
a sample size might get called "large", but that's a relative and qualitative term, so it's not super helpful on its own 🙄 it's better to say what you're trying to achieve with your sample. for example, you could state an implication: this sample is large enough that a single outlying value at ten times the "typical" sample mean (the mean without the outlying value) would influence the mean by no more than X. 🫶
the Appendix of [RFC2330] says a sample size of 128 singletons worked well for goodness-of-fit testing, while a much bigger size (8192 singletons) almost always failed 💀
9. Security Considerations 🔐
the security considerations that apply to any active measurement of live networks are relevant here too. see the Security Considerations section of [RFC4656] for mandatory-to-implement security features meant to mitigate attacks 🛡️
okay, this part is serious. measurement systems doing long-term measurements are more exposed to threats, because their ports stay open longer to do their job, and that measurement activity on those ports is easier to detect 👀 on top of that, using long packet waiting times gives an attacker a better chance to prepare and launch a replay attack 🚨 please don't skip this one, bestie.
10. Acknowledgements 🫶
the authors want to thank Phil Chimento for suggesting conditional distributions for delay, Steve Konish Jr. for his careful review and suggestions, Dave McDysan and Don McLachlan for useful comments based on their long experience with measurement and reporting, Daniel Genin for spotting the non-orthogonality between raw and restricted capacity metrics (and for noticing that we had previously left this fact out 🙈), and Matt Zekauskas for suggestions on organizing the memo for easier consumption 💜
11. References 📚
11.1. Normative References
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
"Framework for IP Performance Metrics", RFC 2330,
May 1998.
[RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring
Connectivity", RFC 2678, September 1999.
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Delay Metric for IPPM", RFC 2679, September 1999.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC3148] Mathis, M. and M. Allman, "A Framework for Defining
Empirical Bulk Transfer Capacity Metrics", RFC 3148,
July 2001.
[RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay
Variation Metric for IP Performance Metrics (IPPM)",
RFC 3393, November 2002.
[RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
performance measurement with periodic streams", RFC 3432,
November 2002.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M.
Zekauskas, "A One-way Active Measurement Protocol
(OWAMP)", RFC 4656, September 2006.
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
November 2006.
[RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity",
RFC 5136, February 2008.11.2. Informative References
[Casner] Casner, S., Alaettinoglu, C., and C. Kuan, "A Fine-
Grained View of High-Performance Networking",
NANOG 22 Conf., May 20-22 2001,
<http://www.nanog.org/presentations/archive/index.php>.
[Cia03] Ciavattone, L., Morton, A., and G. Ramachandran,
"Standardized Active Measurements on a Tier 1 IP
Backbone", IEEE Communications Magazine, Vol. 41
No. 6, pp. 90-97, June 2003.
[IPPM-RPT] Shalunov, S. and M. Swany, "Reporting IP Performance
Metrics to Users", Work in Progress, March 2011.
[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
Applicability Statement", RFC 5481, March 2009.
[RFC5835] Morton, A., Ed., and S. Van den Berghe, Ed., "Framework
for Metric Composition", RFC 5835, April 2010.
[RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage,
"Framework for TCP Throughput Testing", RFC 6349,
August 2011.
[Y.1540] International Telecommunication Union, "Internet protocol
data communication service - IP packet transfer and
availability performance parameters", ITU-T
Recommendation Y.1540, March 2011.
[Y.1541] International Telecommunication Union, "Network
performance objectives for IP-based services", ITU-T
Recommendation Y.1541, December 2011.Authors' Addresses 💌
Al Morton
AT&T Labs
200 Laurel Avenue South
Middletown, NJ 07748
USA
Phone: +1 732 420 1571
Fax: +1 732 368 1192
EMail: acmorton@att.com
URI: http://home.comcast.net/~acmacm/
Gomathi Ramachandran
AT&T Labs
200 Laurel Avenue South
Middletown, New Jersey 07748
USA
Phone: +1 732 420 2353
EMail: gomathi@att.com
Ganga Maguluri
AT&T Labs
200 Laurel Avenue South
Middletown, New Jersey 07748
USA
Phone: +1 732 420 2486
EMail: gmaguluri@att.com📦 where this page comes from
- ietf: rfc6703 2012-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms