RFC 7290
Test Plan and Results for Advancing RFC 2680 on the Standards Track
RFC 7290: Test Plan and Results for Advancing RFC 2680 on the Standards Track 🎀
okay bestie, this is the RFC that says "we tested the rules for measuring packet loss, and here's how we did it" 📊✨ It's published by the IETF (Internet Engineering Task Force) as Request for Comments: 7290, and the ISSN is 2070-1721.
- Authors: L. Ciavattone (AT&T Labs), R. Geib (Deutsche Telekom), A. Morton (AT&T Labs), M. Wieser (Technical University Darmstadt)
- Category: Informational (so it's not a standard, just vibes and receipts 💅)
- Date: July 2014
- Obsoletes / Updates: the header doesn't list any RFCs here 🦋
Abstract 🌸
okay so this memo gives the test plan and results to help advance RFC 2680 along the Standards Track 🚀. RFC 2680 is a performance metric RFC that defines one-way packet loss metrics. The big idea is that the metric definitions are the main thing to focus on, not the implementations 💖. So this memo describes the test procedures used to check specific metric requirement clauses, to see if they were interpreted and implemented the way they were meant to be 🔍. Two completely independent implementations were tested against the key specifications of RFC 2680 🙌
Status of This Memo ✨
This document is not an Internet Standards Track specification. It's published for informational purposes 💜
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, received public review, and was approved for publication by the Internet Engineering Steering Group (IESG). Not every document the IESG approves is a candidate for any level of Internet Standard, so see Section 2 of RFC 5741 for that 🧐
For info about the current status of this document, any errata, and how to give feedback on it, check out http://www.rfc-editor.org/info/rfc7290 🎀
Copyright Notice 🌈
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved 💅
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 📜
This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English 🫶
Table of Contents 📚
1. Introduction ....................................................3
1.1. Requirements Language ......................................4
1.2. RFC 2680 Coverage ..........................................5
2. A Definition-Centric Metric Advancement Process .................5
3. Test Configuration ..............................................5
4. Error Calibration and RFC 2680 ..................................9
4.1. Clock Synchronization Calibration ..........................9
4.2. Packet Loss Determination Error ...........................10
5. Predetermined Limits on Equivalence ............................10
6. Tests to Evaluate RFC 2680 Specifications ......................11
6.1. One-Way Loss: ADK Sample Comparison .......................11
6.1.1. 340B/Periodic Cross-Implementation Results .........12
6.1.2. 64B/Periodic Cross-Implementation Results ..........14
6.1.3. 64B/Poisson Cross-Implementation Results ...........15
6.1.4. Conclusions on the ADK Results for One-Way
Packet Loss ........................................16
6.2. One-Way Loss: Delay Threshold .............................16
6.2.1. NetProbe Results for Loss Threshold ................17
6.2.2. Perfas+ Results for Loss Threshold .................17
6.2.3. Conclusions for Loss Threshold .....................17
6.3. One-Way Loss with Out-of-Order Arrival ....................17
6.4. Poisson Sending Process Evaluation ........................19
6.4.1. NetProbe Results ...................................19
6.4.2. Perfas+ Results ....................................20
6.4.3. Conclusions for Goodness-of-Fit ....................22
6.5. Implementation of Statistics for One-Way Loss .............23
7. Conclusions for a Revision of RFC 2680 .........................23
8. Security Considerations ........................................24
9. Acknowledgements ...............................................24
10. Appendix - Network Configuration and Sample Commands ..........25
11. References ....................................................28
11.1. Normative References .....................................28
11.2. Informative References ...................................291. Introduction 🦋
okay so the IETF IP Performance Metrics (IPPM) working group has been thinking about how to move their metrics along the Standards Track since 2001 🕰️
The renewed effort was all about reducing measurement variability, so that comparing results for equivalence gets way simpler 🧠 And the consensus (captured in [RFC6576]) is that equivalent results from independent implementations of metric specifications are enough evidence that the specifications themselves are clear and unambiguous. It's the metric version of protocol interoperability, and honestly iconic 👑 The advancement process does one of two things: (1) it builds confidence that the metric definitions and supporting material are clearly worded and unambiguous, or (2) it finds ways the metric definitions should be revised for clarity 💅 Comparing the specific implementations themselves is a non-goal, so we're not doing that 🙅♀️
The process also lets you spot options in the metric RFC that weren't implemented, so they can be removed from the advancing spec. (This part is more typical of protocol advancement on the Standards Track 🧚)
The purpose of this memo is to put the current approach for [RFC2680] into practice and write down the results 📝
Specifically, this memo documents the consensus on how many errors are tolerable when deciding whether results are equivalent 🎯 The IPPM working group agreed that the test plan and procedures should include the equivalence threshold, and that this info should be available before cross-implementation comparisons happen. This memo also includes procedures for same-implementation comparisons, which help set that threshold 🌸
Another part of the metric RFC advancement process is documenting the work and results. The procedures of [RFC2026] are expanded in [RFC5657], including sample implementation and interoperability reports. This memo follows the template in [RFC6808] for the report that goes with the protocol action request submitted to the Area Director. That covers a description of the test setup, procedures, results for each implementation, and conclusions 💖
The conclusion is that [RFC2680], with modifications, should be advanced on the Standards Track. The revised text of RFC 2680 [LOSS-METRIC] is ready for review, but it's waiting on work in progress to update the IPPM Framework [RFC2330]. So this memo documents the info needed to support advancing [RFC2680], and approving a revision of RFC 2680 is left for future action ⭐
1.1. Requirements Language ✨
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Some of these key words were used in [RFC2680], but there are no requirements specified in this memo 🎀
1.2. RFC 2680 Coverage 🔍
This plan is meant to cover all the critical requirements and sections of [RFC2680] 💅
Heads up: outside of the boilerplate and the [RFC2119] reference, there are only five relevant instances of the requirement term "MUST" in [RFC2680]. The instance of "MUST" in the Security Considerations section of [RFC2680] is not a basis for implementation equivalence comparisons 🙈
Statements in RFC 2680 that have the character of requirements may be included if the community reaches consensus that the wording implies a requirement. At least one instance of an implied requirement has been found in Section 3.6 of [RFC2680] 🧐
2. A Definition-Centric Metric Advancement Process 👑
The process described in Section 3.5 of [RFC6576] takes as a first principle that the metric definitions, embodied in the text of the RFCs, are the objects that need evaluation and possible revision before advancing to the next step on the Standards Track. This memo follows that process 💖
3. Test Configuration ✨
okay so the testing setup is giving main character energy 💅 One metric implementation was NetProbe version 5.8.5 🦋 (an earlier version is already running in the WIPM system and deployed worldwide [WIPM]). NetProbe sends UDP packets of variable size and can make test streams with Periodic [RFC3432] or Poisson [RFC2330] sample distributions 🌸
The other implementation was Perfas+ version 3.1, made by Deutsche Telekom [Perfas] 🎀 Perfas+ uses UDP unicast packets of variable size (it also supports TCP and multicast, which is so extra 👑). Test streams can use Periodic, Poisson, or uniform sample distributions 💜
Here's the test path. Each implementation's flows go through the Internet and the Layer 2 Tunneling Protocol version 3 (L2TPv3) [RFC3931] tunnel IDs (1 and 2). This is based on Figure 1 of [RFC6576] 🧚
+------------+ +------------+
| Imp 1 | ,---. | Imp 2 |
+------------+ / \ +-------+ +------------+
| V100 ^ V200 / \ | Tunnel| | V300 ^ V400
| | ( ) | Head | | |
+--------+ +------+ | |__| Router| +----------+
|Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet |
|Switch |--|Head |-| | | |Switch |
+-+--+---+ |Router| | | +---+---+--+--+--+----+
|__| +--A---+ ( ) |Network| |__|
\ / |Emulat.|
U-turn \ / |"netem"| U-turn
V300 to V400 `-+-' +-------+ V100 to V200
Implementations ,---. +--------+
+~~~~~~~~~~~/ \~~~~~~| Remote |
+------->-----F2->-| / \ |->---. |
| +---------+ | Tunnel ( ) | | |
| | transmit|-F1->-| ID 1 | | |->. | |
| | Imp 1 | +~~~~~~~~~| |~~~~| | | |
| | receive |-<--+ | | | F1 F2 |
| +---------+ | |Internet | | | | |
*-------<-----+ F1 | | | | | |
+---------+ | | +~~~~~~~~~| |~~~~| | | |
| transmit|-* *-| | | |<-* | |
| Imp 2 | | Tunnel ( ) | | |
| receive |-<-F2-| ID 2 \ / |<----* |
+---------+ +~~~~~~~~~~~\ /~~~~~~| Switch |
`-+-' +--------+Figure 1 shows two views of the same setup 🍓 The upper diagram is all about the VLAN connectivity and where everything is located geographically (where "Imp #" is the sender and receiver of implementation 1 or 2, which is either Perfas+ or NetProbe in this test). The lower diagram shows example flows going between the two measurement implementations. Only two flows are drawn to keep it simple, and the netem emulator is left out (it would sit before or after the Internet, depending on the flow).
The testing uses the L2TPv3 [RFC3931] tunnel between the test sites over the Internet 🌈 The tunnel IP and L2TPv3 headers are there to hide the test equipment addresses and ports from hash functions. Those hash functions would otherwise spread different test streams across parallel network resources, which could change the performance results.
At each end of the tunnel, one pair of VLANs inside the tunnel is looped back, so test traffic gets returned to each test site. That means test streams cross the L2TP tunnel twice, but from the test equipment's point of view they look like one-way tests 🦋
The network emulator is a host running Fedora 14 Linux [FEDORA], with IP forwarding turned on and the "netem" Network emulator loaded and running as part of the Fedora Kernel 2.6.35.11 [NETEM] ⭐ The standard kernel is "tickless", which means it swaps the old periodic timer (250 Hz, with 4 ms uncertainty) interrupts for on-demand interrupts. The netem/Fedora host was connected by bridging Ethernet VLAN interfaces together with "brctl" commands (for example, eth1.100 <-> eth2.100). The netem emulator was activated on one interface (eth1) and only worked on test streams going in one direction. In some tests, separate netem instances ran on each VLAN. See the Appendix for more details 🙈
The links between the netem emulator host, the router, and the switch turned out to be 100BaseTX-HD (100 Mbps half duplex), as reported by "mii-tool" [MII-TOOL] once testing was done. Half duplex wasn't planned, and it probably added a little delay variation that full-duplex mode could have avoided 😭
Each individual test ran with common packet rates (1 pps, 10 pps), Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 500 bytes 💖
For these tests, at least 300 packets were sent from source to destination in each implementation. Periodic streams (as per [RFC3432]) with 1-second spacing were used, except where noted ✨
As required in Section 2.8.1 of [RFC2680], the packet Type-P must be reported. The packet Type-P for this test was IP-UDP with the Best Effort Differentiated Services Code Point (DSCP). These headers were encapsulated according to the L2TPv3 specification [RFC3931], and they were unlikely to change how the packets were treated as they crossed the Internet 🎀
With the L2TPv3 tunnel in use, the metric name for the testing set up here (looking at the IP header that Internet processing can see) is:
Type-IP-protocol-115-One-way-Packet-Loss-<StreamType>-Stream
With the metric parameters from (Section 3.2 of [RFC2680]):
- Src, the IP address of a host (12.3.167.16 or 193.159.144.8)
- Dst, the IP address of a host (193.159.144.8 or 12.3.167.16)
- T0, a time
- Tf, a time
- lambda, a rate in reciprocal seconds
- Thresh, a maximum waiting time in seconds (see Section 2.8.2 of [RFC2680])
Metric Units: a sequence of pairs. The elements of each pair are:
- T, a time, and
- L, either a zero or a one
The values of T in the sequence go up monotonically 📈 T would be a valid parameter of singleton Type-P-One-way-Packet-Loss, and L would be a valid value of Type-P-One-way-Packet-Loss (see Section 3.3 of [RFC2680]) 🧚
Also, Section 2.8.4 of [RFC2680] recommends that the path SHOULD be reported. In this setup, the L2TPv3 tunnels hide most of the path details from the implementations, so a more useful path traceroute can be run by the routers at each location 🔥
When NetProbe is used in production, a traceroute runs in parallel at the start of measurements.
Perfas+ does not support traceroute 💀
Here's what the router trace looked like:
IPLGW#traceroute 193.159.144.8
Type escape sequence to abort.
Tracing the route to 193.159.144.8
1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec
2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec
cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec
3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec
cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec
cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec
4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec
n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec
5 192.205.34.182 [AS 7018] 0 msec
192.205.34.150 [AS 7018] 0 msec
192.205.34.182 [AS 7018] 4 msec
6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec
88 msec
7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec
8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec
9 * * *NetProbe Traceroute 🦋
It was only possible to run the traceroute for the measured path on one of the tunnel-head routers. The normal trace tools on the measurement systems get confused by the L2TPv3 tunnel encapsulation 🫶
4. Error Calibration and RFC 2680 ✨
An implementation is required to report calibration results on clock synchronization per Section 2.8.3 of [RFC2680] (also required in Section 3.7 of [RFC2680] for sample metrics) 🌸
It's also recommended to report the probability that a packet which successfully arrives at the destination network interface is incorrectly designated as lost due to resource exhaustion, per Section 2.8.3 of [RFC2680] 💜
4.1. Clock Synchronization Calibration
For the NetProbe and Perfas+ clock synchronization test results, refer to Section 4 of [RFC6808] 🎀
4.2. Packet Loss Determination Error
Both measurement implementations have resource limits, so it's theoretically possible for those limits to be exceeded and for a packet that actually arrived at the destination to be thrown away by mistake 😭
In previous test efforts [ADV-METRICS], NetProbe produced six multicast streams with an aggregate bit rate over 53 Mbit/s. The goal was to characterize the one-way capacity of an emulator based on NIST Net 🔥 Neither the emulator nor the pair of NetProbe implementations used in this testing dropped any packets in these streams 👑
The maximum load used here between any two NetProbe implementations was 11.5 Mbit/s, split equally across three unicast test streams. We concluded that steady resource usage doesn't add error (extra loss) to the measurements ✨
5. Predetermined Limits on Equivalence 💅
In this section, we give the numerical limits on comparisons between implementations. The point is to declare that the results are equivalent, and that the tested specification is therefore clear 🦋
A key point is that the allowable errors, corrections, and confidence levels only need to be good enough to catch any misinterpretation of the tested specification that would show diverging implementations 🧚
The allowable error also has to be big enough to make up for measured path differences. It wasn't possible to measure fully identical paths in the VLAN-loopback test configuration used, and that practical compromise has to be taken into account 🙈
For Anderson-Darling K-sample (ADK) [ADK] comparisons, the required confidence factor for the cross-implementation comparisons SHALL be the smallest of:
- 0.95 confidence factor at 1-packet resolution, or
- the smallest confidence factor (in combination with resolution) of the two same-implementation comparisons for the same test conditions (if the number of streams is sufficient to allow such comparisons).
For Anderson-Darling Goodness-of-Fit (ADGoF) [RADGOF] comparisons, the required level of significance for the same-implementation Goodness-of-Fit (GoF) SHALL be 0.05 or 5%, as specified in Section 11.4 of [RFC2330]. That's equivalent to a 95% confidence factor 💖
6. Tests to Evaluate RFC 2680 Specifications 🌈
This section describes some results from production network (cross-Internet) tests. The tests used measurement devices implementing IPPM metrics, plus a network emulator to create relevant conditions. The goal was to find out whether implementors interpreted the metric definitions consistently 🍓
The procedures are similar to those in Appendix A.1 of [RFC6576] for one-way delay 💜
6.1. One-Way Loss: ADK Sample Comparison ✨
okay so this test checks whether two implementations give results that look like they came from the same packet loss distribution 🦋 It's basically a big overall check on Section 3 of [RFC2680] ("A Definition for Samples of One-way Packet Loss"). The same-implementation results are what set the bar for the cross-implementation comparisons, so they matter a lot 💅
This test is meant to evaluate the measurements in Sections 2, 3, and 4 of [RFC2680] 🌸
Here's the cute trick: we compare the counts of one-way packet loss from different test streams across two [RFC2680] implementations and check whether they seem to come from the same loss process. This skips a bunch of extra work, because comparing the summary statistics (as defined in Section 4 of [RFC2680]) would mean doing a whole second round of equivalence checks 🙈 We can just check whether the single statistic in Section 4 of [RFC2680] was implemented and report that.
- Configure an L2TPv3 path between the test sites, and set each pair of measurement devices to run tests in their assigned pair of VLANs.
- Measure a sample of one-way packet loss singletons with two or more implementations, using identical options and network emulator settings (if used).
- Measure a sample of one-way packet loss singletons with four or more instances of the same implementations, using identical options. Connectivity differences SHOULD be the same as for cross-implementation testing 🎀
- If fewer than ten test streams are available, skip to step 7.
- Run the ADK comparison procedures (see Appendix B of [RFC6576]), and figure out the resolution and confidence factor for distribution equivalence for each same-implementation comparison and each cross-implementation comparison ✨
- Take the coarsest resolution and confidence factor for distribution equivalence from the same-implementation pairs, or the limit defined in Section 5 above, as a limit on the equivalence threshold for these experimental conditions 👑
- Compare the cross-implementation ADK performance with the equivalence threshold from step 5 to see whether equivalence can be declared 💖
The metric parameters change for each loss test, and they're listed first in each subsection below 🌈
The cross-implementation comparison uses a simple ADK analysis [RTOOL] [RADK], where all NetProbe loss counts are compared with all Perfas+ loss results 🫶
Here's what the results analysis in this section looks like:
- All comparisons used 1-packet resolution.
- No correction factors were applied.
- The 0.95 confidence factor was used (and so was the ADK criterion for t.obs < 1.960 for cross-implementation comparison) 💜
6.1.1. 340B/Periodic Cross-Implementation Results
Tests in this section used:
- IP header + payload = 340 octets
- Periodic sampling at 1 packet per second
- Test duration = 1200 seconds (during April 7, 2011, EDT)
The netem emulator was set for 100 ms constant delay, with a 10% loss ratio. In this experiment, the netem emulator was configured to operate independently on each VLAN, so the emulator itself is a potential source of error when comparing streams that traverse the test path in different directions 😭
=======================================
A07bps_loss <- c(114, 175, 138, 142, 181, 105) (NetProbe)
A07per_loss <- c(115, 128, 136, 127, 139, 138) (Perfas+)
> A07bps_loss <- c(114, 175, 138, 142, 181, 105)
> A07per_loss <- c(115, 128, 136, 127, 139, 138)
>
> A07cross_loss_ADK <- adk.test(A07bps_loss, A07per_loss)
> A07cross_loss_ADK
Anderson-Darling k-sample test.
Number of samples: 2
Sample sizes: 6 6
Total number of values: 12
Number of unique values: 11
Mean of Anderson Darling Criterion: 1
Standard deviation of Anderson Darling Criterion: 0.6569
T = (Anderson Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation
not adj. for ties 0.52043 0.20604 0
adj. for ties 0.62679 0.18607 0
>=======================================
The cross-implementation comparisons pass the ADK criterion (t.obs < 1.960) 🎉
6.1.2. 64B/Periodic Cross-Implementation Results
Tests in this section used:
- IP header + payload = 64 octets
- Periodic sampling at 1 packet per second
- Test duration = 300 seconds (during March 24, 2011, EDT)
The netem emulator was set for 0 ms constant delay, with a 10% loss ratio.
=======================================
> M24per_loss <- c(42,34,35,35) (Perfas+)
> M24apd_23BC_loss <- c(27,39,29,24) (NetProbe)
> M24apd_loss23BC_ADK <- adk.test(M24apd_23BC_loss,M24per_loss)
> M24apd_loss23BC_ADK
Anderson-Darling k-sample test.
Number of samples: 2
Sample sizes: 4 4
Total number of values: 8
Number of unique values: 7
Mean of Anderson Darling Criterion: 1
Standard deviation of Anderson Darling Criterion: 0.60978
T = (Anderson Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation
not adj. for ties 0.76921 0.16200 0
adj. for ties 0.90935 0.14113 0
Warning: At least one sample size is less than 5.
p-values may not be very accurate.
>=======================================
The cross-implementation comparisons pass the ADK criterion 🌸
6.1.3. 64B/Poisson Cross-Implementation Results
Tests in this section used:
- IP header + payload = 64 octets
- Poisson sampling at lambda = 1 packet per second
- Test duration = 1200 seconds (during April 27, 2011, EDT)
The netem configuration was 0 ms delay and 10% loss. There were two passes through an emulator for each stream, and loss emulation was present for 18 minutes of the 20-minute (1200-second) test 🧚
=======================================
A27aps_loss <- c(91,110,113,102,111,109,112,113) (NetProbe)
A27per_loss <- c(95,123,126,114) (Perfas+)
A27cross_loss_ADK <- adk.test(A27aps_loss, A27per_loss)
> A27cross_loss_ADK
Anderson-Darling k-sample test.
Number of samples: 2
Sample sizes: 8 4
Total number of values: 12
Number of unique values: 11
Mean of Anderson Darling Criterion: 1
Standard deviation of Anderson Darling Criterion: 0.65642
T = (Anderson Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation
not adj. for ties 2.15099 0.04145 0
adj. for ties 1.93129 0.05125 0
Warning: At least one sample size is less than 5.
p-values may not be very accurate.
>=======================================
The cross-implementation comparisons barely pass the ADK criterion at 95% = 1.960 when adjusting for ties 😬
6.1.4. Conclusions on the ADK Results for One-Way Packet Loss
We conclude that the two implementations are capable of producing equivalent one-way packet loss measurements based on their interpretation of [RFC2680] 💖
6.2. One-Way Loss: Delay Threshold 🍓
This test checks whether implementations use the same configured maximum waiting time delay from one measurement to another under different delay conditions, and whether they correctly declare packets arriving after the waiting time threshold as lost ⏰
See Section 2.8.2 of [RFC2680] 📖
- Configure an L2TPv3 path between the test sites, and set each pair of measurement devices to run tests in their assigned pair of VLANs.
- Configure the network emulator to add 1 second of one-way constant delay in one direction of transmission.
- Measure (average) one-way delay with two or more implementations, using identical waiting time thresholds (Thresh) for loss set at 3 seconds.
- Configure the network emulator to add 3 seconds of one-way constant delay in one direction of transmission, which is equivalent to 2 seconds of additional one-way delay (or change the path delay while the test is in progress, when there are sufficient packets at the first delay setting) 🌈
- Repeat or continue measurements.
- Observe that the increase measured in step 5 caused all packets with 2 seconds of additional delay to be declared lost, and that all packets that arrive successfully in step 3 are assigned a valid one-way delay ✨
The common parameters used for tests in this section are:
- IP header + payload = 64 octets
- Poisson sampling at lambda = 1 packet per second
- Test duration = 900 seconds total (March 21, 2011 EDT)
The netem emulator settings added constant delays as specified in the procedure above 💜
6.2.1. NetProbe Results for Loss Threshold
In NetProbe, the loss threshold was implemented uniformly over all packets as a post-processing routine. With the loss threshold set at 3 seconds, all packets with one-way delay >3 seconds were marked "Lost" and included in the Lost Packet list with their transmission time (as required in Section 3.3 of [RFC2680]). This resulted in 342 packets designated as lost in one of the test streams (with average delay = 3.091 sec) 🔥
6.2.2. Perfas+ Results for Loss Threshold
Perfas+ uses a fixed loss threshold, which was not adjustable during this study. The loss threshold is approximately one minute, and emulation of a delay of this size was not attempted. However, it is possible to implement any delay threshold desired with a post-processing routine and subsequent analysis. Using this method, 195 packets would be declared lost (with average delay = 3.091 sec) 👑
6.2.3. Conclusions for Loss Threshold
Both implementations assume that any constant delay value desired can be used as the loss threshold, since all delays are stored as a pair <Time, Delay> as required in [RFC2680]. This is a simple way to enforce the constant loss threshold envisioned in [RFC2680] (see Section 2.8.2 of [RFC2680]). We take the position that the assumption of post-processing is compliant, and that the text of the revision of RFC 2680 should be revised slightly to include this point 🦋
6.3. One-Way Loss with Out-of-Order Arrival ✨
okay so bestie, Section 3.6 of [RFC2680] says (with a lowercase "must" 👀) that implementations need to make sure reordered packets get handled correctly 💅. It's basically an implied requirement, because the right packet has to be marked as lost if it doesn't show up before its delay threshold, no matter what, and reordering is always a possibility on IP network paths 🌪️. If you want the IETF definition of reordering for standards-compliant measurements, check [RFC4737] 📖.
The netem emulator can totally make reordering happen because every packet's delay is drawn from an independent distribution 🙈. Here, a big delay (2000 ms) plus big delay variation (1000 ms) was enough to cause reordering. Using the procedure from Section 6.1, netem was set to add 10% loss while the reordering was going on 🎀.
The tests in this section used:
- IP header + payload = 64 octets
- Periodic sampling = 1 packet per second
- Test duration = 600 seconds (during May 2, 2011, EDT)
=======================================
> Y02aps_loss <- c(53,45,67,55) (NetProbe)
> Y02per_loss <- c(59,62,67,69) (Perfas+)
> Y02cross_loss_ADK <- adk.test(Y02aps_loss, Y02per_loss)
> Y02cross_loss_ADK
Anderson-Darling k-sample test.
Number of samples: 2
Sample sizes: 4 4
Total number of values: 8
Number of unique values: 7
Mean of Anderson Darling Criterion: 1
Standard deviation of Anderson Darling Criterion: 0.60978
T = (Anderson Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation
not adj. for ties 1.11282 0.11531 0
adj. for ties 1.19571 0.10616 0
Warning: At least one sample size is less than 5.
p-values may not be very accurate.
>
=======================================The test results show that there was a LOT of reordering going on 😭. Both implementations picked up the big delay variation between neighbouring packets. In NetProbe, the packet arrival order is kept in the raw measurement files, so looking at the arrival packet sequence numbers shows the reordering too 🧚.
Even with all that constant packet reordering in the transmission path, the loss count distributions from the two implementations pass the ADK criterion at 95% = 1.960 ✨👑.
6.4. Poisson Sending Process Evaluation 🎲
Section 3.7 of [RFC2680] says implementations need to make sure their sending process is reasonably close to a classic Poisson distribution when it's in use 💜. You can find way more detail on sample distribution generation and Goodness-of-Fit testing in Section 11.4 of [RFC2330] and in the Appendix of [RFC2330] 📚.
In this section, each implementation's Poisson distribution gets compared to an idealised version that's in the base functionality of the R-tool for Statistical Analysis [RTOOL]. The comparison is done with the Anderson-Darling Goodness-of-Fit test package (ADGofTest) [RADGOF] 🔬. The Goodness-of-Fit criterion from [RFC2330] requires a test statistic value AD <= 2.492 for 5% significance. The Appendix of [RFC2330] also points out that it can be hard to pass the ADGofTest when the sample has lots of packets (when 8192 were used, the test always failed, but smaller sets of the stream passed) 🙈.
Both implementations were set up to produce Poisson distributions with lambda = 1 packet per second, and to assign received packet timestamps in the measurement application (above the UDP layer; see the calibration results in Section 4 of [RFC6808] for error assessment) 🎀.
6.4.1. NetProbe Results
Section 11.4 of [RFC2330] suggests three possible measurement points for evaluating the Poisson distribution. The NetProbe analysis uses "user-level timestamps made just before or after the system call for transmitting the packet" 🌸.
Here's the statistical summary for two NetProbe streams:
=======================================
> summary(a27ms$s1[2:1152])
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0100 0.2900 0.6600 0.9846 1.3800 8.6390
> summary(a27ms$s2[2:1152])
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.010 0.280 0.670 0.979 1.365 8.829
=======================================Both means are near the specified lambda = 1 💖.
Here are the results of the ADGoF tests for these two streams:
=======================================
> ad.test( a27ms$s1[2:101], pexp, 1)
Anderson-Darling GoF Test
data: a27ms$s1[2:101] and pexp
AD = 0.8908, p-value = 0.4197
alternative hypothesis: NA
> ad.test( a27ms$s1[2:1001], pexp, 1)
Anderson-Darling GoF Test
data: a27ms$s1[2:1001] and pexp
AD = 0.9284, p-value = 0.3971
alternative hypothesis: NA
> ad.test( a27ms$s2[2:101], pexp, 1)
Anderson-Darling GoF Test
data: a27ms$s2[2:101] and pexp
AD = 0.3597, p-value = 0.8873
alternative hypothesis: NA
> ad.test( a27ms$s2[2:1001], pexp, 1)
Anderson-Darling GoF Test
data: a27ms$s2[2:1001] and pexp
AD = 0.6913, p-value = 0.5661
alternative hypothesis: NA
=======================================Both the sets of 100 packets and the sets of 1000 packets from the two different streams (s1 and s2) passed the AD <= 2.492 criterion 🥳.
6.4.2. Perfas+ Results
Section 11.4 of [RFC2330] suggests three possible measurement points for evaluating the Poisson distribution. The Perfas+ analysis uses "wire times for the packets as recorded using a packet filter" 🦋.
Heads up though: because access on the Perfas+ side of the test setup was limited, the captures were made after the Perfas+ streams went through the production network. That adds a small amount of unwanted delay variation to the wire times (and possibly some error from packet loss) 😬.
Here's the statistical summary for two Perfas+ streams:
=======================================
> summary(a27pe$p1)
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.004 0.347 0.788 1.054 1.548 4.231
> summary(a27pe$p2)
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0010 0.2710 0.7080 0.9696 1.3740 7.1160
=======================================Both means are near the specified lambda = 1 💅.
Here are the results of the ADGoF tests for these two streams:
=======================================
> ad.test(a27pe$p1, pexp, 1 )
Anderson-Darling GoF Test
data: a27pe$p1 and pexp
AD = 1.1364, p-value = 0.2930
alternative hypothesis: NA
> ad.test(a27pe$p2, pexp, 1 )
Anderson-Darling GoF Test
data: a27pe$p2 and pexp
AD = 0.5041, p-value = 0.7424
alternative hypothesis: NA
> ad.test(a27pe$p1[1:100], pexp, 1 )
Anderson-Darling GoF Test
data: a27pe$p1[1:100] and pexp
AD = 0.7202, p-value = 0.5419
alternative hypothesis: NA
> ad.test(a27pe$p1[101:193], pexp, 1 )
Anderson-Darling GoF Test
data: a27pe$p1[101:193] and pexp
AD = 1.4046, p-value = 0.201
alternative hypothesis: NA
> ad.test(a27pe$p2[1:100], pexp, 1 )
Anderson-Darling GoF Test
data: a27pe$p2[1:100] and pexp
AD = 0.4758, p-value = 0.7712
alternative hypothesis: NA
> ad.test(a27pe$p2[101:193], pexp, 1 )
Anderson-Darling GoF Test
data: a27pe$p2[101:193] and pexp
AD = 0.3381, p-value = 0.9068
alternative hypothesis: NA
>
=======================================Sets of 193, 100, and 93 packets from the two different streams (p1 and p2) all passed the AD <= 2.492 criterion 🎉.
6.4.3. Conclusions for Goodness-of-Fit
Both the NetProbe and Perfas+ implementations produce adequate Poisson distributions according to the Anderson-Darling Goodness-of-Fit test at the 5% significance level (1-alpha = 0.05, or 95% confidence level) 👑🌈.
6.5. Implementation of Statistics for One-Way Loss 📊
We check which statistics were implemented and report on them. We're noting that Section 4 of [RFC2680] doesn't specify the calculations exactly, and only gives some illustrative examples 🤓.
NetProbe Perfas+
Type-P-One-way-Packet-Loss-Average yes yes
(this is more commonly referred
to as "loss ratio")
Implementation of RFC 2680 Section 4 StatisticsImplementations call this metric a loss ratio, and this is an area that will probably get revised so the text matches widespread usage 💖.
7. Conclusions for a Revision of RFC 2680 🎓
This memo concludes that [RFC2680] should be advanced on the Standards Track, and it recommends the following edits to improve the text (which aren't considered significant enough to affect maturity) 🫶.
- Revise
Type-P-One-way-Packet-Loss-AvetoType-P-One-way-Delay-Packet-Loss-Ratio. - Regarding implementation of the loss delay threshold (Section 6.2), the assumption of post-processing is compliant, and the text of the revision of RFC 2680 should be revised slightly to include this point.
- The IETF has reached consensus on guidance for reporting metrics [RFC6703], and this memo should be referenced in a revision of RFC 2680 to incorporate recent experience where appropriate.
We note that there are at least two errata for [RFC2680], and it looks like these minor revisions should be folded into a revision of RFC 2680 🛠️.
The authors who revise [RFC2680] should review all errata filed at the time the document is being written. They shouldn't rely on this document to list all relevant errata updates 🙅♀️.
We recognise BCP 170 [RFC6390], which gives guidelines for developing documents that describe new performance metrics. However, advancing [RFC2680] is fine-tuning of long-standing specifications based on experience that helped shape BCP 170. Material that satisfies some of the requirements of [RFC6390] can also be found in other RFCs, such as the IPPM Framework [RFC2330]. So no specific changes to address the BCP 170 guidelines are recommended for a revision of RFC 2680 🌸.
8. Security Considerations 🚨
The security considerations that apply to any active measurement of live networks are relevant here as well. See [RFC4656] and [RFC5357] 🔐.
9. Acknowledgements 🙏
The authors thank Lars Eggert for his continued encouragement to advance the IPPM metrics during his tenure as AD Advisor 🫶.
Nicole Kowalski supplied the Customer Premises Equipment (CPE) router needed for the NetProbe side of the test setup, and graciously managed her testing even with the issues caused by dual-use of the router. Thanks, Nicole! 💖🎀
The "NetProbe Team" also acknowledges the many useful discussions on statistical interpretation with Ganga Maguluri 🧠.
Constructive comments and helpful reviews were also provided by Bill Cerveny, Joachim Fabini, and Ann Cerveny ⭐.
10. Appendix - Network Configuration and Sample Commands 🛠️✨
okay so this appendix is the behind-the-scenes setup 🎀 It gives you the host config and some sample tc commands for the "netem" network emulator, the same stuff from Section 3 and Figure 1 of this memo. bestie, these details also apply to the test plan in [RFC6808] 💖
The host interface and config are shown below. The output lines are split up because of the 72 character per line limit, so the tc commands got line breaks added 🫶
[system@dell4-4 ~]$ su
Password:
[root@dell4-4 system]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@dell4-4 system]# service iptables stop
iptables: Flushing firewall rules: [ OK ]
iptables: Setting chains to policy ACCEPT: nat filter [ OK ]
iptables: Unloading modules: [ OK ]
[root@dell4-4 system]# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes
[root@dell4-4 system]# ifconfig eth1.300 0.0.0.0 promisc up
[root@dell4-4 system]# ifconfig eth1.400 0.0.0.0 promisc up
[root@dell4-4 system]# ifconfig eth2.400 0.0.0.0 promisc up
[root@dell4-4 system]# ifconfig eth2.300 0.0.0.0 promisc up
[root@dell4-4 system]# brctl addbr br300
[root@dell4-4 system]# brctl addif br300 eth1.300
[root@dell4-4 system]# brctl addif br300 eth2.300
[root@dell4-4 system]# ifconfig br300 up
[root@dell4-4 system]# brctl addbr br400
[root@dell4-4 system]# brctl addif br400 eth1.400
[root@dell4-4 system]# brctl addif br400 eth2.400
[root@dell4-4 system]# ifconfig br400 up
[root@dell4-4 system]# brctl show
bridge name bridge id STP enabled interfaces
br300 8000.0002b3109b8a no eth1.300
eth2.300
br400 8000.0002b3109b8a no eth1.400
eth2.400
virbr0 8000.000000000000 yes
[root@dell4-4 system]# brctl showmacs br300
port no mac addr is local? ageing timer
2 00:02:b3:10:9b:8a yes 0.00
1 00:02:b3:10:9b:99 yes 0.00
1 00:02:b3:c4:c9:7a no 0.52
2 00:02:b3:cf:02:c6 no 0.52
2 00:0b:5f:54:de:81 no 0.01
[root@dell4-4 system]# brctl showmacs br400
port no mac addr is local? ageing timer
2 00:02:b3:10:9b:8a yes 0.00
1 00:02:b3:10:9b:99 yes 0.00
2 00:02:b3:c4:c9:7a no 0.60
1 00:02:b3:cf:02:c6 no 0.42
2 00:0b:5f:54:de:81 no 0.33
[root@dell4-4 system]# tc qdisc add dev eth1.300 root netem
delay 100ms
[root@dell4-4 system]# ifconfig eth1.200 0.0.0.0 promisc up
[root@dell4-4 system]# vconfig add eth1 100
Added VLAN with VID == 100 to IF -:eth1:-
[root@dell4-4 system]# ifconfig eth1.100 0.0.0.0 promisc up
[root@dell4-4 system]# vconfig add eth2 100
Added VLAN with VID == 100 to IF -:eth2:-
[root@dell4-4 system]# ifconfig eth2.100 0.0.0.0 promisc up
[root@dell4-4 system]# ifconfig eth2.200 0.0.0.0 promisc up
[root@dell4-4 system]# brctl addbr br100
[root@dell4-4 system]# brctl addif br100 eth1.100
[root@dell4-4 system]# brctl addif br100 eth2.100
[root@dell4-4 system]# ifconfig br100 up
[root@dell4-4 system]# brctl addbr br200
[root@dell4-4 system]# brctl addif br200 eth1.200
[root@dell4-4 system]# brctl addif br200 eth2.200
[root@dell4-4 system]# ifconfig br200 up
[root@dell4-4 system]# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.0002b3109b8a no eth1.100
eth2.100
br200 8000.0002b3109b8a no eth1.200
eth2.200
br300 8000.0002b3109b8a no eth1.300
eth2.300
br400 8000.0002b3109b8a no eth1.400
eth2.400
virbr0 8000.000000000000 yes
[root@dell4-4 system]# brctl showmacs br100
port no mac addr is local? ageing timer
2 00:02:b3:10:9b:8a yes 0.00
1 00:02:b3:10:9b:99 yes 0.00
1 00:0a:e4:83:89:07 no 0.19
2 00:0b:5f:54:de:81 no 0.91
2 00:e0:ed:0f:72:86 no 1.28
[root@dell4-4 system]# brctl showmacs br200
port no mac addr is local? ageing timer
2 00:02:b3:10:9b:8a yes 0.00
1 00:02:b3:10:9b:99 yes 0.00
2 00:0a:e4:83:89:07 no 1.14
2 00:0b:5f:54:de:81 no 1.87
1 00:e0:ed:0f:72:86 no 0.24
[root@dell4-4 system]# tc qdisc add dev eth1.100 root netem
delay 100ms
[root@dell4-4 system]#
=====================================================================Next up, some sample tc command lines that control netem and its impairments 😈💅
tc qdisc add dev eth1.100 root netem loss 0%
tc qdisc add dev eth1.200 root netem loss 0%
tc qdisc add dev eth1.300 root netem loss 0%
tc qdisc add dev eth1.400 root netem loss 0%
Add delay and delay variation:
tc qdisc change dev eth1.100 root netem delay 100ms 50ms
tc qdisc change dev eth1.200 root netem delay 100ms 50ms
tc qdisc change dev eth1.300 root netem delay 100ms 50ms
tc qdisc change dev eth1.400 root netem delay 100ms 50ms
Add delay, delay variation, and loss:
tc qdisc change dev eth1 root netem delay 2000ms 1000ms loss 10%
=====================================================================11. References 📚🦋
11.1. Normative References
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998.
[RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way Packet Loss Metric for IPPM", RFC 2680, September 1999.
[RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network performance measurement with periodic streams", RFC 3432, November 2002.
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, September 2006.
[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., and J. Perser, "Packet Reordering Metrics", RFC 4737, November 2006.
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, October 2008.
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard", BCP 9, RFC 5657, September 2009.
[RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New Performance Metric Development", BCP 170, RFC 6390, October 2011.
[RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP Performance Metrics (IPPM) Standard Advancement Testing", BCP 176, RFC 6576, March 2012.
[RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting IP Network Performance Metrics: Different Points of View", RFC 6703, August 2012.
[RFC6808] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test Plan and Results Supporting Advancement of RFC 2679 on the Standards Track", RFC 6808, December 2012.
11.2. Informative References
[ADK] Scholz, F. and M. Stephens, "K-Sample Anderson-Darling Tests of Fit, for Continuous and Discrete Cases", University of Washington, Technical Report No. 81, May 1986.
[ADV-METRICS] Morton, A., "Lab Test Results for Advancing Metrics on the Standards Track", Work in Progress, October 2010.
[FEDORA] "Fedora", http://fedoraproject.org/.
[LOSS-METRIC] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A One-Way Loss Metric for IPPM", Work in Progress, July 2014.
[MII-TOOL] Hinds, D., Becker, D., and B. Eckenfels, "Linux System Administrator's Manual", February 2013, http://man7.org/linux/man-pages/man8/mii-tool.8.html.
[NETEM] Linux Foundation, "netem", http://www.linuxfoundation.org/collaborate/workgroups/networking/netem.
[Perfas] Heidemann, C., "Qualitaet in IP-Netzen Messverfahren", published by ITG Fachgruppe, 2nd meeting 5.2.3, November 2001, www.itg523.de/oeffentlich/01nov/Heidemann_QOS_Messverfahren.pdf.
[RADGOF] Bellosta, C., "ADGofTest: Anderson-Darling Goodness-of-Fit Test. R package version 0.3.", R-Package Version 0.3, December 2011, http://cran.r-project.org/web/packages/ADGofTest/index.html.
[RADK] Scholz, F., "ADK: Anderson-Darling K-Sample Test and Combinations of Such Tests. R package version 1.0.", 2008.
[RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005.
[RTOOL] R Development Core Team, "R: A Language and Environment for Statistical Computing", ISBN 3-900051-07-0, 2014, http://www.R-project.org/.
[WIPM] AT&T, "AT&T Global IP Network", 2014, http://ipnetwork.bgtmo.ip.att.net/pws/index.html.
Authors' Addresses 💌
Len Ciavattone
AT&T Labs
200 Laurel Avenue South
Middletown, NJ 07748
USA
Phone: +1 732 420 1239
EMail: lencia@att.com
Ruediger Geib
Deutsche Telekom
Heinrich Hertz Str. 3-7
Darmstadt 64295
Germany
Phone: +49 6151 58 12747
EMail: Ruediger.Geib@telekom.de
Al Morton
AT&T Labs
200 Laurel Avenue South
Middletown, NJ 07748
USA
Phone: +1 732 420 1571
Fax: +1 732 368 1192
EMail: acmorton@att.com
URI: http://home.comcast.net/~acmacm/
Matthias Wieser
Technical University Darmstadt
Darmstadt
Germany
EMail: matthias_michael.wieser@stud.tu-darmstadt.de📦 where this page comes from
- ietf: rfc7290 2014-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms