RFC 6808
Test Plan and Results Supporting Advancement of RFC 2679 on the Standards Track
RFC 6808 📜 Test Plan and Results Supporting Advancement of RFC 2679 on the Standards Track
- Series: Request for Comments (RFC) 6808 🎀
- Organisation: Internet Engineering Task Force (IETF) 🌸
- Category: Informational, so it's published for info, not as a Standards Track spec 📚
- Date: December 2012 🗓️
- ISSN: 2070-1721
- Authors: L. Ciavattone (AT&T Labs), R. Geib (Deutsche Telekom), A. Morton (AT&T Labs), M. Wieser (Technical University Darmstadt) 💜
- Obsoletes / Updates: the title block doesn't list anything it obsoletes or updates 🦋
Abstract ✨
okay so this memo is basically the receipts 🧾 for moving RFC 2679 (the one about one-way delay metrics) up onto the Standards Track, and it follows the process from RFC 6576. the big idea is that the metric definitions are the main thing to check, not the implementations of the metrics 💅. so it describes test procedures for specific metric requirement clauses, to figure out if each requirement was understood and built the way it was meant to be. two totally independent implementations got tested against the key specs of RFC 2679, and this memo also feeds straight into a revision of RFC 2679 🔥
Status of This Memo
this one is not an Internet Standards Track specification, it's just published for informational purposes 🫶
it's a product of the Internet Engineering Task Force (IETF), so it reflects the consensus of the IETF community. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). not every document the IESG approves is up for any level of Internet Standard, so check Section 2 of RFC 5741 if you want the details 👀
for the latest status, any errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc6808 ✨
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date. please actually read those, bestie, because they spell out your rights and restrictions for this document 📜 any Code Components pulled out of this document have to include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Simplified BSD License describes.
this document may contain material from IETF Documents or IETF Contributions that were published or made publicly available before November 10, 2008. the people who control the copyright in some of that material may not have given the IETF Trust permission to allow modifications outside the IETF Standards Process. so unless you get a proper license from those copyright holders, this document can't be modified outside the IETF Standards Process, and derivative works can't be made outside it either. the only exceptions are formatting it for publication as an RFC or translating it into languages other than English 🧚
Table of Contents 🍓
1. Introduction ....................................................3
1.1. Requirements Language ......................................5
2. A Definition-Centric Metric Advancement Process .................5
3. Test Configuration ..............................................5
4. Error Calibration, RFC 2679 .....................................9
4.1. NetProbe Error and Type-P .................................10
4.2. Perfas+ Error and Type-P ..................................12
5. Predetermined Limits on Equivalence ............................12
6. Tests to Evaluate RFC 2679 Specifications ......................13
6.1. One-Way Delay, ADK Sample Comparison: Same- and Cross-
Implementation ............................................13
6.1.1. NetProbe Same-Implementation Results ...............15
6.1.2. Perfas+ Same-Implementation Results ................16
6.1.3. One-Way Delay, Cross-Implementation ADK
Comparison .........................................16
6.1.4. Conclusions on the ADK Results for One-Way Delay ...17
6.1.5. Additional Investigations ..........................17
6.2. One-Way Delay, Loss Threshold, RFC 2679 ...................20
6.2.1. NetProbe Results for Loss Threshold ................21
6.2.2. Perfas+ Results for Loss Threshold .................21
6.2.3. Conclusions for Loss Threshold .....................21
6.3. One-Way Delay, First Bit to Last Bit, RFC 2679 ............21
6.3.1. NetProbe and Perfas+ Results for Serialization .....22
6.3.2. Conclusions for Serialization ......................23
6.4. One-Way Delay, Difference Sample Metric ...................24
6.4.1. NetProbe Results for Differential Delay ............24
6.4.2. Perfas+ Results for Differential Delay .............25
6.4.3. Conclusions for Differential Delay .................25
6.5. Implementation of Statistics for One-Way Delay ............25
7. Conclusions and RFC 2679 Errata ................................26
8. Security Considerations ........................................26
9. Acknowledgements ...............................................27
10. References ....................................................27
10.1. Normative References .....................................27
10.2. Informative References ...................................281. Introduction ✨
the IETF IP Performance Metrics (IPPM) working group has been thinking about how to move their metrics onto the Standards Track since 2001, when Bradner/Paxson/Mankin's memo [METRICS-TEST] first came out 🦋 the original plan was to compare how well the metric implementations performed. that's similar to how protocols usually advance, but it didn't really fit, bestie 🙈 it was hard to reach consensus on exactly how to compare implementations, because lots of legit sources of variation kept showing up in the results even when people tried super hard to keep the network paths equal. on top of that, the parameters (and so the implementation) of each metric could vary a lot. metric definitions needed that flexibility to be customisable and appealing to lots of people, but it made the comparison task really tough 😭
then there was a renewed effort to look at ways to reduce the measurement variability, which would make comparing for equivalence a lot simpler 💖
the consensus process in [RFC6576] says the focus of evaluation should be the metric definitions and not the metric implementations 💅 equivalent test results count as evidence that the metric specifications are clear and unambiguous. honestly, this is now the metric spec version of protocol interoperability ✨ the [RFC6576] advancement process either gives confidence that the metric definitions and their supporting material are clearly worded and unambiguous, or it shows where the metric definitions need revising to get clarity 🌈
the metric RFC advancement process needs the testing and results to be documented. [RFC6576] keeps the testing requirement from the original Standards Track advancement process described in [RFC2026] and [RFC5657], because widespread deployment alone isn't enough to tell whether RFCs that define performance metrics lead to consistent implementations 🧚
the process also lets people identify options that weren't implemented, so those can be removed from the spec that's advancing (this is similar to how protocols advance along the Standards Track) 🍓 all errata have to be considered too.
this memo's purpose is to carry out the advancement process of [RFC6576] for [RFC2679]. it supplies the documentation that goes with the protocol action request submitted to the Area Director. that includes a description of the test setup, results for each implementation, an evaluation of each metric specification, and conclusions 🎀
the big one here is that this memo documents the consensus on how much error is tolerable when judging equivalence in the results 👑 the IPPM working group agreed that the test plan and procedures should include the threshold for deciding equivalence, and that this should be decided before any cross-implementation comparisons happen. this memo also includes procedures for same-implementation comparisons, which may influence that equivalence threshold 💜
the conclusion from testing is that [RFC2679] should advance on the Standards Track with modifications, but the revised text of RFC 2679 isn't ready for review yet. so this memo documents the info needed to support [RFC2679] advancement, and approving a revision of RFC 2769 is left for future action 🫶
quick aside from me: the text says RFC 2769 right there, which is very likely meant to be RFC 2679 🙈 I've kept it exactly as written because the original is the source of truth.
1.1. Requirements Language 🌸
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
2. A Definition-Centric Metric Advancement Process 💖
as a first principle, the process in Section 3.5 of [RFC6576] takes the fact that the metric definitions (embodied in the text of the RFCs) are the objects that need evaluating and possible revision in order to move to the next step on the Standards Track 🔥 this memo follows that process, so the definitions are the main event, bestie ✨
3. Test Configuration ✨
okay so bestie, the first metric implementation was NetProbe version 5.8.5 🎀 (an older version runs in AT&T's IP network performance measurement system and is deployed worldwide [WIPM]). NetProbe sends UDP packets of variable size and can make test streams with Periodic [RFC3432] or Poisson [RFC2330] sample distributions 🦋
The other implementation was Perfas+ version 3.1, made by Deutsche Telekom [Perfas] 💅 Perfas+ also uses UDP unicast packets of variable size (it supports TCP and multicast too!). Test streams can use Periodic, Poisson, or uniform sample distributions 🌸
Figure 1 shows the test path, with each implementation's test flows going through the Internet and the Layer 2 Tunneling Protocol, version 3 (L2TPv3) tunnel IDs (1 and 2). It's based on Figures 2 and 3 of [RFC6576] 🧚
+----+ +----+ +----+ +----+
|Imp1| |Imp1| ,---. |Imp2| |Imp2|
+----+ +----+ / \ +-------+ +----+ +----+
| V100 | V200 / \ | Tunnel| | V300 | V400
| | ( ) | Head | | |
+--------+ +------+ | |__| Router| +----------+
|Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet |
|Switch |--|Head |-| | | |Switch |
+-+--+---+ |Router| | | +---+---+--+--+--+----+
|__| +--A---+ ( ) |Network| |__|
\ / |Emulat.|
U-turn \ / |"netem"| U-turn
V300 to V400 `-+-' +-------+ V100 to V200
Implementations ,---. +--------+
+~~~~~~~~~~~/ \~~~~~~| Remote |
+------->-----F2->-| / \ |->---. |
| +---------+ | Tunnel ( ) | | |
| | transmit|-F1->-| ID 1 ( ) |->. | |
| | Imp 1 | +~~~~~~~~~| |~~~~| | | |
| | receive |-<--+ ( ) | F1 F2 |
| +---------+ | |Internet | | | | |
*-------<-----+ F1 | | | | | |
+---------+ | | +~~~~~~~~~| |~~~~| | | |
| transmit|-* *-| | | |<-* | |
| Imp 2 | | Tunnel ( ) | | |
| receive |-<-F2-| ID 2 \ / |<----* |
+---------+ +~~~~~~~~~~~\ /~~~~~~| Switch |
`-+-' +--------+Figure 1 is a test setup with a bidirectional tunnel 🌈 The top diagram shows the VLAN connectivity and where everything is located geographically. The bottom diagram shows example flows going between the two measurement implementations (only two flows are drawn, to keep it simple).
The testing uses an L2TPv3 [RFC3931] tunnel between the test sites over the Internet 🫶 The tunnel's IP and L2TPv3 headers are meant to hide the test equipment addresses and ports from hash functions. Those hash functions would otherwise spread different test streams across parallel network resources, which could change performance in ways that vary 🙈
At each end of the tunnel, one pair of VLANs inside the tunnel gets looped back, so the test traffic returns to each test site. So the test streams cross the L2TP tunnel twice, but from the test equipment's point of view they look like one-way tests 🔁
The network emulator is a host running Fedora 14 Linux [Fedora14] with IP forwarding turned on. It also has the "netem" Network emulator [netem] loaded and running as part of the Fedora Kernel 2.6.35.11 🔥 Connectivity across the netem/Fedora host was set up by bridging the Ethernet VLAN interfaces together with "brctl" commands (for example, eth1.100 <-> eth2.100). The netem emulator was turned on for one interface (eth1), and it only affects test streams going in one direction. In some tests, separate netem instances ran on each VLAN 💜
When the testing was done, the links between the netem emulator host and the router and switch were found to be 100baseTx-HD (100 Mbps half duplex). Half duplex wasn't planned, but it probably added a small amount of delay variation that full duplex mode could have avoided 😭
Each test ran with common packet rates (1 pps, 10 pps), Poisson/Periodic distributions, and IP packet sizes of 64, 340, and 500 Bytes ✨ These sizes cover a reasonable range while avoiding fragmentation and the complications it brings. That keeps them in line with the idea of "standard formed packets" from Section 15 of [RFC2330] 👑
For these tests, at least 300 packets were sent from Source to Destination in each implementation. Periodic streams (as in [RFC3432]) with 1 second spacing were used, except where noted 🍓
With the L2TPv3 tunnel in use, the metric name for this test setup (looking at the IP header that's visible to Internet processing) is:
Type-IP-protocol-115-One-way-Delay-<StreamType>-StreamWith Metric Parameters from (Section 4.2 of [RFC2679]):
Src, the IP address of a host (12.3.167.16 or 193.159.144.8) 🎀Dst, the IP address of a host (193.159.144.8 or 12.3.167.16) 🎀T0, a time ⏰Tf, a time ⏰lambda, a rate in reciprocal seconds 🦋Thresh, a maximum waiting time in seconds (see Section 3.8.2 of [RFC2679] and Section 4.3 of [RFC2679]) 💖
Metric Units: a sequence of pairs, where each pair has these elements:
T, a time, anddT, either a real number or an undefined number of seconds.
The values of T in the sequence are monotonic increasing. Also note that T would be a valid parameter to Type-P-One-way-Delay, and that dT would be a valid value of Type-P-One-way-Delay 🧠
Also, Section 3.8.4 of [RFC2679] says the path SHOULD be reported 🗺️ In this setup, most of the path details are hidden from the implementations by the L2TPv3 tunnels. So a more useful path trace route can be run by the routers at each location 🫶
When NetProbe is used in production, a traceroute runs in parallel with the measurements and at the start of them 🌸
Perfas+ doesn't support traceroute 🙃
IPLGW#traceroute 193.159.144.8
Type escape sequence to abort.
Tracing the route to 193.159.144.8
1 12.126.218.245 [AS 7018] 0 msec 0 msec 4 msec
2 cr84.n54ny.ip.att.net (12.123.2.158) [AS 7018] 4 msec 4 msec
cr83.n54ny.ip.att.net (12.123.2.26) [AS 7018] 4 msec
3 cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 4 msec
cr2.n54ny.ip.att.net (12.122.115.93) [AS 7018] 0 msec
cr1.n54ny.ip.att.net (12.122.105.49) [AS 7018] 0 msec
4 n54ny02jt.ip.att.net (12.122.80.225) [AS 7018] 4 msec 0 msec
n54ny02jt.ip.att.net (12.122.80.237) [AS 7018] 4 msec
5 192.205.34.182 [AS 7018] 0 msec
192.205.34.150 [AS 7018] 0 msec
192.205.34.182 [AS 7018] 4 msec
6 da-rg12-i.DA.DE.NET.DTAG.DE (62.154.1.30) [AS 3320] 88 msec 88 msec
88 msec
7 217.89.29.62 [AS 3320] 88 msec 88 msec 88 msec
8 217.89.29.55 [AS 3320] 88 msec 88 msec 88 msec
9 * * *It was only possible to run the traceroute for the measured path on one of the tunnel-head routers 🥺 (the normal trace tools of the measurement systems get confused by the L2TPv3 tunnel encapsulation).
4. Error Calibration, RFC 2679 💅
Every implementation is required to report on its error calibration in Section 3.8 of [RFC2679] (it's also required in Section 4.8 for sample metrics) 📏 Sections 3.6, 3.7, and 3.8 of [RFC2679] give the full formulas for the errors and uncertainties used in calibration 🧮 In short, Section 3.7.1 of [RFC2679] describes the total time-varying uncertainty like this:
Esynch(t)+ Rsource + Rdestwhere:
Esynch(t)is an upper bound on the size of clock synchronization uncertainty.RsourceandRdestare the resolution of the source clock and the destination clock, respectively.
Then Section 3.7.2 of [RFC2679] describes the total wire-time uncertainty like this:
Hsource + HdestThose are the upper bounds on host-time to wire-time for the source and the destination, respectively 🔍
Section 3.7.3 of [RFC2679] describes a test with small packets over an isolated minimal network. The results from that test can estimate the systematic and random parts of the sum of the errors and uncertainties above 🧪 In a test with hundreds of singletons, the median is the systematic error. If you subtract the median from every singleton, the variability left over is the random error 💜
The test context, which is the Type-P of the test packets, must also be reported. That's required in Section 3.8 of [RFC2679] and for all the metrics defined there 🎀 Type-P is defined in Section 13 of [RFC2330] (and so are a lot of the terms used below) 🌟
4.1. NetProbe Error and Type-P 💖
okay so the Type-P for this test was IP-UDP with Best Effort Differentiated Services Code Point (DSCP) 🎀. These headers got wrapped up following the L2TPv3 specs [RFC3931], so ngl they may not change the treatment the packets get as they travel across the Internet 🫶.
In general, NetProbe error depends on the specific version and installation details 💅.
NetProbe operates using host-time above the UDP layer 🙈. That's different from the wire-time that [RFC2330] prefers, but it still counts as a source of error under Section 3.7.2 of [RFC2679] ✨.
Accuracy of NetProbe measurements is usually limited by NTP synchronization performance, which is typically taken as ~+/-1 ms error or greater 😭. The installation used in this testing, though, was giving way better than typical NTP 🦋. The primary stratum 1 NTP server sits closely located on a sparsely utilized network management LAN, so it avoids many of the concerns raised in Section 10 of [RFC2330]. Smooth adjustment, long-term drift analysis and compensation, and infrequent adjustment all lead to stability during measurement intervals, which is the main concern 👑.
The resolution of the reported results is 1 us (us = microsecond) in the version of NetProbe tested here, so it contributes to at least +/-1 us error 🌸.
NetProbe implements a timekeeping sanity check on its sending and receiving time-stamping processes 🧚. When a significant process interruption happens, individual test packets get flagged as possibly containing unusual time errors, and they're excluded from the sample used for all "time" metrics 💀.
We performed a NetProbe calibration of the type described in Section 3.7.3 of [RFC2679], using 64-Byte packets over a cross-connect cable 🔥. The results estimate the systematic and random components of the sum of the Hsource + Hdest errors or uncertainties. In a test with 300 singletons conducted over 30 seconds (periodic sample with 100 ms spacing), the median is the systematic error and the remaining variability is the random error ⭐. One set of results is tabulated below:
(Results from the "R" software environment for statistical computing and graphics - http://www.r-project.org/)
> summary(XD4CAL)
CAL1 CAL2 CAL3
Min. : 89.0 Min. : 68.00 Min. : 54.00
1st Qu.: 99.0 1st Qu.: 77.00 1st Qu.: 63.00
Median :110.0 Median : 79.00 Median : 65.00
Mean :116.8 Mean : 83.74 Mean : 69.65
3rd Qu.:127.0 3rd Qu.: 88.00 3rd Qu.: 74.00
Max. :205.0 Max. :177.00 Max. :163.00
>NetProbe Calibration with Cross-Connect Cable, one-way delay values in microseconds (us)
The median or systematic error can be as high as 110 us, and the range of the random error is also on the order of 116 us for all streams 📊.
Also, anticipating the Anderson-Darling K-sample (ADK) [ADK] comparisons to follow, we corrected the CAL2 values for the difference between the means of CAL2 and CAL3 (as permitted in Section 3.2 of [RFC6576]). We found strong support (for the Null Hypothesis) that the samples are from the same distribution, at a resolution of 1 us and with alpha equal to 0.05 and 0.01 🦋.
> XD4CVCAL2 <- XD4CAL$CAL2 - (mean(XD4CAL$CAL2)-mean(XD4CAL$CAL3))
> boxplot(XD4CVCAL2,XD4CAL$CAL3)
> XD4CV2_ADK <- adk.test(XD4CVCAL2, XD4CAL$CAL3)
> XD4CV2_ADK
Anderson-Darling k-sample test.
Number of samples: 2
Sample sizes: 300 300
Total number of values: 600
Number of unique values: 97
Mean of Anderson Darling Criterion: 1
Standard deviation of Anderson Darling Criterion: 0.75896
T = (Anderson-Darling Criterion - mean)/sigma
Null Hypothesis: All samples come from a common population.
t.obs P-value extrapolation
not adj. for ties 0.71734 0.17042 0
adj. for ties -0.39553 0.44589 1
>We used [Rtool] and [Radk] for this 💜.
4.2. Perfas+ Error and Type-P ✨
Perfas+ is configured to use GPS synchronization, and it uses NTP synchronization as a fall-back or default 🛰️. GPS synchronization worked throughout this test with the exception of the calibration stated here, where one implementation was NTP synchronized only. The time stamp accuracy typically is 0.1 ms 🌈.
The resolution of the results reported by Perfas+ is 1 us (us = microsecond) in the version tested here, which contributes to at least +/-1 us error 🎀.
Port 5001 5002 5003
Min. -227 -226 294
Median -169 -167 323
Mean -159 -157 335
Max. 6 -52 376
s 102 102 93Perfas+ Calibration with Cross-Connect Cable, one-way delay values in microseconds (us)
The median or systematic error can be as high as 323 us, and the range of the random error is also less than 232 us for all streams 💅.
5. Predetermined Limits on Equivalence 👑
This section provides the numerical limits on comparisons between implementations. The point is to declare that the results are equivalent and therefore that the tested specification is clear 🫶. These limits have their basis in Section 3.1 of [RFC6576] and the Appendix of [RFC2330], with additional limits representing IP Performance Metrics (IPPM) consensus prior to publication of results.
A key point is that the allowable errors, corrections, and confidence levels only need to be sufficient to detect misinterpretation of the tested specification that results in diverging implementations 🔍.
Also, the allowable error must be sufficient to compensate for measured path differences. It was simply not possible to measure fully identical paths in the VLAN-loopback test configuration used, so this practical compromise must be taken into account 🙈.
For Anderson-Darling K-sample (ADK) comparisons, the required confidence factor for the cross-implementation comparisons SHALL be the smallest of:
- 0.95 confidence factor at 1 ms resolution, or
- the smallest confidence factor (in combination with resolution) of the two same-implementation comparisons for the same test conditions.
A constant time accuracy error of as much as +/-0.5 ms MAY be removed from one implementation's distributions (all singletons) before the ADK comparison is conducted ⏱️.
A constant propagation delay error (due to use of different sub-nets between the switch and measurement devices at each location) of as much as +2 ms MAY be removed from one implementation's distributions (all singletons) before the ADK comparison is conducted 🦋.
For comparisons involving the mean of a sample or other central statistics, the limits on both the time accuracy error and the propagation delay error constants given above also apply 💜.
6. Tests to Evaluate RFC 2679 Specifications 🧪
This section describes some results from real-world (cross-Internet) tests. It uses measurement devices implementing IPPM metrics and a network emulator to create relevant conditions, so we can figure out whether the metric definitions were interpreted consistently by implementors 🌸.
The procedures are slightly modified from the original procedures contained in Appendix A.1 of [RFC6576]. The modifications include the use of the mean statistic for comparisons 🎀.
Note that there are only five instances of the requirement term "MUST" in [RFC2679] outside of the boilerplate and [RFC2119] reference 👀.
6.1. One-Way Delay, ADK Sample Comparison: Same- and Cross-Implementation 🍓
This test determines if implementations produce results that appear to come from a common delay distribution. It's an overall evaluation of Section 4 of [RFC2679], "A Definition for Samples of One-way Delay" 🔥. Same-implementation comparison results help set the threshold of equivalence that gets applied to cross-implementation comparisons.
This test is intended to evaluate measurements in Sections 3 and 4 of [RFC2679] ✨.
By testing the extent to which the distributions of one-way delay singletons from two implementations of [RFC2679] appear to be from the same distribution, we economize on comparisons. Comparing a set of individual summary statistics (as defined in Section 5 of [RFC2679]) would require another set of individual evaluations of equivalence. Instead, we can simply check which statistics were implemented and report on those facts 💅.
- Configure an L2TPv3 path between test sites, and each pair of measurement devices to operate tests in their designated pair of VLANs.
- Measure a sample of one-way delay singletons with two or more implementations, using identical options and network emulator settings (if used).
- Measure a sample of one-way delay singletons with four instances of the same implementations, using identical options, noting that connectivity differences SHOULD be the same as for the cross-implementation testing.
- Apply the ADK comparison procedures (see Appendices A and B of [RFC6576]) and determine the resolution and confidence factor for distribution equivalence of each same-implementation comparison and each cross-implementation comparison.
- Take the coarsest resolution and confidence factor for distribution equivalence from the same-implementation pairs, or the limit defined in Section 5 above, as a limit on the equivalence threshold for these experimental conditions.
- Apply constant correction factors to all singletons of the sample distributions, as described and limited in Section 5 above.
- Compare the cross-implementation ADK performance with the equivalence threshold determined in step 5 to determine if equivalence can be declared 👑.
The common parameters used for tests in this section are:
- IP header + payload = 64 octets
- Periodic sampling at 1 packet per second
- Test duration = 300 seconds (March 29, 2011)
The netem emulator was set for 100 ms average delay, with uniform delay variation of +/-50 ms. In this experiment, the netem emulator was configured to operate independently on each VLAN. Thus, the emulator itself is a potential source of error when comparing streams that traverse the test path in different directions 🫣.
In the result analysis of this section:
- All comparisons used 1 microsecond resolution.
- No correction factors were applied.
- The 0.95 confidence factor (1.960 for paired stream comparison) was used.
6.1.1. NetProbe Same-Implementation Results 💅
okay so, one same-implementation comparison flops the ADK criterion 😭 (s1 <-> sB). the thing is, those two streams went through the test path in opposite directions, so the live network could totally be the reason, and that's a real possibility 🫶
everything else in the same-implementation group passes the ADK criterion ✨
+------------------------------------------------------+
| | | | |
| ti.obs (P) | s1 | s2 | sA |
| | | | |
.............|.............|.............|.............|
| | | | |
| s2 | 0.25 (0.28) | | |
| | | | |
...........................|.............|.............|
| | | | |
| sA | 0.60 (0.19) |-0.80 (0.57) | |
| | | | |
...........................|.............|.............|
| | | | |
| sB | 2.64 (0.03) | 0.07 (0.31) |-0.52 (0.48) |
| | | | |
+------------+-------------+-------------+-------------+
NetProbe ADK results for same-implementation6.1.2. Perfas+ Same-Implementation Results 🎀
every pair comparison passes the ADK criterion, no notes 🙌
+------------------------------------------------------+
| | | | |
| ti.obs (P) | p1 | p2 | p3 |
| | | | |
.............|.............|.............|.............|
| | | | |
| p2 | 0.06 (0.32) | | |
| | | | |
.........................................|.............|
| | | | |
| p3 | 1.09 (0.12) | 0.37 (0.24) | |
| | | | |
...........................|.............|.............|
| | | | |
| p4 |-0.81 (0.57) |-0.13 (0.37) | 1.36 (0.09) |
| | | | |
+------------+-------------+-------------+-------------+
Perfas+ ADK results for same-implementation6.1.3. One-Way Delay, Cross-Implementation ADK Comparison 🦋
for the cross-implementation stuff, we use a combined ADK analysis [Radk]. basically all the NetProbe results get compared with all the Perfas+ results, but only after we check that the combined same-implementation results pass the ADK criterion first 🧚
when 4 (same) samples are compared, the ADK criterion for 0.95 confidence is 1.915, and when all 8 (cross) samples get compared it's 1.85 🔥
Combination of Anderson-Darling K-Sample Tests.
Sample sizes within each data set:
Data set 1 : 299 297 298 300 (NetProbe)
Data set 2 : 300 300 298 300 (Perfas+)
Total sample size per data set: 1194 1198
Number of unique values per data set: 1188 1192
...
Null Hypothesis:
All samples within a data set come from a common distribution.
The common distribution may change between data sets.
NetProbe ti.obs P-value extrapolation
not adj. for ties 0.64999 0.21355 0
adj. for ties 0.64833 0.21392 0
Perfas+
not adj. for ties 0.55968 0.23442 0
adj. for ties 0.55840 0.23473 0
Combined Anderson-Darling Criterion:
tc.obs P-value extrapolation
not adj. for ties 0.85537 0.17967 0
adj. for ties 0.85329 0.18010 0so the combined same-implementation samples and the combined cross-implementation comparison all pass the ADK criterion at P>=0.18 and back up the Null Hypothesis (both data sets come from a common distribution) 🥰
also, the paired ADK comparisons are kinda harsh 💀 the NetProbe s1-sB comparison failed, but the combined data from all four streams passed the ADK criterion easily. no cap, that's giving resilience ✨
6.1.4. Conclusions on the ADK Results for One-Way Delay 👑
this same kind of testing got repeated a bunch of times in March and April 2011 🌸 a bunch of experiments had one test stream from NetProbe or Perfas+ looking different from the rest in paired comparisons, even same-implementation ones. when they pulled out that outlier stream, the remaining streams passed the combined ADK criterion. applying correction factors also led to more comparisons succeeding 💖
so our conclusion is that the two implementations can produce equivalent one-way delay distributions based on how each of them interprets [RFC2679] 🫶✨
6.1.5. Additional Investigations 🔍
on the last day of testing, we ran a series of measurements to see how much emulated delay variation it took to get successful ADK comparisons 🍓 we also checked whether correction factors were needed (as Section 5 permits) and how the measurement sample size mattered (using sub-sets of the complete sample) 🧚
the common parameters for this section:
- IP header + payload = 64 octets
- Periodic sampling at 1 packet per second
- Test duration = 300 seconds at each delay variation setting, for a total of 1200 seconds (May 2, 2011 at 1720 UTC)
the netem emulator was set for 100 ms average delay, with (emulated) uniform delay variation of:
- +/-7.5 ms
- +/-5.0 ms
- +/-2.5 ms
- 0 ms
in this experiment, netem ran independently on each VLAN, so the emulator itself is a possible source of error when you compare streams going through the test path in different directions 🙈
for the analysis in this section:
- All comparisons used 1 microsecond resolution.
- Correction factors were applied as noted (under the column heading "mean adj"). The difference between each sample mean and the lowest mean of the NetProbe or Perfas+ stream samples was subtracted from all values in the sample. ("raw" means no correction factors were used.) All correction factors applied met the limits described in Section 5.
- The 0.95 confidence factor (1.960 for paired stream comparison) was used.
when 8 (cross) samples are compared, the ADK criterion for 0.95 confidence is 1.85. the Combined ADK test statistic ("TC observed") must be less than 1.85 to accept the Null Hypothesis (all samples in the data set are from a common distribution) 🥹
Emulated Delay Sub-Sample size
Variation 0ms
adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj
TC observed 226.6563 67.51559 54.01359 21.56513
P-value 0 0 0 0
Mean std dev (all),us 719 635
Mean diff of means,us 649 0 606 0
Variation +/- 2.5ms
adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj
TC observed 14.50436 -1.60196 3.15935 -1.72104
P-value 0 0.873 0.00799 0.89038
Mean std dev (all),us 1655 1702
Mean diff of means,us 471 0 513 0
Variation +/- 5ms
adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj
TC observed 8.29921 -1.28927 0.37878 -1.81881
P-value 0 0.81601 0.29984 0.90305
Mean std dev (all),us 3023 2991
Mean diff of means,us 582 0 513 0
Variation +/- 7.5ms
adk.combined (all) 300 values 75 values
Adj. for ties raw mean adj raw mean adj
TC observed 2.53759 -0.72985 0.29241 -1.15840
P-value 0.01950 0.66942 0.32585 0.78686
Mean std dev (all),us 4449 4506
Mean diff of means,us 426 0 856 0so from that table, here's the tea ☕
- none of the raw or mean adjusted results pass the ADK criterion with 0 ms emulated delay variation. using the 75 value sub-sample gave the same answer. (same results show up when you compare same-implementation samples for both NetProbe and Perfas+ 🙃)
- once the smallest emulated delay variation (+/-2.5 ms) went in, the mean adjusted samples pass the ADK criterion and the high P-value backs that up. the raw results still don't pass 🥲
- at the higher emulated delay variations (+/-5.0 ms and +/-7.5 ms), the mean adjusted values pass ADK again. the 75-value sub-sample also passed ADK in both the raw and mean adjusted cases 🤭 this suggests sample size may have played a role in our results, as noted in the Appendix of [RFC2330] for Goodness-of-Fit testing 🧐
we also checked 150 value sub-samples, and their ADK conclusions followed the 300 value results 💅 same-implementation analysis gave similar results to the above, except more of the "raw" or uncorrected samples passed the ADK criterion ✨
6.2. One-Way Delay, Loss Threshold, RFC 2679 🍓
this test checks whether implementations use the same configured maximum waiting time delay from one measurement to the next under different delay conditions, and whether they correctly call packets that arrive after the waiting time threshold lost 🙈⏱️
go look at Section 3.5 of [RFC2679], third bullet point, and also Section 3.8.2 of [RFC2679] 📖
- configure an L2TPv3 path between the test sites, and each pair of measurement devices to run tests in their designated pair of VLANs 🌈
- configure the network emulator to add 1.0 sec. one-way constant delay in one direction of transmission 🐢
- measure (average) one-way delay with two or more implementations, using identical waiting time thresholds (Thresh) for loss set at 3 seconds ⏳
- configure the network emulator to add 3 sec. one-way constant delay in one direction of transmission, equivalent to 2 seconds of additional one-way delay (or change the path delay while the test is in progress, when there are sufficient packets at the first delay setting) 🦋
- repeat/continue measurements 🔁
- observe that the increase measured in step 5 caused all packets with 2 sec. additional delay to be declared lost, and that all packets that arrive successfully in step 3 are assigned a valid one-way delay 👑
the common parameters for tests in this section:
- IP header + payload = 64 octets
- Poisson sampling at lambda = 1 packet per second
- Test duration = 900 seconds total (March 21, 2011)
the netem emulator was set to add constant delays as specified in the procedure above ✨
6.2.1. NetProbe Results for Loss Threshold
Okay so NetProbe handles the Loss Threshold as a post-processing routine across every packet 💅 With the Loss Threshold set at 3 seconds, any packet with one-way delay >3 seconds gets marked "Lost" and goes onto the Lost Packet list with its transmission time (as required in Section 3.3 of [RFC2680]). That gave 342 packets marked as lost in one of the test streams (average delay = 3.091 sec.) 😭
6.2.2. Perfas+ Results for Loss Threshold
Perfas+ has a fixed Loss Threshold that couldn't be adjusted during this study 🙈 It's roughly one minute, and nobody tried emulating a delay that big. BUT it's still possible to use any delay threshold you want with a post-processing routine and some analysis afterwards 🧚 Doing it that way, 195 packets would be declared lost (with average delay = 3.091 sec.) ✨
6.2.3. Conclusions for Loss Threshold
Both implementations assume that any constant delay value can be used as the Loss Threshold, since all delays are stored as a pair <Time, Delay> as required in [RFC2679] 💖 This is a simple way to enforce the constant loss threshold envisioned in [RFC2679] (see specific section references above). We take the position that the assumption of post-processing is compliant, and that the text of the RFC should be revised slightly to include this point 🫶
6.3. One-Way Delay, First Bit to Last Bit, RFC 2679
This test checks if implementations register the same relative change in delay from one packet size to another, which shows that the first-to-last time-stamping convention has been followed 🦋 It also tends to cancel out the sources of error that might be present in an implementation, which is very iconic of it ✨
See the requirements of Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330].
- configure an L2TPv3 path between test sites, and each pair of measurement devices to operate tests in their designated pair of VLANs, and ideally including a low-speed link (it was not possible to change the link configuration during testing, so the lowest speed link present was the basis for serialization time comparisons)
- measure (average) one-way delay with two or more implementations, using identical options and equal size small packets (64-octet IP header and payload)
- maintain the same path with additional emulated 100 ms one-way delay
- measure (average) one-way delay with two or more implementations, using identical options and equal size large packets (500 octet IP header and payload)
- observe that the increase measured between steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary 🔥
The common parameters used for tests in this section are:
- IP header + payload = 64 octets
- Periodic sampling at l packet per second
- Test duration = 300 seconds total (April 12)
The netem emulator was set to add constant 100 ms delay 🌸
6.3.1. NetProbe and Perfas+ Results for Serialization
When the IP header + payload size went up from 64 octets to 500 octets, a delay increase was observed 👀
Mean Delays in us
NetProbe
Payload s1 s2 sA sB
500 190893 191179 190892 190971
64 189642 189785 189747 189467
Diff 1251 1394 1145 1505
Perfas
Payload p1 p2 p3 p4
500 190908 190911 191126 190709
64 189706 189752 189763 190220
Diff 1202 1159 1363 489Serialization tests, all values in microseconds
The typical delay increase when the larger packets were used was 1.1 to 1.5 ms (with one outlier) 🎀 The typical measurements indicate that a link with approximately 3 Mbit/s capacity is present on the path.
Through investigation of the facilities involved, it was determined that the lowest speed link was approximately 45 Mbit/s, so the estimated difference should be about 0.077 ms. The observed differences are much higher 😭
The unexpected large delay difference was also the outcome when testing serialization times in a lab environment, using the NIST Net Emulator and NetProbe [ADV-METRICS]. Not me being shocked twice 💀
6.3.2. Conclusions for Serialization
Since it was not possible to confirm the estimated serialization time increases in field tests, we look at the implementations themselves to determine compliance 🔍
NetProbe performs all time stamping above the IP layer, accepting that some compromises must be made to achieve extreme portability and measurement scale 🧚 Therefore, the first-to-last bit convention is supported, because the serialization time is included in the one-way delay measurement, enabling comparison with other implementations 💅
Perfas+ is optimized for its purpose and performs all time stamping close to the interface hardware 👑 The first-to-last bit convention is supported, because the serialization time is included in the one-way delay measurement, enabling comparison with other implementations 🫶
6.4. One-Way Delay, Difference Sample Metric
This test checks if implementations register the same relative increase in delay from one measurement to another under different delay conditions 🌈 It also tends to cancel out the sources of error that may be present in an implementation.
This test is intended to evaluate measurements in Sections 3 and 4 of [RFC2679] ✨
- configure an L2TPv3 path between test sites, and each pair of measurement devices to operate tests in their designated pair of VLANs
- measure (average) one-way delay with two or more implementations, using identical options
- configure the path with X+Y ms one-way delay
- repeat measurements
- observe that the (average) increase measured in steps 2 and 4 is ~Y ms for each implementation. Most of the measurement errors in each system should cancel, if they are stationary 🦋
In this test, X = 1000 ms and Y = 1000 ms.
The common parameters used for tests in this section are:
- IP header + payload = 64 octets
- Poisson sampling at lambda = 1 packet per second
- Test duration = 900 seconds total (March 21, 2011)
The netem emulator was set to add constant delays as specified in the procedure above 🧚
6.4.1. NetProbe Results for Differential Delay
Average pre-increase delay, microseconds 1089868.0
Average post 1 s additional, microseconds 2089686.0
Difference (should be ~= Y = 1 s) 999818.0
Average delays before/after 1 second increaseThe NetProbe implementation observed a 1 second increase with a 182 microsecond error (assuming that the netem emulated delay difference is exact) 🍓
We note that this differential delay test has been run under lab conditions and published in prior work [ADV-METRICS]. The error was 6 microseconds 🌟
6.4.2. Perfas+ Results for Differential Delay
Average pre-increase delay, microseconds 1089794.0
Average post 1 s additional, microseconds 2089801.0
Difference (should be ~= Y = 1 s) 1000007.0
Average delays before/after 1 second increaseThe Perfas+ implementation observed a 1 second increase with a 7 microsecond error 🔥
6.4.3. Conclusions for Differential Delay
Again, the live network conditions seem to have influenced the results, but both implementations measured the same delay increase within their calibration accuracy 💖
6.5. Implementation of Statistics for One-Way Delay
The ADK tests the extent to which the sample distributions of one-way delay singletons from two implementations of [RFC2679] appear to come from the same overall distribution 📊 By testing this way, we save on the number of comparisons, because comparing a set of individual summary statistics (as defined in Section 5 of [RFC2679]) would need another set of individual evaluations of equivalence 🙄 Instead, we can simply check which statistics were implemented and report on those facts. Note that Section 5 of [RFC2679] does not specify the calculations exactly, and gives only some illustrative examples 📝
NetProbe Perfas+
5.1. Type-P-One-way-Delay-Percentile yes no
5.2. Type-P-One-way-Delay-Median yes no
5.3. Type-P-One-way-Delay-Minimum yes yes
5.4. Type-P-One-way-Delay-Inverse-Percentile no no
Implementation of Section 5 StatisticsOnly the Type-P-One-way-Delay-Inverse-Percentile has been ignored in both implementations, so it is a candidate for removal or deprecation in a revision of RFC 2679 (this small discrepancy does not affect candidacy for advancement) 🥺
7. Conclusions and RFC 2679 Errata
The conclusions throughout Section 6 support the advancement of [RFC2679] to the next step of the Standards Track, because its requirements are deemed clear and unambiguous based on evaluation of the test results for two implementations 👑 The results indicate that these implementations produced statistically equivalent results under network conditions that were configured to be as close to identical as possible 🎀
Sections 6.2.3 and 6.5 indicate areas where minor revisions are warranted in RFC 2679 ✨ The IETF has reached consensus on guidance for reporting metrics in [RFC6703], and this memo should be referenced in the revision to RFC 2679 to incorporate recent experience where appropriate 🌸
We note that there is currently one erratum with status "Held for Document Update" for [RFC2679], and it appears this minor revision and additional text should be incorporated in a revision of RFC 2679 🫶
The authors that revise [RFC2679] should review all errata filed at the time the document is being written. They should not rely upon this document to indicate all relevant errata updates. Do your homework, bestie 📚
8. Security Considerations
The security considerations that apply to any active measurement of live networks are relevant here as well. See [RFC4656] and [RFC5357]. Active measurement of live networks is no joke, so treat these security considerations with full seriousness 🚨
9. Acknowledgements
The authors thank Lars Eggert for his continued encouragement to advance the IPPM metrics during his tenure as AD Advisor 💜
Nicole Kowalski supplied the needed CPE router for the NetProbe side of the test setup, and graciously managed her testing in spite of issues caused by dual-use of the router. Thanks Nicole! 🥰
The "NetProbe Team" also acknowledges many useful discussions with Ganga Maguluri 🫶
10. References
10.1. Normative References 💖
okay so the normative refs are the must-read-first crew, bestie 📚✨
- [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. 🎀
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 🔑
- [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. 🌸
- [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way Delay Metric for IPPM", RFC 2679, September 1999. ⏱️
- [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way Packet Loss Metric for IPPM", RFC 2680, September 1999. 📉
- [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network performance measurement with periodic streams", RFC 3432, November 2002. 🦋
- [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, September 2006. 🧚
- [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, October 2008. 💜
- [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard", BCP 9, RFC 5657, September 2009. 🍓
- [RFC6576] Geib, R., Morton, A., Fardid, R., and A. Steinmitz, "IP Performance Metrics (IPPM) Standard Advancement Testing", BCP 176, RFC 6576, March 2012. 🌈
- [RFC6703] Morton, A., Ramachandran, G., and G. Maguluri, "Reporting IP Network Performance Metrics: Different Points of View", RFC 6703, August 2012. 👑
10.2. Informative References 🫶
these ones are the extra-credit reading, no cap 🤓
- [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling Tests of fit, for continuous and discrete cases", University of Washington, Technical Report No. 81, May 1986. 📊
- [ADV-METRICS] Morton, A., "Lab Test Results for Advancing Metrics on the Standards Track", Work in Progress, October 2010. 🧪
- [Fedora14] Fedora Project, "Fedora Project Home Page", 2012,
<http://fedoraproject.org/>. 🐧 - [METRICS-TEST] Bradner, S. and V. Paxson, "Advancement of metrics specifications on the IETF Standards Track", Work in Progress, August 2007. 🛤️
- [Perfas] Heidemann, C., "Qualitaet in IP-Netzen Messverfahren", published by ITG Fachgruppe, 2nd meeting 5.2.3 (NGN), November 2001,
<http://www.itg523.de/oeffentlich/01nov/Heidemann_QOS_Messverfahren.pdf>. 🇩🇪 - [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 🚇
- [Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and Combinations of Such Tests. R package version 1.0.", 2008. 📦
- [Rtool] R Development Core Team, "R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0", 2011,
<http://www.R-project.org/>. 📈 - [WIPM] AT&T, "AT&T Global IP Network", 2012,
<http://ipnetwork.bgtmo.ip.att.net/pws/index.html>. 🌐 - [netem] The Linux Foundation, "netem", 2009,
<http://www.linuxfoundation.org/collaborate/workgroups/networking/netem>. 🐧✨
Authors' Addresses 🎀
the squad behind this doc, iconic every single one 👑
Len Ciavattone
Len Ciavattone
AT&T Labs
200 Laurel Avenue South
Middletown, NJ 07748
USA
Phone: +1 732 420 1239
EMail: lencia@att.comRuediger Geib
Ruediger Geib
Deutsche Telekom
Heinrich Hertz Str. 3-7
Darmstadt, 64295
Germany
Phone: +49 6151 58 12747
EMail: Ruediger.Geib@telekom.deAl Morton
Al Morton
AT&T Labs
200 Laurel Avenue South
Middletown, NJ 07748
USA
Phone: +1 732 420 1571
Fax: +1 732 368 1192
EMail: acmorton@att.com
URI: http://home.comcast.net/~acmacm/Matthias Wieser
Matthias Wieser
Technical University Darmstadt
Darmstadt,
Germany
EMail: matthias_michael.wieser@stud.tu-darmstadt.de📦 where this page comes from
- ietf: rfc6808 2012-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms