RFC 6576
IP Performance Metrics (IPPM) Standard Advancement Testing
RFC 6576: IP Performance Metrics (IPPM) Standard Advancement Testing 💖
okay bestie, let's do the paperwork first 🎀 This is RFC 6576, from the Internet Engineering Task Force (IETF), and it's a Best Current Practice (category: Best Current Practice, aka BCP 176) 🌸 It's dated March 2012, ISSN 2070-1721.
The authors are R. Geib, Ed. (Deutsche Telekom), A. Morton (AT&T Labs), R. Fardid (Cariden Technologies) and A. Steinmitz (Deutsche Telekom) 🫶 And the title block doesn't list anything this one obsoletes or updates ✨
Abstract
okay so this doc is all about checking whether a bunch of separate performance-metric RFCs have been implemented the same way 🔍 It's basically the metric version of interoperability, which you need before an RFC can move up the Standards Track 🚀 The plan is to collect results from different implementations of metric RFCs under the same underlying network conditions, then compare them with statistical methods 📊 The real goal is to judge the metric RFC itself: are its definitions clear and unambiguous for implementors, so it's a good candidate for advancement on the IETF Standards Track? 💅 And yes, this document is an Internet Best Current Practice 🎀
Status of This Memo
this memo documents an Internet Best Current Practice 🌈
this document is a product of the Internet Engineering Task Force (IETF) 💜 It represents the consensus of the IETF community, it got public review, and it was approved for publication by the Internet Engineering Steering Group (IESG) 👑 If you want more about BCPs, check Section 2 of RFC 5741 🧚
info about the current status of this doc, any errata, and how to send feedback is at http://www.rfc-editor.org/info/rfc6576 🦋
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. 🎀
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication 📜 Please read these carefully, because they spell out your rights and restrictions for this document 🙈 Code Components pulled out of this document must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 💀
Table of Contents
1. Introduction ....................................................3
1.1. Requirements Language ......................................5
2. Basic Idea ......................................................5
3. Verification of Conformance to a Metric Specification ...........7
3.1. Tests of an Individual Implementation against a Metric
Specification ..............................................8
3.2. Test Setup Resulting in Identical Live Network
Testing Conditions .........................................9
3.3. Tests of Two or More Different Implementations
against a Metric Specification ............................15
3.4. Clock Synchronization .....................................16
3.5. Recommended Metric Verification Measurement Process .......17
3.6. Proposal to Determine an Equivalence Threshold for
Each Metric Evaluated .....................................20
4. Acknowledgements ...............................................21
5. Contributors ...................................................21
6. Security Considerations ........................................21
7. References .....................................................21
7.1. Normative References ......................................21
7.2. Informative References ....................................23
Appendix A. An Example on a One-Way Delay Metric Validation ......24
A.1. Compliance to Metric Specification Requirements ...........24
A.2. Examples Related to Statistical Tests for One-Way Delay ...25
Appendix B. Anderson-Darling K-sample Reference and 2 Sample
C++ Code .............................................27
Appendix C. Glossary .............................................361. Introduction ✨
okay so the Internet Standards Process, as updated by RFC 6410 [RFC6410], says that widespread deployment and use is enough to show interoperability as a condition for advancing to Internet Standard 🌟 The older rule, requiring interoperability tests before advancing an RFC to the Standard maturity level (as specified in RFC 2026 [RFC2026] and RFC 5657 [RFC5657]), has been removed 🗑️ The new rule works for protocols, but lots of different measurement systems being widely deployed doesn't prove they measure metrics in a standard way 🤔 Section 5.3 of RFC 5657 [RFC5657] explicitly calls out the special case of Standards that aren't "on-the-wire" protocols 🧩 RFC 6410 [RFC6410] doesn't mention this special case, so this document augments the four criteria in Section 2.2 of RFC 6410 [RFC6410] for RFCs that specify performance metrics 💅 The position here is that flexible metric definitions can be proven clear and unambiguous through tests that compare results from independent implementations 🔬 It describes tests that infer whether metric specifications are sufficient, using a definition of metric "interoperability": measuring equivalent results (in a statistical sense) under the same network conditions 📊 The document goes deeper into this problem and its solution 🦋
for a protocol spec, "interoperability" is pretty intuitive. The implementations have to successfully "talk to each other" while exercising all features and options 🫶 To get there, two implementors need to interpret the protocol specs in equivalent ways 💜 For IP Performance Metrics (IPPM), though, this definition of interoperability is only useful for test and control protocols like the One-Way Active Measurement Protocol (OWAMP) [RFC4656] and the Two-Way Active Measurement Protocol (TWAMP) [RFC5357] 🌸
a metric specification RFC describes one or more metric definitions, methods of measurement, and a way to report the results of measurement 🎀 For example, you could have a way to test and report the one-way delay that data packets incur while traveling from one network location to another, using the One-Way Delay Metric 🚀
with metric specs, the conditions that satisfy the "interoperability" requirement are way less obvious 🙈 So there's a need for IETF agreement on practices to judge metric specification "interoperability" in the context of the IETF Standards Process 👑 This memo gives methods that should be suitable for evaluating metric specifications for Standards Track advancement 💖 The methods proposed here MAY be generally applicable to metric specification RFCs beyond those developed under the IPPM Framework [RFC2330] 🌈
here's the tea ☕ A lot of IP metric implementations live inside measurement systems that don't talk to each other, because they were built before OWAMP and TWAMP existed 💀 So the interoperability evaluation the IETF Standards Process asks for can't be figured out by watching independent implementations interact properly across various protocol exchanges 🙅♀️ Instead, checking that different implementations give statistically equivalent results under controlled measurement conditions takes the place of interoperability observations 🧪 Even when evaluating OWAMP and TWAMP RFCs for Standards Track advancement, the methods here are still useful for evaluating measurement results, because their validity wouldn't be confirmed by protocol interoperability testing 🔍
the Standards advancement process aims to build confidence that the metric definitions and supporting material are clearly worded and unambiguous, or to reveal ways the definitions can be revised to become clear 🧠 It also lets you spot options that were never implemented, so they can be removed from the spec that's advancing ✂️ So the product of this process is information about the metric specification RFC itself: which specs or definitions are clear and unambiguous, and which aren't 💅 (It's not an evaluation of the implementations that help out in the process 🫧)
this document defines a process for verifying that implementations (or, practically, measurement systems) have interpreted the metric specifications in equivalent ways and produce equivalent results 🎀
testing for statistical equivalence means making sure the test setups are identical, or at least that you're aware of any differences, to the best extent possible 🌷 So producing identical test conditions is a core goal of this memo 💜 Another important part of the process is testing individual implementations against specific requirements in the metric specs, using customized tests for each requirement 🧁 These tests can tell apart equivalent interpretations of each specific requirement 🔥
conclusions on equivalence come from two measures 📝
first, implementations are compared against individual metric specifications, so that differences in implementation are minimized or at least known 🔎
second, a test setup is proposed that ensures identical networking conditions, so unknowns are minimized and comparisons are simplified 🌈 The resulting separate data sets may be seen as samples taken from the same underlying distribution, and statistical methods are used to verify that the results are equivalent 📈 To illustrate how the process and methods work, Appendix A provides an evaluation of the One-Way Delay Metric [RFC2679] 🦋 Test setups will vary with the metrics being validated, but the general methodology for determining equivalent results won't 🫶 Documents defining test setups for other metrics should be developed once the process proposed here has been agreed and approved 💅
the metric RFC advancement process starts with a request for protocol action, accompanied by a memo that documents the supporting tests and results 📬 The procedures of [RFC2026] are expanded in [RFC5657], including sample implementation and interoperability reports 🧾 [TESTPLAN] can be a template for a metric RFC report that goes with the protocol action request to the Area Director 🗂️ It covers a description of the test setup, procedures, results for each implementation, and conclusions ✨
1.1. Requirements Language 🎀
okay so the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be read as described in RFC 2119 [RFC2119] 💖
the key words are the boss words of this doc, so read them carefully bestie 👑
2. Basic Idea ✨
okay so the big idea is pretty simple bestie 🧚 an implementation of a standards-compliant metric is expected to meet the requirements of the related metric specification. so before we compare two metric implementations, we check each one individually against the spec first 💅
most metric specs leave implementors some freedom on the non-fundamental stuff of an individual metric (or its options). and if you want to compare different measurement results with a statistical test, you're assuming the test path and conditions are identical, which means you need to know every difference in the overall test setup 👀 so the metric spec options that implementors choose have to be documented. it is RECOMMENDED to use identical metric options for any test proposed here (the exception is if a variable parameter of the metric definition isn't configurable in one or more implementations) 🌸 and calibrations specified by metric standards SHOULD be performed to further identify (and possibly reduce) potential sources of error in the test setup ✨
the IPPM Framework [RFC2330] wants a "methodology for a metric should have the property that it is repeatable: if the methodology is used multiple times under identical conditions, it should result in consistent measurements". so an implementation is expected to repeatedly measure a metric with consistent results (repeatability with the same result) 🦋 small deviations in the test setup are expected to lead to small deviations in results only. to characterize statistical equivalence when deviations are small, [RFC2330] and [RFC2679] suggest applying a 95% confidence interval. quoting RFC 2679, "95 percent was chosen because ... a particular confidence level should be specified so that the results of independent implementations can be compared" 🫶
two different implementations are expected to produce statistically equivalent results if they both measure a metric under the same networking conditions. in stats language: separate metric implementations collect separate samples from the same underlying statistical process (the same network conditions). the hypothesis being tested is the expectation that both samples do not expose statistically different properties. this needs careful test design, so here's the checklist 📝
- the measurement test setup must be self-consistent to the largest possible extent. to minimize the influence of the test and measurement setup on the result, network conditions and paths MUST be identical for the compared implementations to the largest possible degree. this includes both the stability and non-ambiguity of routes taken by the measurement packets. see [RFC2330] for a discussion on self-consistency 🧚
- to minimize the influence of implementation options on the result, metric implementations SHOULD use identical options and parameters for the metric under evaluation 💜
- the sample size must be large enough to minimize its influence on the consistency of the test results. this consideration may be especially important if two implementations measure with different average packet transmission rates 🍓
- the implementation with the lowest average packet transmission rate determines the smallest temporal interval for which samples can be compared ⏱️
- repeat comparisons with several independent metric samples to avoid random indications of compatibility (or the lack of it) 🌈
the metric specifications themselves are the main focus of evaluation, not the implementations of metrics 👑 the documentation produced by the advancement process should identify which metric definitions and supporting material were found to be clearly worded and unambiguous, OR it should identify ways in which the metric specification text should be revised to achieve clarity and unified interpretation 💖
the process should also let us spot options that were never implemented, so they can be removed from the advancing specification (this is more of a protocol advancement along the Standards Track thing) 🎀
and lowkey important: this document does NOT propose basing interoperability indications of performance-metric implementations on comparisons of individual singletons 🙈 individual singletons can get hit by lots of statistical effects while they're being measured, and comparing two singletons from different implementations may result in failures with higher probability than comparing samples. so no singleton showdowns, periodt 💅
3. Verification of Conformance to a Metric Specification 🎀
this section explains how to verify compliance of two or more IPPM implementations against a metric specification 🦋 this document only proposes a general methodology. compliance criteria for a specific metric implementation need to be defined for each individual metric specification. the only exception is the statistical test comparing two metric implementations that are simultaneously tested, and that test works without metric-specific decision criteria ✨
there are several testing options for comparing two or more implementations:
- use a single test lab to compare the implementations and emulate the Internet with an impairment generator 🧪
- use a single test lab to compare the implementations and measure across the Internet 🌐
- use remotely separated test labs to compare the implementations and emulate the Internet with two "identically" configured impairment generators 👯
- use remotely separated test labs to compare the implementations and measure across the Internet 🛰️
- use remotely separated test labs to compare the implementations, measure across the Internet, and include a single impairment generator to impact all measurement flows in a non-discriminatory way 💅
the first two approaches work, but they cost more than the others (due to travel and/or shipping plus installation) 💸 for the third option, making sure two identically configured impairment generators exist requires well-defined test cases and possibly identical hardware and software 🛠️
as documented in a test report [TESTPLAN], the last option was required to prove compatibility of two delay metric implementations 🏆 an impairment generator is probably required when testing compatibility of most other metrics, so it is therefore RECOMMENDED to include an impairment generator in metric test setups 💜
3.1. Tests of an Individual Implementation against a Metric Specification
okay this is the part where we check ONE implementation against the spec 🔍 a metric implementation is compliant with a metric specification if it supports the requirements classified as "MUST" and "REQUIRED" in the related metric specification. an implementation that implements all requirements is fully compliant with the specification, and the degree of compliance SHOULD be noted in the conclusions of the report 📋
also, supported options of a metric implementation SHOULD be documented in enough detail to evaluate whether the specification was correctly interpreted 🧐 documenting the chosen options should minimize (and recognize) differences in the test setup if two metric implementations get compared. this documentation is also used to validate or clarify the wording of the metric specification option, and to remove options that saw no implementation or that are badly specified from the metric specification 🗑️ this documentation SHOULD be included for all implementation-relevant specifications of a metric picked for a comparison, even the ones not explicitly marked as "MUST" or "REQUIRED" in the RFC text. it applies to these sections of all metric specifications:
- Singleton Definition of the Metric. 🎯
- Sample Definition of the Metric. 🧩
- Statistics Definition of the Metric. As statistics are compared by the test specified here, this documentation is required even if the metric specification has no Statistics Definition 📊
- Timing- and Synchronization-related specification (if relevant for the Metric). ⏰
- Any other technical part present or missing in the metric specification, which is relevant for implementing the Metric. 🔧
[RFC2330] and [RFC2679] both push precision as a goal for IPPM metric implementations ✨ a single IPPM-conforming implementation should, under otherwise identical network conditions, produce precise results for repeated measurements of the same metric 💖
RFC 2330 prefers the "empirical distribution function" (EDF) to describe collections of measurements 📈 RFC 2330 determines, that "unless otherwise stated, IPPM goodness-of-fit tests are done using 5% significance". the goodness-of-fit test figures out how precisely two or more samples from a metric implementation belong to the same underlying distribution (of measured network performance events) 🔬 the goodness-of-fit test suggested for the metric test is the Anderson-Darling K sample test (ADK sample test, K stands for the number of samples to be compared) [ADK]. heads up: RFC 2330 and RFC 2679 apply an Anderson-Darling goodness-of-fit test too 🙌
the results of a repeated test with a single implementation MUST pass an ADK sample test with a confidence level of 95%. the conditions under which the ADK test passed with the specified confidence level MUST be documented. said another way, the requirement is to document the set of parameters with the smallest deviation at which the results of the tested metric implementation pass an ADK test with a confidence level of 95% 🎯 the minimum resolution available in the reported results from each implementation MUST be taken into account in the ADK test ⚖️
the test conditions to document for a passed metric test include:
- The metric resolution at which a test was passed (e.g., the resolution of timestamps). ⏱️
- The parameters modified by an impairment generator. 🎛️
- The impairment generator parameter settings. 🧾
3.2. Test Setup Resulting in Identical Live Network Testing Conditions
okay so testing for metric compliance across LIVE networks under identical conditions is tricky, bestie 😭 two major issues make it hard. one is the general point that metric definition implementations cannot be conveniently examined in field measurement scenarios. the other is broadly described as "parallelism in devices and networks", which includes mechanisms like load balancing (see [RFC4928]) 🌀
this section proposes two measures to deal with both issues ✨ tunneling mechanisms can be used to avoid parallel processing of different flows in the network. measuring with separate parallel probe flows results in repeated collection of data 🧵 if you combine both measures, Wide Area Network (WAN) conditions are identical for a number of independent measurement flows, no matter what the network conditions are in detail 🦋
any measurement setup must be made so the probing traffic itself doesn't impede the metric measurement 🙅♀️ the created measurement load must not result in congestion at the access link connecting the measurement implementation to the WAN. the created measurement load must not overload the measurement implementation itself, e.g., by causing a high CPU load or by causing timestamp imprecision due to unwanted queuing while transmitting or receiving test packets 🔥
Okay so when you tunnel a bunch of flows headed for ONE physical port on a network element, every packet takes the same path 🦋 That's the whole vibe. Using tunnels to dodge the sneaky influence of normal routing during measurements is a known move in the literature, see e.g., GRE-encapsulated multicast probing [GU-Duffield] 🧚 An existing IP-in-IP tunnel protocol can be applied to stop Equal-Cost Multi-Path (ECMP) routing from splitting up different measurement streams, if it checks these boxes ✨
- Inner IP packets from different measurement implementations get mapped into a single tunnel with a single outer IP origin and destination address, plus origin and destination port numbers that are identical for all packets. 💅
- An easily accessible tunneling protocol lets you run a metric test from more test sites 🌸
- Low operational overhead may enable a broader audience to set up a metric test with the desired properties 🎀
- The tunneling protocol should be reliable and stable in setup and operation so it doesn't cause disturbances or influence the test results 🫶
- The tunneling protocol should not incur any extra cost for those interested in setting up a metric test 💖
Here's a test setup with two layer 2 tunnels and two flows between two linecards of one implementation, and it's giving very clean energy 👑
Implementation ,---. +--------+
+~~~~~~~~~~~/ \~~~~~~| Remote |
+------->-----F2->-| / \ |->---+ |
| +---------+ | Tunnel 1( ) | | |
| | transmit|-F1->-| ( ) |->+ | |
| | LC1 | +~~~~~~~~~| |~~~~| | | |
| | receive |-<--+ ( ) | F1 F2 |
| +---------+ | |Internet | | | | |
*-------<-----+ F2 | | | | | |
+---------+ | | +~~~~~~~~~| |~~~~| | | |
| transmit|-* *-| | | |--+<-* |
| LC2 | | Tunnel 2( ) | | |
| receive |-<-F1-| \ / |<-* |
+---------+ +~~~~~~~~~~~\ /~~~~~~| Router |
`-+-' +--------+For simplicity, only two linecards of one implementation and two flows F between them are shown.
Figure 1: Illustration of a Test Setup with Two Layer 2 Tunnels
Figure 2 shows the network elements you need to set up layer 2 tunnels like the ones in Figure 1 🧸
Implementation
+-----+ ,---.
| LC1 | / \
+-----+ / \ +------+
| +-------+ ( ) +-------+ |Remote|
+--------+ | | | | | | | |
|Ethernet| | Tunnel| |Internet | | Tunnel| | |
|Switch |--| Head |--| |--| Head |--| |
+--------+ | Router| | | | Router| | |
| | | ( ) | | |Router|
+-----+ +-------+ \ / +-------+ +------+
| LC2 | \ /
+-----+ `-+-'Figure 2: Illustration of a Hardware Setup to Realize the Test Setup Illustrated by Figure 1 with Layer 2 Tunnels or Pseudowires
The test setup that was successfully used during a delay metric test [TESTPLAN] is shown in Figure 3 🌈 Note that this setup allows a metric test between two remote sites 🦋
+----+ +----+ +----+ +----+
|LC10| |LC11| ,---. |LC20| |LC21|
+----+ +----+ / \ +-------+ +----+ +----+
| V10 | V11 / \ | Tunnel| | V20 | V21
| | ( ) | Head | | |
+--------+ +------+ | | | Router|__+----------+
|Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet |
|Switch |--|Head |-| | | |Switch |
+-+--+---+ |Router| | | +---+---+ +--+--+----+
|__| +--A---+ ( )--|Option.| |__|
\ / |Impair.|
Bridge \ / |Gener. | Bridge
V20 to V21 `-+-? +-------+ V10 to V11Figure 3: Example of Test Setup Successfully Used during a Delay Metic Test
In Figure 3, LC10 identifies measurement clients / linecards. V10 and the others denote VLANs. All VLANs use the same tunnel from A to B and in the reverse direction. The remote site VLANs are U-bridged at the local site Ethernet switch. The measurement packets of site 1 travel tunnel A->B first, get U-bridged at site 2, and then travel tunnel B->A second. Measurement packets of site 2 travel tunnel B->A first, get U-bridged at site 1, and then travel tunnel A->B second. So all measurement packets pass the same tunnel segments, just in a different segment order 🔥
If tunneling is applied, two tunnels MUST carry all test traffic between the test site and the remote site 🎀 For example, if 802.1Q Virtual LANs (VLANs) are used and the measurement streams are carried in different VLANs, the IP tunnel or pseudowires are set up in physical port mode, so you don't end up with a pseudowire per VLAN (which may see different paths due to ECMP routing), see [RFC4448] 💜 The remote router and the Ethernet switch shown in Figure 3 have to support 802.1Q in this setup.
The IP packet size of the metric implementation SHOULD be chosen small enough to avoid fragmentation due to the added Ethernet and tunnel headers 🍓 Otherwise, the impact of tunnel overhead on fragmentation and interface MTU size must be understood and taken into account (see [RFC4459]) 🙈
An Ethernet port mode IP tunnel carrying several 802.1Q VLANs, each containing measurement traffic from a single measurement system, was successfully applied when testing compatibility of two metric implementations [TESTPLAN] 🦋 Ethernet over Layer 2 Tunneling Protocol Version 3 (L2TPv3) [RFC4719] was picked for this test ✨
If VLANs and Ethernet over L2TPv3 tunnels are applied, these headers may have to be counted when you calculate total packet length 🧮
- Ethernet 802.1Q: 22 bytes. 💖
- L2TPv3 Header: 4-16 bytes for L2TPv3 data messages over IP; 16-28 bytes for L2TPv3 data messages over UDP. 🌸
- IPv4 Header (outer IP header): 20 bytes. 🌷
- MPLS Labels may be added by a carrier. Each MPLS Label has a length of 4 bytes. At the time of this writing, between 1 and 4 Labels seems to be a fair guess of what's expected. 🫣
If Ethernet over L2TPv3 feels unsuitable, interested parties may look into one or more of these tunneling protocols: IP in IP [RFC2003] or Generic Routing Encapsulation (GRE) [RFC2784]. RFC 4928 [RFC4928] proposes measures for avoiding ECMP treatment in MPLS networks 🧠
L2TP is a commodity tunneling protocol [RFC2661] 🍼 At the time of this writing, L2TPv3 [RFC3931] is the latest version of L2TP. If you go with L2TPv3, software-based implementations of this protocol are not suitable for the test setup, because they may cause incalculable delay shifts 😭
Ethernet pseudowires may also be set up on MPLS networks [RFC4448]. There's no technical issue with this solution, but MPLS interfaces are mostly found in the network provider domain, so not all of the above criteria for picking a tunneling protocol are met 🥺
Note that setting up a metric test environment is not a plug-and-play situation 💀 Skilled networking engineers should be consulted and involved if you want a setup between remote sites 🫶
Passing or failing an ADK test with 2 samples could just be a random result! (Note that [RFC2330] defines a sample as a set of singleton metric values produced by a measurement stream, and we keep using that terminology here 🤓) The error margin of a statistical test is higher when the number of samples it's based on is low. The number of samples taken affects the so-called "degree of freedom" of a statistical test, and a higher degree of freedom gives more reliable results 📈 To pass an ADK with higher probability, the number of samples collected per implementation under identical networking conditions SHOULD be greater than 2. Hardware and load constraints may put an upper limit on the number of simultaneous measurement streams 😩 The ADK test lets you combine different samples (see Section 9 of [ADK]) and then run a 2-sample test between the combined samples. At least 4 samples per implementation captured under identical networking conditions is RECOMMENDED when comparing different metric implementations with a statistical test 🏆
It is RECOMMENDED that tests be carried out by establishing N different parallel measurement flows 🧚♀️ Two or three linecards per implementation that send or receive measurement flows should be sufficient to create 4 or more parallel measurement flows. Other options are to separate flows by DiffServ marks (without deploying any Quality of Service (QoS) in the inner or outer tunnel), or to use a single Constant Bitrate (CBR) flow and evaluate whether every n-th singleton belongs to a specific measurement flow 🦄 Also, a practical test showed that ADK passed with 4 samples even when a 2-sample test failed [TESTPLAN] 🎉
Some extra guidelines for calculating and comparing samples in a metric test are:
- Comparing different probes of a common underlying distribution in terms of metrics that characterize a communication network requires respecting the temporal nature, since the assumption of a common underlying distribution may hold only over time. Any singletons or samples to be compared must be captured within the same time interval ⏰
- If statistical events like rates are used to characterize measured metrics of a time interval, a minimum of 5 singletons of a relevant metric should be picked to ensure a minimum confidence in the reported value. The error margin of the determined rate depends on the number of singletons (refer to statistical textbooks on student's t-test). For example, any packet loss measurement interval to be compared with another implementation's results should contain at least five lost packets, so you can be confident the observed loss rate wasn't caused by a small number of random packet drops 📉
- The minimum number of singletons or samples to be compared by an Anderson-Darling test should be 100 per tested metric implementation. Note that the Anderson-Darling test detects small differences in distributions fairly well, and it will fail for a high number of compared results (RFC 2330 mentions an example with 8192 measurements where an Anderson-Darling test always failed) 😭🔥
Okay so the Anderson-Darling test is super sensitive to accuracy and bias differences between implementations or test conditions 🔬 It's giving "small things make big waves." Different packet sizes can add a constant delay gap between samples, which can make the averages look different even when nothing is actually wrong. So samples MAY be calibrated by the difference of their average values 💖 But bestie, any calibration like that MUST be documented in the test result 📝
3.3. Tests of Two or More Different Implementations against a Metric Specification
[RFC2330] expects that "a methodology for a given metric exhibits continuity if, for small variations in conditions, it results in small variations in the resulting measurements. Slightly more precisely, for every positive epsilon, there exists a positive delta, such that if two sets of conditions are within delta of each other, then the resulting measurements will be within epsilon of each other". 🧠✨ In this metric test, a small variation in conditions can be seen as different implementations measuring the same metric along the same path 🛤️
IPPM metric specifications, however, allow implementor options to the largest possible degree 🎛️ Two implementors can't be expected to pick 100% identical options, so it's lowkey impossible. Testers SHOULD pick the same metric measurement configurations for their systems when comparing their implementations by a metric test 🫶
Sometimes a goodness-of-fit test may not be possible or shows disappointing results 😭 To figure out the difficulties from different metric implementation options, the individual options picked for every compared metric implementation should be documented as specified in Section 3.5 📋 If the failure comes from a lack of specification clarity or multiple legitimate interpretations of the definition text, the text should be modified and the resulting memo proposed for consensus and (possible) advancement to Internet Standard 👑
The same statistical test used to quantify precision of a single metric implementation must be used to compare metric result equivalence for different implementations 🔍 To document compatibility, the smallest measurement resolution at which the compared implementations passed the ADK sample test must be documented 🎀
For different implementations of the same metric, "variations in conditions" are reasonably expected 🌈 The ADK test comparing samples of different implementations may end up with a lower precision than the test for precision in the same-implementation comparison. Not me being like "that's normal, don't panic" 💅
3.4. Clock Synchronization
Clock synchronization effects need special attention, bestie 🕰️ The accuracy of one-way active delay measurements for any metric implementation depends on clock synchronization between the source and destination of tests. Ideally, one-way active delay measurement [RFC2679] test endpoints either have direct access to independent GPS or CDMA-based time sources, or indirect access to nearby NTP primary (stratum 1) time sources equipped with GPS receivers 🛰️ Access to these time sources may not be available at all test locations on different Internet paths, for a variety of reasons that are out of scope of this document 🙈
When secondary (stratum 2 and above) time sources are used with NTP running across the same network whose metrics are subject to comparative implementation tests, network impairments can affect clock synchronization and distort sample one-way values and their interval statistics 😬 Discarding sample one-way delay values for any implementation is recommended when one of the following reliability conditions is met:
- Delay is measured and is finite in one direction but not the other.
- Absolute value of the difference between the sum of one-way measurements in both directions and the round-trip measurement is greater than X% of the latter value.
Checking the second condition needs round-trip time (RTT) measurement for reference, e.g., based on TWAMP [RFC5357] in conjunction with one-way delay measurement 🔁
Specifying X% is a balancing act 🤹 It has to identify unreliable one-way delay samples without misidentifying reliable samples across a wide range of Internet path RTTs, and that requires further study 📚
An IPPM-compliant metric implementation of an RFC that requires synchronized clocks is expected to provide precise measurement results ✨
IF an implementation publishes a specification of its precision, such as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then the specification should be met over a useful measurement duration 💯 For example, if the metric is measured along an Internet path that is stable and not congested, the precision specification should be met over durations of an hour or more 🌸
3.5. Recommended Metric Verification Measurement Process 🔍✨
Okay so, to meet their obligations under the IETF Standards Process, the IESG has to be convinced of three things 👑: each metric specification advanced to Internet Standard status is clearly written, there are enough verified equivalent implementations, and any options that got implemented are documented.
Metrics here are designed to measure some characteristic of a data network. The goal of any metric definition should be to specify it in a way that can reliably measure that exact characteristic in a repeatable way across multiple independent implementations 🎯.
Each metric, statistic, or option being validated MUST be compared against a reference measurement or another implementation, as this document specifies 💅.
Finally, the metric definitions in the text of the RFCs are the things that need evaluating and possibly revising before they can advance to Internet Standard 🧐.
IF two (or more) implementations do not measure an equivalent metric as this document specifies,
AND sources of measurement error do not adequately explain the lack of agreement,
THEN the details of each implementation should be audited, along with the exact definition text, to determine whether there is a lack of clarity that caused the implementations to vary in a way that affects how well the results correspond 🕵️♀️.
IF there was a lack of clarity or multiple legitimate interpretations of the definition text, THEN the text should be modified, and the resulting memo proposed for consensus and (possible) advancement along the Standards Track 📝.
Finally, all the findings MUST be documented in a report that can support advancement to Internet Standard, as described here (similar to the reports described in [RFC5657]) 📋. The list of measurement devices used in testing satisfies the implementation requirement, while the test results show how good each specification in the metric RFC is (the surrogate for feature interoperability) 🦋.
The whole process of advancing a metric specification to a Standard, as defined by this document, is illustrated in Figure 4 🧚.
,---.
/ \
( Start )
\ / Implementations
`-+-' +-------+
| /| 1 `.
+---+----+ / +-------+ `.-----------+ ,-------.
| RFC | / |Check for | ,' was RFC `. YES
| | / |Equivalence.... clause x ------+
| |/ +-------+ |under | `. clear? ,' |
| Metric \.....| 2 ....relevant | `---+---' +----+-----+
| Metric |\ +-------+ |identical | No | |Report |
| Metric | \ |network | +--+----+ |results + |
| ... | \ |conditions | |Modify | |Advance |
| | \ +-------+ | | |Spec +--+RFC |
+--------+ \| n |.'+-----------+ +-------+ |request |
+-------+ +----------+Figure 4: Illustration of the Metric Standardization Process
Any recommendation to advance a metric specification MUST be accompanied by an implementation report 📑. The implementation report needs to include the tests performed, the applied test setup, the specific metrics in the RFC, and reports of the tests performed with two or more implementations. The test plan needs to specify the precision reached for each measured metric, and so define what "statistically equivalent" means for the specific metrics being tested 🧮.
Ideally, the test plan would co-evolve with the development of the metric, since that's when participants have the clearest context in their minds about the different subtleties that can come up 🧠💖.
In particular, the implementation report MUST include the following at minimum:
- The metric compared and the RFC specifying it. This includes statements as required by Section 3.1 ("Tests of an Individual Implementation against a Metric Specification") of this document.
- The measurement configuration and setup.
- A complete specification of the measurement stream (mean rate, statistical distribution of packets, packet size or mean packet size, and their distribution), Differentiated Services Code Point (DSCP), and any other measurement stream properties that could result in deviating results. Deviations in results can also be caused if the chosen IP addresses and ports of different implementations result in different layer 2 or layer 3 paths due to operation of Equal Cost Multi-Path routing in an operational network 🛣️.
- The duration of each measurement used for a metric validation, the number of measurement points collected for each metric during each measurement interval (i.e., the probe size), and the level of confidence derived from this probe size for each measurement interval.
- The result of the statistical tests performed for each metric validation, as required by Section 3.3 ("Tests of Two or More Different Implementations against a Metric Specification") of this document.
- A parameterization of laboratory conditions and applied traffic and network conditions, allowing readers of the implementation report to reproduce these laboratory conditions.
- The documentation helping to improve metric specifications defined by this section 🌸.
All of the tests for each set SHOULD be run in a test setup as specified in Section 3.2 ("Test Setup Resulting in Identical Live Network Testing Conditions" 🎀.
If a different test setup is chosen, it is recommended to avoid effects that falsify validation results because of real data networks (like parallelism in devices and networks). Data networks may forward packets differently in the case of:
- Different packet sizes chosen for different metric implementations. A proposed countermeasure is selecting the same packet size when validating results of two samples, or a sample against an original distribution.
- Selection of differing IP addresses and ports used by different metric implementations during metric validation tests. If ECMP is applied at the IP or MPLS level, different paths can result (note that it may be impossible to detect an MPLS ECMP path from an IP endpoint 🙈). A proposed countermeasure is to connect the measurement equipment being compared through a NAT device, or to establish a single tunnel to transport all measurement traffic. The aim is to have the same IP addresses and port for all measurement packets, or to avoid ECMP-based local routing diversion by using a layer 2 tunnel.
- Different IP options.
- Different DSCP.
- If the N measurements are captured using sequential measurements instead of simultaneous ones, then the following factors come into play: time-varying paths and load conditions ⏳.
3.6. Proposal to Determine an Equivalence Threshold for Each Metric Evaluated 🦋
This section describes a proposal for maximum error of equivalence, based on performance comparison of identical implementations. This comparison may be useful for both ADK and non-ADK comparisons 🔥.
Each metric is tested by two or more implementations (cross-implementation testing) 👯♀️.
Each metric is also tested twice, simultaneously, by the same implementation, using different Src/Dst Address pairs and other differences, such that the connectivity differences of the cross-implementation tests are also experienced and measured by the same implementation 🪞.
Comparative results for the same implementation represent a bound on cross-implementation equivalence. This should be particularly useful when the metric does not produce a continuous distribution of singleton values, such as with a loss metric or a duplication metric. Appendix A indicates how the ADK will work for one-way delay and should be likewise applicable to distributions of delay variation 📈.
Appendix B discusses two possible ways to perform the ADK analysis: the R statistical language [Rtool] with the ADK package [Radk], and C++ code 💻.
Conclusion: the implementation with the largest difference in homogeneous comparison results is the lower bound on the equivalence threshold, noting that there may be other systematic errors to account for when comparing implementations 🧁.
Thus, when evaluating equivalence in cross-implementation results:
Maximum_Error = Same_Implementation_Error + Systematic_Errorand only the systematic error needs to be decided beforehand ✨.
In the case of ADK comparison, the largest same-implementation resolution of distribution equivalence can be used as a limit on cross-implementation resolutions (at the same confidence level) 💜.
7.1. Normative References
okay bestie, these are the ones the spec leans on to work at all 💅 I'm keeping every single detail, titles and all, so you can go look up any of them 🎀
[RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996. 🌸
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 🔥
[RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. 🦋
[RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", RFC 2661, August 1999. 🧚
[RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way Delay Metric for IPPM", RFC 2679, September 1999. ⭐
[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, March 2000. 💜
[RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 🌈
[RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, "Encapsulation Methods for Transport of Ethernet over MPLS Networks", RFC 4448, April 2006. 🍓
[RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, September 2006. ✨
[RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport of Ethernet Frames over Layer 2 Tunneling Protocol Version 3 (L2TPv3)", RFC 4719, November 2006. 🎀
[RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal Cost Multipath Treatment in MPLS Networks", BCP 128, RFC 4928, June 2007. 🌸
[RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard", BCP 9, RFC 5657, September 2009. 👑
[RFC6410] Housley, R., Crocker, D., and E. Burger, "Reducing the Standards Track to Two Maturity Levels", BCP 9, RFC 6410, October 2011. 🫶
7.2. Informative References
these ones are the extra reading, the stuff that's helpful to know but not part of the core deal 🦋
[ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling Tests of Fit, for Continuous and Discrete Cases", University of Washington, Technical Report No. 81, May 1986. 📚
[GU-Duffield] Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE Encapsulated Multicast Probing: A Scalable Technique for Measuring One-Way Loss", SIGMETRICS'07 San Diego, California, USA, June 2007. 🌈
[METRICTEST] Bradner, S. and V. Paxson, "Advancement of metrics specifications on the IETF Standards Track", Work in Progress, August 2007. 🚧
[RFC2026] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. 💅
[RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the-Network Tunneling", RFC 4459, April 2006. 🍓
[RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", RFC 5357, October 2008. 🧚
[Radk] Scholz, F., "adk: Anderson-Darling K-Sample Test and Combinations of Such Tests. R package version 1.0", 2008. 💜
[Rtool] R Development Core Team, "R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0", 2011, <http://www.R-project.org/>. 📊
[TESTPLAN] Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test Plan and Results for Advancing RFC 2679 on the Standards Track", Work in Progress, March 2012. 🌸
Appendix A. An Example on a One-Way Delay Metric Validation ✨
The text of this appendix is not binding. It is an example of what parts of a One-Way Delay Metric test could look like. 🎀
A.1. Compliance to Metric Specification Requirements
okay this part is the test checklist, and it's giving very detailed lab-coat energy 🥼💖
One-Way Delay, Loss Threshold, RFC 2679
This test determines if implementations use the same configured maximum waiting time delay from one measurement to another under different delay conditions and correctly declare packets arriving in excess of the waiting time threshold as lost. See Sections 3.5 (3rd bullet point) and 3.8.2 of [RFC2679].
- Configure a path with 1-second one-way constant delay.
- Measure one-way delay with 2 or more implementations, using identical waiting time thresholds for loss set at 2 seconds.
- Configure the path with 3-second one-way delay.
- Repeat measurements.
- Observe that the increase measured in step 4 caused all packets to be declared lost and that all packets that arrive successfully in step 2 are assigned a valid one-way delay.
One-Way Delay, First Bit to Last Bit, RFC 2679
This test determines if implementations register the same relative increase in delay from one measurement to another under different delay conditions. This test tends to cancel the sources of error that may be present in an implementation. See Section 3.7.2 of [RFC2679] and Section 10.2 of [RFC2330]. 🦋
- Configure a path with X ms one-way constant delay and ideally include a low-speed link.
- Measure one-way delay with 2 or more implementations, using identical options and equal size small packets (e.g., 100 octet IP payload).
- Maintain the same path with X ms one-way delay.
- Measure one-way delay with 2 or more implementations, using identical options and equal size large packets (e.g., 1500 octet IP payload).
- Observe that the increase measured in steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.
One-Way Delay, RFC 2679
This test determines if implementations register the same relative increase in delay from one measurement to another under different delay conditions. This test tends to cancel the sources of error that may be present in an implementation. This test is intended to evaluate measurements in Sections 3 and 4 of [RFC2679]. 🌈
- Configure a path with X ms one-way constant delay.
- Measure one-way delay with 2 or more implementations, using identical options.
- Configure the path with X+Y ms one-way delay.
- Repeat measurements.
- Observe that the increase measured in steps 2 and 4 is ~Y ms for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.
Error Calibration, RFC 2679
This is a simple check to determine if an implementation reports the error calibration as required in Section 4.8 of [RFC2679]. Note that the context (Type-P) must also be reported. 🫶
A.2. Examples Related to Statistical Tests for One-Way Delay
this is where the stats get spicy, but I'll walk you through it 🔥📊
A one-way delay measurement may pass an ADK test with a timestamp result of 1 ms. The same test may fail if timestamps with a resolution of 100 microseconds are evaluated. The implementation is then conforming to the metric specification up to a timestamp resolution of 1 ms. ⏱️
Let's assume another one-way delay measurement comparison between implementation 1 probing with a frequency of 2 probes per second and implementation 2 probing at a rate of 2 probes every 3 minutes. To ensure reasonable confidence in results, sample metrics are calculated from at least 5 singletons per compared time interval. This means that sample delay values are calculated for each system for identical 6-minute intervals for the duration of the whole test. 💖
Per 6-minute interval, the sample metric is calculated from 720 singletons for implementation 1 and from 6 singletons for implementation 2. Note that if outliers are not filtered, moving averages are an option for an evaluation too. The minimum move of an averaging interval is three minutes in this example. 🧚
The data in Table 1 may result from measuring one-way delay with implementation 1 (see column Implemnt_1) and implementation 2 (see column Implemnt_2). Each data point in the table represents a (rounded) average of the sampled delay values per interval. The resolution of the clock is one micro-second. The difference in the delay values may result, e.g., from different probe packet sizes. 🍓
+------------+------------+-----------------------------+
| Implemnt_1 | Implemnt_2 | Implemnt_2 - Delta_Averages |
+------------+------------+-----------------------------+
| 5000 | 6549 | 4997 |
| 5008 | 6555 | 5003 |
| 5012 | 6564 | 5012 |
| 5015 | 6565 | 5013 |
| 5019 | 6568 | 5016 |
| 5022 | 6570 | 5018 |
| 5024 | 6573 | 5021 |
| 5026 | 6575 | 5023 |
| 5027 | 6577 | 5025 |
| 5029 | 6580 | 5028 |
| 5030 | 6585 | 5033 |
| 5032 | 6586 | 5034 |
| 5034 | 6587 | 5035 |
| 5036 | 6588 | 5036 |
| 5038 | 6589 | 5037 |
| 5039 | 6591 | 5039 |
| 5041 | 6592 | 5040 |
| 5043 | 6599 | 5047 |
| 5046 | 6606 | 5054 |
| 5054 | 6612 | 5060 |
+------------+------------+-----------------------------+
Table 1Average values of sample metrics captured during identical time intervals are compared. This excludes random differences caused by differing probing intervals or differing temporal distance of singletons resulting from their Poisson-distributed sending times. 📈
In the example, 20 values have been picked (note that at least 100 values are recommended for a single run of a real test). Data must be ordered by ascending rank. The data of Implemnt_1 and Implemnt_2 as shown in the first two columns of Table 1 clearly fails an ADK test with 95% confidence. 😭
The results of Implemnt_2 are now reduced by the difference of the averages of column 2 (rounded to 6581 us) and column 1 (rounded to 5029 us), which is 1552 us. The result may be found in column 3 of Table 1. Comparing column 1 and column 3 of the table by an ADK test shows that the data contained in these columns passes an ADK test with 95% confidence. ✨
Comment: Extensive averaging was used in this example because of the vastly different sampling frequencies. As a result, the distributions compared do not exactly align with a metric in [RFC2679] but illustrate the ADK process adequately. 🫶
The R tool is a language and command-line environment for statistical computing and plotting [Rtool] 📊. With the optional "adk" package installed [Radk], it can run individual and combined sample ADK computations 🦋. You'll still need to check the package docs and the original paper [ADK] to interpret the results, and honestly? That's how it should be 💅
The C++ code below does an AD2-sample comparison once it's compiled and given two column vectors in a file, with white space separating the values 🍓. This version has modifications Wes Eddy made in Sept 2011 so it can use the vectors and run as a stand-alone module ✨. You can check the result on the command line with $ echo $?, or swap the last line for a printf of adk_result if you'd rather 💖
/*
Copyright (c) 2012 IETF Trust and the persons identified
as authors of the code. All rights reserved.
Redistribution and use in source and binary forms, with
or without modification, is permitted pursuant to, and subject
to the license terms contained in, the Simplified BSD License
set forth in Section 4.c of the IETF Trust's Legal Provisions
Relating to IETF Documents (http://trustee.ietf.org/license-info).
*/
/* Routines for computing the Anderson-Darling 2 sample
* test statistic.
*
* Implemented based on the description in
* "Anderson-Darling K Sample Test" Heckert, Alan and
* Filliben, James, editors, Dataplot Reference Manual,
* Chapter 15 Auxiliary, NIST, 2004.
* Official Reference by 2010
* Heckert, N. A. (2001). Dataplot website at the
* National Institute of Standards and Technology:
* http://www.itl.nist.gov/div898/software/dataplot.html/
* June 2001.
*/
#include <iostream>
#include <fstream>
#include <vector>
#include <sstream>
using namespace std;
int main() {
vector<double> vec1, vec2;
double adk_result;
static int k, val_st_z_samp1, val_st_z_samp2,
val_eq_z_samp1, val_eq_z_samp2,
j, n_total, n_sample1, n_sample2, L,
max_number_samples, line, maxnumber_z;
static int column_1, column_2;
static double adk, n_value, z, sum_adk_samp1,
sum_adk_samp2, z_aux;
static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux;
static bool next_z_sample2, equal_z_both_samples;
static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2,
old_eq_line1;
static double adk_criterium = 1.993;
/* vec1 and vec2 to be initialized with sample 1 and
* sample 2 values in ascending order */
while (!cin.eof()) {
double f1, f2;
cin >> f1;
cin >> f2;
vec1.push_back(f1);
vec2.push_back(f2);
}
k = 2;
n_sample1 = vec1.size() - 1;
n_sample2 = vec2.size() - 1;
// -1 because vec[0] is a dummy value
n_total = n_sample1 + n_sample2;
/* value equal to the line with a value = zj in sample 1.
* Here j=1, so the line is 1.
*/
val_eq_z_samp1 = 1;
/* value equal to the line with a value = zj in sample 2.
* Here j=1, so the line is 1.
*/
val_eq_z_samp2 = 1;
/* value equal to the last line with a value < zj
* in sample 1. Here j=1, so the line is 0.
*/
val_st_z_samp1 = 0;
/* value equal to the last line with a value < zj
* in sample 1. Here j=1, so the line is 0.
*/
val_st_z_samp2 = 0;
sum_adk_samp1 = 0;
sum_adk_samp2 = 0;
j = 1;
// as mentioned above, j=1
equal_z_both_samples = false;
next_z_sample2 = false;
//assuming the next z to be of sample 1
stop_loop1 = n_sample1 + 1;
// + 1 because vec[0] is a dummy, see n_sample1 declaration
stop_loop2 = n_sample2 + 1;
stop_loop3 = n_total + 1;
/* The required z values are calculated until all values
* of both samples have been taken into account. See the
* lines above for the stoploop values. Construct required
* to avoid a mathematical operation in the while condition.
*/
while (((stop_loop1 > val_eq_z_samp1)
|| (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j)
{
if(val_eq_z_samp1 < n_sample1+1)
{
/* here, a preliminary zj value is set.
* See below how to calculate the actual zj.
*/
z = vec1[val_eq_z_samp1];
/* this while sequence calculates the number of values
* equal to z.
*/
while ((val_eq_z_samp1+1 < n_sample1)
&& z == vec1[val_eq_z_samp1+1] )
{
val_eq_z_samp1++;
}
}
else
{
val_eq_z_samp1 = 0;
val_st_z_samp1 = n_sample1;
// this should be val_eq_z_samp1 - 1 = n_sample1
}
if(val_eq_z_samp2 < n_sample2+1)
{
z_aux = vec2[val_eq_z_samp2];;
/* this while sequence calculates the number of values
* equal to z_aux
*/
while ((val_eq_z_samp2+1 < n_sample2)
&& z_aux == vec2[val_eq_z_samp2+1] )
{
val_eq_z_samp2++;
}
/* the smaller of the two actual data values is picked
* as the next zj.
*/
if(z > z_aux)
{
z = z_aux;
next_z_sample2 = true;
}
else
{
if (z == z_aux)
{
equal_z_both_samples = true;
}
/* This is the case if the last value of column1 is
* smaller than the remaining values of column2.
*/
if (val_eq_z_samp1 == 0)
{
z = z_aux;
next_z_sample2 = true;
}
}
}
else
{
val_eq_z_samp2 = 0;
val_st_z_samp2 = n_sample2;
// this should be val_eq_z_samp2 - 1 = n_sample2
}
/* in the following, sum j = 1 to L is calculated for
* sample 1 and sample 2.
*/
if (equal_z_both_samples)
{
/* hj is the number of values in the combined sample
* equal to zj
*/
hj = val_eq_z_samp1 - val_st_z_samp1
+ val_eq_z_samp2 - val_st_z_samp2;
/* H_j is the number of values in the combined sample
* smaller than zj plus one half the number of
* values in the combined sample equal to zj
* (that's hj/2).
*/
H_j = val_st_z_samp1 + val_st_z_samp2
+ hj / 2;
/* F1j is the number of values in the 1st sample
* that are less than zj plus one half the number
* of values in this sample that are equal to zj.
*/
F1j = val_st_z_samp1 + (double)
(val_eq_z_samp1 - val_st_z_samp1) / 2;
/* F2j is the number of values in the 1st sample
* that are less than zj plus one half the number
* of values in this sample that are equal to zj.
*/
F2j = val_st_z_samp2 + (double)
(val_eq_z_samp2 - val_st_z_samp2) / 2;
/* set the line of values equal to zj to the
* actual line of the last value picked for zj.
*/
val_st_z_samp1 = val_eq_z_samp1;
/* Set the line of values equal to zj to the actual
* line of the last value picked for zj of each
* sample. This is required as data smaller than zj
* is accounted differently than values equal to zj.
*/
val_st_z_samp2 = val_eq_z_samp2;
/* next the lines of the next values z, i.e., zj+1
* are addressed.
*/
val_eq_z_samp1++;
/* next the lines of the next values z, i.e.,
* zj+1 are addressed
*/
val_eq_z_samp2++;
}
else
{
/* the smaller z value was contained in sample 2;
* hence, this value is the zj to base the following
* calculations on.
*/
if (next_z_sample2)
{
/* hj is the number of values in the combined
* sample equal to zj; in this case, these are
* within sample 2 only.
*/
hj = val_eq_z_samp2 - val_st_z_samp2;
/* H_j is the number of values in the combined sample
* smaller than zj plus one half the number of
* values in the combined sample equal to zj
* (that's hj/2).
*/
H_j = val_st_z_samp1 + val_st_z_samp2
+ hj / 2;
/* F1j is the number of values in the 1st sample that
* are less than zj plus one half the number of values in
* this sample that are equal to zj.
* As val_eq_z_samp2 < val_eq_z_samp1, these are the
* val_st_z_samp1 only.
*/
F1j = val_st_z_samp1;
/* F2j is the number of values in the 1st sample that
* are less than zj plus one half the number of values in
* this sample that are equal to zj. The latter are from
* sample 2 only in this case.
*/
F2j = val_st_z_samp2 + (double)
(val_eq_z_samp2 - val_st_z_samp2) / 2;
/* Set the line of values equal to zj to the actual line
* of the last value picked for zj of sample 2 only in
* this case.
*/
val_st_z_samp2 = val_eq_z_samp2;
/* next the line of the next value z, i.e., zj+1 is
* addressed. Here, only sample 2 must be addressed.
*/
val_eq_z_samp2++;
if (val_eq_z_samp1 == 0)
{
val_eq_z_samp1 = stop_loop1;
}
}
/* the smaller z value was contained in sample 2;
* hence, this value is the zj to base the following
* calculations on.
*/
else
{
/* hj is the number of values in the combined
* sample equal to zj; in this case, these are
* within sample 1 only.
*/
hj = val_eq_z_samp1 - val_st_z_samp1;
/* H_j is the number of values in the combined
* sample smaller than zj plus one half the number
* of values in the combined sample equal to zj
* (that's hj/2).
*/
H_j = val_st_z_samp1 + val_st_z_samp2
+ hj / 2;
/* F1j is the number of values in the 1st sample that
* are less than zj plus; in this case, these are within
* sample 1 only one half the number of values in this
* sample that are equal to zj. The latter are from
* sample 1 only in this case.
*/
F1j = val_st_z_samp1 + (double)
(val_eq_z_samp1 - val_st_z_samp1) / 2; /* F2j is the number of values in the 1st sample that
* are less than zj plus one half the number of values
* in this sample that are equal to zj. As
* val_eq_z_samp1 < val_eq_z_samp2, these are the
* val_st_z_samp2 only.
*/
F2j = val_st_z_samp2;
/* Set the line of values equal to zj to the actual line
* of the last value picked for zj of sample 1 only in
* this case.
*/
val_st_z_samp1 = val_eq_z_samp1;
/* next the line of the next value z, i.e., zj+1 is
* addressed. Here, only sample 1 must be addressed.
*/
val_eq_z_samp1++;
if (val_eq_z_samp2 == 0)
{
val_eq_z_samp2 = stop_loop2;
}
}
}
denom_1_aux = n_total * F1j - n_sample1 * H_j;
denom_2_aux = n_total * F2j - n_sample2 * H_j;
sum_adk_samp1 = sum_adk_samp1 + hj
* (denom_1_aux * denom_1_aux) /
(H_j * (n_total - H_j)
- n_total * hj / 4);
sum_adk_samp2 = sum_adk_samp2 + hj
* (denom_2_aux * denom_2_aux) /
(H_j * (n_total - H_j)
- n_total * hj / 4);
next_z_sample2 = false;
equal_z_both_samples = false;
/* index to count the z. It is only required to prevent
* the while slope to execute endless
*/
j++;
}
// calculating the adk value is the final step.
adk_result = (double) (n_total - 1) / (n_total
* n_total * (k - 1))
* (sum_adk_samp1 / n_sample1
+ sum_adk_samp2 / n_sample2);
/* if(adk_result <= adk_criterium)
* adk_2_sample test is passed
*/
return adk_result <= adk_criterium;
}Appendix C. Glossary 📖
okay bestie, here's the glossary, the little cheat sheet for all the jargon we've been slinging around this whole doc 🧚✨
+-------------+-----------------------------------------------------+
| ADK | Anderson-Darling K-Sample test, a test used to |
| | check whether two samples have the same statistical |
| | distribution. |
| ECMP | Equal Cost Multipath, a load-balancing mechanism |
| | evaluating MPLS Labels stacks, IP addresses, and |
| | ports. |
| EDF | The "empirical distribution function" of a set of |
| | scalar measurements is a function F(x), which for |
| | any x gives the fractional proportion of the total |
| | measurements that were smaller than or equal to x. |
| Metric | A measured quantity related to the performance and |
| | reliability of the Internet, expressed by a value. |
| | This could be a singleton (single value), a sample |
| | of single values, or a statistic based on a sample |
| | of singletons. |
| OWAMP | One-Way Active Measurement Protocol, a protocol for |
| | communication between IPPM measurement systems |
| | specified by IPPM. |
| OWD | One-Way Delay, a performance metric specified by |
| | IPPM. |
| Sample | A sample metric is derived from a given singleton |
| metric | metric by evaluating a number of distinct instances |
| | together. |
| Singleton | A singleton metric is, in a sense, one atomic |
| metric | measurement of this metric. |
| Statistical | A 'statistical' metric is derived from a given |
| metric | sample metric by computing some statistic of the |
| | values defined by the singleton metric on the |
| | sample. |
| TWAMP | Two-way Active Measurement Protocol, a protocol for |
| | communication between IPPM measurement systems |
| | specified by IPPM. |
+-------------+-----------------------------------------------------+Authors' Addresses 🎀
and that's everyone who wrote this thing, with the ways to reach them 💌
Ruediger Geib (editor)
Deutsche Telekom
Heinrich Hertz Str. 3-7
Darmstadt 64295
Germany
Phone: +49 6151 58 12747
EMail: Ruediger.Geib@telekom.de
Al Morton
AT&T Labs
200 Laurel Avenue South
Middletown, NJ 07748
USA
Phone: +1 732 420 1571
Fax: +1 732 368 1192
EMail: acmorton@att.com
URI: http://home.comcast.net/~acmacm/
Reza Fardid
Cariden Technologies
888 Villa Street, Suite 500
Mountain View, CA 94041
USA
Phone:
EMail: rfardid@cariden.com
Alexander Steinmitz
Deutsche Telekom
Memmelsdorfer Str. 209b
Bamberg 96052
Germany
Phone:
EMail: Alexander.Steinmitz@telekom.de📦 where this page comes from
- ietf: rfc6576 2012-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms