RFC 9971

Multiple Loss Ratio Search

📅 published August 2026, from ietf

🎀 RFC 9971: Multiple Loss Ratio Search ✨

Internet Engineering Task Force (IETF) · Request for Comments: 9971 · Category: Informational · ISSN: 2070-1721 · August 2026 💅

Okay bestie, the title block tea: this one is written by M. Konstantynowicz and V. Polak, both from Cisco Systems 🧚. It's Informational, so it's published for info and it is NOT a Standards Track spec 🙈. It doesn't obsolete anything, and it doesn't update anything either. It just sits next to [RFC2544] being its own thing 🌸.

Abstract

This doc gives us an alternative to throughput from "Benchmarking Methodology for Network Interconnect Devices" ([RFC2544]) called Multiple Loss Ratio Search, or MLRsearch for short 🦋. The goals are to minimize Search Duration, support multiple loss ratio goals, and make results more repeatable and comparable. Literally the whole point is less waiting, more consistent numbers 🔥.

Why does this matter? Testing data plane stuff is getting messy, especially for software networking on COTS CPU hardware versus purpose-built ASIC / NPU / FPGA hardware 💀. Those software boxes are noisy and need a smarter way to test, and that's the gap MLRsearch fills 💖.

Status of This Memo

Not a Standards Track spec, just published for info 🌈. It IS a product of the IETF and reflects consensus of the IETF community. It got public review and was approved by the IESG. Not everything the IESG approves becomes an Internet Standard, so check Section 2 of RFC 7841 if you're curious 🫶.

For current status, errata, and how to send feedback, head to https://www.rfc-editor.org/info/rfc9971 ✨.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as of the publication date. Read those carefully, because they spell out your rights and restrictions 💅. Any Code Components pulled out of this doc must include Revised BSD License text (see Section 4.e of the Trust Legal Provisions) and come without warranty, as the Revised BSD License describes 🔥.

Table of Contents

1.  Introduction
  1.1.  Purpose
  1.2.  Positioning Within BMWG Methodologies
2.  Overview of RFC 2544 Problems
  2.1.  Binary Search
  2.2.  Long Search Duration
  2.3.  DUT in SUT
  2.4.  Repeatability and Comparability
  2.5.  Throughput with Non-Zero Loss
  2.6.  Inconsistent Trial Results
3.  Requirements Language
4.  MLRsearch Specification
  4.1.  Scope
    4.1.1.  Relationship to RFC 2544
    4.1.2.  Applicability of Other Specifications
    4.1.3.  Out of Scope
  4.2.  Architecture Overview
    4.2.1.  Search
    4.2.2.  Search Duration
    4.2.3.  Test Report
    4.2.4.  Behavior Correctness
  4.3.  Quantities
    4.3.1.  Current and Final Values
  4.4.  Existing Terms
    4.4.1.  SUT
    4.4.2.  DUT
    4.4.3.  Trial
    4.4.4.  Load Quantities
    4.4.5.  Forwarding Rate Quantities
  4.5.  Trial Terms
    4.5.1.  Trial Duration
    4.5.2.  Trial Load
    4.5.3.  Trial Input
    4.5.4.  Traffic Profile
    4.5.5.  Trial Forwarding Ratio
    4.5.6.  Trial Loss Ratio
    4.5.7.  Trial Forwarding Rate
    4.5.8.  Trial Effective Duration
    4.5.9.  Trial Output
    4.5.10. Trial Result
  4.6.  Goal Terms
    4.6.1.  Goal Final Trial Duration
    4.6.2.  Goal Duration Sum
    4.6.3.  Goal Loss Ratio
    4.6.4.  Goal Exceed Ratio
    4.6.5.  Goal Width
    4.6.6.  Goal Initial Trial Duration
    4.6.7.  Search Goal
    4.6.8.  Controller Input
  4.7.  Auxiliary Terms
    4.7.1.  Trial Classification
    4.7.2.  Load Classification
  4.8.  Result Terms
    4.8.1.  Relevant Upper Bound
    4.8.2.  Relevant Lower Bound
    4.8.3.  Conditional Throughput
    4.8.4.  Goal Results
    4.8.5.  Search Result
    4.8.6.  Controller Output
  4.9.  Architecture Terms
    4.9.1.  Measurer
    4.9.2.  Controller
    4.9.3.  Manager
  4.10. Compliance
    4.10.1.  Test Procedure Compliant with MLRsearch
    4.10.2.  MLRsearch Compliant with RFC 2544
    4.10.3.  MLRsearch Compliant with TST009
5.  Methodology Rationale and Design Considerations
  5.1.  Binary Search Commonalities
  5.2.  Stopping Conditions and Precision
  5.3.  Loss Ratios and Loss Inversion
    5.3.1.  Single Goal and Hard Bounds
    5.3.2.  Loss Inversion
    5.3.3.  Conservativeness and Relevant Bounds
    5.3.4.  Consequences
  5.4.  Exceed Ratio and Multiple Trials
  5.5.  Short Trials and Duration Selection
  5.6.  Generalized Throughput
    5.6.1.  Hard Performance Limit
    5.6.2.  Performance Variability
6.  MLRsearch Logic
  6.1.  Load Classification Logic
  6.2.  Conditional Throughput Logic
    6.2.1.  Conditional Throughput and Load Classification
  6.3.  SUT Behaviors
    6.3.1.  Expert Predictions
    6.3.2.  Exceed Probability
    6.3.3.  Trial Duration Dependence
7.  IANA Considerations
8.  Security Considerations
9.  References
  9.1.  Normative References
  9.2.  Informative References
Appendix A.  Load Classification Code
Appendix B.  Conditional Throughput Code
Appendix C.  Example Search
  C.1.  Example Goals
  C.2.  Example Trial Results
  C.3.  Load Classification Computations
    C.3.1.  Point 1
    C.3.2.  Point 2
    C.3.3.  Point 3
    C.3.4.  Point 4
    C.3.5.  Point 5
    C.3.6.  Point 6
  C.4.  Conditional Throughput Computations
    C.4.1.  Goal 2
    C.4.2.  Goal 3
    C.4.3.  Goal 4
Acknowledgements
Authors' Addresses

1. Introduction ✨

1.1. Purpose 🎯

So the whole point of this doc is to describe MLRsearch, which is optimized for finding data plane throughput on software networking devices and functions 💖.

Using "Binary Search" (Section 2.1) on software Devices Under Test (DUTs) causes a few problems 😭:

Binary Search takes forever, because most Trials happen way far from the Throughput it eventually finds 💀.
The required final Trial Duration and the pauses between Trials stretch out the whole Search Duration 🙈.
Software DUTs give noisy Trial Results, so the Throughput values that could be found spread out a lot 🌪️.
Throughput needs exactly zero lost frames, but industry best practice often tolerates some low, non-zero loss (see [Y.1564] and test-equipment manuals) 🫠.
Throughput gets fuzzy when Trial Results are inconsistent. For example, a later Trial at a higher Load can show a smaller Trial Loss Ratio, and then Throughput can't be pinned to one clear value 🤷‍♀️.

To fix these, early MLRsearch versions used these upgrades 🎀:

Run multiple Short Trials instead of one long Trial per Load.
- Optionally, tolerate a percentage of Trial Results with higher Trial Loss Ratios.
Search for multiple Search Goals, each with a different Goal Loss Ratio.
- In principle, any Trial Result can affect each Search Goal 🦋.
Add multiple coarse targets for each Search Goal, where the earlier ones spend less time on Trials.
- Earlier targets also aim for less precision.
- Use Forwarding Rate at Maximum Offered Load (FRMOL), as defined in Section 3.6.2 of [RFC2285], to initialize bounds.
Clarify how to handle Inconsistent Trial Results.
- Reported Throughput should be smaller than the smallest Load with high loss.
- Measure smaller Load candidates first.
Apply several time-saving Load selection heuristics that deliberately stop the bounds from narrowing unnecessarily 🧠.

Enhancements 1, 2, and partly 4 are formalized as the MLRsearch Specification in this doc. The rest are treated as implementation details and are out of scope here. This gets high comparability without blocking future improvements 🌈.

MLRsearch config supports both conservative and aggressive settings. Results that are unconditionally compliant with [RFC2544] are possible with conservative enough settings, but you don't get much improvement on Search Duration and repeatability (see "MLRsearch Compliant with RFC 2544" (Section 4.10.2)). Aggressive settings give shorter Search Durations and better repeatability, but those results are NOT compliant with [RFC2544] 😭. This doc only offers soft recommendations for settings, but check "Overview of RFC 2544 Problems" (Section 2) to see how the different settings affect result quality 💅.

This document does not change or obsolete any part of [RFC2544] 🫶.

Heads up: this is part 1 of 15, so more is coming. Keep scrolling, bestie 💜

1.2. Positioning Within BMWG Methodologies ✨

okay so the Benchmarking Methodology Working Group (BMWG) 💅 makes recommendations (RFCs) that describe different benchmarking methodologies for use in a controlled laboratory environment. A huge number of these benchmarks are built on the terminology from [RFC1242] and the foundational methodology from [RFC2544] 🦋. A pattern has emerged where BMWG documents reference the methodology of [RFC2544], then add specific requirements for testing particular network systems or protocols, without changing the core benchmark definitions. It's giving consistency, bestie 🫶

While BMWG documents are formal recommendations, they're widely treated as industry norms so that results stay comparable between different labs 🧪. The set of benchmarks defined in [RFC2544], in particular, became a de facto standard for performance testing 👑. In this context, the MLRsearch Specification formally defines a new class of benchmarks that fits within the wider framework of [RFC2544]; see "Scope" (Section 4.1) 🎀.

A primary consideration in the design of MLRsearch is the trade-off between configurability and comparability ⚖️. The methodology's flexibility, especially the ability to define various sets of Search Goals, supports both single-goal and multi-goal benchmarks in a unified way. That's powerful for detailed characterization and internal testing 💜. However, this same flexibility is detrimental to inter-lab comparability unless a specific, common set of Search Goals is agreed upon 💀.

Therefore, MLRsearch should not be seen as a direct extension nor a replacement for the [RFC2544] Throughput benchmark 🙈. Instead, this document provides a foundational methodology that future BMWG documents can use to define new, specific, and comparable benchmarks by mandating particular Search Goal configurations ✨. For operators of existing test procedures, it's worth noting that many test setups measuring [RFC2544] Throughput can be adapted to produce results compliant with the MLRsearch Specification. This often works without affecting Trials, merely by augmenting the content of the final Test Report 🍓.

2. Overview of RFC 2544 Problems 🌸

This section is the tea on the problems that make various performance testing methodologies hard to use, mainly the Binary Search for unconditionally compliant [RFC2544] Throughput 🫠.

2.1. Binary Search 🔍

While [RFC2544] gives you some flexibility when searching for Throughput, one particular algorithm gets used all the time as a starting point. It's the simplest one that still offers reasonable effectivity ✨.

This algorithm is based on (balanced) binary search over sorted arrays, but it doesn't have a specific name when searching for Throughput. "Trial duration" (Section 24 of [RFC2544]) mentions binary search only in quotes, without giving any specifics 🙃. In this document, we call that algorithm the Binary Search, since that's the title of Section 12.3.2 of [TST009], which describes a variant of it 💖.

Here is a simplified description of the algorithm 🧁:

Initialize the lower-bound variable to a line-rate Load.
Initialize the upper-bound variable to a loss-free Load.
Compute a midpoint, the arithmetic mean of current bounds.
Run a single 60-second Trial at the midpoint (for [RFC2544] unconditional compliance).
If loss is zero, set the lower-bound to the midpoint; else, set the upper bound to the midpoint.
Repeat (computing new midpoint) until the gap between the bounds meets the desired precision.
Return the final lower-bound as the Throughput.

The description in [TST009] has two more requirements (stopping condition and rounding, both based on the Offered Load Step Size Parameter), but those are not required in this document 🙅. Small modifications related to initial bounds are also allowed 🎀.

The Loads currently held in the two variables are called tightest bounds, especially when discussing older Trial Results (logically still bounds) 💅.

2.2. Long Search Duration ⏳

The proliferation of software DUTs, with frequent software updates and a number of different frame processing modes and configurations, has increased both the number of performance tests required to verify the DUT update and the frequency of running those tests. This makes the overall test execution time even more important than before 😩.

The definition of Throughput test methodology per [RFC2544] restricts the potential for time-efficiency improvements. The Binary Search, when used in a manner unconditionally compliant with [RFC2544] Throughput methodology, is excessively slow due to two main factors 🐌.

First, a significant amount of time is spent on Trials with Loads that, in retrospect, are far from the final determined Throughput 🕰️.

Second, [RFC2544] does not specify any stopping condition for Throughput search. So users of testing equipment implementing the procedure already have access to a limited trade-off between Search Duration and achieved precision, since each one of the full 60-second Trials halves the interval of possible results 🔪.

As such, not many Trials can be removed without a substantial loss of precision 😭.

2.3. DUT in SUT 🧩

Section 19 of [RFC2544] specifies a test setup with an external tester stimulating the networking system. It treats that system either as a single Device Under Test (DUT) or as a system of devices, a System Under Test (SUT) 🎀.

[RFC2285] defines these terms as follows 📖:

DUT:   The network frame forwarding device to which stimulus is
       offered and response measured (Section 3.1.1 of [RFC2285]).

SUT:   The collective set of network devices to which stimulus is
       offered as a single entity and response measured
       (Section 3.1.2 of [RFC2285]).

For software-based data plane forwarding running on commodity x86/ARM CPUs, the SUT comprises not only the forwarding application itself (the DUT), but also the entire execution environment 🖥️. That includes host hardware, firmware and kernel/hypervisor services, plus any other software workloads that share the same CPUs, memory, and I/O resources.

Given that a SUT is a shared multi-tenant environment, the DUT might inadvertently experience interference from the operating system or from other software operating on the same server 😬.

Some of this interference can be mitigated. For instance, in multi-core CPU systems, pinning DUT program threads to specific CPU cores and isolating those cores can prevent context switching 🧘.

Despite taking all feasible precautions, some adverse effects may still impact the DUT's network performance. In this document, these effects are collectively referred to as SUT noise, even if the effects are not as unpredictable as what other engineering disciplines call noise 📢.

A DUT can also exhibit fluctuating performance itself, for reasons not related to the rest of SUT. For example, this can be due to pauses in execution as needed for internal stateful processing. In many cases, this may be an expected per-design behavior, since it would be observable even in a hypothetical scenario where all sources of SUT noise are eliminated. Such behavior affects Trial Results in a way similar to SUT noise. As the two phenomena are hard to distinguish, in this document the term noise is used to encompass both the internal performance fluctuations of the DUT and the genuine noise of the SUT 🌀.

A simple model of SUT performance consists of an idealized noiseless performance and additional noise effects. For a specific SUT, the noiseless performance is assumed to be constant, with all observed performance variations being attributed to noise. The impact of the noise can vary in time, sometimes wildly, even within a single Trial. The noise can sometimes be negligible, but it frequently lowers the observed SUT performance as observed in Trial Results 📉.

In this simple model, a SUT does not have a single performance value; it has a spectrum. One end of the spectrum is the idealized noiseless performance, and the other end can be called a noiseful performance. In practice, Trial Results close to the noiseful end of the spectrum happen only rarely. The worse a possible performance is, the more rarely it is seen in a Trial. Therefore, the extreme noiseful end of the SUT spectrum is not observable among Trial Results 🌈.

Furthermore, the extreme noiseless end of the SUT spectrum is unlikely to be observable. This time that's because minor noise events almost always occur during each Trial, nudging the measured performance slightly below the theoretical maximum 🥲.

Unless specified otherwise, this document's focus is on the potentially observable ends of the SUT performance spectrum, as opposed to the extreme ones 👀.

When focusing on the DUT, the benchmarking effort should ideally aim to eliminate only the SUT noise from SUT measurements. However, this is currently not feasible in practice, as there are no realistic enough models that would be capable to distinguish SUT noise from DUT fluctuations (based on the available literature at the time of writing) 🤷.

If the SUT execution environments and any co-resident workloads place only negligible demands on SUT shared resources, so that the DUT remains the principal performance limiter, the DUT's ideal noiseless performance is defined as the noiseless end of the SUT performance spectrum 💅.

Note that by this definition, DUT noiseless performance also minimizes the impact of DUT fluctuations, as much as realistically possible for a given Trial Duration 🫧.

The MLRsearch methodology aims to solve the DUT-in-SUT problem by estimating the noiseless end of the SUT performance spectrum using a limited number of Trial Results 🧠.

Improvements to the Throughput search algorithm, aimed at better dealing with software networking SUT and DUT setups, should adopt methods that explicitly model SUT-generated noise. Those methods should derive surrogate metrics that approximate the (proxies for) DUT noiseless performance across a range of SUT noise-tolerance levels 🦋.

2.4. Repeatability and Comparability

okay so repeatability is a whole mood 💅 [RFC2544] doesn't suggest repeating a Throughput search at all 💀 And if you only have ONE discovered Throughput value, you literally can't tell how repeatable it is 🙈 Then unsatisfactory repeatability leads to unacceptable comparability, because different benchmarking teams might get different Throughput values for the same SUT, and that gap can be bigger than the expected differences from search precision 😭 Repeatability also matters when the test procedure stays the same but the SUT gets varied in small ways. Like, when you're developing software-based DUTs, you need repeatability to catch tiny regressions 🔥

[RFC2544] Throughput requirements (60-second Trial and no tolerance of a single frame loss) affect the Throughput result as follows.

The SUT behavior close to the noiseful end of its performance spectrum is made of rare moments of significantly low performance. But the long Trial Duration makes those moments not so rare on the Trial level 🫠 So the Binary Search results tend to spread away from the noiseless end of the SUT performance spectrum more often and more widely than shorter Trials would, which causes unacceptable Throughput repeatability 💔

The repeatability problem can be better addressed by defining a search procedure that identifies a consistent level of performance, even if it doesn't meet the strict definition of Throughput test methodology in [RFC2544] ✨

According to the SUT performance spectrum model, better repeatability will be at the noiseless end of the spectrum. So solutions to the DUT-in-SUT problem will also help with the repeatability problem 🧚

Conversely, any alteration to [RFC2544] Throughput search that improves repeatability should be considered as less dependent on the SUT noise 🌸

An alternative option is to simply run a search multiple times and report some statistics (e.g., average and standard deviation and/or percentiles like p95).

This can be used for a subset of tests deemed more important, but it makes the Search Duration problem even more pronounced 😩

2.5. Throughput with Non-Zero Loss

Section 3.17 of [RFC1242] defines Throughput as:

The maximum rate at which none of the offered frames are dropped by the device.

Then, it says:

Since even the loss of one frame in a data stream can cause significant delays while waiting for the higher level protocols to time out, it is useful to know the actual maximum data rate that the device can support.

However, many benchmarking teams accept a low, non-zero Goal Loss Ratio for their Load search 🍓

There are many motivations, bestie:

Networking protocols tolerate frame loss better, compared to the time when [RFC1242] and [RFC2544] were specified 🦋
Increased link speeds require Trials sending more frames within the same duration, increasing the chance of a small SUT performance fluctuation being enough to cause frame loss 🚀
Because noise-related drops usually arrive in small bursts, their impact on the Trial Loss Ratio is diluted by the longer intervals in which the SUT operates close to its noiseless performance; consequently, the Trial Loss Ratio as an average over the Trial can still end up below the specified Goal Loss Ratio 🌈
If an approximation of the SUT noise impact on the Trial Loss Ratio is known, it can be set as the Goal Loss Ratio 🎀

For more information, see Section 5 of [Lencze-Shima] (and the references there) for a few synthetic examples, confirming that each protocol and application can have different realistic Goal Loss Ratios 👑

Regardless of the validity of all similar motivations, support for non-zero Goal Loss Ratios makes a search algorithm applicable for a wider range of use cases than the approach defined in [RFC2544] ✨

Furthermore, allowing users to specify multiple Goal Loss Ratios, and enabling a single Search to find all relevant bounds, significantly enhances the usefulness of the search algorithm 💖

Searching for multiple Search Goals also helps to describe the SUT performance spectrum better than the result of a single Search Goal. For example, the repeated wide gap between zero and non-zero loss Loads indicates the noise has a large impact on the observed performance, which is not evident from the result of a single goal Load search procedure 👀

It is easy to modify the Binary Search to find a Lower Bound for the Load that satisfies a single non-zero Goal Loss Ratio. But how to search for multiple goals at once is not that obvious; hence, the support for multiple Search Goals remains a problem 😤

At the time of writing, there does not seem to be a consensus in the industry on which Goal Loss Ratio is the best. For users, performance of higher protocol layers is important, for example, goodput of TCP connection (TCP throughput [RFC6349]), but the relationship between goodput and Trial Loss Ratio is not simple 🧐 Refer to [Lencze-Kovacs-Shima] for examples of various corner cases, Section 3 of [RFC6349] for Goal Loss Ratios acceptable for an accurate measurement of TCP throughput, and [Ott-Mathis-Semke-Mahdavi] for models and computations of TCP performance in presence of packet loss 📚

2.6. Inconsistent Trial Results

While performing Throughput search by executing a sequence of measurement Trials, there is a risk of encountering inconsistencies between Trial Results 😬

Examples include but are not limited to:

A Trial at the same Load (same or different Trial Duration) results in a different Trial Loss Ratio.
A Trial at a larger Load (same or different Trial Duration) results in a lower Trial Loss Ratio.

The Binary Search never encounters inconsistent Trials. But [RFC2544] hints about the possibility of Inconsistent Trial Results in two places in its text 👀 The first place is Section 24 of [RFC2544], where full Trial Durations are required, presumably because they can be inconsistent with the Trial Results from shorter Trial Durations. The second place is Section 26.3 of [RFC2544], where two successive zero-loss Trials are recommended, presumably because after one zero-loss Trial Result, there can be a subsequent inconsistent non-zero-loss Trial Result 🕵️‍♀️

A robust Throughput search algorithm needs to decide how to continue the search in the presence of such inconsistencies. Definitions of Throughput and its test methodology in [RFC1242] and [RFC2544] are not specific enough to imply a unique way of handling such inconsistencies 🤷‍♀️

Ideally, there will be a definition of a new metric that both generalizes Throughput for non-zero Goal Loss Ratio (and other possible repeatability enhancements) while being precise enough to force a specific way to resolve Trial Result inconsistencies. But until such a definition is agreed upon, the correct way to handle Inconsistent Trial Results remains an open problem 🥺

Relevant Lower Bound is the MLRsearch term that addresses this problem 💜

3. Requirements Language ✨

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 📣

This document is categorized as an Informational RFC. While it does not mandate the adoption of the MLRsearch methodology, it uses the normative language of BCP 14 [RFC2119] [RFC8174] to provide an unambiguous specification. This ensures that if a test procedure or Test Report claims compliance with the MLRsearch Specification, it MUST adhere to all the absolute requirements defined herein. The use of normative language is intended to promote repeatable and comparable results among those who choose to implement this methodology 🫶

4. MLRsearch Specification 👑

This section provides all technical definitions needed for evaluating whether a particular test procedure complies with the MLRsearch Specification 📋

Some terms used in the specification are capitalized. This is a stylistic choice for this document, reminding the reader that the term is introduced, defined, or explained elsewhere in the document. Lowercase variants are equally valid; capitalization was already applied in earlier sections where required 💅

This document does not separate terminology from methodology. Terms are fully specified and discussed in their own subsections, under sections with the word "Terms" in their titles. This way, the list of terms is visible in the table of contents 🗂️

Each term-defining subsection contains a short Definition paragraph containing a minimal definition and all strict requirements, followed by Discussion paragraphs focusing on important consequences and recommendations. Requirements about how other components can use the defined term are also included in the discussion 🎀

4.1. Scope

This document specifies the Multiple Loss Ratio Search (MLRsearch) methodology. The MLRsearch Specification details a new class of benchmarks by listing all terminology definitions and methodology requirements. The definitions support multi-goal benchmarks, with single-goal ones as a subset 🌟

The normative scope of this specification includes:

The terminology for all required quantities and their attributes.
An abstract architecture consisting of functional components (Manager, Controller, and Measurer) and the requirements for their inputs and outputs.
The required structure and attributes of the Controller Input, including one or more Search Goals.
The required logic for Load Classification, which determines whether a given Load qualifies as a Lower Bound or an Upper Bound for a Search Goal.
The required structure and attributes of the Controller Output, including a Goal Result for each Search Goal.

4.1.1. Relationship to RFC 2544

The MLRsearch Specification is an independent methodology and does not change nor obsolete any part of [RFC2544] 🦋

This specification permits deviations from the Trial procedure as described in [RFC2544]. Any deviation from the procedure in [RFC2544] must be documented explicitly in the Test Report, and such variations remain outside the scope of the original benchmarks in [RFC2544] 📝

A specific single-goal MLRsearch benchmark can be configured to be compliant with [RFC2544] Throughput, and most procedures reporting [RFC2544] Throughput can be adapted to also satisfy MLRsearch requirements for a specific Search Goal 🧚‍♀️

4.1.2. Applicability of Other Specifications

Methodology extensions from other BMWG documents that specify details for testing particular DUTs, configurations, or protocols (e.g., by defining a particular Traffic Profile) are considered orthogonal to MLRsearch and are applicable to a benchmark conducted using MLRsearch methodology 🌈

4.1.3. Out of Scope 🙈

okay so bestie, here's what's NOT in the normative scope of this doc. no cap, these are the things the spec is just like "nope, not my problem" about 💅

The internal heuristics or algorithms the Controller uses to pick Trial Inputs are implementation specific. whatever your Controller does to choose them is its own business ✨
What happens when Offered Load turns out to be noticeably different from Intended Load is not prescribed. the only thing mentioned is the "use smaller Max Load" recommendation 🦋
The potential for, and effects of, interference between different Search Goals in a multi-goal search are outside the normative scope of this spec. lowkey a plot twist 😭
This spec does not mandate any single, universal Search Goal configuration for every use case. picking Search Goal parameters is left to the operator of the test procedure, or it may be recommended in future publications 🌈

ngl there are a few weak recommendations about Search Goal possibilities too, but they only apply under specific intents (like compatibility with other methodologies). they're not considered universal enough yet 🫶

4.2. Architecture Overview 🎀

okay so the normative text only uses terms that were already introduced, but explanatory passages sometimes need terms defined later on. so this informative section gives you a quick, top-down sketch of the whole MLRsearch architecture so the first read-through stays clear 💖

the architecture is modelled as a set of abstract, interacting components. they talk to each other in an imperative-programming style: one component _calls_ another, passing inputs (arguments) and getting outputs (return values). this notation is purely conceptual, so actual implementations don't need to exchange explicit messages. when the text contrasts alternative behaviors, it's talking about different implementations of the same component 🧚

a test procedure is considered _compliant_ with the MLRsearch Specification if it can be conceptually broken down into the abstract components defined here, AND each component satisfies the requirements defined for its matching MLRsearch Specification section. main character energy 👑

the _Measurer_ component performs Trials, the _Controller_ component selects Trial Durations and Loads, and the _Manager_ component preconfigures the involved entities and produces the _Test Report_. the Test Report explicitly states _Search Goals_ (as Controller Input) and the matching _Goal Results_ (Controller Output) 📝

this makes up one _benchmark_ (single-goal or multi-goal). repeated or slightly different benchmarks are done by calling the Controller once for each benchmark ✨

4.2.1. Search 🔥

for one benchmark, the Manager calls a Controller once, and then the Controller keeps invoking the Measurer over and over until it decides it has enough info to return its outputs 💅

the part where the Controller invokes the Measurer is called the _Search_. any work the Manager does either before calling the Controller or after the Controller returns falls outside the Search, so it's not part of it 🦋

the MLRsearch Specification prescribes Regular Goal Results and recommends matching Search completion conditions. Irregular Goal Results are also allowed, but they have different requirements, and their stopping conditions are out of scope. the Search Result is the combo of regular or irregular results, one for each Search Goal 🌸

Search Results are based on _Load Classification_. once a chosen _Load_ has been measured enough, it can either achieve or fail each Search Goal (separately), which makes it a Lower Bound or an Upper Bound for that Search Goal 💜

when the _Relevant Lower Bound_ is close enough to the _Relevant Upper Bound_ according to the _Goal Width_, the _Regular Goal Result_ is found. the Search stops when all Regular Goal Results are found, or when all remaining Search Goals are proven to only have _Irregular Goal Results_ 🎀

4.2.2. Search Duration 🌈

even if the time the Controller takes to compute _Trial Input_ attributes for the next Trial is negligible, Trials themselves take some time to finish, and SUT behavior affects how quickly the stopping conditions can be satisfied 😭

for large scale tests where lots of SUT configurations and Traffic Profiles need benchmarking, it matters a lot to optimize the overall duration of one Search, since the time cost of initial configuration tends to stay constant. there are three considerations worth discussing for this _Search Duration_ optimization 🍓

firstly, SUT behavior can be probabilistic, so the Search Duration of each benchmark can be too. the quantity to optimize is the expected value of Search Duration, and it should be as short as possible within the intended reliability 💖

secondly, Search Duration probability distributions with smaller standard deviations are preferred, because they make it easier to plan running multiple benchmarks one-by-one on the same SUT 🫶

thirdly, any prior knowledge about SUT behavior can turn into average time gains if the Controller implementation supports the matching heuristics. without that knowledge, implementations should focus on the worst possible results, since lowering Search Duration for those helps the standard deviation, and maybe the expected value too ⭐

most optimization heuristics are outside the scope of this document, but lots of design decisions were shaped by generic optimizations found in practice so far 🧚

4.2.3. Test Report 📝

a primary responsibility of the Manager is producing a Test Report, which is the final and formal output of the test procedure 💅

this document does NOT provide a single, complete, normative definition for the structure of the Test Report. for example, the Test Report may contain results for a single benchmark, or it could aggregate results from many benchmarks 🌸

instead, normative requirements for the content of the Test Report are spread throughout this document, alongside the definitions of the quantities and procedures they apply to. heads up: any clause requiring a quantity to be "reported" or "stated in the Test Report" is a normative requirement on the content of this final artifact 🚨

even where not stated explicitly, the "Reporting format" paragraphs in [RFC2544] still count as requirements on Test Reports if they apply to an MLRsearch benchmark 🦋

4.2.4. Behavior Correctness 💀

the MLRsearch Specification by itself does NOT guarantee that the Search ends in finite time, because the freedom the Controller has for Load selection also allows for clearly deficient choices 😭

for deeper insights on these matters, check out [FDio-CSIT-MLRsearch] 🔥

the primary MLRsearch implementation, used as the prototype for this specification, is [PyPI-MLRsearch] 🌈

4.3. Quantities 🍓

the MLRsearch Specification relies on several specific _quantities_ 💜

in physics, a single magnitude can be expressed in multiple ways as a product of a _unit_ and a _value_. this document uses the computer science vocabulary instead, where the term "quantity" can describe multiple variables or constants at once 🧚

one _instance_ of a scalar quantity is called a _value_, and the corresponding unit is only mentioned when needed. where context allows, a shorter phrase is used, for example, "Load" instead of "value of the Load quantity" 🎀

in general, the MLRsearch Specification does NOT prescribe particular units to be used, but it is REQUIRED for the Test Report to state all the units. for example, ratio quantities can be dimensionless numbers between zero and one, but they may be expressed as percentages instead 🫶

in rare cases, a _primary unit_ is mentioned, usually when comparing across different Traffic Profiles would otherwise be ambiguous. some units MAY be preferred by some implementations, for example, when they round values to integers 💅

for convenience, a group of quantities can be treated as a _composite_ quantity. one constituent of a composite quantity is called an _attribute_, and a group of attribute values is called an _instance_ of that composite quantity 🌸

some attributes may depend on others and can be computed from them. such quantities are called _derived_ quantities 🦋

4.3.1. Current and Final Values 🌈

some quantities are defined so their values can be computed in the middle of a Search. other quantities can only be computed after the Search ends. some quantities only matter after a Search ends, but their values can still be computed before it ends 👑

for a quantity that can be computed before a Search ends, the adjective _current_ marks a value of that quantity that's available before the Search ends. when such a value is relevant for the Search Result, the adjective _final_ denotes the value of that quantity at the end of the Search ✨

if a time evolution of such a dynamic quantity is guided by configuration quantities, those adjectives can tell them apart. for example, if the current value of "duration" (a dynamic quantity) increases from "initial duration" to "final duration" (configuration quantities), all the quoted names are separate but related quantities. as the naming suggests, the final value of the duration quantity is expected to equal the "final duration" value 💖

4.4. Existing Terms 📚

this specification relies on the following three documents, which you should consult before trying to use this document 🫶

"Benchmarking Terminology for Network Interconnection Devices" [RFC1242] contains basic term definitions 🧚
"Benchmarking Terminology for LAN Switching Devices" [RFC2285] includes more terms and discussions, and it describes some known network benchmarking situations more precisely 💜
"Benchmarking Methodology for Network Interconnect Devices" [RFC2544] contains discussions about terms and additional methodology requirements 🎀

references to several central terms from the above documents are given in the following subsections. some of those terms need no further discussion, but some need a bit of elaboration to put them in proper context 🌸

4.4.1. SUT 🍓

SUT is defined in Section 3.1.2 of [RFC2285] as follows 💅

Definition:

The collective set of network devices to which stimulus is offered as a single entity and response measured. ✨

Discussion:

A SUT consisting of a single network device is allowed by this definition 🦋
in software-based networking, a SUT may comprise a multitude of networking applications and the entire host hardware and software execution environment 🌈
the SUT is the only entity that can be benchmarked directly, even though only the performance of some of its sub-components is of interest 🔥
for example, Section 6.1 of [RFC2544] describes a single SUT consisting of DUT 1, DUT 2, and a WAN link between them 💖

4.4.2. DUT 💅

okay so DUT is defined in Section 3.1.1 of [RFC2285] like this 👇

Definition: The network forwarding device to which stimulus is offered and response measured.

Discussion: Contrary to SUT, the DUT stimulus and response are frequently initiated and observed only indirectly, on different parts of SUT. 🦋

DUT is a sub-component of SUT. It's only indirectly mentioned in the MLRsearch Specification, but it's totally key for why the spec exists 🔥. The device can be a software-based networking function running on commodity x86/ARM CPUs (vs. purpose-built ASICs, NPUs, and FPGAs).

When the MLRsearch Specification mentions SUT configuration, it implies adequate configuration of the constituent DUTs, which are subject to the requirements from Section 7 of [RFC2544]. 🧚

A well-designed SUT SHOULD have the primary DUT as its performance bottleneck. The ways to achieve and verify that are outside the scope of the MLRsearch Specification. A typical exception to this rule is when the DUT has a surprisingly good performance. For example, achieving the line rate does not make the SUT badly designed 🙈. This situation is super common in practice: the DUT is the bottleneck for traffic made of small frames, but it hits the line rate with large frames.

4.4.3. Trial 🍓

A trial is the part of the test described in Section 23 of [RFC2544].

Definition: A particular test consists of multiple trials. Each trial returns one piece of information, for example, the loss rate at a particular input frame rate. Each trial consists of a number of phases:

a) If the DUT is a router, send the routing update to the "input" port and pause two seconds to be sure that the routing has settled.
b) Send the "learning frames" to the "output" port and wait 2 seconds to be sure that the learning has settled. Bridge learning frames are frames with source addresses that are the same as the destination addresses used by the test frames. Learning frames for other protocols are used to prime the address resolution tables in the DUT. The formats of the learning frame that should be used are shown in the Test Frame Formats document.
c) Run the test trial.
d) Wait for two seconds for any residual frames to be received.
e) Wait for at least five seconds for the DUT to restabilize.

Discussion: The traffic is sent only in phase c) and received in phases c) and d). 📦

Trials are the only stimuli the SUT is expected to experience during the Search. ✨

It's useful to think of the traffic as sent and received by a tester, as implicitly defined in Section 6 of [RFC2544].

The definition above describes some traits, but it does not use key words as defined in BCP 14 [RFC2119] [RFC8174] to signify the strength of the requirements. For the purposes of the MLRsearch Specification, the test procedure MAY deviate from the description in [RFC2544], but any such deviation MUST be described explicitly in the Test Report. It is still RECOMMENDED to not deviate from the description, because any deviation weakens comparability. 🥺

An example of a deviation from [RFC2544] is using shorter wait times, compared to those described in phases a), b), d), and e).

[RFC2544] seems to treat phase b) as any type of configuration that cannot be configured only once (by the Manager, before the Search starts), since some crucial SUT state could time out during the Search. It is RECOMMENDED to interpret the "learning frames" to mean any such time-sensitive per-Trial configuration method, with bridge Media Access Control (MAC) address learning being only one possible example. Appendix C.2.4.1 of [RFC2544] lists another example: ARP with a wait time of 5 seconds. 💜

Some methodologies describe recurring tests. If those are based on Trials, they are treated as multiple independent Trials. That means there are no "recurring trials" treated as a single measurement in MLRsearch, but the Controller MAY repeat the same Trial Inputs to get multiple independent Trial Outputs. 🌈

4.4.4. Load Quantities 🎀

Several existing quantities describe the intensity of the traffic entering a SUT. MLRsearch needs a quantity that is useful not only as an input for a Trial, but also as a basis for several related quantities that aren't tied to a single Trial. 👑

The following subsections refer to existing terms before introducing their combination, which is used as such a base quantity in this document.

4.4.4.1. Intended Load

Defined in "Intended load (Iload)" (Section 3.5.1 of [RFC2285]).

4.4.4.2. Offered Load

Defined in "Offered load (Oload)" (Section 3.5.2 of [RFC2285]).

4.4.4.3. Constant Load

Defined in "Constant Load" (Section 3.4 of [RFC1242]).

4.4.4.4. Load

Definition: Load is a global per-interface Intended Load usable for a Trial.

Discussion: For specification purposes, Load is also assumed to be Constant Load. 💖

Just like the existing definitions, this quantity is related to one (input or output) SUT interface.

In the common case of bidirectional traffic, as described in "Bidirectional traffic" (Section 14 of [RFC2544]), Load is the "data rate" per each direction, which is half of the "sum of data rates". ⭐

The Test Report MAY present the aggregate Load across multiple interfaces, treating it as the same magnitude expressed using different units. Each reported Load MUST unambiguously state whether it refers to (i) a single interface, (ii) a specified subset of interfaces (such as all logical interfaces mapped to one physical port), or (iii) the total across every interface. For any aggregate Load value, the report MUST also give the fixed conversion factor that links the per-interface and multi-interface Load values. 📏

The per-interface Load remains the primary unit, consistent with the prevailing practice described in [RFC1242], [RFC2544], and [RFC2285].

The previous paragraph also applies to other terms related to Load. For example, tests with symmetric bidirectional traffic MAY report Load-related values as "bidirectional load" (double of "unidirectional load" if that is distinguished in the same Test Report). 🦋

Besides Trial Load, the Relevant Lower Bound will be another example of a newly defined quantity based on Load.

4.4.5. Forwarding Rate Quantities 🌸

Similar to Load as a main quantity acting as an input for a Trial, Forwarding Rate is an existing quantity describing an output from a Trial.

Besides Trial Forwarding Rate, the Conditional Throughput will be another example of a newly defined quantity based on Forwarding Rate.

4.4.5.1. Forwarding Rate

Defined in "Forwarding rate (FR)" (Section 3.6.1 of [RFC2285]).

4.4.5.2. Throughput

Defined in "Throughput" (Section 3.17 of [RFC1242]).

The methodology of measuring Throughput is codified in "Throughput" (Section 26.1 of [RFC2544]).

This document frequently uses the phrase [RFC2544] Throughput when referring not only to the Throughput quantity itself, but also to the methodology requirements for benchmarking it. 🧁

4.5. Trial Terms ✨

This section defines new terms and redefines existing terms for quantities relevant as inputs or outputs of a Trial, as handled by the Measurer component. It also includes any derived quantities related to results of one Trial. 🫶

4.5.1. Trial Duration ⏱️

Definition: Trial Duration is the intended duration of phase c) of a Trial.

Discussion: The value MUST be positive.

While any positive real value may be provided, some Measurer implementations MAY limit possible values, for example, by rounding down to the nearest integer in seconds. In that case, it is RECOMMENDED to give such inputs to the Controller so that the Controller implementation only generates the accepted values. 🙌

4.5.2. Trial Load 🏋️

Definition: Trial Load is the Load used in a specific Trial.

Discussion: This is a Load-related quantity that is tied to a single Trial, to distinguish it from later derived quantities that are not tied that way. 💅

For specification purposes, it is assumed that this is a Constant Load by default. Informally, Trial Load is a single number that can "scale" any traffic pattern, as long as the intuition of a load intended against a single interface can be applied.

It MAY be possible to use a Trial Load to describe non-constant traffic (using average load when the traffic consists of repeated bursts of frames, for example, as suggested in Section 21 of [RFC2544]). In the case of a non-constant load, the Test Report MUST explicitly mention exactly how non-constant the traffic is and how it reacts to the Trial Load value. But the rest of the MLRsearch Specification assumes that is not the case, to avoid discussing corner cases (for example, which values are possible within medium limitations). 🤓

Similarly, traffic patterns where different interfaces are subject to different loads MAY be described by a single Trial Load value (for example, using the largest Intended Load among interfaces). But again, the Test Report MUST explicitly describe how the traffic pattern reacts to the Trial Load value, and this specification does not discuss all the implications of that approach.

Traffic patterns where a single Trial Load does not describe their scaling cannot be used for MLRsearch benchmarks. 🚫

Similarly to Trial Duration, some Measurers MAY limit the possible values of Trial Load. Contrary to Trial Duration, documenting such behavior in the Test Report is OPTIONAL. This is because the load differences are negligible (and frequently undocumented) in practice. 🤷‍♀️

The Controller MAY select the Trial Load and Trial Duration values in a way that would not be possible to achieve using any integer number of data frames.

Given that the Trial Load is a quantity based on Load, the Test Report MAY express this quantity using multi-interface values, as the sum of Intended Loads over input interfaces. 🧮

4.5.3. Trial Input 🎁

Definition: Trial Input is a composite quantity, consisting of exactly two attributes: Trial Duration and Trial Load.

Discussion: When talking about a set of Trials, it's common to say Trial Inputs to denote all corresponding Trial Input instances.

One Trial Input instance acts as the input for one call of the Measurer component.

Unlike other composite quantities, MLRsearch implementations MUST NOT add optional attributes into Trial Input. This improves interoperability between various implementations of a Controller and a Measurer. 🔒

Note that both attributes are intended quantities, as only those can be fully controlled by the Controller. The actual offered quantities, as realized by the Measurer, can be different (and must be different if not multiplying into an integer number of frames), but questions around those offered quantities are generally outside of the scope of this document. 💫

4.5.4. Traffic Profile 🎀

Definition: Okay so, Traffic Profile is a composite quantity that bundles up every attribute except Trial Load and Trial Duration that's needed to uniquely pin down the Trial being run 💅

Discussion: All the attributes MUST stay constant during the Search 🔒. The Manager basically configures the whole bundle on the Measurer before the Search even starts ✨. That's why the Traffic Profile is not part of the Trial Input 🧚

Specifying the traffic properties in the Traffic Profile is the Manager's job 💜, but the actual configuration mechanisms are out of scope for this document. Implementations are allowed to include this data in Controller calls to the Measurer, as long as its impact is functionally constant 🫶.

Manager and Measurer implementations should know their shared capabilities well enough that a Traffic Profile instance uniquely defines the traffic during the Search. Typically, they're tightly integrated, so this is usually fine 🌸.

Integration between independent Manager and Measurer implementations is also out of scope. An example standardization effort is described in [Vassilev] 🦋.

Examples of common traffic properties include:

Data link frame size:
- Fixed sizes as listed in Section 3.5 of [RFC1242] and in Section 9 of [RFC2544]
- "Internet Mix" (IMIX) mixed frame sizes as defined in [RFC6985]
Frame formats and protocol addresses:
- Sections 8 and 12 of [RFC2544] and Appendix C of [RFC2544]

Other traffic properties that somehow need to be specified in Traffic Profile, and MUST be mentioned in Test Report if they apply to the benchmark, include:

symmetric bidirectional traffic from Section 14 of [RFC2544],
fully meshed traffic from Section 3.3.3 of [RFC2285],
modifiers from Section 11 of [RFC2544], and
IP version mixing from Section 5.3 of [RFC8219].

4.5.5. Trial Forwarding Ratio ✨

Definition: Trial Forwarding Ratio is a dimensionless floating point quantity 🔢. Its value MUST range between 0.0 and 1.0, both inclusive 💖. You get it by dividing the number of frames successfully forwarded by the SUT by the total number of frames expected to be forwarded during the Trial 🍓.

Discussion: For most Traffic Profiles, "expected to be forwarded" means "intended to get received by a SUT from the tester" 👑. This SHOULD be the default interpretation. But if that's not the case, the Test Report MUST describe the Traffic Profile in enough detail to imply how the Trial Forwarding Ratio should be computed 📝.

The Trial Forwarding Ratio MAY be expressed in other units (e.g., as a percentage) in the Test Report 💅.

Note that, unlike Load terms, frame counts used to compute the Trial Forwarding Ratio are generally aggregates over all SUT output interfaces, since most test procedures verify all outgoing frames 👀. The procedure for [RFC2544] Throughput counts received frames, so it implies bidirectional counts for bidirectional traffic, even though the final value is the "rate" that is still per-interface 🤯. For example, in a test with symmetric bidirectional traffic, if one direction is forwarded without any losses but the opposite direction doesn't forward at all, the Trial Forwarding Ratio would be 0.5 (50%) 😭.

In future extensions, more general ways to compute the Trial Forwarding Ratio may be allowed. But the current MLRsearch Specification relies on this specific averaged counters approach 🌈.

4.5.6. Trial Loss Ratio 💀

Definition: Trial Loss Ratio is equal to one minus the Trial Forwarding Ratio 🧮.

Discussion: When expressing ratio values as percentages, the Trial Loss Ratio is equal to 100% minus the Trial Forwarding Ratio 💯.

This is almost identical to Frame Loss Rate in Section 3.6 of [RFC1242] 🙈. The only minor differences are that Trial Loss Ratio doesn't need to be expressed as a percentage, and it's explicitly based on averaged frame counts when more than one traffic direction is present ✨.

4.5.7. Trial Forwarding Rate 🦋

Definition: Trial Forwarding Rate is a derived quantity, computed by multiplying the Trial Load by the Trial Forwarding Ratio 🧠.

Discussion: Despite the similar name, this quantity differs substantially from Forwarding Rate 🙅‍♀️.

Under the method described in [RFC2285], each output interface is measured separately, so every interface may report a different Forwarding Rate 🫠. The Trial Forwarding Rate, by contrast, uses a single set of frame counts. That means it yields one value representing the whole system while still keeping a direct relation to the per-interface Load 🎯.

When the Traffic Profile is symmetric and bidirectional, as defined in Section 14 of [RFC2544], the Trial Forwarding Rate is numerically equal to the arithmetic average of the individual per-interface Forwarding Rates 🌸.

For more complex traffic patterns, such as many-to-one (mentioned in "Partially meshed traffic", Section 3.3.2 of [RFC2285]), the meaning of Trial Forwarding Rate gets less straightforward 😵‍💫. For example, if two input interfaces each receive one million frames per second (fps), and a single interface outputs 1.4 million fps, then the Trial Load is 1 million fps, the Trial Loss Ratio is 30%, and the Trial Forwarding Rate is 0.7 million fps 🍓.

Because Trial Forwarding Rate is anchored to the Load defined for one interface, a Test Report MAY show it either as the single averaged figure just described or as the sum of the separate per-interface Forwarding Rates 💅. For the example above, the _aggregate_ Trial Forwarding Rate is 1.4 million fps 👑.

4.5.8. Trial Effective Duration ⏱️

Definition: Trial Effective Duration is a time quantity related to a Trial. By default, it is equal to the Trial Duration 🕰️.

Discussion: This is an OPTIONAL feature 🎀. If the Measurer doesn't return any Trial Effective Duration value, the Controller MUST use the Trial Duration value from its Trial Input instead 🫶.

The Trial Effective Duration MAY be any positive time quantity chosen by the Measurer to be used for time-based decisions in the Controller 🧚.

The Test Report MUST explain how the Measurer computes the returned Trial Effective Duration values if they are not always equal to the Trial Duration 📝.

This feature can be beneficial for time-critical benchmarks designed to manage the overall Search Duration, rather than solely the traffic portion of it ⭐. One approach is to measure the duration of the whole Trial (including all wait times) and use that as the Trial Effective Duration 🌈.

This is also a way for the Measurer to tell the Controller about its surprising behavior, for example, when rounding the Trial Duration value 🙈.

4.5.9. Trial Output 💜

Definition: Trial Output is a composite quantity made of several attributes. The REQUIRED attributes are Trial Loss Ratio, Trial Effective Duration, and Trial Forwarding Rate 🔥.

Discussion: When referring to more than one Trial, the plural term _Trial Outputs_ is used to collectively describe multiple Trial Output instances 🌸.

Measurer implementations MAY provide additional attributes. Controller implementations SHOULD ignore any such optional attribute they aren't familiar with. If a Controller implementation passes selected Trial Outputs to the Manager, those instances SHOULD contain all such additional attributes 🎁.

An example of an optional attribute is the aggregate count of frames expected to be forwarded during the Trial, especially if it isn't (a rounded-down value) implied by Trial Load and Trial Duration 🍓.

While Section 3.5.2 of [RFC2285] requires the Offered Load to be reported for Forwarding Rate measurements, it is not required in the MLRsearch Specification 🤷‍♀️. That's because Search Results don't depend on it, and technically the Measurer isn't performing a direct Forwarding Rate measurement 🦋.

4.5.10. Trial Result 🌈

Definition: Trial Result is a composite quantity, consisting of the Trial Input and the Trial Output as REQUIRED attributes 💖.

Discussion: When referring to more than one Trial, the plural term _Trial Results_ is used to collectively describe multiple Trial Result instances ✨.

4.6. Goal Terms 👑

This section defines new terms for quantities relevant (directly or indirectly) to the inputs and outputs of the Controller component 🧚.

Several goal attributes are defined before introducing the main composite quantity: the _Search Goal_ 🎯.

Unlike other sections, definitions in subsections of this section are necessarily vague 🫠, since their fundamental meaning is to act as coefficients in formulas for Controller Output attributes, which aren't defined yet 🧮.

The discussions in this section relate the attributes to concepts mentioned in "Overview of RFC 2544 Problems" (Section 2). These discussion paragraphs are short and informal, and they mostly reference later sections, where the impact on Search Results is discussed after introducing the complete set of Auxiliary Terms 📚.

4.6.1. Goal Final Trial Duration ⏳

Definition: This is the minimal value for Trial Duration that should be reached. The value MUST be positive 💅.

Discussion: Certain Trials must reach this minimum Trial Duration before a Load can be classified as a Lower Bound 🙌.

The Controller MAY choose shorter Trial Durations, and those Trial Results can be enough for classification as an Upper Bound 🦋.

If such shorter Trials prove that no Lower Bound exists, this Trial Duration value may never be reached, which shortens the Search Duration ✨.

It is RECOMMENDED for all Search Goals to share the same Goal Final Trial Duration value. Otherwise, Trial Durations larger than the Goal Final Trial Duration may occur, weakening the assumptions the "Load Classification Logic" (Section 6.1) is based on 🤓.

Note that this is a soft limitation for the Trial Duration values selected by the Controller. The Measurer may keep returning different Effective Trial Durations in Trial Results, so Controller implementations MUST be ready to handle such situations 🚨.

4.6.2. Goal Duration Sum ⏱️✨

Definition: Okay so this is a threshold for a specific sum of Trial Effective Duration values 🎀. The value MUST be positive 💖.

Discussion: Basically, this sets how many Trials you need at a specific Trial Load and Goal Final Trial Duration during the Search 🧚. It's giving "enough data, no more, no less" 💅.

If the Goal Duration Sum is bigger than the Goal Final Trial Duration, you might need multiple Trials at the same Load 🔁. Like, that's normal 🫶.

Check out "MLRsearch Compliant with TST009" (Section 4.10.3) if you want to see an example where doing multiple Trials at the same Load is totally the plan 📋.

Making the Goal Duration Sum shorter than the Goal Final Trial Duration (for the same Search Goal) could make the Search faster ⚡, but it is NOT RECOMMENDED 🙅‍♀️. You'd be trading speed for repeatability, and that's a hard no from me 😭.

In practice, the Search might spend less than the Goal Duration Sum measuring a Load when the Trial Results are super one-sided 🎯. But it could also spend more when the results are balanced and include Trials shorter than the Goal Final Trial Duration 🌈. Plot twist, but make it make sense 🤷‍♀️.

4.6.3. Goal Loss Ratio 🍓

Definition: This is a threshold for Trial Loss Ratio values 🎀. The value MUST be non-negative and smaller than one 💜.

Discussion: A Trial with a Trial Loss Ratio bigger than this value is a sign that the SUT might not be able to handle this Load well enough 😬.

Check out "Throughput with Non-Zero Loss" (Section 2.5) for reasons why users might want this value above zero 🧐.

Since you might need multiple Trials for one Load, Load Classification can get more complicated than just comparing one Trial Loss Ratio to the Goal Loss Ratio 🌀. Not me overthinking it 💀.

4.6.4. Goal Exceed Ratio 🦋

Definition: This is a threshold for a particular ratio of sums of Trial Effective Durations 📐. The value MUST be non-negative and smaller than one 🫧.

Discussion: Informally, this controls the ratio of Trial Results that exceed the Goal Loss Ratio 🎯. More specifically, up to this proportion of Trial Results with a Trial Loss Ratio above the Goal Loss Ratio is tolerated at a Lower Bound 🙈. This is the full impact if every Trial is measured at the Goal Final Trial Duration. The actual full logic is more complicated, since shorter Trials are allowed 🌸.

For explainability reasons, the RECOMMENDED value for the Goal Exceed Ratio is 0.5 (50%) 💖. In practice, that also leads to the smallest variation in overall Search Duration ✨. Iconic 👑.

Check out "Exceed Ratio and Multiple Trials" (Section 5.4) for more details 📖.

4.6.5. Goal Width 📏

Definition: This is a threshold for deciding whether two Load values are close enough 🫶. This is an OPTIONAL attribute. If present, the value MUST be positive 💅.

Discussion: Informally, this acts as a stopping condition, controlling the precision of the Goal Result 🛑. The Search stops once every Search Goal has reached its precision 🎉.

Implementations without this attribute MUST provide the Controller with other means to control the Search _stopping conditions_ 🧯. No skipping this part, bestie 🙅‍♀️.

_Absolute Load difference_ (in fps) and _relative Load difference_ (in percents) are two popular choices 📊. Implementations MAY choose different ways to specify Goal Width 🦄.

The Test Report MUST make it clear what specific quantity is used as the Goal Width 📝. Be clear, be iconic 💜.

It is RECOMMENDED to express the Goal Width as a relative difference and to set it to a value not lower than the Goal Loss Ratio 🍰. Check out "Performance Variability" (Section 5.6.2) for more on the reasoning 🧠.

4.6.6. Goal Initial Trial Duration 🐣

Definition: This is the minimal Trial Duration value suggested for this goal 🌷. This is an OPTIONAL attribute. If present, this value MUST be positive ⭐.

Discussion: This is an example of an optional Search Goal attribute 🧩.

A typical default value is equal to the Goal Final Trial Duration 🎀.

Informally, this is the shortest Trial Duration the Controller should select when focusing on this Search Goal 🏃‍♀️.

Note that the Controller might still select shorter Trial Durations, for example, when it's focusing on a different Search Goal 🔀. Plot twist again 🌈.

The Goal Initial Trial Duration is a way for a user to discourage Trials with Trial Durations that are too small to be reliable for a particular SUT and a given Search Goal 🙅‍♀️. Pure protection energy 🛡️.

As with the Goal Final Trial Duration, the Controller implementation MUST be able to handle possibly lower Effective Trial Duration values returned by the Measurer 🫡. No crashing out 😭.

4.6.7. Search Goal 🎯

Definition: The Search Goal is a composite quantity made up of several attributes, and some of them are required 🧁.

The REQUIRED attributes are Goal Final Trial Duration, Goal Duration Sum, Goal Loss Ratio, and Goal Exceed Ratio 📋✨.

Discussion: Typical optional attributes are Goal Initial Trial Duration and Goal Width 🎀.

Implementations MAY add their own attributes 🛠️. Those additional attributes may be required by an implementation even if they are not required by the MLRsearch Specification 🤓. However, it is RECOMMENDED for those implementations to support missing attributes by providing typical default values 💖.

For example, implementations with Goal Initial Trial Durations might also require users to specify "how quickly" Trial Durations should increase 🚀.

Check out "Compliance" (Section 4.10) for the important Search Goals 📚.

4.6.8. Controller Input 🎛️

Definition: Controller Input is a composite quantity that's required as an input for the Controller 🫧. The only REQUIRED attribute is a list of Search Goals 📝.

Discussion: MLRsearch implementations MAY use additional attributes 🌸. Those additional attributes may be required by an implementation even if they are not required by the MLRsearch Specification 🙃.

Formally, the Manager does not apply any Controller configuration apart from one Controller Input instance 🗂️.

For example, the Traffic Profile is conceptually configured on the Measurer by the Manager, without explicit help from the Controller 🧚‍♀️.

The order of Search Goals in a list SHOULD NOT have a big impact on Controller Output 🙅‍♀️. That said, MLRsearch implementations MAY base their behavior on the order of Search Goals in a list 👀.

4.6.8.1. Max Load 🏔️

Definition: Max Load is an OPTIONAL attribute of Controller Input 💎. It is the maximal value the Controller is allowed to select for Trial Loads 🔝.

Discussion: Max Load is an example of an optional attribute (outside the list of Search Goals) that some MLRsearch implementations require 🧩.

If Max Load is provided, the Controller MUST NOT select Trial Loads larger than that value 🚫. Hard boundary, no exceptions 💅.

In theory, each Search Goal could have its own Max Load 🤔. But since all Trial Results possibly affect all Search Goals, it makes more sense for one single Max Load to apply to all Search Goals 🫶.

Max Load is a super frequently used configuration parameter 📈. It's already governed (as Maximum Frame Rate) by Section 20 of [RFC2544] and (as Maximum Offered Load) by Section 3.5.3 of [RFC2285] 📜. Some implementations MAY detect or discover it, instead of requiring a user-supplied value 🕵️‍♀️.

In the MLRsearch Specification, one reason for listing the Relevant Upper Bound as a required attribute is that it makes the Search Result independent of the Max Load value 💜.

Given that Max Load is a quantity based on Load, the Test Report MAY express this quantity using multi-interface values, as the sum of per-interface maximal Loads 🧮.

4.6.8.2. Min Load 🐾

Definition: Min Load is an OPTIONAL attribute of Controller Input 🌱. It is the minimal value the Controller is allowed to use for Trial Loads 🔻.

Discussion: Min Load is another example of an optional attribute that some MLRsearch implementations require 🎀. Similar to Max Load, it makes more sense to prescribe one common value, instead of using a different value for each Search Goal 🤝.

If Min Load is provided, the Controller MUST NOT select Trial Loads smaller than that value 🙅‍♀️.

Min Load is mainly useful for saving time by failing early 🏁. It results in an Irregular Goal Result when the Min Load gets classified as an Upper Bound 🙈.

For implementations, it is RECOMMENDED to require Min Load to be non-zero and large enough to allow at least one frame to be forwarded, even at the shortest allowed Trial Duration 📦. That way the Trial Loss Ratio is always well-defined, and the implementation can apply a relative Goal Width safely 🛡️✨.

Given that Min Load is a quantity based on Load, the Test Report MAY express this quantity using multi-interface values, as the sum of per-interface minimal Loads 🧠.

4.7. Auxiliary Terms 🧸

While the terms in this section are not strictly needed when formulating MLRsearch requirements, they simplify the language used in discussion paragraphs and explanation sections 💬. Makes everything way easier to talk about 🫶.

4.7.1. Trial Classification 🏷️

When one Trial Result instance is compared to one Search Goal instance, several relations can be named using short adjectives 💅.

As Trial Results do not affect each other, this Trial Classification does not change during a Search 🔒.

4.7.1.1. High-Loss Trial 🔥

A Trial with a Trial Loss Ratio larger than a Goal Loss Ratio is called a _High-Loss Trial_, with respect to the given Search Goal (or _lossy Trial_, if the Goal Loss Ratio is zero) 💀.

4.7.1.2. Low-Loss Trial 🌿

If a Trial is not a High-Loss Trial, it is called a _Low-Loss Trial_ (or _zero-loss Trial_, if the Goal Loss Ratio is zero) 🍃.

4.7.1.3. Short Trial ⏳

A Trial with a Trial Duration shorter than the Goal Final Trial Duration is called a _Short Trial_ with respect to the given Search Goal 🐇.

4.7.1.4. Full-Length Trial 📏

A Trial that is not a Short Trial is called a _Full-Length Trial_ 🎞️.

Note that this includes Trial Durations larger than the Goal Final Trial Duration 🙌.

4.7.1.5. Long Trial 🐢

A Trial with a Trial Duration longer than the Goal Final Trial Duration is called a _Long Trial_ 🌙.

4.7.2. Load Classification 📊

When a set of all Trial Result instances performed so far at one Load is compared to one Search Goal instance, their relation can be named using the concept of a _bound_ 🧁.

In general, such bounds are a current quantity, even though cases of a Load changing its classification more than once during the Search are rare in practice 🦋.

4.7.2.1. Upper Bound 💖

Definition: Okay so an Upper Bound is a Load that a "Load Classification Code" (Appendix A) algorithm labels as one, for the given Search Goal, at the current moment of the Search. 🎀

Discussion: Bestie, here's the tea 🧚. Every Trial Result measured so far at this Load (and any Trial Duration) is certain to fail to uphold all the requirements of the Search Goal, mainly the Goal Loss Ratio together with the Goal Exceed Ratio. And "certain to fail" is judged against any possible Trial Results within the time left until the Goal Duration Sum is reached. It's giving no-way-out energy 💀.

One Search Goal can have lots of different Loads that count as its Upper Bounds 👑. As the Search goes on and more Trials get measured, any Load can become an Upper Bound in principle ✨.

Now for the plot twist 🙈: a Load can stop being an Upper Bound, but only when more than a Goal Duration Sum of Trials has been measured (for example, because another Search Goal needs more Trials at this Load). Informally, the previous Upper Bound got invalidated 😭. In practice, the Load usually turns into a Lower Bound instead 🦋.

4.7.2.2. Lower Bound 🌸

Definition: A Lower Bound is a Load that a "Load Classification Code" (Appendix A) algorithm labels as one, for the given Search Goal, at the current moment of the Search. 💜

Discussion: So this is the mirror image of the last one 🪞. Every Trial Result measured so far at this Load (and any Trial Duration) is certain to uphold all the requirements of the Search Goal, mainly the Goal Loss Ratio together with the Goal Exceed Ratio. "Certain to uphold" is judged against any possible Trial Results within the time left until the Goal Duration Sum is reached. Literally the safe zone ✨.

One Search Goal can have multiple different Loads classified as its Lower Bounds 🍓. As the Search goes on and more Trials get measured, any Load value can become a Lower Bound in principle.

Here's a fun rule 🎀: no Load can be both an Upper Bound and a Lower Bound for the same Search Goal at the same time. But it IS possible for a larger Load to be a Lower Bound while a smaller Load is an Upper Bound at the same time. Wild, I know 😭.

Also, a Load can stop being a Lower Bound, but only when more than a Goal Duration Sum of Trials has been measured (for example, because another Search Goal needs more Trials at this Load). Informally, the previous Lower Bound got invalidated 💀. In practice, the Load usually becomes an Upper Bound instead 🦋.

4.7.2.3. Undecided 🧸

Definition: A Load is called Undecided if it is currently neither an Upper Bound nor a Lower Bound for the given Search Goal. 🫶

Discussion: Any Load that hasn't been measured so far is Undecided. Easy 🌈.

It's possible for a Load to go from an Upper Bound to Undecided by adding Short Low-Loss Trials. That's yet another reason for users to avoid using Search Goals with different Goal Final Trial Durations 🙅‍♀️✨.

4.8. Result Terms 🎀

Before we get into the full structure of a Controller Output, we need the composite quantity called Goal Result 💅. The following subsections define its attributes first, and then describe the Goal Result quantity itself.

There's a correspondence between Search Goals and Goal Results 🔗. Most of the following subsections refer to a given Search Goal when defining their terms. And at the end of the Search, every Search Goal instance has its own matching Goal Result instance 🫶.

4.8.1. Relevant Upper Bound 👑

Definition: The Relevant Upper Bound is the smallest Load classified as an Upper Bound for a given Search Goal at the end of the Search. 🔥

Discussion: If no measured Load had enough High-Loss Trials, the Relevant Upper Bound MAY be non-existent. For example, that happens when Max Load is classified as a Lower Bound 🥲.

Conversely, when the Relevant Upper Bound does exist, it is not affected by the Max Load value 💖.

Since the Relevant Upper Bound is a quantity based on Load, Test Report MAY express this quantity using multi-interface values, as the sum of Intended Loads over input interfaces 🧚.

4.8.2. Relevant Lower Bound 🌸

Definition: The Relevant Lower Bound is the largest Load among those smaller than the Relevant Upper Bound that got classified as a Lower Bound for a given Search Goal at the end of the Search. 💜

Discussion: If no Load had enough Low-Loss Full-Length Trials, the Relevant Lower Bound MAY be non-existent 😭.

Strictly speaking, if the Relevant Upper Bound doesn't exist, the Relevant Lower Bound doesn't exist either 💀. In a typical case, Max Load is classified as a Lower Bound, which makes it impossible to increase the Load to keep searching for an actual Upper Bound. So it's not clear whether a larger value would be found for a Relevant Lower Bound if larger Loads were available for selection 🤷‍♀️.

Since the Relevant Lower Bound is a quantity based on Load, Test Report MAY express this quantity using multi-interface values, as the sum of Intended Loads over input interfaces 🎀.

4.8.3. Conditional Throughput 🦋

Definition: Conditional Throughput is a value computed at the Relevant Lower Bound according to the algorithm defined in "Conditional Throughput Code" (Appendix B). ✨

Discussion: The Relevant Lower Bound is only defined at the end of the Search, and so is the Conditional Throughput. But the algorithm can be applied at any time on any current Lower Bound, so the final Conditional Throughput value may show up sooner than the end of a Search 🏃‍♀️.

Informally, Conditional Throughput should be a typical Trial Forwarding Rate, the kind you'd expect to see at the Relevant Lower Bound of a given Search Goal 🌈.

But honestly? It's frequently just a conservative estimate of that 🙈. MLRsearch implementations tend to stop measuring more Trials as soon as they confirm the value can't get any worse than this estimate within the Goal Duration Sum. Stop when you're sure, periodt 💅.

This quantity is RECOMMENDED to be used when evaluating repeatability and comparability of different MLRsearch implementations. Refer to "Generalized Throughput" (Section 5.6) for more details 🫶.

Since Conditional Throughput is a quantity based on Load, Test Report MAY express this quantity using multi-interface values, as the sum of per-interface Forwarding Rates 🍓.

4.8.4. Goal Results 🎀

The MLRsearch Specification is built on a set of requirements for a regular result. But in practice, it isn't always possible for such a result instance to exist, so irregular results need to be supported too 🫠.

4.8.4.1. Regular Goal Result 💖

Definition: Regular Goal Result is a composite quantity made of several attributes. Relevant Upper Bound and Relevant Lower Bound are REQUIRED attributes. Conditional Throughput is a RECOMMENDED attribute. 👑

Discussion: Implementations MAY add their own attributes 🧚.

Test Report MUST display the Relevant Lower Bound. Displaying the Relevant Upper Bound is RECOMMENDED, especially if the implementation doesn't use Goal Width 💅.

Generally, the stopping conditions for the corresponding Search Goal MUST be satisfied to produce a Regular Goal Result. Specifically, if an implementation offers Goal Width as a Search Goal attribute, the distance between the Relevant Lower Bound and the Relevant Upper Bound MUST NOT be larger than the Goal Width 🔥.

For stopping conditions, check out "Goal Width" (Section 4.6.5) and "Stopping Conditions and Precision" (Section 5.2) 📚.

4.8.4.2. Irregular Goal Result 🌈

Definition: Irregular Goal Result is a composite quantity. All attributes are OPTIONAL. 🫶

Discussion: It is RECOMMENDED to report any useful quantity, even if it doesn't satisfy all the requirements 🙌. For example, if Max Load is classified as a Lower Bound, it's fine to report it as an "effective" Relevant Lower Bound (not a real one, since that requires a Relevant Upper Bound, which doesn't exist in this case) and compute Conditional Throughput for it 🦋. In this case, only the missing Relevant Upper Bound signals that the instance is irregular.

Similarly, if both Relevant Bounds exist, it is RECOMMENDED to include them as Irregular Goal Result attributes, and let the Manager decide if they're too far apart for Test Report purposes 🧠.

If Test Report displays some Irregular Goal Result attribute values, they MUST be clearly marked as coming from irregular results 🚩.

The implementation MAY define additional attributes, for example explicit flags for expected situations, so the Manager logic can be simpler 🍓.

4.8.4.3. Goal Result 💜

Definition: Goal Result is a composite quantity. Each instance is either a Regular Goal Result or an Irregular Goal Result. 🎀

Discussion: The Manager MUST be able of distinguishing whether the instance is regular or not, without accessing data outside the instance 👀.

4.8.5. Search Result 🌸

Definition: The Search Result is a single composite object that maps each Search Goal instance to a corresponding Goal Result instance. ✨

Discussion: As an alternative to mapping, the Search Result MAY be represented as an ordered list of Goal Results. In that case, the list MUST appear in the exact same order as the list of their corresponding Search Goals, as specified in the Controller Input instance 📋.

When the Search Result is expressed as a mapping, it MUST contain an entry for every Search Goal supplied in the Controller Input 💅.

Identical Goal Result instances MAY be listed for different Search Goals, but their status as regular or irregular MAY be different. For example, this happens if two Search Goals differ only in the Goal Width, and the Relevant Lower Bound is close enough to the Relevant Upper Bound according to only one of them 🙈🦋.

4.8.6. Controller Output 💖

Definition: The Controller Output is a composite quantity returned from the Controller to the Manager at the end of the Search. The Search Result instance is its only REQUIRED attribute. 👑

Discussion: The MLRsearch implementation MAY return additional data in the Controller Output, for example the number of Trials performed and the total Search Duration 📊.

4.9. Architecture Terms 🧚

The MLRsearch architecture consists of three main system components: the Manager, the Controller, and the Measurer 🏗️. These were introduced in "Architecture Overview" (Section 4.2), and the following sections finalize their definitions using terms from the previous sections 🌈.

Note that the architecture also implies the presence of other components, such as the SUT and the tester (as a sub-component of the Measurer) 🔌.

Communication protocols and interfaces between components are left unspecified. For example, when the MLRsearch Specification uses the verb "to call" to describe how the Controller interacts with the Measurer, it's possible that the Controller notifies the Manager to call the Measurer indirectly instead. Doing this means Measurer implementations can be fully independent from Controller implementations, for example developed in different programming languages 🫶💜.

4.9.1. Measurer ✨

Definition: okay so the Measurer is the functional element that, when you call it with a Trial Input instance, it performs one Trial and hands back a Trial Output instance. 🎀

Discussion: this definition assumes the Measurer is already initialized 💅. In practice there can be extra steps before the Search, e.g., when the Manager sets up the Traffic Profile (either on the Measurer or on its tester sub-component directly) and does a warm-up (if the tester or the test procedure needs one) 🌸.

It's the Measurer implementation's job to uphold any requirements and assumptions in the MLRsearch Specification, e.g., the Trial Forwarding Ratio not being larger than one 🫶.

Implementers get some freedom 🦋. For example, Section 10 of [RFC2544] gives some suggestions (but not requirements) about duplicated or reordered frames. Implementations are RECOMMENDED to document their behavior around those freedoms in as much detail as possible 📝✨.

It is RECOMMENDED to benchmark the test equipment first 🧚. Connect the sender and receiver directly (no SUT in the path), find a Load that keeps the Offered Load close enough to the Intended Load, and use that value as the Max Load. When you measure the real SUT, it is RECOMMENDED to turn any severe gap between the Intended Load and the Offered Load into a higher Trial Loss Ratio 👑.

Neither of these two recommendations became a mandatory requirement, and honestly that makes sense 🙈. It's really hard to say when the difference is "severe" in a way that can be separated from the other Measurer freedoms.

For a sample situation where the Offered Load can't keep up with the Intended Load, and what that does to the Search Result, check out "Hard Performance Limit" (Section 5.6.1) 💖.

4.9.2. Controller

Definition: the Controller is the functional element that, once it gets a Controller Input instance, keeps generating Trial Input instances for the Measurer and collecting the matching Trial Output instances 🔁. The loop keeps going until the stopping conditions are met, and then the Controller produces a final Controller Output instance and terminates 🦋.

Discussion: the Controller has a lot of freedom when picking Trial Inputs, and implementations want to hit all the Search Goals in the shortest average Search Duration ⏱️✨.

The Controller's job of optimizing the overall Search Duration is what makes MLRsearch algorithms different from simpler search procedures 💅.

Each implementation can have different stopping conditions, and Goal Width is just one example 🌈. In practice, implementation details don't affect comparability that much, as long as the Goal Result instances being compared are regular 🫶.

4.9.3. Manager

Definition: the Manager is the functional element that's responsible for provisioning the other components, calling a Controller component once, and creating the Test Report following the reporting format defined in Section 26 of [RFC2544] 📋💜.

Discussion: the Manager MUST initialize the SUT (including constituent DUTs) and the Measurer (and the tester if it's independent from the Measurer) with their intended configurations before calling the Controller 🔥.

Also, Section 7 of [RFC2544] already puts requirements on DUT setups, and it's giving strict 💀:

It is expected that all of the tests will be run without changing the configuration or setup of the DUT in any way other than that required to do the specific test. For example, it is not acceptable to change the size of frame handling buffers between tests of frame handling rates or to disable all but one transport protocol when testing the throughput of that protocol.

It is REQUIRED for the Test Report to cover all the SUT configuration details, including a description of a default DUT configuration that's common for most tests, plus any configuration changes a specific test needs 📑🎀.

For example, Section 5.1.1 of [RFC5180] recommends testing jumbo frames if the SUT can forward them, even though they're outside the scope of the 802.3 IEEE standard [IEEE.802.3df] 🌸. In this case it's fine for the SUT default configuration to not support jumbo frames, and to only enable that support when testing jumbo Traffic Profiles. Jumbo frames usually need different packet buffer requirements and can have higher processing overhead, so that makes sense 🧠. Non-jumbo frame sizes should also be tested on the jumbo-enabled setup 💅.

The Manager doesn't need to be able to tweak any Search Goal attributes, but it MUST report all applied attributes even if it didn't tweak them 📝✨.

A human or automated user invokes the Manager once to launch a single Search and receive its Test Report 🦋. Every new invocation is treated as a fresh, independent Search. How the system behaves across multiple calls (like combining or comparing their Search Results) is explicitly out of scope for this document 🙅‍♀️.

4.10. Compliance

This section talks about compliance relations between MLRsearch and other test procedures 🎀.

4.10.1. Test Procedure Compliant with MLRsearch

Any networking measurement setup that can be understood as functional elements satisfying all the requirements for the Measurer, the Controller, and the Manager is compliant with the MLRsearch Specification 🌈.

These components are abstractions that show up in any testing procedure 🧚. For example, one component can act as both the Manager and the Controller. As long as the values of all required attributes of Search Goals and Goal Results are visible in the Test Report, the Controller Input and Controller Output instances are implied 💖.

Another example: any setup for conditionally (or unconditionally) compliant [RFC2544] Throughput testing can be understood as an MLRsearch architecture, as long as there's enough data to reconstruct the Relevant Upper Bound 👑.

Check out "MLRsearch Compliant with RFC 2544" (Section 4.10.2) for an equivalent Search Goal 🫶.

Any test procedure that can be understood as one call to the Manager of the MLRsearch architecture is said to be compliant with the MLRsearch Specification ✨.

4.10.2. MLRsearch Compliant with RFC 2544

Compliance with RFC 2544 is governed by [RFC2544], so MLRsearch rules can't change that 🙅‍♀️. They can still give recommendations to improve comparability 💅.

The following Search Goal instance is RECOMMENDED to make the corresponding Search Result unconditionally compliant with Section 24 of [RFC2544] 🔥:

Goal Final Trial Duration = 60 seconds
Goal Duration Sum = 60 seconds
Goal Loss Ratio = 0%
Goal Exceed Ratio = 0%

The presence of other Search Goals doesn't affect the compliance of this Goal Result 🌸. In this case, the Relevant Lower Bound and the Conditional Throughput are equal to each other, and the value is the Throughput 💜.

A Goal Duration Sum smaller than Goal Final Trial Duration should have no effect on the results 🧚. A non-zero Goal Exceed Ratio would needlessly prolong the Search when Short Trials of both loss types are present 🙈.

Goal Loss Ratio and Goal Exceed Ratio are enough to make the Search Goal conditionally compliant. Adding Goal Final Trial Duration makes it unconditionally compliant 🎀.

A small enough Goal Duration Sum stops MLRsearch from repeating zero-loss Full-Length Trials 🦋. Allowing those would make it less clear whether the result is compliant with Section 24 of [RFC2544], or whether it's a new, stricter methodology 🤔.

4.10.3. MLRsearch Compliant with TST009

One of the alternatives to [RFC2544] is Binary Search With Loss Verification, described in Section 12.3.3 of [TST009] 🌷.

The point of that Search is to repeat High-Loss Trials, hoping for zero loss on the second try. That way the Goal Results land closer to the noiseless end of the performance spectrum, which makes them more repeatable and comparable 💖.

Only the variant with "z = infinity" is achievable with MLRsearch 🌈.

For example, for the "max(r) = 2" variant, the following Search Goal instance is RECOMMENDED (for comparability reasons) to get a compatible Search Result 🫶:

Goal Final Trial Duration = 60 seconds
Goal Duration Sum = 120 seconds
Goal Loss Ratio = 0%
Goal Exceed Ratio = 50%

If the first 60-second Trial has zero loss, MLRsearch can stop measuring at that Load, because even a second High-Loss Trial would still fit within the Goal Exceed Ratio of 50% 🙌.

But if the first Trial is High-Loss, MLRsearch also needs to run the second Trial to classify that Load 😭. The Goal Duration Sum is twice as long as the Goal Final Trial Duration, so a third Full-Length Trial is never needed 💅✨.

5. Methodology Rationale and Design Considerations 🧠

This section explains the "why" behind MLRsearch 🦋. Building on the normative specification in MLRsearch Specification, it contrasts MLRsearch with the classic single-ratio Binary Search in [RFC2544] and walks through the key design choices: search mechanics, stopping-rule precision, Loss Inversion for multiple goals, Goal Exceed Ratio handling, Short Trial strategies, and the generalized throughput concept 💜. Together, these show how the methodology reduces test time, supports multiple Goal Loss Ratios, and improves repeatability 👑.

5.1. Binary Search Commonalities

A typical search implementation for [RFC2544], such as Binary Search, tracks only the two tightest bounds (in variables lower-bound and upper-bound) 🔍. To start, the search needs both Max Load and Min Load values. Then one Trial confirms that Max Load is an Upper Bound, and one Trial confirms that Min Load is a Lower Bound 🌸.

After that, Trial Load is chosen as the mean of the current tightest upper bound and the current tightest lower bound. It then becomes a new tightest bound depending on the Trial Loss Ratio 🎀.

After some number of Trials, the tightest lower bound becomes the Throughput. But [RFC2544] doesn't say when, if ever, the search should stop 🤷‍♀️. In practice, the search stops either at some distance between the tightest upper bound and the tightest lower bound, or after some number of Trials 🫶.

For a given pair of Max Load and Min Load values, there's a one-to-one correspondence between the number of Trials and the final distance between the tightest bounds. So the search always takes the same time, assuming the initial bounds are confirmed 🧚✨.

5.2. Stopping Conditions and Precision 🎯

Okay so the MLRsearch Specification says you have to list both the Relevant Upper Bound and the Relevant Lower Bound for each Search Goal 💅. The gap between those two bounds tells you whether the precision you need for the Regular Goal Result is achieved, so it's not necessary to report which stopping condition was used. Iconic, right? ✨

MLRsearch implementations may use Goal Width to get direct control over Regular Goal Result precision, and indirect control over the Search Duration 🦋.

Other MLRsearch implementations may use different stopping conditions, like one based on the Search Duration. That trades precision control for Search Duration control 🌸.

Also bestie, there's no strict correspondence between Search Duration and Goal Width values, because of various possible time optimizations 🙈. Noisy SUT performance makes the average Search Duration go up, and its variance too 😭.

5.3. Loss Ratios and Loss Inversion 💜

The biggest difference between MLRsearch and Binary Search is what the search is actually going for 👑. [RFC2544] has a single goal, which is classifying one Full-Length Trial as either zero loss or non-zero loss. MLRsearch can search for multiple Search Goals at once, and they usually differ in their Goal Loss Ratios ✨.

5.3.1. Single Goal and Hard Bounds 🧚

Each bound in Binary Search is "hard," which means all further Trial Loads are smaller than any current upper bound and larger than any current lower bound 💖.

This is also possible for MLRsearch implementations when the Search is started with only one Search Goal 🎀.

5.3.2. Loss Inversion 🌈

The MLRsearch Specification supports multiple Search Goals, which makes the Search procedure more complicated than Binary Search with a single goal. Most of those complications don't affect the final Search Results much, except for one phenomenon: Loss Inversion 💀.

Depending on the Search Goal attributes, Load Classification results may be resistant to small amounts of Inconsistent Trial Results. But for larger amounts, a Load that gets classified as an Upper Bound for one Search Goal may still be a Lower Bound for another Search Goal 😭. Because of that other Search Goal, MLRsearch will probably perform subsequent Trials at Loads even larger than the original value 🔥.

This raises questions that any multi-goal search algorithm has to answer, like: What do you do when all those Trials at larger Loads happen to have zero loss? Does that mean the earlier Upper Bound wasn't real? Or does it mean the later Low-Loss Trials don't count toward a Lower Bound? Literally so many questions 🙈

The situation where a smaller Load is classified as an Upper Bound while a larger Load is classified as a Lower Bound (for the same Search Goal) is called Loss Inversion 🦋.

And here's the thing: only single-goal search algorithms can have hard bounds that shield them from Loss Inversion 💅.

5.3.3. Conservativeness and Relevant Bounds 🫶

MLRsearch is conservative when it deals with Loss Inversion. The Upper Bound is considered real, and the Lower Bound is considered a fluke, at least when computing the Goal Result 🛡️.

This gets formalized using the definitions of "Relevant Upper Bound" (Section 4.8.1) and "Relevant Lower Bound" (Section 4.8.2) ✨.

The Relevant Upper Bound (for a specific Search Goal) is the smallest Load classified as an Upper Bound. The Relevant Lower Bound is not simply the largest among the Lower Bounds, though. It's the largest Load among the Loads that are Lower Bounds while also being smaller than the Relevant Upper Bound 👑.

With these definitions, the Relevant Lower Bound is always smaller than the Relevant Upper Bound (if both exist). The two Relevant Bounds get used the same way as the two tightest bounds in Binary Search. When they meet the stopping conditions, the Relevant Upper Bound and the Relevant Lower Bound are used in the Goal Result 💖.

5.3.4. Consequences 🍓

Because of how the Relevant Upper Bound and Relevant Lower Bound are defined, every Trial Result can affect any current Relevant Upper Bound or Relevant Lower Bound that's larger than that Trial Load. It does this by becoming a new Relevant Upper Bound 🌸.

This also applies when that Load is measured before another Load gets enough measurements to become a current Relevant Lower Bound or Relevant Upper Bound 🧚.

It also means that if the SUT being tested (or the Traffic Generator being used) needs a warm-up, it should be warmed up before starting the Search. Otherwise, the first few measurements could become unjustly limiting 😭.

For MLRsearch implementations, this means it's better to measure at smaller Loads first, so any Lower Bounds found earlier are less likely to get invalidated later 🫶.

5.4. Exceed Ratio and Multiple Trials 🦋

The idea of performing multiple Trials at the same Trial Load comes from a model where some Trial Results (the ones with a high Trial Loss Ratio) are affected by infrequent effects. Those effects cause unsatisfactory repeatability of [RFC2544] Throughput results 🙈. Check out "DUT in SUT" (Section 2.3) for a discussion about the noiseful and noiseless ends of the SUT performance spectrum. Stable results are closer to the noiseless end, so MLRsearch may need to allow some frequency of High-Loss Trials in order to ignore the rare but big effects near the noiseful end 💅.

For MLRsearch to do this kind of Trial Result filtering, it needs a configuration option that tells it how frequent that "infrequent" big Trial Loss Ratio can be. This option is called the Goal Exceed Ratio 🎀. It tells MLRsearch what ratio of Trials (more specifically, what ratio of Trial Effective Duration seconds) can have a Trial Loss Ratio larger than the Goal Loss Ratio and still be classified as a Lower Bound ✨.

A zero Goal Exceed Ratio means all Trials must have a Trial Loss Ratio equal to or lower than the Goal Loss Ratio 💀.

When more than one Trial is meant to classify a Load, MLRsearch also needs something to control how many Trials are needed. That's why each Search Goal also has an attribute called Goal Duration Sum 👑.

The meaning of a Goal Duration Sum is this: when a Load has Full-Length Trials whose Trial Effective Durations add up to a value at least as big as the Goal Duration Sum, the Load is guaranteed to be classified as either an Upper Bound or a Lower Bound for that Search Goal 🌈.

5.5. Short Trials and Duration Selection ⭐

MLRsearch requires each Search Goal to specify its Goal Final Trial Duration 🧚.

Section 24 of [RFC2544] already anticipates possible time savings when Short Trials are used 💖.

An MLRsearch implementation MAY expose configuration parameters that decide whether, when, and how Short Trials are used. The exact heuristics and controls are left to the discretion of the implementer 🎀.

MLRsearch implementations are free to use any logic they like to select Trial Inputs. But comparability between MLRsearch implementations is only assured when the Load Classification Logic handles any possible set of Trial Results in the same way 🫶.

The presence of Short Trial Results complicates the Load Classification Logic. See more details in "Load Classification Logic" (Section 6.1) 🦋.

The Load Classification algorithm is designed to avoid any unneeded Trials. Still, for explainability reasons, it is recommended that users pick Controller Inputs that make all Trial Durations selected by the Controller the same. For example, you can set any Goal Initial Trial Duration to a single value that's also used for all Goal Final Trial Durations 🍓.

5.6. Generalized Throughput 🌸

Testing equipment takes the Intended Load as an input parameter for a Trial measurement. So any load search algorithm needs to deal with Intended Load values internally 🧚.

But when Search Goals have a non-zero Goal Loss Ratio, the Load usually doesn't match what a user intuitively thinks a throughput is 😭. The Trial Forwarding Rate is better, but it's not obvious how to generalize it for Loads with multiple Trials and a non-zero Goal Loss Ratio 🙈.

The clearest illustration for adopting a generalized throughput definition is the presence of a hard performance limit 💅.

5.6.1. Hard Performance Limit 🔥

Even if the bandwidth of a medium allows higher traffic forwarding performance, the SUT interfaces may have their own additional limitations. For example, there might be a specific frames-per-second limit on the Network Interface Card (NIC), which is a super common occurrence 💀.

Those limitations should be known and provided as Max Load ✨.

But if Max Load is set larger than what the interface can receive or transmit, a hard limit behavior gets observed in Trial Results 👑.

Let's say the hard limit is at one hundred million frames per second (100 Mfps), Max Load is larger, and the Goal Loss Ratio is 0.5%. If the DUT introduces no additional losses, a 0.5% Trial Loss Ratio will be achieved at the Relevant Lower Bound of 100.5025 Mfps 🦋.

Reporting a throughput that exceeds the SUT's verified hard limit would be counterintuitive, so the Throughput metric should be generalized to reflect realistic, limit-aware performance 🫶.

MLRsearch defines one such generalization, the "Conditional Throughput" (Section 4.8.3). It's the Trial Forwarding Rate from one of the Full-Length Trials performed at the Relevant Lower Bound. For the algorithm to determine exactly which Trial, check out "Conditional Throughput Code" (Appendix B) 🌈.

In the hard limit example, a 100.5025 Mfps Load will still have only a 100.0 Mfps Trial Forwarding Rate, which nicely confirms the known limitation 🎀.

5.6.2. Performance Variability 🍓

With a non-zero Goal Loss Ratio and no hard performance limits, Low-Loss Trials at the same Load may achieve different Trial Forwarding Rate values, simply because of DUT performance variability 🌸.

By comparing the best case (all Relevant Lower Bound Trials have zero loss) with the worst case (all Trial Loss Ratios at the Relevant Lower Bound are equal to the Goal Loss Ratio), you can prove that Conditional Throughput values may have a relative difference up to the Goal Loss Ratio 🧚.

Setting the Goal Width below the Goal Loss Ratio may cause the Conditional Throughput for a larger Goal Loss Ratio to end up smaller than the Conditional Throughput for a Search Goal with a lower Goal Loss Ratio. That's counterintuitive, considering they come from the same Search 😭. So it is RECOMMENDED to set the Goal Width to a value no lower than the Goal Loss Ratio of the next higher loss Search Goal 💅.

Conditional Throughput can fluctuate from one run to the next, but it still offers a more nuanced basis for comparison than the Relevant Lower Bound. This is especially true when deterministic Load selection gives the same Relevant Lower Bound value across multiple runs 💖.

6. MLRsearch Logic 🧠

This section uses informal language to describe two aspects of MLRsearch logic: Load Classification and Conditional Throughput. It reflects the formal pseudocode in "Load Classification Code" (Appendix A) and "Conditional Throughput Code" (Appendix B) 💜.

The logic is equivalent to the pseudocode in the appendices, but it isn't identical. The pseudocode is designed to be short, and it often combines multiple operations into one expression. The logic in this section lists each operation separately and uses more intuitive names for the intermediate values 🫶.

A detailed description of an example Search is in "Example Search" (Appendix C) 🦋.

Okay bestie, let's get into it! 💖 We're picking up with the stuff about how MLRsearch decides what a Load is worth, and it's giving lots of careful bookkeeping 🧠✨

6.1. Load Classification Logic 🎀

For clarity of explanation, variables are tagged as (I)nput, (T)emporary, and (O)utput. So every little number has a label, bestie, so you always know if it's something we're handed, something we're juggling in the middle, or the final answer 💅

Collect Trial Results:
- Take all Trial Results (I) measured at a given Load. 📦
Aggregate Trial Durations:
- Full-Length High-Loss sum (T) is the sum of Trial Effective Duration values of all Full-Length High-Loss Trials (I). 🔥
- Full-Length Low-Loss sum (T) is the sum of Trial Effective Duration values of all Full-Length Low-Loss Trials (I). 🌸
- Short High-Loss sum is the sum (T) of Trial Effective Duration values of all Short High-Loss Trials (I). ✨
- Short Low-Loss sum is the sum (T) of Trial Effective Duration values of all Short Low-Loss Trials (I). 🦋
Derive goal-based ratios:
- Subceed ratio (T) is one minus the Goal Exceed Ratio (I). 💜
- Exceed coefficient (T) is the Goal Exceed Ratio divided by the subceed ratio. 🧚
Balance Short Trial effects:
- Balancing sum (T) is the Short Low-Loss sum multiplied by the exceed coefficient. ⭐
- Excess sum (T) is the Short High-Loss sum minus the balancing sum. 🍓
- Positive excess sum (T) is the maximum of zero and the excess sum. 🌈
Compute effective duration totals:
- Effective High-Loss sum (T) is the Full-Length High-Loss sum plus the positive excess sum. 👑
- Effective full sum (T) is the effective High-Loss sum plus the Full-Length Low-Loss sum. 💖
- Effective whole sum (T) is the larger of the effective full sum and the Goal Duration Sum. 🫶
- Missing sum (T) is the effective whole sum minus the effective full sum. 🙈
Estimate exceed ratios:
- Pessimistic High-Loss sum (T) is the effective High-Loss sum plus the missing sum. 😭
- Optimistic exceed ratio (T) is the effective High-Loss sum divided by the effective whole sum. 💅
- Pessimistic exceed ratio (T) is the pessimistic High-Loss sum divided by the effective whole sum. 🔥
Classify the Load:
- The Load is classified as an Upper Bound (O) if the optimistic exceed ratio is larger than the Goal Exceed Ratio. 🌸
- The Load is classified as a Lower Bound (O) if the pessimistic exceed ratio is not larger than the Goal Exceed Ratio. ✨
- The Load is classified as Undecided (O) otherwise. 🦋

6.2. Conditional Throughput Logic 🧁

Same vibe: every step is labelled, and this one figures out how much traffic the SUT can actually keep up with 💕

Collect Trial Results:
- Take all Trial Results (I) measured at a given Load. 📦
Sum of Full-Length Durations:
- Full-Length High-Loss sum (T) is the sum of Trial Effective Duration values of all Full-Length High-Loss Trials (I). 🔥
- Full-Length Low-Loss sum (T) is the sum of Trial Effective Duration values of all Full-Length Low-Loss Trials (I). 🌸
- Full-Length sum (T) is the Full-Length High-Loss sum (I) plus the Full-Length Low-Loss sum (I). 🫶
Derive initial thresholds:
- Subceed ratio (T) is one minus the Goal Exceed Ratio (I). 💜
- Remaining sum (T) is initially the Full-Length sum multiplied by the subceed ratio. 🧚
- Current loss ratio (T) is initially 100%. 💖
Iterate through ordered Trials:
- For each Full-Length Trial Result, sorted in increasing order by Trial Loss Ratio:
  - If the remaining sum is not larger than zero, exit the loop. 🛑
  - Set the current loss ratio to this Trial's Trial Loss Ratio (I). ✨
  - Decrease the remaining sum by this Trial's Trial Effective Duration (I). 🌈
Compute Conditional Throughput:
- Current forwarding ratio (T) is one minus the current loss ratio. 👑
- Conditional Throughput (T) is the current forwarding ratio multiplied by the Load value. 💅

6.2.1. Conditional Throughput and Load Classification ✨

Conditional Throughput and results of Load Classification overlap but are not identical, so don't mix them up, bestie 🙈

When a Load is marked as a Relevant Lower Bound, its Conditional Throughput is taken from a Trial whose Trial Loss Ratio is not larger than the Goal Loss Ratio. 🦋
The reverse is not guaranteed: If the Goal Width is narrower than the Goal Loss Ratio, Conditional Throughput can still end up higher than the Relevant Upper Bound. 💀

6.3. SUT Behaviors 🌷

In "DUT in SUT" (Section 2.3), the notion of noise is introduced. This section uses new terms to describe possible SUT behaviors more precisely. 🧠

From a measurement point of view, noise is visible as Inconsistent Trial Results. See "Inconsistent Trial Results" (Section 2.6) for general points and "Loss Inversion" (Section 5.3.2) for specifics when comparing different Load values. 🔍

Load Classification and Conditional Throughput apply to a single Load value, but even the set of Trial Results measured at that Trial Load value may appear inconsistent. 🌀

As MLRsearch aims to save time, it executes only a small number of Trials, getting only a limited amount of information about SUT behavior. It is useful to introduce a _SUT expert_ point of view to contrast with that limited information. 🧚‍♀️

6.3.1. Expert Predictions 🔮

Imagine that before the Search starts, a human expert had unlimited time to measure SUT and obtain all reliable information about it. The information is not perfect, as there is still random noise influencing SUT. But the expert is familiar with possible noise events, even the rare ones, and thus, the expert can do probabilistic predictions about future Trial Outputs. 🥹

When several outcomes are possible, the expert can assess the probability of each outcome. 🎯

6.3.2. Exceed Probability 🌟

When the Controller selects a new Trial Duration and Trial Load, and just before the Measurer starts performing the Trial, the SUT expert can envision possible Trial Results. 👀

With respect to a particular Search Goal, the possibilities can be summarized into a single number: _Exceed Probability_. It is the probability (according to the expert) that the measured Trial Loss Ratio will be higher than the Goal Loss Ratio. 💫

6.3.3. Trial Duration Dependence 💞

When comparing Exceed Probability values for the same Trial Load value but different Trial Duration values, there are several patterns that commonly occur in practice. 📊

6.3.3.1. Strong Increase 🚨

Exceed Probability is very low for Short Trials but very high for Full-Length Trials. This SUT behavior is undesirable and may hint at a faulty SUT, e.g., SUT leaks resources and is unable to sustain the desired performance. 😱

But this behavior is also seen when SUT uses large amount of buffers. This is the main reason users may want to set a large Goal Final Trial Duration. 🧺

6.3.3.2. Mild Increase 🌤️

Short Trials are slightly less likely to become High-Loss according to the Goal Loss Ratio, but the slope is modest. This mild increase is typical when noise is dominated by rare, large loss spikes: During a Full-Length Trial, the good-performing periods cannot fully offset the heavy frame loss that occurs in the brief low-performing bursts. 🌊

6.3.3.3. Independence 🕊️

Short Trials have basically the same Exceed Probability as Full-Length Trials. This is possible only if loss spikes are small (so other parts can compensate) and if the Goal Loss Ratio is more than zero (otherwise, other parts cannot compensate at all). 🤝

6.3.3.4. Decrease 📉

Short Trials have a larger Exceed Probability than Full-Length Trials. This can only be possible for a non-zero Goal Loss Ratio, for example, if the SUT needs to "warm up" to the best performance within each Trial, which is not commonly seen in practice. 🔥

7. IANA Considerations 🌸

This document has no IANA actions. Nothing to register here, bestie, so we move right along 💅

8. Security Considerations 🔐

Benchmarking activities as described in this document are limited to the technology characterization of a DUT/SUT using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above. 🧪

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. 🚨

Further, benchmarking is performed on an "opaque" basis, relying solely on measurements observable external to the DUT/SUT. 👀

The DUT/SUT SHOULD NOT include features that serve only to boost benchmark scores, such as a dedicated "fast-track" test mode that is never used in normal operation. 🙅‍♀️

Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. 🛡️

9. References 📚

9.1. Normative References 📖

[RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <https://www.rfc-editor.org/info/rfc1242>. 💜
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>. 🌷
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, DOI 10.17487/RFC2285, February 1998, <https://www.rfc-editor.org/info/rfc2285>. 🧁
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <https://www.rfc-editor.org/info/rfc2544>. 🦋
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>. ✨

Part 11 of 15 done, and we're doing amazing, bestie! 💖🎀

9.2. Informative References 📚✨

okay bestie, these are the informative refs, aka the sources the doc points to for extra context 🤓💅

[FDio-CSIT-MLRsearch] "FD.io CSIT Test Methodology - MLRsearch", October 2023, https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/.
[IEEE.802.3df] IEEE, "IEEE Standard for Ethernet Amendment 9: Media Access Control Parameters for 800 Gb/s and Physical Layers and Management Parameters for 400 Gb/s and 800 Gb/s Operation", March 2024, https://standards.ieee.org/ieee/802.3df/11107/.
[Lencze-Kovacs-Shima] Lencse, G., Kovács, Á., and K. Shima, "Gaming with the Throughput and the Latency Benchmarking Measurement Procedures of RFC 2544", International Journal of Advances in Telecommunications, Electrotechnics, Signals and Systems, vol. 9, no. 2, pp. 10-17, DOI 10.11601/ijates.v9i2.288, June 2020, https://doi.org/10.11601/ijates.v9i2.288.
[Lencze-Shima] Lencse, G. and K. Shima, "An Upgrade to Benchmarking Methodology for Network Interconnect Devices", Work in Progress, Internet-Draft, draft-lencse-bmwg-rfc2544-bis-00, 20 May 2020, https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00.
[Ott-Mathis-Semke-Mahdavi] Mathis, M., Semke, J., Mahdavi, J., and T. Ott, "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm", ACM SIGCOMM Computer Communication Review, vol. 27, no. 3, pp. 67-82, July 1997, https://www.cs.cornell.edu/people/egs/cornellonly/syslunch/fall02/ott.pdf.
[PyPI-MLRsearch] Python Package Index, "MLRsearch 1.2.1", October 2023, https://pypi.org/project/MLRsearch/1.2.1/.
[RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. Dugatkin, "IPv6 Benchmarking Methodology for Network Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 2008, https://www.rfc-editor.org/info/rfc5180.
[RFC6349] Constantine, B., Forget, G., Geib, R., and R. Schrage, "Framework for TCP Throughput Testing", RFC 6349, DOI 10.17487/RFC6349, August 2011, https://www.rfc-editor.org/info/rfc6349.
[RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet Sizes for Additional Testing", RFC 6985, DOI 10.17487/RFC6985, July 2013, https://www.rfc-editor.org/info/rfc6985.
[RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking Methodology for IPv6 Transition Technologies", RFC 8219, DOI 10.17487/RFC8219, August 2017, https://www.rfc-editor.org/info/rfc8219.
[TST009] ETSI, "Network Functions Virtualisation (NFV) Release 3; Testing; Specification of Networking Benchmarks and Measurement Methods for NFVI", ETSI GS NFV-TST 009 V3.4.1, December 2020, https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf.
[Vassilev] Vassilev, V., "A YANG Data Model for Network Tester Management", Work in Progress, Internet-Draft, draft-ietf-bmwg-network-tester-cfg-17, 3 July 2026, https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-network-tester-cfg-17.
[Y.1564] ITU-T, "Ethernet service activation test methodology", ITU-T Recommendation Y.1564, February 2016, https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-Y.1564-201602-I!!PDF-E&type=items.

Appendix A. Load Classification Code 🧮💖

okay this appendix is the recipe for how to do the Load Classification 🍓✨

Any Load value can be classified, according to a given Search Goal. 🎯

The algorithm uses (some subsets of) the set of all available Trial Results from Trials measured at a given Load at the end of the Search. 📊

The block at the end of this appendix holds pseudocode that computes two values, stored in variables named optimistic_is_lower and pessimistic_is_lower. 🧠

Even though it's written as pseudocode, the listing is syntactically valid Python and you can run it without changing anything. 🐍🫶

If both variables come out true, the Load gets classified as a Lower Bound for that Search Goal. If both are false, it's an Upper Bound. Anything else means it's Undecided. 🤷‍♀️

Some variable names got shortened so the expressions fit on one line. Variables holding sum quantities end in _s instead of _sum, and variables holding effective quantities start with effect_ instead of effective_. ✂️

The pseudocode expects these variables to hold these values:

goal_duration_s: The Goal Duration Sum value of the given Search Goal. ⏱️
goal_exceed_ratio: The Goal Exceed Ratio value of the given Search Goal. 📈
full_length_low_loss_s: Sum of Trial Effective Durations across Trials with the Trial Duration at least equal to the Goal Final Trial Duration and with the Trial Loss Ratio not higher than the Goal Loss Ratio (across Full-Length Low-Loss Trials). 💚
full_length_high_loss_s: Sum of Trial Effective Durations across Trials with the Trial Duration at least equal to the Goal Final Trial Duration and with the Trial Loss Ratio higher than the Goal Loss Ratio (across Full-Length High-Loss Trials). 💔
short_low_loss_s: Sum of Trial Effective Durations across Trials with the Trial Duration shorter than the Goal Final Trial Duration and with the Trial Loss Ratio not higher than the Goal Loss Ratio (across Short Low-Loss Trials). 🌸
short_high_loss_s: Sum of Trial Effective Durations across Trials with the Trial Duration shorter than the Goal Final Trial Duration and with the Trial Loss Ratio higher than the Goal Loss Ratio (across Short High-Loss Trials). 🥀

The code also works fine when there are no Trial Results at the given Load. 🙌

<CODE BEGINS>
exceed_coefficient = goal_exceed_ratio / (1.0 - goal_exceed_ratio)
balancing_s = short_low_loss_s * exceed_coefficient
positive_excess_s = max(0.0, short_high_loss_s - balancing_s)
effect_high_loss_s = full_length_high_loss_s + positive_excess_s
effect_full_length_s = full_length_low_loss_s + effect_high_loss_s
effect_whole_s = max(effect_full_length_s, goal_duration_s)
quantile_duration_s = effect_whole_s * goal_exceed_ratio
pessimistic_high_loss_s = effect_whole_s - full_length_low_loss_s
pessimistic_is_lower = pessimistic_high_loss_s <= quantile_duration_s
optimistic_is_lower = effect_high_loss_s <= quantile_duration_s
<CODE ENDS>

Appendix B. Conditional Throughput Code 🩷⚡

this appendix is an example of how to compute Conditional Throughput 💅🔥

Any Load value can be the basis for this computation, but only the Relevant Lower Bound (at the end of the Search) gives you the value called the Conditional Throughput for a given Search Goal. 👑

Same as before, the algorithm uses (some subsets of) the set of all available Trial Results from Trials measured at a given Load at the end of the Search. 📊

The block at the end of this appendix holds pseudocode that computes a value stored as the conditional_throughput variable. 🎀

Also valid Python, so it runs as-is. 🐍✨

Some variable names are shortened to fit expressions on one line. Variables holding sum quantities end in _s instead of _sum, and variables holding effective quantities start with effect_ instead of effective_. ✂️

The pseudocode expects these variables to hold these values:

goal_duration_s: The Goal Duration Sum value of the given Search Goal. ⏱️
goal_exceed_ratio: The Goal Exceed Ratio value of the given Search Goal. 📈
full_length_low_loss_s: Sum of Trial Effective Durations across Trials with the Trial Duration at least equal to the Goal Final Trial Duration and with the Trial Loss Ratio not higher than the Goal Loss Ratio (across Full-Length Low-Loss Trials). 💚
full_length_high_loss_s: Sum of Trial Effective Durations across Trials with the Trial Duration at least equal to the Goal Final Trial Duration and with the Trial Loss Ratio higher than the Goal Loss Ratio (across Full-Length High-Loss Trials). 💔
full_length_trials: An iterable of all Trial Results from Trials with the Trial Duration at least equal to the Goal Final Trial Duration (all Full-Length Trials), sorted by increasing the Trial Loss Ratio. Each item, trial, is a composite with two attributes available:
- trial.loss_ratio: The Trial Loss Ratio as measured for this Trial. 📉
- trial.effect_duration: The Trial Effective Duration of this Trial. ⏳

The code only works correctly when there's at least one Trial Result measured at a given Load. 🙏

<CODE BEGINS>
full_length_s = full_length_low_loss_s + full_length_high_loss_s
whole_s = max(goal_duration_s, full_length_s)
remaining = whole_s * (1.0 - goal_exceed_ratio)
quantile_loss_ratio = None
for trial in full_length_trials:
    if quantile_loss_ratio is None or remaining > 0.0:
        quantile_loss_ratio = trial.loss_ratio
        remaining -= trial.effect_duration
    else:
        break
else:
    if remaining > 0.0:
        quantile_loss_ratio = 1.0
conditional_throughput = intended_load * (1.0 - quantile_loss_ratio)
<CODE ENDS>

Appendix C. Example Search 🧁🔍

this example is a made-up run of the Search part of the MLRsearch test procedure, started with multiple Search Goals. 🎬 Several points in time get picked to show how the logic works, with specific sets of Trial Results available at each one. The Trial Results aren't super realistic on purpose, because the goal is to show several corner cases of the logic. 🙈

In all Trials, the Trial Effective Duration is equal to the Trial Duration. ⏱️

Only one Load is in focus here, and its value is one million frames per second (1 Mfps). Trial Results at other Loads aren't mentioned, because the parts of the logic shown here don't depend on them. In practice there would be Trial Results at other Load values too, since MLRsearch will look for a Lower Bound smaller than any Upper Bound it has found. 🪜

At any given moment, exactly one Search Goal is designated as in focus. That designation only affects the Trial Duration chosen for new Trials. It doesn't change the rest of the decision logic. 🎯

An MLRsearch implementation is free to evaluate several Search Goals at the same time. The focus mechanism is optional, and it only shows up here to demonstrate that a Load can still be classified against Search Goals that aren't currently in focus. 🌈💜

C.1. Example Goals ✨

okay so the example Search has four Search Goals, and each one gets a readable name plus a dense code 🧠 the name is for humans, the code is for showing Search Goal attribute values. 💖

Also, the variable _exceed coefficient_ doesn't depend on Trial Results, so it gets precomputed right here 🎀

   Goal 1:
      name:  RFC2544
      Goal Final Trial Duration:  60s
      Goal Duration Sum:  60s
      Goal Loss Ratio:  0%
      Goal Exceed Ratio:  0%
      exceed coefficient:  0% / (100% / 0%) = 0.0
      code:  60f60d0l0e

   Goal 2:
      name:  TST009
      Goal Final Trial Duration:  60s
      Goal Duration Sum:  120s
      Goal Loss Ratio:  0%
      Goal Exceed Ratio:  50%
      exceed coefficient:  50% / (100% - 50%) = 1.0
      code:  60f120d0l50e

   Goal 3:
      name:  1s final
      Goal Final Trial Duration:  1s
      Goal Duration Sum:  120s
      Goal Loss Ratio:  0.5%
      Goal Exceed Ratio:  50%
      exceed coefficient:  50% / (100% - 50%) = 1.0
      code:  1f120d.5l50e

   Goal 4:
      name:  20% exceed
      Goal Final Trial Duration:  60s
      Goal Duration Sum:  60s
      Goal Loss Ratio:  0.5%
      Goal Exceed Ratio:  20%
      exceed coefficient:  20% / (100% - 20%) = 0.25
      code:  60f60d0.5l20e

Goals 1 and 2 are the compliance-critical ones 🫶 Goals 3 and 4 are the plot twists that cover the rarer cases. 🌈

C.2. Example Trial Results 🦋

Next up are six sets of Trial Results, which are the example Search's receipts 🧾 Each set is a point in time that tells you which Trial Results got added since the previous point. 🕰️

Every point has its own readable name and dense code. The code is what shows Trial Output attribute values and how many times identical Trial Results got added. 💅

   Point 1:
      name:  first short good
      goal in focus:  1s final (1f120d.5l50e)
      added Trial Results:  59 Trials, each 1 second and 0% loss
      code:  59x1s0l

   Point 2:
      name:  first short bad
      goal in focus:  1s final (1f120d.5l50e)
      added Trial Result:  one Trial, 1 second, 1% loss
      code:  59x1s0l+1x1s1l

   Point 3:
      name:  last short bad
      goal in focus:  1s final (1f120d.5l50e)
      added Trial Results:  59 Trials, 1 second each, 1% loss each
      code:  59x1s0l+60x1s1l

   Point 4:
      name:  last short good
      goal in focus:  1s final (1f120d.5l50e)
      added Trial Results:  one Trial, 1 second, 0% loss
      code:  60x1s0l+60x1s1l

   Point 5:
      name:  first long bad
      goal in focus:  TST009 (60f120d0l50e)
      added Trial Results:  one Trial, 60 seconds, 0.1% loss
      code:  60x1s0l+60x1s1l+1x60s.1l

   Point 6:
      name:  first long good
      goal in focus:  TST009 (60f120d0l50e)
      added Trial Results:  one Trial, 60 seconds, 0% loss
      code:  60x1s0l+60x1s1l+1x60s.1l+1x60s0l

Comments on the naming, because the names are doing a LOT of work 😭

When a name contains short, the added Trial had a Trial Duration of 1 second. That's a Short Trial for 3 of the Search Goals, but a Full-Length Trial for the 1s final goal. 🍓
When a name contains long, the added Trial had a Trial Duration of 60 seconds. That's a Full-Length Trial for 3 Search Goals, but a Long Trial for the 1s final goal. 🧚
When a name contains good, the added Trial is a Low-Loss Trial for all the Search Goals. ✨
When a name contains short bad, the added Trial is a High-Loss Trial for all the Search Goals. 💀
When a name contains long bad, the added Trial is a High-Loss Trial for the goals RFC2544 and TST009, but a Low-Loss Trial for the two other Search Goals. 🙈

C.3. Load Classification Computations 🧮

This section shows how Load Classification Logic is applied, by listing all the temporary values at one specific time point. It's basically the math diary, and it's giving detailed 👑

C.3.1. Point 1

This is the first short good point. The code for the available Trial Results is 59x1s0l. 🌸

    +==============+==========+============+============+=============+
    |Goal name     |RFC2544   |TST009      |1s final    |20% exceed   |
    +==============+==========+============+============+=============+
    |Goal code     |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
    +--------------+----------+------------+------------+-------------+
    |Full-Length   |0s        |0s          |0s          |0s           |
    |High-Loss sum |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Full-Length   |0s        |0s          |59s         |0s           |
    |Low-Loss sum  |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Short High-   |0s        |0s          |0s          |0s           |
    |Loss sum      |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Short Low-Loss|59s       |59s         |0s          |59s          |
    |sum           |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Balancing sum |0s        |59s         |0s          |14.75s       |
    +--------------+----------+------------+------------+-------------+
    |Excess sum    |0s        |-59s        |0s          |-14.75s      |
    +--------------+----------+------------+------------+-------------+
    |Positive      |0s        |0s          |0s          |0s           |
    |excess sum    |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Effective     |0s        |0s          |0s          |0s           |
    |High-Loss sum |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Effective full|0s        |0s          |59s         |0s           |
    |sum           |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Effective     |60s       |120s        |120s        |60s          |
    |whole sum     |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Missing sum   |60s       |120s        |61s         |60s          |
    +--------------+----------+------------+------------+-------------+
    |Pessimistic   |60s       |120s        |61s         |60s          |
    |High-Loss sum |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Optimistic    |0%        |0%          |0%          |0%           |
    |exceed ratio  |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Pessimistic   |100%      |100%        |50.833%     |100%         |
    |exceed ratio  |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Classification|Undecided |Undecided   |Undecided   |Undecided    |
    |result        |          |            |            |             |
    +--------------+----------+------------+------------+-------------+

                                  Table 1

This is the last point in time where every Search Goal still has this Load as Undecided. That's the end of the Undecided era 🥀

C.3.2. Point 2

This is the first short bad point. The code for the available Trial Results is 59x1s0l+1x1s1l. Things are about to get spicy 🌶️

    +==============+==========+============+============+=============+
    |Goal name     |RFC2544   |TST009      |1s final    |20% exceed   |
    +==============+==========+============+============+=============+
    |Goal code     |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
    +--------------+----------+------------+------------+-------------+
    |Full-Length   |0s        |0s          |1s          |0s           |
    |High-Loss sum |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Full-Length   |0s        |0s          |59s         |0s           |
    |Low-Loss sum  |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Short High-   |1s        |1s          |0s          |1s           |
    |Loss sum      |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Short Low-Loss|59s       |59s         |0s          |59s          |
    |sum           |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Balancing sum |0s        |59s         |0s          |14.75s       |
    +--------------+----------+------------+------------+-------------+
    |Excess sum    |1s        |-58s        |0s          |-13.75s      |
    +--------------+----------+------------+------------+-------------+
    |Positive      |1s        |0s          |0s          |0s           |
    |excess sum    |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Effective     |1s        |0s          |1s          |0s           |
    |High-Loss sum |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Effective full|1s        |0s          |60s         |0s           |
    |sum           |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Effective     |60s       |120s        |120s        |60s          |
    |whole sum     |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Missing sum   |59s       |120s        |60s         |60s          |
    +--------------+----------+------------+------------+-------------+
    |Pessimistic   |60s       |120s        |61s         |60s          |
    |High-Loss sum |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Optimistic    |1.667%    |0%          |0.833%      |0%           |
    |exceed ratio  |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Pessimistic   |100%      |100%        |50.833%     |100%         |
    |exceed ratio  |          |            |            |             |
    +--------------+----------+------------+------------+-------------+
    |Classification|Upper     |Undecided   |Undecided   |Undecided    |
    |result        |Bound     |            |            |             |
    +--------------+----------+------------+------------+-------------+

                                  Table 2

Here's the tea ☕ Because the RFC2544 goal has a zero Goal Loss Ratio, its Exceed Probability has to take a mild or strong increase. That means the one lossy Trial would still be lossy even if it were measured at a 60-second Trial Duration. 🫠

And because the RFC2544 goal has a zero Goal Exceed Ratio, one High-Loss Trial is enough to stop this Load from becoming a Lower Bound for that goal. So this Load gets classified as an Upper Bound for RFC2544 this early. Iconic behaviour, honestly 👑

This is a great example of how much time you can save compared to using 60-second Trials. ⏰💜

C.3.3. Point 3 ✨

okay so this is the "last short bad" point 💀 The code for available Trial Results is 59x1s0l+60x1s1l.

+==============+==========+============+============+=============+
|Goal name     |RFC2544   |TST009      |1s final    |20% exceed   |
+==============+==========+============+============+=============+
|Goal code     |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+
|Full-Length   |0s        |0s          |60s         |0s           |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Full-Length   |0s        |0s          |59s         |0s           |
|Low-Loss sum  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Short High-   |60s       |60s         |0s          |60s          |
|Loss sum      |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Short Low-Loss|59s       |59s         |0s          |59s          |
|sum           |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Balancing sum |0s        |59s         |0s          |14.75s       |
+--------------+----------+------------+------------+-------------+
|Excess sum    |60s       |1s          |0s          |45.25s       |
+--------------+----------+------------+------------+-------------+
|Positive      |60s       |1s          |0s          |45.25s       |
|excess sum    |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective     |60s       |1s          |60s         |45.25s       |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective full|60s       |1s          |119s        |45.25s       |
|sum           |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective     |60s       |120s        |120s        |60s          |
|whole sum     |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Missing sum   |0s        |119s        |1s          |14.75s       |
+--------------+----------+------------+------------+-------------+
|Pessimistic   |60s       |120s        |61s         |60s          |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Optimistic    |100%      |0.833%      |50%         |75.417%      |
|exceed ratio  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Pessimistic   |100%      |100%        |50.833%     |100%         |
|exceed ratio  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Classification|Upper     |Undecided   |Undecided   |Upper Bound  |
|result        |Bound     |            |            |             |
+--------------+----------+------------+------------+-------------+

Table 3

This is the last point for the "1s final" goal to have this Load still Undecided 🫶 Only one 1-second Trial is missing within the 120-second Goal Duration Sum, but its Trial Result will decide the classification result. 🔥

The "20% exceed" goal started to classify this Load as an Upper Bound somewhere between points 2 and 3. 💅

C.3.4. Point 4 🌸

this is the "last short good" point 🥹 The code for available Trial Results is 60x1s0l+60x1s1l.

+==============+==========+============+============+=============+
|Goal name     |RFC2544   |TST009      |1s final    |20% exceed   |
+==============+==========+============+============+=============+
|Goal code     |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+
|Full-Length   |0s        |0s          |60s         |0s           |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Full-Length   |0s        |0s          |60s         |0s           |
|Low-Loss sum  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Short High-   |60s       |60s         |0s          |60s          |
|Loss sum      |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Short Low-Loss|60s       |60s         |0s          |60s          |
|sum           |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Balancing sum |0s        |60s         |0s          |15s          |
+--------------+----------+------------+------------+-------------+
|Excess sum    |60s       |0s          |0s          |45s          |
+--------------+----------+------------+------------+-------------+
|Positive      |60s       |0s          |0s          |45s          |
|excess sum    |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective     |60s       |0s          |60s         |45s          |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective full|60s       |0s          |120s        |45s          |
|sum           |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective     |60s       |120s        |120s        |60s          |
|whole sum     |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Missing sum   |0s        |120s        |0s          |15s          |
+--------------+----------+------------+------------+-------------+
|Pessimistic   |60s       |120s        |60s         |60s          |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Optimistic    |100%      |0%          |50%         |75%          |
|exceed ratio  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Pessimistic   |100%      |100%        |50%         |100%         |
|exceed ratio  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Classification|Upper     |Undecided   |Lower Bound |Upper Bound  |
|result        |Bound     |            |            |             |
+--------------+----------+------------+------------+-------------+

Table 4

The one missing Trial for "1s final" was Low-Loss; half of Trial Results are Low-Loss, which exactly matches the 50% Goal Exceed Ratio. 🧚 This shows time savings are not guaranteed. 💀

C.3.5. Point 5 🦋

this is the "first long bad" point 👑 The code for available Trial Results is 60x1s0l+60x1s1l+1x60s.1l.

+==============+==========+============+============+=============+
|Goal name     |RFC2544   |TST009      |1s final    |20% exceed   |
+==============+==========+============+============+=============+
|Goal code     |60f60d0l0e|60f120d0l50e|1f120d.5l50e|60f60d0.5l20e|
+--------------+----------+------------+------------+-------------+
|Full-Length   |60s       |60s         |60s         |0s           |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Full-Length   |0s        |0s          |120s        |60s          |
|Low-Loss sum  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Short High-   |60s       |60s         |0s          |60s          |
|Loss sum      |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Short Low-Loss|60s       |60s         |0s          |60s          |
|sum           |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Balancing sum |0s        |60s         |0s          |15s          |
+--------------+----------+------------+------------+-------------+
|Excess sum    |60s       |0s          |0s          |45s          |
+--------------+----------+------------+------------+-------------+
|Positive      |60s       |0s          |0s          |45s          |
|excess sum    |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective     |120s      |60s         |60s         |45s          |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective full|120s      |60s         |180s        |105s         |
|sum           |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Effective     |120s      |120s        |180s        |105s         |
|whole sum     |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Missing sum   |0s        |60s         |0s          |0s           |
+--------------+----------+------------+------------+-------------+
|Pessimistic   |120s      |120s        |60s         |45s          |
|High-Loss sum |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Optimistic    |100%      |50%         |33.333%     |42.857%      |
|exceed ratio  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Pessimistic   |100%      |100%        |33.333%     |42.857%      |
|exceed ratio  |          |            |            |             |
+--------------+----------+------------+------------+-------------+
|Classification|Upper     |Undecided   |Lower Bound |Lower Bound  |
|result        |Bound     |            |            |             |
+--------------+----------+------------+------------+-------------+

Table 5

As designed for the "TST009" goal, one Full-Length High-Loss Trial can be tolerated. 🙈 120s worth of 1-second Trials is not useful, as this is allowed when Exceed Probability does not depend on Trial Duration. As the Goal Loss Ratio is zero, it is not possible for 60-second Trials to compensate for losses seen in 1-second Trial Results. But Load Classification Logic does not have that knowledge hardcoded, so the optimistic exceed ratio is still only 50%. 🤓

But the 0.1% Trial Loss Ratio is lower than the "20% exceed" Goal Loss Ratio, so this unexpected Full-Length Low-Loss Trial changed the classification result of this Load to Lower Bound. 😭🫶

C.3.6. Point 6

okay so this is the "first long good" point 😭🎀 The code for the available Trial Results is: 60x1s0l+60x1s1l+1x60s.1l+1x60s0l.

+==============+==========+============+============+=============+

Goal name

RFC2544

TST009

1s final

20% exceed

+==============+==========+============+============+=============+

Goal code

60f60d0l0e

60f120d0l50e

1f120d.5l50e

60f60d0.5l20e

+--------------+----------+------------+------------+-------------+

Full-Length	60s	60s	60s	0s
High-Loss sum

+--------------+----------+------------+------------+-------------+

Full-Length	60s	60s	180s	120s
Low-Loss sum

+--------------+----------+------------+------------+-------------+

Short High-	60s	60s	0s	60s
Loss sum

+--------------+----------+------------+------------+-------------+

Short Low-Loss	60s	60s	0s	60s
sum

+--------------+----------+------------+------------+-------------+

Balancing sum

60s

15s

+--------------+----------+------------+------------+-------------+

Excess sum

60s

45s

+--------------+----------+------------+------------+-------------+

Positive	60s	0s	0s	45s
excess sum

+--------------+----------+------------+------------+-------------+

Effective	120s	60s	60s	45s
High-Loss sum

+--------------+----------+------------+------------+-------------+

Effective full	180s	120s	240s	165s
sum

+--------------+----------+------------+------------+-------------+

Effective	180s	120s	240s	165s
whole sum

+--------------+----------+------------+------------+-------------+

Missing sum

+--------------+----------+------------+------------+-------------+

Pessimistic	120s	60s	60s	45s
High-Loss sum

+--------------+----------+------------+------------+-------------+

Optimistic	66.667%	50%	25%	27.273%
exceed ratio

+--------------+----------+------------+------------+-------------+

Pessimistic	66.667%	50%	25%	27.273%
exceed ratio

+--------------+----------+------------+------------+-------------+

Classification	Upper	Lower Bound	Lower Bound	Lower Bound
result	Bound

+--------------+----------+------------+------------+-------------+

Table 6

This is the Low-Loss Trial the "TST009" goal was waiting for 🦋 This Load is now classified for all Search Goals, so the Search may end. Or, more realistically, it can focus on larger Loads only, because the three Search Goals will want an Upper Bound (unless this Load is Max Load) 💅

C.4. Conditional Throughput Computations ✨

At the end of this hypothetical Search, the "RFC2544" goal labels the Load as an Upper Bound, which makes it ineligible for Conditional Throughput computation 🙈 By contrast, the other three Search Goals treat the same Load as a Lower Bound. If it is also accepted as their Relevant Lower Bound, Conditional Throughput values can be computed for each of them 🫶

(The Load under discussion is one million frames per second.)

C.4.1. Goal 2

The Conditional Throughput is computed from a sorted list of Full-Length Trial Results. Since the "TST009" Goal Final Trial Duration is 60 seconds, only two of 122 Trials count as Full-Length Trials. One has a Trial Loss Ratio of 0%, and the other has 0.1% 🌸

Full-Length High-Loss sum is 60 seconds.
Full-Length Low-Loss sum is 60 seconds.
Full-Length sum is 120 seconds.
Subceed ratio is 50%.
Remaining sum is initially 0.5x12s = 60 seconds.
Current loss ratio is initially 100%.
For first Trial Result (duration 60s, loss 0%):
- Remaining sum is larger than zero, so we do not exit the loop.
- Set current loss ratio to this Trial's Trial Loss Ratio, which is 0%.
- Decrease the remaining sum by this Trial's Trial Effective Duration.
- New remaining sum is 60s - 60s = 0s.
For second Trial Result (duration 60s, loss 0.1%):
- Remaining sum is not larger than zero, so we exit the loop.
Current loss ratio was most recently set to 0%.
Current forwarding ratio is one minus the current loss ratio, so 100%.
Conditional Throughput is the current forwarding ratio multiplied by the Load value.
Conditional Throughput is one million frames per second. 🎉

C.4.2. Goal 3

The "1s final" has a Goal Final Trial Duration of 1 second, so all 122 Trial Results count as Full-Length Trials 🧚 They are ordered like this:

60 1-second 0% loss Trials,
1 60-second 0% loss Trial,
1 60-second 0.1% loss Trial, and
60 1-second 1% loss Trials.

The Conditional Throughput value does not depend on the order of 0% loss Trials 💜

Full-Length High-Loss sum is 60 seconds.
Full-Length Low-Loss sum is 180 seconds.
Full-Length sum is 240 seconds.
Subceed ratio is 50%.
Remaining sum is initially 0.5x240s = 120 seconds.
Current loss ratio is initially 100%.
For first 61 Trial Results (duration varies, loss 0%):
- Remaining sum is larger than zero, so we do not exit the loop.
- Set current loss ratio to this Trial's Trial Loss Ratio, which is 0%.
- Decrease the remaining sum by this Trial's Trial Effective Duration.
- New remaining sum varies.
After 61 Trials, the duration of 60x1s + 1x60s has been subtracted from 120s, leaving 0s.
For 62nd Trial Result (duration 60s, loss 0.1%):
- Remaining sum is not larger than zero, so we exit the loop.
Current loss ratio was most recently set to 0%.
Current forwarding ratio is one minus the current loss ratio, so 100%.
Conditional Throughput is the current forwarding ratio multiplied by the Load value.
Conditional Throughput is one million frames per second. 👑

C.4.3. Goal 4

The Conditional Throughput is computed from a sorted list of Full-Length Trial Results. Because the "20% exceed" Goal Final Trial Duration is 60 seconds, only two of 122 Trials count as Full-Length Trials. One has a Trial Loss Ratio of 0%, and the other has 0.1% 🍓

Full-Length High-Loss sum is 60 seconds.
Full-Length Low-Loss sum is 60 seconds.
Full-Length sum is 120 seconds.
Subceed ratio is 80%.
Remaining sum is initially 0.8x120s = 96 seconds.
Current loss ratio is initially 100%.
For first Trial Result (duration 60s, loss 0%):
- Remaining sum is larger than zero, so we do not exit the loop.
- Set current loss ratio to this Trial's Trial Loss Ratio, which is 0%.
- Decrease the remaining sum by this Trial's Trial Effective Duration.
- New remaining sum is 96s - 60s = 36s.
For second Trial Result (duration 60s, loss 0.1%):
- Remaining sum is larger than zero, so we do not exit the loop.
- Set current loss ratio to this Trial's Trial Loss Ratio, which is 0.1%.
- Decrease the remaining sum by this Trial's Trial Effective Duration.
- New remaining sum is 36s - 60s = -24s.
No more Trials (and remaining sum is not larger than zero), so we exit the loop.
Current loss ratio was most recently set to 0.1%.
Current forwarding ratio is one minus the current loss ratio, so 99.9%.
Conditional Throughput is the current forwarding ratio multiplied by the Load value.
Conditional Throughput is 999 thousand frames per second.

Due to a stricter Goal Exceed Ratio, this Conditional Throughput is smaller than the Conditional Throughput of the other two Search Goals 🥺✨

Acknowledgements 🫶

Special wholehearted gratitude and thanks to the late Al Morton for his thorough reviews filled with very specific feedback and constructive guidelines. Thank you, Al, for the close collaboration over the years, your mentorship, and your continuous, unwavering encouragement full of empathy and an energizing positive attitude. Al, you are dearly missed 💜🌈

Thanks to Gábor Lencse, Giuseppe Fioccola, Carsten Rossenhövel, and BMWG contributors for good discussions and thorough reviews, guiding and helping us to improve the clarity and formality of this document 🎀

Many thanks to the Linux Foundation FastData I/O (FD.io) project, specifically committers and contributors to the two core projects of FD.io: VPP and CSIT. It was there that the need for MLRsearch originally came up, and it was in FD.io CSIT labs where the first MLRsearch open-source code got prototyped and then, over the years, productized. Thanks also go to Alec Hothan of the OPNFV NFVbench project for a thorough review and numerous useful comments and suggestions in the earlier draft versions of this document 🌸

We are equally indebted to Mohamed Boucadair for a very thorough and detailed AD review, for providing many good comments and suggestions, and for helping us make this document complete 🧚

Our appreciation is also extended to Shawn Emery, Yoshifumi Nishida, David Dong, Nabeel Cocker, Lars Eggert, Jen Linkova, Mike Bishop, and Éric Vyncke for their reviews and valuable comments 💖

Authors' Addresses 💅

Maciek Konstantynowicz Cisco Systems Email: mkonstan@cisco.com

Vratko Polak Cisco Systems Email: vrpolak@cisco.com

📦 where this page comes from

ietf: rfc9971 2026-08
💖 girlified by claude-haiku-5-5 on 2026-10-19: a language model's rewrite of the original, under its terms