RFC 8238

Data Center Benchmarking Terminology

📅 published August 2017, from ietf

Abstract ✨

okay bestie, this informational doc is basically the glossary for data center benchmarking 🎀. It sets up definitions, explains how to measure things, and introduces new terms for testing data center network gear like switches and routers. It's the groundwork for the test methodology doc, RFC 8239, so you need this one first. A bunch of these terms could also work for network equipment outside the data center, since the tech from data centers is spreading everywhere now 🌸.

Status of This Memo 📜

This doc is not an Internet Standards Track spec. It's published just for info 🦋.

It's an IETF product and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you're curious 🧐.

For the current status, errata, and how to send feedback, go to http://www.rfc-editor.org/info/rfc8238 💌.

Copyright (c) 2017 IETF Trust and the people listed as document authors. All rights reserved 💜.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. Please actually read those, because they spell out your rights and restrictions 🙈. Any Code Components pulled out of this doc must include the Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions. They come without warranty, as described in that Simplified BSD License 🫶.

Table of Contents 🗺️

1. Introduction ....................................................4
   1.1. Requirements Language ......................................5
   1.2. Definition Format ..........................................5
2. Latency .........................................................5
   2.1. Definition .................................................5
   2.2. Discussion .................................................7
   2.3. Measurement Units ..........................................7
3. Jitter ..........................................................8
   3.1. Definition .................................................8
   3.2. Discussion .................................................8
   3.3. Measurement Units ..........................................8
4. Calibration of the Physical Layer ...............................9
   4.1. Definition .................................................9
   4.2. Discussion .................................................9
   4.3. Measurement Units ..........................................9
5. Line Rate ......................................................10
   5.1. Definition ................................................10
   5.2. Discussion ................................................10
   5.3. Measurement Units .........................................11
6. Buffering ......................................................12
   6.1. Buffer ....................................................12
        6.1.1. Definition .........................................12
        6.1.2. Discussion .........................................14
        6.1.3. Measurement Units ..................................14
   6.2. Incast ....................................................15
        6.2.1. Definition .........................................15
        6.2.2. Discussion .........................................15
        6.2.3. Measurement Units ..................................16
7. Application Throughput: Data Center Goodput ....................16
   7.1. Definition ................................................16
   7.2. Discussion ................................................16
   7.3. Measurement Units .........................................16
8. Security Considerations ........................................17
9. IANA Considerations ............................................18
10. References ....................................................18
   10.1. Normative References .....................................18
   10.2. Informative References ...................................19
Acknowledgments ...................................................20
Authors' Addresses ................................................20

1. Introduction 🌈

Traffic in the data center is NOT uniform, and it keeps changing 😭. Whatever apps are running decides what it looks like. In one data center it might be mostly east-west traffic (server to server inside the data center), while another is mostly north-south (from outside the data center to a server), and some have both. Traffic can come in bursts and can be many-to-one, many-to-many, or one-to-many. Each flow might be small and latency sensitive, or big and throughput sensitive, and it can be a mix of UDP and TCP. All of that can live in one cluster and go through a single network device at the same time 🤯.

Network devices have leaned on [RFC1242], [RFC2432], [RFC2544], [RFC2889], and [RFC3918] for benchmarking for ages 📚. Those benchmarks mostly focus on different latency stuff and the max throughput of the Device Under Test (DUT) being benchmarked. They're pretty good at measuring theoretical max throughput, forwarding rates, and latency under test conditions, but they don't reflect the real traffic patterns that can affect these devices 🙃. The data center devices this doc covers are switches and routers 💅.

Typical data center networking devices are characterized by:

  • High port density (48 ports or more) 🍓.
  • High speed (currently, up to 100 GB/s per port) 🔥.
  • High throughput (line rate on all ports for Layer 2 and/or Layer 3) 🚀.
  • Low latency (in the microsecond or nanosecond range) ⚡.
  • Low amount of buffer (in the MB range per networking device) 🧁.
  • Layer 2 and Layer 3 forwarding capability (Layer 3 not mandatory) 🦋.

This doc defines a set of definitions, metrics, and new terminology, including congestion scenarios and switch buffer analysis. It also redefines some basic definitions so they can represent a wide mix of traffic conditions ✨. The test methodologies are defined in [RFC8239] 💖.

1.1. Requirements Language 🔑

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🫶.

1.2. Definition Format 📝

  • Term to be defined (e.g., "latency") 🏷️.
  • Definition: The specific definition for the term 🎯.
  • Discussion: A brief discussion about the term, its application, and any restrictions on measurement procedures 💬.
  • Measurement Units: Methodology for measurements and units used to report measurements of the term in question, if applicable 📏.

2. Latency ⏱️

2.1. Definition 💭

Latency is the amount of time it takes a frame to transit the DUT ⏳. It's measured in units of time (seconds, milliseconds, microseconds, and so on) 🕰️. The whole point of measuring latency is to understand the impact of adding a device to the communication path 🧐.

Latency intervals can be assessed between different combinations of events, no matter what kind of switching device it is (bit forwarding, aka cut-through; or a store-and-forward device) 🔀. [RFC1242] defined latency differently for each of these device types 📖.

The traditional latency measurement definitions are:

  • FILO (First In Last Out):

The time interval starting when the end of the first bit of the input frame reaches the input port and ending when the last bit of the output frame is seen on the output port 🥇.

  • FIFO (First In First Out):

The time interval starting when the end of the first bit of the input frame reaches the input port and ending when the start of the first bit of the output frame is seen on the output port. Latency (as defined in [RFC1242]) for bit-forwarding devices uses these events 🧃.

  • LILO (Last In Last Out):

The time interval starting when the last bit of the input frame reaches the input port and the last bit of the output frame is seen on the output port 🫠.

  • LIFO (Last In First Out):

The time interval starting when the last bit of the input frame reaches the input port and ending when the first bit of the output frame is seen on the output port. Latency (as defined in [RFC1242]) for store-and-forward devices uses these events 📦.

Another way to sum up those four definitions is to look at the bit positions in the order they normally happen: input to output 👀.

  • FILO is FL (First bit Last bit) 🌷.
  • FIFO is FF (First bit First bit) 🌸.
  • LILO is LL (Last bit Last bit) 💜.
  • LIFO is LF (Last bit First bit) 🎀.

This definition, explained in this section for data center switch benchmarking, is used instead of the previous "latency" definition in RFC 1242, Section 3.8. Here's that old definition quoted 📜:

For store and forward devices: The time interval starting when the last bit of the input frame reaches the input port and ending when the first bit of the output frame is seen on the output port. For bit forwarding devices: The time interval starting when the end of the first bit of the input frame reaches the input port and ending when the start of the first bit of the output frame is seen on the output port.

To cover both device types, plus the hybrids that have shown up, switch latency measurements made under this document MUST be measured with the FILO events 🔥. FILO includes the switch's latency, the frame's latency, and the serialization delay. It's the full "whole" latency going through the DUT 🌈. For latency-sensitive apps that can work with the initial bytes of the frame, FIFO (or, for bit-forwarding devices, latency per RFC 1242) MAY be used 🦋. In all cases, the event combinations used in latency measurements MUST be reported 📣.

2.2. Discussion 💅

okay so, like I said in Section 2.1, FILO is the most important measuring definition 👑

Not every DUT is strictly cut-through or store-and-forward 🫶 Data center DUTs are often store-and-forward for smaller packet sizes and then switch over to cut-through behavior at specific larger packet sizes ✨ The packet size where that behavior changes MAY be configurable, depending on the DUT manufacturer 🎀 FILO covers both scenarios, store-and-forward and cut-through, so the threshold for the change does not matter for benchmarking 💖

LIFO can be used with store-and-forward switches, but not with cut-through switches 🙈 It gives negative latency values for larger packet sizes because LIFO removes the serialization delay 💀 So this mechanism MUST NOT be used when comparing the latencies of two different DUTs 🚫

2.3. Measurement Units 📏

The measuring methods to use for benchmarking purposes are as follows:

  1. FILO MUST be used as a measuring method, because it includes the latency of the packet 🍓 Today, applications commonly need to read the whole packet to process the information and take an action.
  2. FIFO MAY be used for certain applications able to process the data as the first bits arrive, for example with a Field-Programmable Gate Array (FPGA) 🧚
  3. LIFO MUST NOT be used, because unlike all the other methods, it subtracts the latency of the packet 😭

3. Jitter 🦋

3.1. Definition 🌸

In the context of the data center, jitter is synonymous with the common term "delay variation" 💜 It is derived from multiple measurements of one-way delay, as described in RFC 3393. The mandatory definition of "delay variation" is the Packet Delay Variation (PDV) as defined in Section 4.2 of [RFC5481]. When considering a stream of packets, the delays of all packets are subtracted from the minimum delay over all packets in the stream ✨ This helps you assess the range of delay variation (Max - Min) or a high percentile of PDV (99th percentile, for robustness against outliers) 🔥

When First-bit to Last-bit timestamps are used for delay measurement, delay variation MUST be measured using packets or frames of the same size, since the definition of latency includes the serialization time for each packet 🫠 Otherwise, if using First-bit to First-bit, the size restriction does not apply 🙌

3.2. Discussion 🧠

In addition to a PDV range and/or a high percentile of PDV, Inter-Packet Delay Variation (IPDV) as defined in Section 4.1 of [RFC5481] (differences between two consecutive packets) MAY be used to figure out how packet spacing has changed during transfer 👀 For example, you can check whether a packet stream has become closely spaced or "bursty" 💅 However, the absolute value of IPDV SHOULD NOT be used, because it "collapses" the "bursty" and "dispersed" sides of the IPDV distribution together 🙅‍♀️

3.3. Measurement Units 🎀

The measurement of delay variation is expressed in units of seconds ⏱️ A PDV histogram MAY be provided for the population of packets measured 📊

4. Calibration of the Physical Layer 🌈

4.1. Definition 🛠️

Calibration of the physical layer consists of defining and measuring the latency of the physical devices used to perform tests on the DUT 🧪

It includes the list of all physical-layer components used, as specified here:

  • Type of device used to generate traffic / measure traffic.
  • Type of line cards used on the traffic generator.
  • Type of transceivers on the traffic generator.
  • Type of transceivers on the DUT.
  • Type of cables.
  • Length of cables.
  • Software name and version of the traffic generator and DUT.
  • A list of enabled features on the DUT MAY be provided and is recommended (especially in the case of control-plane protocols, such as the Link Layer Discovery Protocol and Spanning Tree) 🫶 A comprehensive configuration file MAY be provided to this effect 📁

4.2. Discussion 💭

Calibration of the physical layer contributes to end-to-end latency and should be taken into account when evaluating the DUT 👀 Small variations in the physical components of the test may impact the latency being measured, so they MUST be described when presenting results 📝

4.3. Measurement Units 📐

It is RECOMMENDED that all cables used for testing (1) be of the same type and length and (2) come from the same vendor whenever possible 🧦 It is a MUST to document the cable specifications listed in Section 4.1, along with the test results 🚨 The test report MUST specify whether or not the cable latency has been subtracted from the test measurements 🧮 The accuracy of the traffic-generator measurements MUST be provided (for current test equipment, this is usually a value within a range of 20 ns) 🎯

5. Line Rate 🏎️

5.1. Definition 💨

The transmit timing, or maximum transmitted data rate, is controlled by the "transmit clock" in the DUT ⏰ The receive timing (maximum ingress data rate) is derived from the transmit clock of the connected interface 🔁

The line rate or physical-layer frame rate is the maximum capacity to send frames of a specific size at the transmit clock frequency of the DUT 🚀

The term "nominal value of line rate" defines the maximum speed capability for the given port, for example (expressed as Gigabit Ethernet), 1 GE, 10 GE, 40 GE, 100 GE 🌟

The frequency ("clock rate") of the transmit clock in any two connected interfaces will never be precisely the same, so a tolerance is needed 🙃 This is expressed by a Parts Per Million (PPM) value 🔢 The IEEE standards allow a specific +/- variance in the transmit clock rate, and Ethernet is designed to allow for small, normal variations between the two clock rates 🫧 This results in a tolerance of the line-rate value when traffic is generated from test equipment to a DUT 💜

Line rate SHOULD be measured in frames per second (FPS) 🎯

5.2. Discussion 🍭

For a transmit clock source, most Ethernet switches use "clock modules" (also called "oscillator modules") that are sealed, internally temperature-compensated, and very accurate 🧊 The output frequency of these modules is not adjustable because it is not necessary 🤷‍♀️ Many test sets, however, offer a software-controlled adjustment of the transmit clock rate 🎛️ These adjustments SHOULD be used to "compensate" the test equipment, so it does not send more than the line rate of the DUT 🙌

To allow for the minor variations typically found in the clock rate of commercially available clock modules and other crystal-based oscillators, Ethernet standards specify the maximum transmit clock-rate variation to be not more than +/- 100 PPM from a calculated center frequency 📏 Therefore, a DUT must be able to accept frames at a rate within +/- 100 PPM to comply with the standards ✅

Very few clock circuits are precisely +/- 0.0 PPM because:

  1. The Ethernet standards allow a maximum variance of +/- 100 PPM over time. Therefore, it is normal for the frequency of the oscillator circuits to experience variation over time and over a wide temperature range, among other external factors 🌡️
  2. The crystals, or clock modules, usually have a specific +/- PPM variance that is significantly better than +/- 100 PPM. Oftentimes, this is +/- 30 PPM or better in order to be considered a "certification instrument" 🏅

When testing an Ethernet switch throughput at "line rate", any specific switch will have a clock-rate variance 😮‍💨 If a test set is running +1 PPM faster than a switch under test and a sustained line-rate test is performed, you can observe a gradual increase in latency and, eventually, packet drops as buffers fill and overflow in the switch 📈💥 Depending on how much clock variance there is between the two connected systems, the effect may show up after the traffic stream has been running for a few hundred microseconds, a few milliseconds, or seconds ⌛ The same low latency and no packet loss can be demonstrated by setting the test set's link occupancy to slightly less than 100 percent link occupancy 🫶 Typically, 99 percent link occupancy produces excellent low latency and no packet loss 🌸 No Ethernet switch or router will have a transmit clock rate of exactly +/- 0.0 PPM, and very few (if any) test sets have a clock rate that is precisely +/- 0.0 PPM 🤯

Test-set equipment manufacturers are well aware of the standards and allow a software-controlled +/- 100 PPM "offset" (clock-rate adjustment) to compensate for normal variations in the clock speed of DUTs 🧑‍🔧 This offset adjustment lets engineers determine the approximate speed at which the connected device is operating and verify that it is within parameters allowed by standards 🔍

5.3. Measurement Units 🧮

"Line rate" can be measured in terms of "frame rate" 💖

Frame Rate = Transmit-Clock-Frequency /
   (Frame-Length*8 + Minimum_Gap + Preamble + Start-Frame Delimiter)

Minimum_Gap represents the interframe gap 🫧 This formula "scales up" or "scales down" to represent 1 GB Ethernet, 10 GB Ethernet, and so on ✨

Example for 1 GB Ethernet speed with 64-byte frames:

Frame Rate = 1,000,000,000 / (64*8 + 96 + 56 + 8)

           = 1,000,000,000 / 672

           = 1,488,095.2 FPS

Considering the allowance of +/- 100 PPM, a switch may "legally" transmit traffic at a frame rate between 1,487,946.4 FPS and 1,488,244 FPS 😳 Each 1 PPM variation in clock rate will translate to a frame-rate increase or decrease of 1.488 FPS 📉📈

In a production network, it is very unlikely that you would see precise line rate over a very brief period 🥹 There is no observable difference between dropping packets at 99% of line rate and 100% of line rate 🙃

Line rate can be measured at 100% of line rate with a -100 PPM adjustment 🎯

Line rate SHOULD be measured at 99.98% with a 0 PPM adjustment 💅

The PPM adjustment SHOULD only be used for a line-rate measurement 🔥

6. Buffering 🫧

6.1. Buffer 🧺

6.1.1. Definition ✨

okay so buffer size is the total amount of frame-buffering memory a DUT has available 🧠 we measure it in B (bytes), KB (kilobytes), MB (megabytes), or GB (gigabytes). when you report it, you also need the frame MTU (Maximum Transmission Unit) you used, plus the CoS (Class of Service) or DSCP (Differentiated Services Code Point) value, because QoS implementations often carve up the buffers 💅 see Section 3 of [RFC8239] for the deets 🎀

Example:

The Buffer Size of the DUT when sending 1518-byte frames is 18 MB.

Port Buffer Size is the amount of buffer for a single ingress port, a single egress port, or a combo of ingress and egress buffering locations for one port 🌸 we list all three locations because the DUT's buffering scheme can be unknown or untested, and knowing where the buffer lives helps explain the architecture and the total buffer size 🧩 the Port Buffer Size is an informational value that MAY be provided by the DUT vendor. it is NOT something benchmarking tests. benchmarking uses the Maximum Port Buffer Size or Maximum Buffer Size methodology instead 🦋

Maximum Port Buffer Size is usually the same as the Port Buffer Size. but on a "SoC" (switch on chip) architecture, there's a port buffer plus a shared buffer pool that every port can use 🍓 so the Maximum Port Buffer Size there is the port buffer plus the largest share of the shared buffer that this port is allowed to take. it's measured in B (bytes), KB (kilobytes), MB (megabytes), or GB (gigabytes), and it needs the frame MTU and the CoS or DSCP bit value set for the test, too 💜

Example:

A DUT has been measured to have 3 KB of port buffer for 1518-byte frames, and a total of 4.7 MB of maximum port buffer for 1518-byte frames and a CoS of 0.

Maximum DUT Buffer Size is the total buffer size a DUT can be measured to have 👑 it's most likely different from the Maximum Port Buffer Size, and it can also be different from the sum of the Maximum Port Buffer Sizes. it needs the frame MTU and the CoS or DSCP value set during the test, too ✨

Example:

A DUT has been measured to have 3 KB of port buffer for 1518-byte frames and a total of 4.7 MB of maximum port buffer for 1518-byte frames.  The DUT has a Maximum Buffer Size of 18 MB at 1500 B and a CoS of 0.

Burst is a fixed number of packets sent over a percentage of line rate for a defined port speed 🌈 the frames are evenly spread across the interval T. a constant, C, can be defined to give the average time between two evenly spaced consecutive packets ⭐

Microburst is a kind of burst where packets get dropped even though there's no sustained or noticeable congestion on the link or device 🙈 one giveaway is when the burst is NOT evenly spread over T and the gaps are less than the constant C (C = the average time between two evenly spaced consecutive packets). bestie, that's a whole vibe 💀

Intensity of Microburst is a percentage, from 1 to 100%, showing how strong the microburst is. higher number = more intense 🔥

I=[1-[ (Tp2-Tp1)+(Tp3-Tp2)+....(TpN-Tp(n-1) ] / Sum(packets)]]*100
ok but these definitions aren't here to say what the ideal buffer size is 🙅‍♀️ they're here to explain how to measure it. a bigger buffer is NOT automatically better, and it can cause bufferbloat issues 😭

6.1.2. Discussion 💅

when you measure buffering on a DUT, you have to understand what every single port is doing 👀 that gives you the data for the total buffering on the switch. the buffer efficiency terms help you figure out the best packet size for the buffer, or how much of the buffer you really get for a specific packet size 🧁 this section doesn't cover how to run the test itself. instead, it explains the buffer definitions and which metrics should be provided for complete data center device-buffering benchmarking 🫶

6.1.3. Measurement Units 📏

when you measure the DUT buffer:

  • The buffer size MUST be measured.
  • The port buffer size MAY be provided for each port.
  • The maximum port buffer size MUST be measured.
  • The maximum DUT buffer size MUST be measured.
  • The intensity of the microburst MAY be mentioned when a microburst test is performed.
  • The CoS or DSCP value set during the test SHOULD be provided.

6.2. Incast 🧲

6.2.1. Definition 💖

"Incast" is super common in the data center 🏢 it describes many-to-one or many-to-many traffic patterns. it measures the number of ingress and egress ports and how synchronized they are 🎯 usually that means lots of server ports (the "many") sending traffic to one shared uplink (many-to-one), or to multiple uplinks (many-to-many). the pattern works for any network where lots of incoming ports send traffic to one or a few uplinks 🌸

Synchronous arrival time: when two or more frames of sizes L1 and L2 arrive at their own ingress port, or at multiple ingress ports, and any of their bits overlap in time on the DUT, then L1 and L2 have synchronous arrival times 💫 this is called "Incast", whether it's many-to-one (simpler) or many-to-many 🦋

Asynchronous arrival time: literally anything that doesn't count as synchronous arrival time 🙃

Percentage of synchronization: how much overlap (in bits) there is between frames of sizes L1, L2..Ln 📊

Example:

Two 64-byte frames of length L1 and L2 arrive at ingress port 1 and port 2 of the DUT.  There is an overlap of 6.4 bytes between the two, where the L1 and L2 frames were on their respective ingress ports at the same time.  Therefore, the percentage of synchronization is 10%.

Stateful traffic is packets exchanged with a stateful protocol, like TCP 🧠

Stateless traffic is packets exchanged with a stateless protocol, like UDP 🪽

6.2.2. Discussion 🫧

in this scenario, the DUT uses its buffers 🧺 with ingress buffering, the ingress port buffers get used along with virtual output queues (when they're available). with egress buffering, the egress buffer of the one outgoing port gets used instead 🔁

no matter where the buffer memory sits in the switch architecture, Incast creates buffer utilization 💥 when one or more frames arrive at the DUT with synchronous arrival times, they're considered to be forming an Incast 🎀

6.2.3. Measurement Units 📐

It is a MUST to measure the number of ingress and egress ports.

It is a MUST to have a non-null percentage of synchronization, which MUST be specified.

7. Application Throughput: Data Center Goodput 🛍️

7.1. Definition 🌷

in data center networking, a balanced network means you get maximal throughput with minimal loss at any given time 🎯 this is captured by Goodput [TCP-INCAST]. Goodput is the application-level throughput 🧁 for standard TCP applications, even a very small loss can totally tank application throughput 😭 [RFC2647] defines Goodput, and the definition in this document is a variant of that one 💅

Goodput is the number of bits per unit of time forwarded to the correct destination interface of the DUT, minus any bits retransmitted ✨

7.2. Discussion 💬

in data center benchmarking, Goodput is a value that SHOULD be measured 📈 it gives you a realistic idea of how much of the available bandwidth is actually being used. the goal in data center environments is to maximize Goodput while minimizing loss 🫶

7.3. Measurement Units 🧮

The Goodput, G, is then measured by the following formula:

G = (S/F) x V bytes per second
  • S represents the payload bytes, not including packet or TCP headers.
  • F is the frame size.
  • V is the speed of the media in bytes per second.

Example: A TCP file transfer over HTTP on 10 GB/s media.

The file cannot be transferred over Ethernet as a single continuous stream.  It must be broken down into individual frames of 1500 B when the standard MTU is used.  Each packet requires 20 B of IP header information and 20 B of TCP header information; therefore, 1460 B are available per packet for the file transfer.  Linux-based systems are further limited to 1448 B, as they also carry a 12 B timestamp.  Finally, in this example the date is transmitted over Ethernet, which adds 26 B of overhead per packet to 1500 B, increasing it to 1526 B.
G = 1460/1526 x 10 Gbit/s, which is 9.567 Gbit/s or 1.196 GB/s.
Please note: This example does not take into consideration the additional Ethernet overhead, such as the interframe gap (a minimum of 96 bit times), nor does it account for collisions (which have a variable impact, depending on the network load).

when you do Goodput measurements, please document these in addition to the items listed in Section 4.1:

  • The TCP stack used.
  • OS versions.
  • Network Interface Card (NIC) firmware version and model.

for example, Windows TCP stacks and different Linux versions can change TCP-based test results 🪟🐧

8. Security Considerations 🔒

Benchmarking activities as described in this memo are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above.

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. 🚨 this is the big one, don't skip it 😤

Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT 🖤

Special capabilities SHOULD NOT exist in the DUT specifically for benchmarking purposes. Any implications for network security arising from the DUT SHOULD be identical in the lab and in production networks 🛡️

9. IANA Considerations 🧚

This document does not require any IANA actions.

10. References 📚

10.1. Normative References 💖

okay bestie, these are the sources the spec leans on hardest 💅

[RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, https://www.rfc-editor.org/info/rfc1242.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, https://www.rfc-editor.org/info/rfc2119.

[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, https://www.rfc-editor.org/info/rfc2544.

[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, March 2009, https://www.rfc-editor.org/info/rfc5481.

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, https://www.rfc-editor.org/info/rfc8174.

[RFC8239] Avramov, L. and J. Rapp, "Data Center Benchmarking Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, https://www.rfc-editor.org/info/rfc8239.

10.2. Informative References 🦋

these ones are the background reading. super useful, not strictly required 🤓

[RFC2432] Dubray, K., "Terminology for IP Multicast Benchmarking", RFC 2432, DOI 10.17487/RFC2432, October 1998, https://www.rfc-editor.org/info/rfc2432.

[RFC2647] Newman, D., "Benchmarking Terminology for Firewall Performance", RFC 2647, DOI 10.17487/RFC2647, August 1999, https://www.rfc-editor.org/info/rfc2647.

[RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN Switching Devices", RFC 2889, DOI 10.17487/RFC2889, August 2000, https://www.rfc-editor.org/info/rfc2889.

[RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October 2004, https://www.rfc-editor.org/info/rfc3918.

[TCP-INCAST] Chen, Y., Griffith, R., Zats, D., Joseph, A., and R. Katz, "Understanding TCP Incast and Its Implications for Big Data Workloads", April 2012, http://yanpeichen.com/professional/usenixLoginIncastReady.pdf.

Acknowledgments 🫶

the authors want to thank Al Morton, Scott Bradner, Ian Cox, and Tim Stevenson for their reviews and feedback 🌸 the real MVPs 👑

Authors' Addresses 🧚

Lucien Avramov
Google
1600 Amphitheatre Parkway
Mountain View, CA  94043
United States of America

Email: lucien.avramov@gmail.com


Jacob Rapp
VMware
3401 Hillview Ave.
Palo Alto, CA  94304
United States of America

Email: jhrapp@gmail.com
📦 where this page comes from