RFC 8239

Data Center Benchmarking Methodology

📅 published August 2017, from ietf

🎀 Data Center Benchmarking Methodology 🎀

Okay bestie, this is RFC 8239, an IETF Informational document (not Standards Track, it's published just for info 💅). It came out in August 2017 under ISSN 2070-1721. The authors are L. Avramov (Google) and J. Rapp (VMware). The title block doesn't list anything it obsoletes or updates, so it stands on its own ✨.

Abstract 🌸

Okay so the whole point of this doc is to set up a test and evaluation methodology, plus measurement techniques, for physical network equipment in the data center 🖥️. You need RFC 8238 first, because it has the terminology that's considered normative (aka the official vocab you have to use 📖). A lot of these terms and methods could be useful outside the data center too, since the tech that started there is spreading everywhere 🌍.

Status of This Memo 🧾

This doc is not an Internet Standards Track spec. It's published for informational purposes 🦋.

It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every doc the IESG approves is a candidate for any level of Internet Standard, so check out Section 2 of RFC 7841 for the deets 🤓.

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc8239 💌.

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved 💜.

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date it was published. Please read these carefully, because they spell out your rights and restrictions 📜. Code Components pulled from this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License 🫶.

Table of Contents 🗂️

1. Introduction ....................................................3
   1.1. Requirements Language ......................................4
   1.2. Methodology Format and Repeatability Recommendation ........4
2. Line-Rate Testing ...............................................4
   2.1. Objective ..................................................4
   2.2. Methodology ................................................4
   2.3. Reporting Format ...........................................5
3. Buffering Testing ...............................................6
   3.1. Objective ..................................................6
   3.2. Methodology ................................................7
   3.3. Reporting Format ...........................................9
4. Microburst Testing .............................................10
   4.1. Objective .................................................10
   4.2. Methodology ...............................................10
   4.3. Reporting Format ..........................................11
5. Head-of-Line Blocking ..........................................12
   5.1. Objective .................................................12
   5.2. Methodology ...............................................12
   5.3. Reporting Format ..........................................14
6. Incast Stateful and Stateless Traffic ..........................15
   6.1. Objective .................................................15
   6.2. Methodology ...............................................15
   6.3. Reporting Format ..........................................17
7. Security Considerations ........................................17
8. IANA Considerations ............................................17
9. References .....................................................18
   9.1. Normative References ......................................18
   9.2. Informative References ....................................18
Acknowledgments ...................................................19
Authors' Addresses ................................................19

1. Introduction ✨

Traffic patterns in the data center are not uniform, and they're constantly changing 🔄. They're dictated by the nature and variety of the apps running there. In one data center it might be mostly east-west traffic (server to server inside the DC), and in another it's mostly north-south (from outside the DC to a server). Some combine both 🌀. Traffic can be bursty and can be many-to-one, many-to-many, or one-to-many 💥. Each flow might be small and latency sensitive, or large and throughput sensitive, and it can be a mix of UDP and TCP at the same time 🧁. All of that can coexist in one cluster and go through one network device at once. Honestly, chaotic but iconic 💀.

Benchmarking tests for network devices have long used [RFC1242], [RFC2432], [RFC2544], [RFC2889], and [RFC3918]. They've mostly focused on various latency attributes and throughput [RFC2889] of the Device Under Test (DUT) being benchmarked 🎯. These standards are good at measuring theoretical throughput, forwarding rates, and latency under test conditions. BUT they don't represent the real traffic patterns that might affect these devices 😤.

Typical data center networking devices are characterized by the following 👑:

  • High port density (48 ports or more).
  • High speed (currently, up to 100 GB/s per port).
  • High throughput (line rate on all ports for Layer 2 and/or Layer 3).
  • Low latency (in the microsecond or nanosecond range).
  • Low amount of buffer (in the MB range per networking device).
  • Layer 2 and Layer 3 forwarding capability (Layer 3 not mandatory).

This doc gives a methodology for benchmarking data center physical network equipment DUTs. That includes congestion scenarios, switch buffer analysis, microbursts, and head-of-line blocking, all while using a wide mix of traffic conditions 🌈. [RFC8238] is a prerequisite for this doc, since it has the normative terminology 📚.

1.1. Requirements Language 💅

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 🎀.

1.2. Methodology Format and Repeatability Recommendation 🔁

Sections 2 through 6 of this doc follow this format 📝:

  • Objective
  • Methodology
  • Reporting Format

For every test methodology here, repeatability of the results is critical 😭. The recommendation is to run enough iterations of the test and make sure the result is consistent. This matters extra for the tests in Section 3, because buffering testing has historically been the least reliable 🙈. The number of iterations SHOULD be explicitly reported. The relative standard deviation SHOULD be below 10% 📊.

2. Line-Rate Testing 🚀

2.1. Objective 🎯

The goal here is a "maximum rate" test for the performance values of throughput, latency, and jitter 💫. It's meant to provide (1) the tests to perform and (2) a methodology for checking that a DUT can forward packets at line rate under non-congested conditions 🏁.

2.2. Methodology 🧪

A traffic generator SHOULD be connected to all ports on the DUT. Two tests MUST be conducted: (1) a port-pair test [RFC2544] [RFC3918] and (2) a test using a full-mesh DUT [RFC2889] [RFC3918] 🔌.

For all tests, the traffic generator's sending rate MUST be less than or equal to 99.98% of the nominal value of the line rate (with no further Parts Per Million (PPM) adjustment to account for interface clock tolerances). This makes sure the DUT gets stressed in reasonable worst-case conditions (see [RFC8238], Section 5 for more details) 🔥. Test results at a lower rate MAY be provided, so you can see how performance (latency and jitter) improves when the rate is below 99.98% 📉. The receiving rate of the traffic SHOULD be captured during this test as a percentage of line rate 📈.

The test MUST provide the minimum, average, and maximum of the latency distribution, for the exact same iteration of the test ⏱️.

The test MUST provide the minimum, average, and maximum of the jitter distribution, for the exact same iteration of the test 🫨.

BUT if a traffic generator can't be connected to all ports on the DUT, then a snake test MUST be used for line-rate testing. Latency and jitter get excluded there, since they'd become irrelevant 🐍. Here's how the snake test goes:

  • Connect the first and last port of the DUT to a traffic generator.
  • Connect, back to back and sequentially, all the ports in between: port 2 to port 3, port 4 to port 5, etc., up to port N-2 to port N-1, where N is the total number of ports of the DUT 🔗.
  • Configure port 1 and port 2 in the same VLAN X, port 3 and port 4 in the same VLAN Y, etc., and port N-1 and port N in the same VLAN Z 🎀.

Honestly the snake test is so clever 🧠. It lets you test line rate for Layer 2 and Layer 3 [RFC2544] [RFC3918] when you only have a traffic generator with two ports ✨. Latency and jitter are not to be considered for this test 🙅‍♀️.

2.3. Reporting Format ✨

okay so the report MUST include the following 💅

  • Physical-layer calibration information, as defined in [RFC8238], Section 4.
  • Number of ports used. 🔌
  • Reading for "throughput received as a percentage of bandwidth", while sending 99.98% of the nominal value of the line rate on each port, for each packet size from 64 bytes to 9216 bytes. As guidance, with a packet-size increment of 64 bytes between each iteration being ideal, 256-byte and 512-byte packets are also often used. The most common packet-size ordering for the report is 64 bytes, 128 bytes, 256 bytes, 512 bytes, 1024 bytes, 1518 bytes, 4096 bytes, 8000 bytes, and 9216 bytes.

The pattern for testing can be expressed using [RFC6985]. 🧩

  • Throughput needs to be expressed as a percentage of total transmitted frames. 📊
  • Packet drops MUST be expressed as a count of packets and SHOULD be expressed as a percentage of line rate. 🙈
  • For latency and jitter, values are expressed in units of time (usually microseconds or nanoseconds), reading across packet sizes from 64 bytes to 9216 bytes. ⏱️
  • For latency and jitter, provide minimum, average, and maximum values. If different iterations are done to gather the minimum, average, and maximum values, this SHOULD be specified in the report, along with a justification for why the information could not have been gathered in the same test iteration. 🫶
  • For jitter, a histogram describing the population of packets measured per latency or latency buckets is RECOMMENDED. 📈
  • The tests for throughput, latency, and jitter MAY be conducted as individual independent trials, with proper documentation provided in the report, but SHOULD be conducted at the same time. 🎀
  • The methodology assumes that the DUT has at least nine ports, as certain methodologies require nine or more ports. 🦋

3. Buffering Testing 🧺

3.1. Objective 🎯

The objective of this test is to measure the size of the buffer of a DUT under typical/many/multiple conditions. Buffer architectures between multiple DUTs can differ and include egress buffering, shared egress buffering SoC (Switch-on-Chip), ingress buffering, or a combination thereof. The test methodology covers the buffer measurement, regardless of buffer architecture used in the DUT. 💜

3.2. Methodology 🛠️

A traffic generator MUST be connected to all ports on the DUT. The methodology for measuring buffering for a data center switch is based on using known congestion of known fixed packet size, along with maximum latency value measurements. The maximum latency will increase until the first packet drop occurs. At this point, the maximum latency value will remain constant. This is the point of inflection of this maximum latency change to a constant value. There MUST be multiple ingress ports receiving a known amount of frames at a known fixed size, destined for the same egress port in order to create a known congestion condition. The total amount of packets sent from the oversubscribed port minus one, multiplied by the packet size, represents the maximum port buffer size at the measured inflection point. 🌊

Note that the tests described in procedures 1), 2), 3), and 4) in this section have iterations called "first iteration", "second iteration", and "last iteration". The idea is to show the first two iterations so the reader understands the logic of how to keep incrementing the iterations. The last iteration shows the end state of the variables. 🧠

  1. Measure the highest buffer efficiency.
    • First iteration: Ingress port 1 sending 64-byte packets at line rate to egress port 2, while port 3 is sending a known low amount of oversubscription traffic (1% recommended) with the same packet size of 64 bytes to egress port 2. Measure the buffer size value of the number of frames sent from the port sending the oversubscribed traffic up to the inflection point multiplied by the frame size.
    • Second iteration: Ingress port 1 sending 65-byte packets at line rate to egress port 2, while port 3 is sending a known low amount of oversubscription traffic (1% recommended) with the same packet size of 65 bytes to egress port 2. Measure the buffer size value of the number of frames sent from the port sending the oversubscribed traffic up to the inflection point multiplied by the frame size.
    • Last iteration: Ingress port 1 sending packets of size B bytes at line rate to egress port 2, while port 3 is sending a known low amount of oversubscription traffic (1% recommended) with the same packet size of B bytes to egress port 2. Measure the buffer size value of the number of frames sent from the port sending the oversubscribed traffic up to the inflection point multiplied by the frame size.

When the B value is found to provide the largest buffer size, then size B allows the highest buffer efficiency. 🏆

  1. Measure maximum port buffer size.

At fixed packet size B as determined in procedure 1), for a fixed default Differentiated Services Code Point (DSCP) / Class of Service (CoS) value of 0 and for unicast traffic, proceed with the following:

  • First iteration: Ingress port 1 sending line rate to egress port 2, while port 3 is sending a known low amount of oversubscription traffic (1% recommended) with the same packet size to egress port 2. Measure the buffer size value by multiplying the number of extra frames sent by the frame size.
  • Second iteration: Ingress port 2 sending line rate to egress port 3, while port 4 is sending a known low amount of oversubscription traffic (1% recommended) with the same packet size to egress port 3. Measure the buffer size value by multiplying the number of extra frames sent by the frame size.
  • Last iteration: Ingress port N-2 sending line rate to egress port N-1, while port N is sending a known low amount of oversubscription traffic (1% recommended) with the same packet size to egress port N. Measure the buffer size value by multiplying the number of extra frames sent by the frame size.

This test series MAY be repeated using all different DSCP/CoS values of traffic, and then using multicast traffic, in order to find out if there is any DSCP/CoS impact on the buffer size. 🔍

  1. Measure maximum port pair buffer sizes.
    • First iteration: Ingress port 1 sending line rate to egress port 2, ingress port 3 sending line rate to egress port 4, etc. Ingress port N-1 and port N will oversubscribe, at 1% of line rate, egress port 2 and port 3, respectively. Measure the buffer size value by multiplying the number of extra frames sent by the frame size for each egress port.
    • Second iteration: Ingress port 1 sending line rate to egress port 2, ingress port 3 sending line rate to egress port 4, etc. Ingress port N-1 and port N will oversubscribe, at 1% of line rate, egress port 4 and port 5, respectively. Measure the buffer size value by multiplying the number of extra frames sent by the frame size for each egress port.
    • Last iteration: Ingress port 1 sending line rate to egress port 2, ingress port 3 sending line rate to egress port 4, etc. Ingress port N-1 and port N will oversubscribe, at 1% of line rate, egress port N-3 and port N-2, respectively. Measure the buffer size value by multiplying the number of extra frames sent by the frame size for each egress port.

This test series MAY be repeated using all different DSCP/CoS values of traffic and then using multicast traffic. 💅

  1. Measure maximum DUT buffer size with many-to-one ports.
    • First iteration: Ingress ports 1,2,... N-1 each sending [(1/[N-1])*99.98]+[1/[N-1]] % of line rate per port to egress port N.
    • Second iteration: Ingress ports 2,... N each sending [(1/[N-1])*99.98]+[1/[N-1]] % of line rate per port to egress port 1.
    • Last iteration: Ingress ports N,1,2...N-2 each sending [(1/[N-1])*99.98]+[1/[N-1]] % of line rate per port to egress port N-1.

This test series MAY be repeated using all different CoS values of traffic and then using multicast traffic. 🌸

Unicast traffic, and then multicast traffic, SHOULD be used in order to determine the proportion of buffer for the documented selection of tests. Also, the CoS value for the packets SHOULD be provided for each test iteration, as the buffer allocation size MAY differ per CoS value. It is RECOMMENDED that the ingress and egress ports be varied in a random but documented fashion in multiple tests in order to measure the buffer size for each port of the DUT. 🎲

3.3. Reporting Format 📝

The report MUST include the following:

  • The packet size used for the most efficient buffer used, along with the DSCP/CoS value.
  • The maximum port buffer size for each port.
  • The maximum DUT buffer size.
  • The packet size used in the test.
  • The amount of oversubscription, if different than 1%.
  • The number of ingress and egress ports, along with their location on the DUT.
  • The repeatability of the test needs to be indicated: the number of iterations of the same test and the percentage of variation between results for each of the tests (min, max, avg). 🔁

The percentage of variation is a metric providing a sense of how big the difference is between the measured value and the previous values. ✨

For example, for a latency test where the minimum latency is measured, the percentage of variation (PV) of the minimum latency will indicate by how much this value has varied between the current test executed and the previous one. 🧁

PV = ((x2-x1)/x1)*100, where x2 is the minimum latency value in the
current test and x1 is the minimum latency value obtained in the
previous test.

The same formula is used for maximum and average variations measured. 💖

4. Microburst Testing 💥

4.1. Objective 🎀

The objective of this test is to find the maximum amount of packet bursts that a DUT can sustain under various configurations.

This test provides additional methodology that supplements the tests described in [RFC1242], [RFC2432], [RFC2544], [RFC2889], and [RFC3918]. 🌈

  • All bursts should be sent with 100% intensity. Note: "Intensity" is defined in [RFC8238], Section 6.1.1. 🔥
  • All ports of the DUT must be used for this test. 🫧
  • It is recommended that all ports be tested simultaneously. 👑

4.2. Methodology 🎀

Okay bestie, first things first: a traffic generator MUST be connected to all ports on the DUT 🔌 (DUT just means "device under test," aka the box we're poking at 👀). To cause congestion, two or more ingress ports MUST send bursts of packets that all head to the same egress port 💥. The simplest setup is 2 ingress ports and 1 egress port, aka the "2 to 1" setup 🦋

The burst MUST be sent at an intensity of 100% (see [RFC8238], Section 6.1.1), which means the packets go out with a minimum interpacket gap ⚡. How many packets go in the burst is a trial variable, and it keeps climbing until you measure a non-zero packet loss 📈. Then you add up the packets from all the senders to figure out the maximum microburst amount the DUT can handle 💅

It is RECOMMENDED that you vary the ingress and egress ports across multiple tests so you can measure the maximum microburst capacity 🌈

The intensity of a microburst (see [RFC8238], Section 6.1.1) MAY be varied so you can get the microburst capacity at different ingress rates 🍓

It is RECOMMENDED that all ports on the DUT get tested simultaneously, in lots of different configurations, so you understand every combo of ingress ports, egress ports, and intensities 🧚

Here's an example, because examples are everything 🫶

  • First iteration: N-1 ingress ports sending to one egress port.
  • Second iteration: N-2 ingress ports sending to two egress ports.
  • Last iteration: Two ingress ports sending to N-2 egress ports.

4.3. Reporting Format 📝

The report MUST include the following:

  • The maximum number of packets received per ingress port, with the maximum burst size that got zero packet loss ✨
  • The packet size used in the test 📦
  • The number of ingress and egress ports, plus where they're located on the DUT 📍
  • The repeatability of the test, which needs to be indicated: the number of iterations of the same test and the percentage of variation between results (min, max, avg) 📊

5. Head-of-Line Blocking 🚧

5.1. Objective 🎯

Head-of-line blocking (HOLB) is a performance-limiting drama 😤 where packets get stuck behind the first packet in line, which is waiting to go out a different output port. This is defined in RFC 2889, Section 5.5 ("Congestion Control"). This section expands on RFC 2889 for data center benchmarking 🏢

The goal of this test is to understand how the DUT behaves in a HOLB scenario and to measure the packet loss 📉

Here's how this HOLB test is different from RFC 2889 🔍

  • This HOLB test starts with eight ports in two groups of four ports each, instead of four ports (compared with Section 5.5 of RFC 2889) 👯
  • This HOLB test shifts all the port numbers by one in a second iteration. This part is new compared to the HOLB test in RFC 2889 🔄 The shifting keeps going until every port has been the first in a group. The reason is to make sure every permutation gets tested, so we catch behavior differences in the DUT's SoC 🧠
  • Another test inside this HOLB test expands the group so traffic is split across four ports instead of two (25% per port instead of 50%) 🍰
  • Section 5.3 lists requirements that supplement the requirements in RFC 2889, Section 5.5 📋

5.2. Methodology 🧩

To cause congestion in the form of HOLB, we use groups of four ports. Each group has two ingress ports and two egress ports 👭 The first ingress port MUST have two flows configured, each going to a different egress port. The second ingress port congests the second egress port by sending at line rate 🏎️ The point is to check whether the flow to the first egress port, which is not oversubscribed, sees any loss 👀

A traffic generator MUST be connected to at least eight ports on the DUT, and SHOULD be connected using all the DUT ports 🔌

Heads up: the tests in procedures 1) and 2) below have iterations called "first iteration," "second iteration," and "last iteration." We show the first two so you get the logic of how to keep incrementing, and the last one shows the end state of the variables 🎀

  1. Measure two groups with eight DUT ports. 💕
    • First iteration: Measure the packet loss for two groups with consecutive ports.
      • Group one looks like this: ingress port 1 sends 50% of its traffic to egress port 3 and 50% to egress port 4. Ingress port 2 sends line rate to egress port 4. Then you measure the loss on the traffic going from ingress port 1 to egress port 3.
      • Group two looks like this: ingress port 5 sends 50% of its traffic to egress port 7 and 50% to egress port 8. Ingress port 6 sends line rate to egress port 8. Then you measure the loss on the traffic going from ingress port 5 to egress port 7.
    • Second iteration: Repeat the first iteration by shifting all the ports from N to N+1.
      • Group one looks like this: ingress port 2 sends 50% of its traffic to egress port 4 and 50% to egress port 5. Ingress port 3 sends line rate to egress port 5. Then you measure the loss on the traffic going from ingress port 2 to egress port 4.
      • Group two looks like this: ingress port 6 sends 50% of its traffic to egress port 8 and 50% to egress port 9. Ingress port 7 sends line rate to egress port 9. Then you measure the loss on the traffic going from ingress port 6 to egress port 8.
    • Last iteration: This is when the first port of the first group is connected to the last DUT port, and the last port of the second group is connected to the seventh port of the DUT.

Then measure the loss for the traffic going from ingress port N to egress port 2, and from ingress port 4 to egress port 6 📉

  1. Measure with N/4 groups with N DUT ports. 🌟

The traffic from each ingress port gets split across four egress ports (100/4 = 25%) 🍕

  • First iteration: Expand to fully use all the DUT ports in increments of four. Repeat the method from procedure 1) with every group of ports possible on the device, and measure the loss for each port group 🧮
  • Second iteration: Shift by +1 the start of each consecutive port of the port groups 🔁
  • Last iteration: Shift by N-1 the start of each consecutive port of the port groups, and measure the loss for each port group 🏁

5.3. Reporting Format 📝

For each test, the report MUST include the following:

  • The port configuration, including the number and location of the ingress and egress ports on the DUT 📍
  • Whether HOLB was observed, in accordance with the HOLB test described in Section 5 👀
  • Percent of traffic loss 📉
  • The repeatability of the test, which needs to be indicated: the number of iterations of the same test and the percentage of variation between results (min, max, avg) 📊

6. Incast Stateful and Stateless Traffic 🌊

6.1. Objective 🎯

The goal here is to measure TCP Goodput [TCP-INCAST] and latency when you mix large and small flows 💖 The test simulates a mixed environment: stateful flows that need high goodput, and stateless flows that need low latency. Stateful flows are made with TCP traffic, and stateless flows are made with UDP traffic ✨

6.2. Methodology 🧪

To simulate how stateless and stateful traffic affect the DUT, there MUST be multiple ingress ports receiving traffic destined for the same egress port 🚦 There also MAY be a mix of stateful and stateless traffic arriving on a single ingress port 🎭 The simplest setup is two ingress ports receiving traffic destined for the same egress port 💕

One ingress port MUST maintain a TCP connection, through the ingress port, to a receiver connected to an egress port 🔗 The TCP stream MUST be sent at the maximum rate the traffic generator allows 🏎️ While the TCP traffic is flowing through the DUT, stateless traffic gets sent to a receiver on that same egress port. The stateless traffic MUST be a microburst of 100% intensity 💥

It is RECOMMENDED that you vary the ingress and egress ports across multiple tests so you can measure the maximum microburst capacity 🌈

The intensity of a microburst MAY be varied so you can get the microburst capacity at various ingress rates 🍓

It is RECOMMENDED that all ports on the DUT get used in the test 🫶

The tests below have iterations called "first iteration," "second iteration," and "last iteration." We show the first two so you understand how to keep incrementing, and the last one shows the end state of the variables 🎀

For example 👑

Stateful traffic port variation (TCP traffic):

TCP traffic needs to be generated for this test. During the iterations, the number of egress ports MAY vary as well 🦋

  • First iteration: One ingress port receiving stateful TCP traffic, and one ingress port receiving stateless traffic destined to one egress port.
  • Second iteration: Two ingress ports receiving stateful TCP traffic, and one ingress port receiving stateless traffic destined to one egress port.
  • Last iteration: N-2 ingress ports receiving stateful TCP traffic, and one ingress port receiving stateless traffic destined to one egress port.

Stateless traffic port variation (UDP traffic):

UDP traffic needs to be generated for this test. During the iterations, the number of egress ports MAY vary as well 🌸

  • First iteration: One ingress port receiving stateful TCP traffic, and one ingress port receiving stateless traffic destined to one egress port.
  • Second iteration: One ingress port receiving stateful TCP traffic, and two ingress ports receiving stateless traffic destined to one egress port.
  • Last iteration: One ingress port receiving stateful TCP traffic, and N-2 ingress ports receiving stateless traffic destined to one egress port.

6.3. Reporting Format 📝

The report MUST include the following:

  • The number of ingress and egress ports, along with the designation of stateful or stateless flow assignment 🏷️
  • Stateful flow goodput 💧
  • Stateless flow latency ⚡
  • The repeatability of the test, which needs to be indicated: the number of iterations of the same test and the percentage of variation between results (min, max, avg) 📊 periodt 🌈

7. Security Considerations 🔒✨

okay so bestie, this section is literally the most important vibe check in the whole doc 💅 benchmarking here is only for technology characterization using controlled stimuli in a lab, with dedicated address space and all the constraints from the sections above. no cap, keep it in the lab 🧪🫶

the test topology is its own independent setup, and it MUST NOT be connected to devices that could forward the test traffic into a production network or misroute traffic to the test management network. 😭 this is a hard no, periodt 🚫

also, benchmarking is "black-box" 🖤 which means it only relies on what you can observe from outside the DUT. no peeking inside 👀

special capabilities SHOULD NOT exist in the DUT just for benchmarking purposes 🎀. and any network security implications from the DUT SHOULD be identical in the lab and in production networks. the vibe should be consistent everywhere, iykyk 💖

8. IANA Considerations 🌸

this document does not require any IANA actions. zero registry drama 🧚

9. References 📚

9.1. Normative References 💜

[RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <https://www.rfc-editor.org/info/rfc1242>.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.

[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <https://www.rfc-editor.org/info/rfc2544>.

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8238] Avramov, L. and J. Rapp, "Data Center Benchmarking Terminology", RFC 8238, DOI 10.17487/RFC8238, August 2017, <https://www.rfc-editor.org/info/rfc8238>.

9.2. Informative References 🌈

[RFC2432] Dubray, K., "Terminology for IP Multicast Benchmarking", RFC 2432, DOI 10.17487/RFC2432, October 1998, <https://www.rfc-editor.org/info/rfc2432>.

[RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN Switching Devices", RFC 2889, DOI 10.17487/RFC2889, August 2000, <https://www.rfc-editor.org/info/rfc2889>.

[RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October 2004, <https://www.rfc-editor.org/info/rfc3918>.

[RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet Sizes for Additional Testing", RFC 6985, DOI 10.17487/RFC6985, July 2013, <https://www.rfc-editor.org/info/rfc6985>.

[TCP-INCAST] Chen, Y., Griffith, R., Zats, D., Joseph, A., and R. Katz, "Understanding TCP Incast and Its Implications for Big Data Workloads", April 2012, <http://yanpeichen.com/professional/usenixLoginIncastReady.pdf>.

Acknowledgments 🫶

the authors would like to thank Al Morton and Scott Bradner for their reviews and feedback. so grateful, the real MVPs 🏆✨

Authors' Addresses 💌

Lucien Avramov Google 1600 Amphitheatre Parkway Mountain View, CA 94043 United States of America

Email: lucien.avramov@gmail.com

Jacob Rapp VMware 3401 Hillview Ave. Palo Alto, CA 94304 United States of America

Email: jhrapp@gmail.com

📦 where this page comes from