RFC 7640

Traffic Management Benchmarking

📅 published September 2015, from ietf

Traffic Management Benchmarking 🎀

Title block, the tea ☕💅

  • Who wrote it: B. Constantine (JDSU) and R. Krishnan (Dell Inc.) 👩‍💻
  • Published by: the Internet Engineering Task Force (IETF) 🌈
  • RFC number: 7640 ✨
  • Category: Informational 🦋
  • ISSN: 2070-1721
  • Date: September 2015 📅
  • Obsoletes or updates: the title block doesn't list anything, so this RFC doesn't obsolete or update any other document. 🙅‍♀️

Abstract 🌸

okay so this framework is basically a practical way to benchmark the traffic management capabilities of networking devices (think policing, shaping, etc.) 💅. The goals are a repeatable test method that objectively compares how well a device's traffic management performs, plus a way to benchmark traffic management using representative application traffic 🦋.

Status of This Memo 🌷

this document is not an Internet Standards Track specification 🙈. It's published for informational purposes, so no pressure 🫶.

It is a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every document the IESG approves is a candidate for any level of Internet Standard, so check Section 2 of RFC 5741 if you want the details 📚.

Info about the current status of this document, any errata, and how to give feedback lives at http://www.rfc-editor.org/info/rfc7640 🔗.

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they describe your rights and restrictions for this document 📜. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 🫧.

Table of Contents 📖

   1. Introduction ....................................................3
      1.1. Traffic Management Overview ................................3
      1.2. Lab Configuration and Testing Overview .....................5
   2. Conventions Used in This Document ...............................6
   3. Scope and Goals .................................................7
   4. Traffic Benchmarking Metrics ...................................10
      4.1. Metrics for Stateless Traffic Tests .......................10
      4.2. Metrics for Stateful Traffic Tests ........................12
   5. Tester Capabilities ............................................13
      5.1. Stateless Test Traffic Generation .........................13
           5.1.1. Burst Hunt with Stateless Traffic ..................14
      5.2. Stateful Test Pattern Generation ..........................14
           5.2.1. TCP Test Pattern Definitions .......................15
   6. Traffic Benchmarking Methodology ...............................17
      6.1. Policing Tests ............................................17
           6.1.1. Policer Individual Tests ...........................18
           6.1.2. Policer Capacity Tests .............................19
                  6.1.2.1. Maximum Policers on Single Physical Port ..20
                  6.1.2.2. Single Policer on All Physical Ports ......22
                  6.1.2.3. Maximum Policers on All Physical Ports ....22
      6.2. Queue/Scheduler Tests .....................................23
           6.2.1. Queue/Scheduler Individual Tests ...................23
                  6.2.1.1. Testing Queue/Scheduler with
                           Stateless Traffic .........................23
                  6.2.1.2. Testing Queue/Scheduler with
                           Stateful Traffic ..........................25
           6.2.2. Queue/Scheduler Capacity Tests .....................28
                  6.2.2.1. Multiple Queues, Single Port Active .......28
                           6.2.2.1.1. Strict Priority on
                                      Egress Port ....................28
                           6.2.2.1.2. Strict Priority + WFQ on
                                      Egress Port ....................29
                  6.2.2.2. Single Queue per Port, All Ports Active ...30
                  6.2.2.3. Multiple Queues per Port, All
                           Ports Active ..............................31
      6.3. Shaper Tests ..............................................32
           6.3.1. Shaper Individual Tests ............................32
                  6.3.1.1. Testing Shaper with Stateless Traffic .....33
                  6.3.1.2. Testing Shaper with Stateful Traffic ......34
           6.3.2. Shaper Capacity Tests ..............................36
                  6.3.2.1. Single Queue Shaped, All Physical
                           Ports Active ..............................37
                  6.3.2.2. All Queues Shaped, Single Port Active .....37
                  6.3.2.3. All Queues Shaped, All Ports Active .......39

      6.4. Concurrent Capacity Load Tests ............................40
   7. Security Considerations ........................................40
   8. References .....................................................41
      8.1. Normative References ......................................41
      8.2. Informative References ....................................42
   Appendix A. Open Source Tools for Traffic Management Testing ......44
   Appendix B. Stateful TCP Test Patterns ............................45
   Acknowledgments ...................................................51
   Authors' Addresses ................................................51

1. Introduction 💖

Traffic management (policing, shaping, and friends) is becoming a seriously important piece of implementing network Quality of Service (QoS) ✨.

Right now there's no framework for benchmarking these features, even though some standards cover specific areas, as described in Section 1.1 🥺.

This document gives you a framework for running repeatable traffic management benchmarks on devices and systems in a lab environment 🧪.

To be specific, the framework defines methods for characterizing the capacity of these traffic management features in network devices: classification, policing, queuing/scheduling, and traffic shaping 👑.

You can also use this benchmarking framework as a test procedure to help tune traffic management parameters before service activation 🎯. On top of Layer 2/3 (Ethernet/IP) benchmarking, this document proposes Layer 4 (TCP) test patterns so the benchmarks reflect real end-user traffic more realistically 🫶.

1.1. Traffic Management Overview 🦋

Generally speaking, a device with traffic management capabilities does the following:

  • Traffic classification 🏷️: identifies traffic according to various configuration rules (for example, IEEE 802.1Q Virtual LAN (VLAN), Differentiated Services Code Point (DSCP)) and marks that traffic internally to the network device. Multiple external priorities (DSCP, 802.1p, etc.) can map to the same priority in the device.
  • Traffic policing 🚦: limits the rate of traffic entering a network device according to the traffic classification. If traffic exceeds the provisioned limits, it's either dropped or remarked and forwarded to the next network device 💀.
  • Traffic scheduling 📬: provides traffic classification within the network device by directing packets into various types of queues, then applies a dispatching algorithm to decide the forwarding sequence of packets.
  • Traffic shaping 🧘‍♀️: controls traffic by actively buffering and smoothing the output rate, trying to adapt bursty traffic to the configured limits.
  • Active Queue Management (AQM) 🌊: monitors the status of internal queues and proactively drops (or remarks) packets, which makes hosts using congestion-aware protocols "back off" and ease queue congestion [RFC7567]. Classic traffic management techniques, on the other hand, reactively drop (or remark) packets based on queue-full conditions. The benchmarking scenarios for AQM are different and are outside the scope of this testing framework 🙅‍♀️.

Even though AQM is outside this framework's scope, the TCP metrics and TCP test patterns (defined in Sections 4.2 and 5.2, respectively) could be useful for testing new AQM algorithms aimed at alleviating "bufferbloat" 🍓. Examples of these algorithms include Controlled Delay [CoDel] and Proportional Integral controller Enhanced [PIE] 🧠.

Here's a generic model of the traffic management capabilities within a network device. It isn't meant to cover every manufacturer's variation, but it gives you the context for this test framework 🖼️.

 |----------|   |----------------|   |--------------|   |----------|
 |          |   |                |   |              |   |          |
 |Interface |   |Ingress Actions |   |Egress Actions|   |Interface |
 |Ingress   |   |(classification,|   |(scheduling,  |   |Egress    |
 |Queues    |   | marking,       |   | shaping,     |   |Queues    |
 |          |-->| policing, or   |-->| active queue |-->|          |
 |          |   | shaping)       |   | management,  |   |          |
 |          |   |                |   | remarking)   |   |          |
 |----------|   |----------------|   |--------------|   |----------|

Figure 1: Generic Traffic Management Capabilities of a Network Device

Ingress actions like classification are defined in [RFC4689] and cover things like IP addresses, port numbers, and DSCP 🔍. For marking, [RFC2697] and [RFC2698] define a Single Rate Three Color Marker and a Two Rate Three Color Marker, respectively 🌈.

The Metro Ethernet Forum (MEF) specifies policing and shaping in terms of ingress and egress subscriber/provider conditioning functions, as described in MEF 12.2 [MEF-12.2]. It also specifies ingress and bandwidth profile attributes in MEF 10.3 [MEF-10.3] and MEF 26.1 [MEF-26.1] 💜.

1.2. Lab Configuration and Testing Overview ✨

okay so first, the lab setup 🧪 bestie this diagram is giving main character energy:

  +--------------+     +-------+     +----------+    +-----------+
  | Transmitting |     |       |     |          |    | Receiving |
  | Test Host    |     |       |     |          |    | Test Host |
  |              |-----| Device|---->| Network  |--->|           |
  |              |     | Under |     | Delay    |    |           |
  |              |     | Test  |     | Emulator |    |           |
  |              |<----|       |<----|          |<---|           |
  |              |     |       |     |          |    |           |
  +--------------+     +-------+     +----------+    +-----------+

          Figure 2: Lab Setup for Traffic Management Tests

so the framework covers unidirectional AND bidirectional traffic management tests 🔄 (in the bidirectional ones the transmitting and receiving roles just swap on the return path). iconic 💅

this framework describes the tests and metrics for each of these traffic management functions:

  • Classification
  • Policing
  • Queuing/scheduling
  • Shaping

the tests are split into individual tests and rated capacity tests 🎀. the individual tests benchmark each traffic management function using the metrics in Section 4. the capacity tests check how those functions hold up when a bunch of individual tests and their flows are running at the same time 🔥

that means testing multiple interfaces at once with the specific traffic management function turned on, and cranking the load up to the capacity limit of each interface. literally so intense 😭

for example, say a device is specified to shape on all of its egress ports 🚀. first you run the individual test to benchmark the shaping against the metrics in Section 4. then you run the capacity test to check shaping on every interface at the same time with maximum traffic load. main character moment 👑

the Network Delay Emulator (NDE) is REQUIRED for TCP stateful tests so that TCP can use a TCP window of significant size in its control loop 🌊. the NDE needs to be the right kind of thing for this. it's really important 🫶

also, the NDE SHOULD be passive in nature (e.g., a fiber spool) 🧵. this is recommended so the NDE doesn't mess with the test flows the way an active delay element (a test impairment generator) might. if a fiber spool isn't practical because of the latency you want, then an active NDE MUST be independently verified to add the configured delay without loss. in other words, you take the DUT out of the picture and benchmark the NDE on its own 🧪✨

and the NDE SHOULD be used ONLY as emulated delay. most NDEs can do per-flow delay actions to emulate QoS prioritization, but for this framework the NDE's only job is to add delay to all packets (emulate network latency) 🙅‍♀️. so to benchmark the NDE, you test the maximum offered load against these frame sizes: 128, 256, 512, 768, 1024, 1500, and 9600 bytes. the delay accuracy at each of those packet sizes can then be used to calibrate the range of expected Bandwidth-Delay Product (BDP) for the TCP stateful tests 📏💖

2. Conventions Used in This Document 💅

the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 📖 (these are the BCP 14 words, so read them carefully bestie)

here are the acronyms used in this doc 🧠:

   AQM: Active Queue Management

   BB: Bottleneck Bandwidth

   BDP: Bandwidth-Delay Product

   BSA: Burst Size Achieved

   CBS: Committed Burst Size
   CIR: Committed Information Rate

   DUT: Device Under Test

   EBS: Excess Burst Size

   EIR: Excess Information Rate

   NDE: Network Delay Emulator

   QL: Queue Length

   QoS: Quality of Service

   RTT: Round-Trip Time

   SBB: Shaper Burst Bytes

   SBI: Shaper Burst Interval

   SP: Strict Priority

   SR: Shaper Rate

   SSB: Send Socket Buffer

   SUT: System Under Test

   Ti: Transmission Interval

   TTP: TCP Test Pattern

   TTPET: TCP Test Pattern Execution Time

3. Scope and Goals 🎀

the scope of this work is to develop a framework for benchmarking and testing the traffic management capabilities of network devices in the lab environment 🧪. these network devices may include, but are not limited to:

  • Switches (including Layer 2/3 devices)
  • Routers
  • Firewalls
  • General Layer 4-7 appliances (Proxies, WAN Accelerators, etc.)

basically, any network device that does traffic management as defined in Section 1.1 can be benchmarked or tested with this framework 🌸

the main goal is to measure the maximum forwarding performance that's still within the provisioned traffic limits a network device can sustain without dropping or impairing packets, and without messing up the accuracy of multiple traffic management functions running together 📊. this is the benchmark you use to compare devices. iykyk 🦋

inside this framework, the metrics are defined for each traffic management test, but they do NOT come with pass/fail criteria, because that's not within the charter of the BMWG 🙅‍♀️. the framework gives you the test methods and metrics for repeatable testing, which lets you compare measured performance between DUTs. so repeatable is the whole vibe 🔁💜

as mentioned in Section 1.2, these methods describe the individual tests and metrics for several management functions. it's also in scope for this framework to benchmark each function in terms of overall rated capacity 💪. that means concurrent testing of multiple interfaces with the specific traffic management function enabled, up to the capacity limit of each interface.

it is NOT within the scope of this framework to specify the procedure for testing multiple configurations of traffic management functions concurrently 🙈. the combinations are almost unbounded, and finding functional "break points" would be almost impossible ☠️

however, Section 6.4 has suggestions for some profiles of concurrent functions that would be useful to benchmark ✨. the key requirement for any concurrent test function is that tests MUST produce reliable and repeatable results 🫶

also, it's NOT in scope to do conformance testing 🚫. the tests in this framework benchmark the traffic management functions according to the metrics in Section 4, and they do not address conformance to any standards related to traffic management.

the current specs don't pin down exact behavior or implementation, and the specs that DO exist (cited in Section 1.1) let implementations vary when it comes to short-term rate accuracy and other factors 🌈. this is a primary driver for the framework: giving us an objective way to compare vendor traffic management functions. this is the whole reason this exists 💅

another goal is to come up with methods that use flows with congestion-aware transport (TCP) as part of the traffic load, while still producing repeatable results in the isolated test environment 🧚. this framework will derive stateful test patterns (TCP or application layer) that can further benchmark applicable traffic management techniques like queuing/scheduling and traffic shaping. when the network device is stateful (like a firewall, for example), it's important to test stateful test pattern traffic, along with stateless UDP traffic in specific test scenarios (like applications using TCP transport alongside UDP VoIP) 🔥

as mentioned earlier in this document, repeatability is critical, especially with stateful TCP traffic 😭. so the stateful tests will use TCP test patterns to emulate applications. the framework also gives guidelines for application modeling and open source tools to get that repeatable stimulus 🛠️. last but not least, TCP metrics from [RFC6349] MUST be measured for each stateful test, which gives you a way to compare each repeated test 📈💖

even though this framework targets testing TCP applications (like web, email, database, etc.), it could also be applied to the Stream Control Transmission Protocol (SCTP) in terms of test patterns 🦋. WebRTC, Signaling System 7 (SS7) signaling, and 3GPP are SCTP-based applications that could be modeled with this framework to benchmark SCTP's effect on traffic management performance 🌟

and fyi, at the time of writing, this framework does NOT address tcpcrypt (encrypted TCP) test patterns 🔒. that said, the metrics in Section 4.2 can still be used, because they're based on TCP retransmission and RTT measurements (not on the payload). so if tcpcrypt becomes popular, it would be natural for benchmarkers to consider encrypted TCP patterns and add them to test cases 👑

4. Traffic Benchmarking Metrics 🦋

okay so the metrics you measure during the benchmarks are split into two (2) sections 📊: packet-layer metrics for the stateless traffic testing, and TCP-layer metrics for the stateful traffic testing. two categories, zero drama 💅

4.1. Metrics for Stateless Traffic Tests ✨

okay so stateless traffic tests need a sequence number and a timestamp stuffed into the payload so we can do lost-packet analysis 📦. For delay analysis you can either put the timestamps right in the packets or store them somewhere else, like packet captures 🦋. This framework doesn't pick a packet format for carrying the sequence or timing info, so that's up to you 💅.

That said, [RFC4737] and [RFC4689] give recommendations for sequence tracking, and they also define in-sequence and out-of-order packets. Major reading material 📚🎀.

The following metrics MUST be measured during the stateless traffic benchmarking components of the tests 🔥:

  • Burst Size Achieved (BSA): For the traffic policing and network queue tests, the tester gets set up to send bursts that test either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of a policer, or the queue/buffer size configured in the DUT 🌸. BSA measures the actual burst size that shows up at the DUT's egress port with no lost packets. For example, say a DUT's configured CBS is 64 KB, but after the burst test only a 63 KB burst gets through without packet loss. Then 63 KB is the BSA 🍓. You should also record the average Packet Delay Variation (PDV) (see below) that packets experienced at the BSA burst size. This metric SHALL be reported in units of bytes, KB, or MB 💜.
  • Lost Packets (LP): For all traffic management tests, the tester sends test packets into the DUT ingress port, and then counts how many packets come out the egress port 📊. The difference between packets sent into the ingress port and packets received at the egress port is the number of lost packets, as measured at the egress port 😭. These packets must have unique identifiers so that only the test packets get measured. When multiple flows go from the ingress port to the egress port (like IP conversations), each flow must have sequence numbers within its own stream of test packets 🫶.

[RFC6703] and [RFC2680] talk about needing to set a time threshold to wait before a packet gets declared as lost ⏳. This threshold MUST be reported, and the results have to be reported as an integer that cannot be negative 🙈.

  • Out-of-Sequence (OOS): On top of LP, the test packets also have to be watched for sequence 👀. [RFC4689] defines the general function of sequence tracking, plus the definitions for in-sequence and out-of-order packets. Out-of-order packets get counted per [RFC4737]. This metric SHALL be reported as an integer that cannot be negative 🌈.
  • Packet Delay (PD): PD is the difference between the timestamp of the packets received at the egress port and the timestamp of the packets sent into the ingress port, as specified in [RFC1242] ⭐. The transmitting host and the receiving host clocks must be in time sync, which you can get with NTP, GPS, etc. ⏰. This metric SHALL be reported as a real number of seconds, and a negative measurement usually means there's a time synchronization problem between the test devices 💀.
  • Packet Delay Variation (PDV): PDV is the variation between the timestamps of the packets received at the egress port, as specified in [RFC5481] 🦋. Per [RFC5481], PDV is the variation of one-way delay across many packets in the traffic flow. When you apply the measurement formula from [RFC5481], select the high percentile of 99%. The units of measure will be a real number of seconds, and a negative value isn't possible for PDV (it would mean a measurement error) 🧚.
  • Shaper Rate (SR): SR is the average DUT output rate (bps) over the test interval 💖. It only applies to the traffic-shaping tests 🎀.
  • Shaper Burst Bytes (SBB): A traffic shaper sends packets in "trains" of different sizes, and these frames go out "back-to-back" with respect to the mandatory interframe gap 🚂. This metric describes how the shaper emits traffic. Some shapers send larger bursts per interval, and a burst of one packet covers the less common case where a shaper sends a constant-bitrate stream of single packets 💅. This metric SHALL be reported in units of bytes, KB, or MB. SBB only applies to the traffic-shaping tests 🌸.
  • Shaper Burst Interval (SBI): SBI is the time between bursts emitted by the shaper, measured at the DUT egress port ⏱️. This metric SHALL be reported as a real number of seconds. SBI only applies to the traffic-shaping tests 🫶.

4.2. Metrics for Stateful Traffic Tests 🦋

The stateful metrics are based on [RFC6349] TCP metrics, and they MUST include the following 👑:

  • TCP Test Pattern Execution Time (TTPET): [RFC6349] defined the TCP Transfer Time for bulk transfers, which is basically the measured time to transfer bytes across single or concurrent TCP connections ⏳. The TCP test patterns used in traffic management tests cover bulk transfer and interactive applications. The interactive patterns include things like HTTP business applications and database applications 💼. TTPET is the measure of the time for one execution of a TCP Test Pattern (TTP). The average, minimum, and maximum times get measured or calculated and expressed as a real number of seconds 📈.

Here's an example, because it helps so much 🙈. Say an interactive HTTP TTP session should take 5 seconds on a GigE network with 0.5-millisecond latency. Across ten (10) executions of this TTP, the TTPET results might show an average of 6.5 seconds, a minimum of 5.0 seconds, and a maximum of 7.9 seconds 🍓.

  • TCP Efficiency: After the TTP runs, TCP Efficiency is the percentage of bytes that were NOT retransmitted 💅. Here's the formula:
                         Transmitted Bytes - Retransmitted Bytes
     TCP Efficiency % =  ---------------------------------------  X 100
                                  Transmitted Bytes

"Transmitted Bytes" is the total number of TCP bytes to be transmitted, counting both the original bytes and the retransmitted bytes 🔁. To avoid any mix-up where a reordered packet gets mistaken for a retransmitted one (which can happen with packet decode interpretation), these retransmitted bytes should be recorded from the perspective of the sender's TCP/IP stack 🧠.

  • Buffer Delay: Buffer Delay is the increase in RTT during a TCP test compared to the baseline DUT RTT (the non-congested, inherent latency) 🌊. RTT and the technique for measuring it (average versus baseline) are defined in [RFC6349]. Referencing [RFC6349], the average RTT is the total of all the RTTs measured during the actual test, sampled every second, divided by the test duration in seconds ✨.
                                      Total RTTs during transfer
     Average RTT during transfer =  ------------------------------
                                     Transfer duration in seconds
                     Average RTT during transfer - Baseline RTT
   Buffer Delay % =  ------------------------------------------  X 100
                                 Baseline RTT

Even though [RFC6349] doesn't explicitly say so, retransmitted packets should not be used in RTT measurements 🙅‍♀️. Also, the test results should record the average RTT in milliseconds across the entire test duration, along with the number of samples 📝.

5. Tester Capabilities 🎀

The testing capabilities of the traffic management test environment are split into two (2) sections: stateless traffic testing and stateful traffic testing 💖.

5.1. Stateless Test Traffic Generation 🚀

The test device MUST be capable of generating traffic at up to the link speed of the DUT 🔥. The test device must be calibrated to verify that it won't drop any packets 🧐. The test device's inherent PD and PDV must also be calibrated and subtracted from the PD and PDV metrics 🧮. The test device must support the encapsulation being tested, for example IEEE 802.1Q VLAN, IEEE 802.1ad Q-in-Q, or Multiprotocol Label Switching (MPLS) 🏷️. The test device must also let you control the classification techniques defined in [RFC4689] (for example IP address, DSCP, or classification of Type of Service) 🌈.

The open source tool "iperf" can generate stateless UDP traffic and is discussed in Appendix A 🛠️. Since iperf is a software-based tool, there are performance limitations at higher link speeds (like 1 GigE or 10 GigE) 😬. Careful calibration of any test environment using iperf is important 💅. At higher link speeds, using hardware-based packet test equipment is recommended 💜.

5.1.1. Burst Hunt with Stateless Traffic 🦋

A central theme for the traffic management tests is benchmarking the specified burst parameter of a traffic management function, since burst parameters listed in Service Level Agreements (SLAs) are specified in bytes 📏. For testing efficiency, including a burst hunt feature is recommended, because it automates the manual process of figuring out the maximum burst size a traffic management function can support 🤖.

The burst hunt algorithm should start at the target burst size (the maximum burst size supported by the traffic management function) and send single bursts until it can determine the largest burst that can pass without loss 🎯. If the target burst size passes, the test is complete ✅. The "hunt" part kicks in when the target burst size doesn't make it: the algorithm drops down to a configured minimum burst size and incrementally increases the burst until it finds the maximum burst the DUT supports 🕵️‍♀️. The recommended granularity for the incremental burst size increase is 1 KB 📐.

For a policer function, if the burst size passes, the burst should be increased in increments of 1 KB to verify that the policer is truly configured properly (or enabled at all) 🧐.

5.2. Stateful Test Pattern Generation 💅

okay so the TCP test host is giving a lot of the same energy as the TCP test host from [RFC6349] 🎀 The TCP test device can be a regular computer or a dedicated communications test instrument, but either way it must be capable of emulating both a client and a server ✨

For any test using stateful TCP test traffic, the Network Delay Emulator (the NDE function, as shown in the lab setup diagram in Section 1.2) must be used in order to provide a meaningful BDP 🧚 As we talked about in Section 1.2, the target traffic rate and configured RTT MUST be verified independently, using just the NDE for all stateful tests. This makes sure the NDE can add delay without inducing any packet loss 🫶

The TCP test host MUST be capable of generating and receiving stateful TCP test traffic at the full link speed of the DUT. As a general rule of thumb, testing TCP throughput at rates greater than 500 Mbps may require high-performance server hardware or dedicated hardware-based test tools 💀

The TCP test host MUST allow the adjustment of both Send and Receive Socket Buffer sizes. The Socket Buffers must be large enough to fill the BDP for bulk transfer of TCP test application traffic 🌸

Measuring RTT and retransmissions per connection will generally require a dedicated communications test instrument. Without dedicated hardware-based test tools, these measurements may need to be conducted with packet capture tools. In other words, you run the TCP throughput tests, then analyze the RTT and retransmissions in the packet captures 📸

The TCP implementation used by the test host MUST be specified in the test results (e.g., TCP New Reno, TCP options supported). Additionally, the test results SHALL provide specific congestion control algorithm details, as per [RFC3148] 📋

While [RFC6349] defined how to run throughput tests for TCP bulk transfers, this traffic management framework takes TCP test execution into interactive TCP application traffic. Examples include email, HTTP, and business applications 💜 This interactive traffic is bidirectional and can be chatty, meaning there are many turns in the traffic communication during a single transaction. Bulk transfer is much more of a one-way flow, so it's a totally different vibe 🦋

The test device must not only support bulk TCP transfer application traffic but MUST also support chatty traffic. A valid stress test SHOULD include both traffic types. That's because chatty applications are non-uniform and bursty, while bulk transfers are relatively uniform (the bulk transfer smoothly settles into an equilibrium state under lossless conditions) 🔥

iperf is an excellent choice for TCP bulk transfer testing, but the "netperf" open source tool lets you control client and server request/response behavior. The netperf-wrapper tool is a Python script that runs multiple simultaneous netperf instances and aggregates the results 🌈 Appendix A gives an overview of netperf/netperf-wrapper, as well as iperf. As with any software-based tool, the performance must be qualified to the link speed being tested. Hardware-based test equipment should be considered for reliable results at higher link speeds (e.g., 1 GigE, 10 GigE) ⭐

5.2.1. TCP Test Pattern Definitions

As mentioned in the goals of this framework, techniques are defined to specify TCP traffic test patterns. These benchmark traffic management techniques and produce repeatable results 🎀 Some network devices, such as firewalls, will not process stateless test traffic. This is another reason why stateful TCP test traffic must be used 🙈

An application could be fully emulated up to Layer 7, but this framework proposes using stateful TCP test patterns instead. That gives granular, repeatable control over the benchmarks 💖 Here's a simple web-browsing (HTTP) example:

                             GET URL

             Client      ------------------------->   Web
                                                  |
             Web             200 OK        100 ms |
                                                  |
             Browser     <-------------------------   Server

            Figure 3: Simple Flow Diagram for a Web Application

So the Client Web Browser (client) requests a URL, and then the Web Server delivers the web page content back to the client, after a server delay of 100 milliseconds 🧚 This asynchronous "request/response" behavior is basically baked into most TCP-based applications. Think email (SMTP), file transfers (FTP and Server Message Block (SMB)), databases (SQL), web applications (SOAP), and Representational State Transfer (REST) 🌸 The impact on network elements comes from tons of clients and a mix of bursty traffic, which really stresses traffic management functions. The actual emulation of specific application protocols is not required. TCP test patterns can be defined to mimic the application network traffic flows and still produce repeatable results ✨

[3GPP2-C_R1002-A] proposed application modeling techniques. It gives examples of how to model HTTP, FTP, and Wireless Application Protocol (WAP) applications at the TCP layer 🦋 These models use various mathematical distributions for the request/response bytes and inter-request gap times. The model definition formats described in [3GPP2-C_R1002-A] are the basis for the guidelines in Appendix B, and they're also similar to formats used by network modeling tools 💅 Packet captures can also be used to characterize application traffic and to specify some of the test patterns listed in Appendix B 📸

This framework doesn't lock you into a fixed set of TCP test patterns, but it does provide test cases that SHOULD be performed. See Appendix B 💜 Some of these examples reflect those in [CA-Benchmark], which suggests traffic mixes for a variety of representative application profiles. Others are simply well-known application traffic types, like HTTP 🌈

6. Traffic Benchmarking Methodology 🔥

The traffic benchmarking methodology uses the test setup from Section 1.2 and the metrics defined in Section 4 👑

Each test SHOULD compare the network device's internal statistics (available via command line management interface, SNMP, etc.) to the measured metrics defined in Section 4. This checks how accurate the internal traffic management counters are under individual test conditions and capacity test conditions, as defined in Sections 4.1 and 4.2 🧐 This comparison isn't about real-time statistics. It's about the cumulative statistics reported after the test has finished and the device counters have updated. (It's super common for device counters to update after an interval of 10 seconds or more ⏳)

From a device configuration standpoint, scheduling and shaping functionality can be applied to logical ports (e.g., Link Aggregation (LAG)). This results in the same scheduling and shaping configuration being applied to all of the member physical ports. This document only focuses on tests at a physical-port level 🎀

The following sections provide the objective, procedure, metrics, and reporting format for each test. For all test steps, the following global parameters must be specified:

  • Test Runs (Tr): The number of times the test needs to be run to ensure accurate and repeatable results. The recommended value is a minimum of 10 🍓
  • Test Duration (Td): The duration of a test iteration, expressed in seconds. The recommended minimum value is 60 seconds ⭐

The variability in the test results MUST be measured between test runs. If the variation is a significant portion of the measured values, the next step may be to revise the methods to achieve better consistency 🦋

6.1. Policing Tests

A policer is defined as the entity performing the policy function 🚨 The whole point of policing tests is to verify the policer performance (i.e., CIR/CBS and EIR/EBS parameters). The tests verify that the network device can handle the CIR with CBS and the EIR with EBS. They use back-to-back packet-testing concepts from [RFC2544], but adapted to burst size algorithms and terminology 💖 [MEF-14], [MEF-19], and [MEF-37] provide some bases for specific components of this test. The burst hunt algorithm defined in Section 5.1.1 can also be used to automate the measurement of the CBS value 🧚

The tests are split into two (2) sections: individual policer tests, and then full-capacity policing tests. It's important to benchmark the basic functionality of the individual policer first, and then move on to the fully rated capacity of the device 👑 That capacity may include the number of policing policies per device and the number of policers simultaneously active across all ports 🌈

6.1.1. Policer Individual Tests 💅

Objective: Okay bestie, we're testing a policer as defined by [RFC4115] or [MEF-10.3], depending on what the equipment's spec says 🎀. Checking that the policer lets the specified CBS and EBS bursts through is only part of it. The policer test MUST also verify that the policer will remark or drop excess packets, and pass traffic at the specified CBS/EBS values ✨.

Test Summary: Policing tests should use stateless traffic 🫶. Stateful TCP test traffic gets lowkey wrecked by a policer when there's no traffic shaping. So even though TCP could be used, benchmarking a policer with stateless traffic is more accurate 💀.

As an example of a policer as defined by [RFC4115], imagine a CBS/EBS of 64 KB and a CIR/EIR of 100 Mbps on a 1 GigE physical link (in color-blind mode) 🍓. A stateless traffic burst of 64 KB gets sent into the policer at the GigE rate. That's roughly a 0.512-millisecond burst time (64 KB at 1 GigE). The traffic generator has to space these bursts so the aggregate throughput doesn't go over the CIR. The Ti between bursts works out to CBS * 8 / CIR = 5.12 milliseconds in this example 🦋.

Test Metrics: The metrics in Section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL be measured at the egress port and recorded 📊.

Procedure:

  1. Configure the DUT policing parameters for the desired CIR/EIR and CBS/EBS values to be tested ⚙️.
  2. Configure the tester to generate a stateless traffic burst equal to CBS and an interval equal to Ti (CBS in bits/CIR) 🧚.
  3. Compliant Traffic Test: Generate bursts of CBS + EBS traffic into the policer ingress port, and measure the metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV) at the egress port and across the entire Td (default 60-second duration) ⏱️.
  4. Excess Traffic Test: Generate bursts of greater than CBS + EBS bytes into the policer ingress port, and verify that the policer only allowed the BSA bytes to exit the egress 👀. The excess burst MUST be recorded. The recommended value is 1000 bytes. Additional tests beyond the simple color-blind example might include color-aware mode, configurations where EIR is greater than CIR, etc. 🌈

Reporting Format: The policer individual report MUST contain all results for each CIR/EIR/CBS/EBS test run. A recommended format is as follows 📝:

***********************************************************

Test Configuration Summary: Tr, Td

DUT Configuration Summary: CIR, EIR, CBS, EBS

The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):

-  Compliant Traffic Test: BSA, LP, OOS, PD, and PDV

-  Excess Traffic Test: BSA

***********************************************************

6.1.2. Policer Capacity Tests 👑

Objective: The whole point of capacity tests is checking how the policer performs in a scaled environment, with multiple ingress customer policers spread across multiple physical ports 🌸. This test benchmarks the maximum number of active policers the device manufacturer specifies.

Test Summary: The specified policing function capacity is usually written as the number of policers active on each individual physical port, plus the number of unique policer rates being used 🎀. For all of the capacity tests, the benchmarking test procedure and reporting format from Section 6.1.1 for a single policer MUST be applied to each of the physical-port policers. No skipping, periodt 💅.

For example, a Layer 2 switching device might say each of its 32 physical ports can be policed using a pool of policing service policies 🦋. The device could carry one customer's traffic on each physical port, with a single policer instantiated per physical port. Or a single physical port could carry multiple customers, in which case lots of customer flows get policed at the same time on one port, with separate policers per customer on that port 🫶.

Test Metrics: The metrics in Section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL be measured at the egress port and recorded 📊.

The following sections provide the specific test scenarios, procedures, and reporting formats for each policer capacity test ✨.

6.1.2.1. Maximum Policers on Single Physical Port 🧁

Test Summary: The first policer capacity test benchmarks a single physical port with the maximum number of policers on that port 🔥.

Assume there are multiple categories of ingress policers at rates r1, r2, ..., rn. There are multiple customers on one physical port, and each customer could be represented by a single-tagged VLAN, a double-tagged VLAN, a Virtual Private LAN Service (VPLS) instance, etc. Each customer maps to a different policer, and each of those policers can be running at one of the rates r1, r2, ..., rn 💜.

An example configuration would be:

  • Y1 customers, policer rate r1
  • Y2 customers, policer rate r2
  • Y3 customers, policer rate r3 ...
  • Yn customers, policer rate rn

Some bandwidth on the physical port is set aside for other traffic (anything that isn't customer traffic), and that includes network control protocol traffic 🙈. There's a separate policer for the other traffic too. Typical deployments have three categories of policers, but some deployments have more or fewer than three categories of ingress policers 🌈.

Procedure:

  1. Configure the DUT policing parameters for the desired CIR/EIR and CBS/EBS values for each policer rate (r1-rn) to be tested ⚙️.
  2. Configure the tester to generate a stateless traffic burst equal to CBS and an interval equal to Ti (CBS in bits/CIR) for each customer stream (Y1-Yn). The encapsulation for each customer must also be configured according to the service tested (VLAN, VPLS, IP mapping, etc.) 🧚.
  3. Compliant Traffic Test: Generate bursts of CBS + EBS traffic into the policer ingress port for each customer traffic stream, and measure the metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV) at the egress port for each stream and across the entire Td (default 30-second duration) ⏱️.
  4. Excess Traffic Test: Generate bursts of greater than CBS + EBS bytes into the policer ingress port for each customer traffic stream, and verify that the policer only allowed the BSA bytes to exit the egress for each stream. The excess burst MUST be recorded. The recommended value is 1000 bytes 👀.

Reporting Format: The policer individual report MUST contain all results for each CIR/EIR/CBS/EBS test run, per customer traffic stream. A recommended format is as follows 📝:

*****************************************************************

Test Configuration Summary: Tr, Td

Customer Traffic Stream Encapsulation: Map each stream to VLAN,
VPLS, IP address

DUT Configuration Summary per Customer Traffic Stream: CIR, EIR,
CBS, EBS
The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):

-  Customer Stream Y1-Yn (see note) Compliant Traffic Test:
   BSA, LP, OOS, PD, and PDV

-  Customer Stream Y1-Yn (see note) Excess Traffic Test: BSA

*****************************************************************

Note: For each test run, there will be two (2) rows for each customer stream: the Compliant Traffic Test result and the Excess Traffic Test result 🎀.

6.1.2.2. Single Policer on All Physical Ports 🌟

Test Summary: The second policer capacity test is a single policer function per physical port, with all physical ports active 🔥. There's one policer per physical port, and that policer can be running at one of the rates r1, r2, ..., rn. Every physical port on the networking device is active, so it's a full house 🏠.

Procedure: The procedure for this test is identical to the procedure listed in Section 6.1.1 🫶. The configured parameters must be reported per port, and the test report must include results per measured egress port 📊.

6.1.2.3. Maximum Policers on All Physical Ports 🦋

The third policer capacity test is a combination of the first and second capacity tests. In other words, it's maximum policers active per physical port AND all physical ports active at once 💖.

Procedure: The procedure for this test is identical to the procedure listed in Section 6.1.2.1 💅. The configured parameters must be reported per port, and the test report must include per-stream results per measured egress port ✨.

6.2. Queue/Scheduler Tests 🧚

Queues and traffic scheduling are super closely related, because a queue's priority decides how the traffic scheduler transmits packets out of the egress port 🫶.

Since device queues and buffers are generally an egress function, this test framework discusses testing at the egress. (The technique can also be applied to ingress-side queues, though 👀.)

Just like the policing tests, these tests are split into two sections: individual queue/scheduler function tests, and then full-capacity tests 🌈.

6.2.1. Queue/Scheduler Individual Tests 🎀

The various scheduling techniques include FIFO, Strict Priority (SP) queuing, and Weighted Fair Queuing (WFQ), along with other variations 💜. This test framework recommends testing with a minimum of three techniques. Benchmarking other device-scheduling algorithms is left up to the discretion of the tester, so you do you 💅.

6.2.1.1. Testing Queue/Scheduler with Stateless Traffic ✨

Objective: okay so the goal here is to check that the queue and scheduling setup can handle stateless traffic bursts up to the queue depth 💅 that's the whole vibe.

Test Summary: a queue is memory based 🧠, unlike a policer, which is token or credit based 🪙. Still, the same concepts from Section 6.1 totally apply to testing queues 🎀

First, set the device's queue to the size you want in KB (that's the Queue Length, or QL). Then send stateless traffic at it to test that QL 🦋

A queue should be able to handle repeated bursts, with gaps between them that are proportional to the Bottleneck Bandwidth (BB). That gap is called the transmission interval (Ti) 💜 You can define Ti for the bursts, and it's based on the QL and BB of the egress interface:

Ti = QL * 8 / BB

Bestie, this looks a lot like the Ti you need when pushing traffic into a policer (QL = CBS, BB = CIR) 👀 And the burst hunt algorithm from Section 5.1.1 can also automate measuring the queue value, which is giving automation energy 🔥

The stateless traffic burst SHALL be transmitted at the link speed and spaced within the transmission interval (Ti). The metrics defined in Section 4.1 SHALL be measured at the egress port and recorded. The primary intent is to verify the BSA and make sure no packets are dropped 🙈

The scheduler needs characterizing too, so we can see how well the device schedules queues by priority. For example, two priority levels that include SP and FIFO queuing 👑 When the flow load is greater than the egress port speed, the higher-priority packets should get transmitted without drops (and keep latency low), while the lower-priority (or best-effort) queue may be dropped 💀

Test Metrics: the metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL be measured at the egress port and recorded 📊

Procedure:

  1. Configure the DUT QL and scheduling technique parameters (FIFO, SP, etc.). 🛠️
  2. Configure the tester to generate a stateless traffic burst equal to QL and an interval equal to Ti (QL in bits/BB). 🎯
  3. Generate bursts of QL traffic into the DUT, and measure the metrics defined in Section 4.1 (LP, OOS, PD, and PDV) at the egress port and across the entire Td (default 30-second duration). ⏱️

Reporting Format: the Queue/Scheduler Stateless Traffic individual report MUST contain all results for each QL/BB test run. Here's the recommended layout 📝

****************************************************************

Test Configuration Summary: Tr, Td

DUT Configuration Summary: Scheduling technique (i.e., FIFO, SP,
WFQ, etc.), BB, and QL
The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):

-  LP, OOS, PD, and PDV

****************************************************************

6.2.1.2. Testing Queue/Scheduler with Stateful Traffic 🌸

Objective: this one checks that the queue and scheduling technique can handle stateful traffic bursts up to the queue depth. It's the more realistic version, so we're going full main character 🎬

Test Background and Summary: to get a more realistic benchmark, and to test queues in Layer 4 devices like firewalls, stateful traffic testing is recommended for the queue tests 🧱 Stateful traffic tests also use the Network Delay Emulator (NDE) from the network setup configuration in Section 1.2 🌐

The BDP of the TCP test traffic must be calibrated to the QL of the device queue. Referencing [RFC6349], the BDP is equal to:

BB * RTT / 8 (in bytes)

The NDE must be configured to an RTT value that's large enough for the BDP to be greater than QL. Here's an example scenario 🍓

  • Ingress link = GigE
  • Egress link = 100 Mbps (BB)
  • QL = 32 KB
RTT(min) = QL * 8 / BB and would equal 2.56 ms
   (and the BDP = 32 KB)

In this example, one (1) TCP connection with a window size / SSB of 32 KB would be needed to test the QL of 32 KB. This Bulk Transfer Test can be done with iperf, as described in Appendix A 🧚

Two types of TCP tests MUST be performed. Think of them as the two core looks you can't skip 💅

Two types of TCP tests MUST be performed: the Bulk Transfer Test and the Micro Burst Test Pattern, as documented in Appendix B. The Bulk Transfer Test only bursts during the TCP Slow Start (or Congestion Avoidance) state, while the Micro Burst Test Pattern emulates application-layer bursting, which may happen any time during the TCP connection 💥

Other types of tests SHOULD include the following: simple web sites, complex web sites, business applications, email, and SMB/CIFS (Common Internet File System) file copy. All of these are also documented in Appendix B 📧

Test Metrics: the test results will be recorded per the stateful metrics defined in Section 4.2, primarily the TCP Test Pattern Execution Time (TTPET), TCP Efficiency, and Buffer Delay 📈

Procedure:

  1. Configure the DUT QL and scheduling technique parameters (FIFO, SP, etc.). 🛠️
  2. Configure the test generator* with a profile of an emulated application traffic mixture. ✨
    • The application mixture MUST be defined in terms of percentage of the total bandwidth to be tested.
    • The rate of transmission for each application within the mixture MUST also be configurable.

* To ensure repeatable results, the test generator MUST be capable of generating precise TCP test patterns for each application specified. 🔁

  1. Generate application traffic between the ingress (client side) and egress (server side) ports of the DUT, and measure the metrics (TTPET, TCP Efficiency, and Buffer Delay) per application stream and at the ingress and egress ports (across the entire Td, default 60-second duration) 🚀

A couple of items need clarifying about application measurements 🤓 An application session may be made up of a single TCP connection or multiple TCP connections.

If an application session uses a single TCP connection, the application throughput and metrics have a 1-1 relationship to the TCP connection measurements 🫶

If an application session (e.g., an HTTP-based application) uses multiple TCP connections, then all of those TCP connections are aggregated in the application throughput measurement and metrics for that application 🧩

Then there's the case of multiple instances of an application session (i.e., multiple FTPs emulating multiple clients). Here the test should measure and record each FTP application session independently, and tabulate the minimum, maximum, and average for all FTP sessions 📋

Finally, application throughput measurements are based on Layer 4 TCP throughput and do not include bytes retransmitted. The TCP Efficiency metric MUST be measured during the test, because it gives you a measure of "goodput" during each test 🍯

Reporting Format: the Queue/Scheduler Stateful Traffic individual report MUST contain all results for each traffic scheduler and QL/BB test run. Here's the recommended format 🎀

******************************************************************

Test Configuration Summary: Tr, Td

DUT Configuration Summary: Scheduling technique (i.e., FIFO, SP,
WFQ, etc.), BB, and QL

Application Mixture and Intensities: These are the percentages
configured for each application type.

The results table should contain entries for each test run, with
minimum, maximum, and average per application session, as follows
(Test #1 to Test #Tr):

-  Throughput (bps) and TTPET for each application session

-  Bytes In and Bytes Out for each application session

-  TCP Efficiency and Buffer Delay for each application session

******************************************************************

6.2.2. Queue/Scheduler Capacity Tests 🏋️‍♀️

Objective: these capacity tests benchmark queue and scheduler performance in a scaled-up environment, with multiple queues and schedulers active on multiple egress physical ports 🌈 They'll benchmark the maximum number of queues and schedulers that the device manufacturer specifies. Each priority in the system maps to its own separate queue 👑

Test Metrics: the metrics defined in Section 4.1 (BSA, LP, OOS, PD, and PDV) SHALL be measured at the egress port and recorded 📊

The specific test scenarios, procedures, and reporting formats for each queue/scheduler capacity test are in the sections below 🧁

6.2.2.1. Multiple Queues, Single Port Active 🚦

For the first capacity test, multiple queues per port get tested on a single physical port. All of the queues (typically eight) are active on that one physical port. Traffic from multiple ingress physical ports is directed to the same egress physical port, which causes oversubscription on the egress physical port 🔥

There are lots of priority schemes and combinations of priorities that the scheduler can manage. The sections below specify the priority schemes that should be tested 🧠

6.2.2.1.1. Strict Priority on Egress Port 👑

Test Summary: for this test, SP scheduling on the egress physical port should be tested. The benchmarking methodologies from Sections 6.2.1.1 (stateless) and 6.2.1.2 (stateful), meaning the procedure, metrics, and reporting format, should be applied here 💅 For a given priority, each ingress physical port should get a fair share of the egress physical-port bandwidth ⚖️

Since this is a capacity test, the configuration and report results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also include:

Configuration:

  • The number of physical ingress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port

Report Results:

  • For each ingress port traffic stream, the achieved throughput rate and metrics at the egress port 🎉

6.2.2.1.2. Strict Priority + WFQ on Egress Port ✨

Test Summary: okay bestie, for this one we turn on SP and WFQ at the same time in the scheduler, and they both have to live on one single egress port 🎀 The methodologies from Section 6.2.1.1 (stateless) and 6.2.1.2 (stateful) (procedure, metrics, and reporting format) should be applied here too. The egress port bandwidth-sharing among the weighted queues should be proportional to the assigned weights, so the weights are literally the vibe check for who gets how much 💜 For a given priority, each ingress physical port should get a fair share of the egress physical-port bandwidth 🦋

Since this is a capacity test, the configuration and report results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also include:

Configuration:

  • The number of physical ingress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port

Report Results:

  • For each ingress port traffic stream, the achieved throughput rate and metrics at each queue of the egress port queue (both the SP and WFQ) 📊

Example:

  • Egress Port SP Queue: throughput and metrics for ingress streams 1-n
  • Egress Port WFQ: throughput and metrics for ingress streams 1-n

6.2.2.2. Single Queue per Port, All Ports Active 🌸

Test Summary: this one is giving traffic jam 🚗💨 Traffic from multiple ingress physical ports gets sent to the same egress physical port, which causes oversubscription on that egress port. At the same time, the same amount of traffic goes to each egress physical port 🫶

The benchmarking methodologies from Sections 6.2.1.1 (stateless) and 6.2.1.2 (stateful) (procedure, metrics, and reporting format) should be applied here. Each ingress physical port should get a fair share of the egress physical-port bandwidth. Also, each egress physical port should receive the same amount of traffic ✨

Since this is a capacity test, the configuration and report results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also include:

Configuration:

  • The number of ingress ports active during the test
  • The number of egress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port

Report Results:

  • For each egress port, the achieved throughput rate and metrics at the egress port queue for each ingress port stream 💅

Example:

  • Egress Port 1: throughput and metrics for ingress streams 1-n
  • Egress Port n: throughput and metrics for ingress streams 1-n

6.2.2.3. Multiple Queues per Port, All Ports Active 🧚

Test Summary: okay now we're going full chaos mode 🌈 Traffic from multiple ingress physical ports is sent to all the queues of each egress physical port. That means oversubscription on the egress physical ports, and the same amount of traffic goes to each egress physical port, so everybody's getting an equal slice of the drama 💀

The benchmarking methodologies from Sections 6.2.1.1 (stateless) and 6.2.1.2 (stateful) (procedure, metrics, and reporting format) should be applied here. For a given priority, each ingress physical port should get a fair share of the egress physical-port bandwidth. Additionally, each egress physical port should receive the same amount of traffic 🍓

Since this is a capacity test, the configuration and report results format (see Sections 6.2.1.1 and 6.2.1.2) MUST also include:

Configuration:

  • The number of physical ingress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port

Report Results:

  • For each egress port, the achieved throughput rate and metrics at each egress port queue for each ingress port stream 🔥

Example:

  • Egress Port 1, SP Queue: throughput and metrics for ingress streams 1-n
  • Egress Port 2, WFQ: throughput and metrics for ingress streams 1-n

...

  • Egress Port n, SP Queue: throughput and metrics for ingress streams 1-n
  • Egress Port n, WFQ: throughput and metrics for ingress streams 1-n

6.3. Shaper Tests 🎀

Like a queue, a traffic shaper is memory based, but it comes with the added intelligence of an active traffic scheduler 🧠 The same concepts from Section 6.2 (queue testing) can be applied to testing a network device shaper, so if you already get queues, you're honestly already halfway there 😭

Again, the tests are split into two sections: individual shaper benchmark tests first, then full-capacity shaper benchmark tests 💖

6.3.1. Shaper Individual Tests 🦋

A traffic shaper generally has three (3) components that you can configure:

  • Ingress Queue bytes
  • Shaper Rate (SR), bps
  • Burst Committed (Bc) and Burst Excess (Be), bytes

The Ingress Queue holds burst traffic, and then the shaper meters traffic out of the egress port according to the SR and Bc/Be parameters. Shapers generally transmit into policers, so the goal is for the emitted traffic to conform to the policer's limits 🫶

6.3.1.1. Testing Shaper with Stateless Traffic ✨

Objective: test a shaper by transmitting stateless traffic bursts into the shaper ingress port and verifying that the egress traffic is shaped according to the shaper traffic profile 💅

Test Summary: the stateless traffic must be burst into the DUT ingress port and must not exceed the Ingress Queue. The burst can be a single burst or multiple bursts. If multiple bursts are transmitted, then the transmission interval (Ti) must be large enough so that the SR is not exceeded. Let me show you an example so this clicks 💡

In this example, the shaper's ingress and egress ports are both full-duplex Gigabit Ethernet. The Ingress Queue is configured to be 512,000 bytes, the SR = 50 Mbps, and both Bc and Be are configured to be 32,000 bytes. For a single-burst test, the transmitting test device would burst 512,000 bytes maximum into the ingress port and then stop transmitting 🛑

If a multiple-burst test is to be conducted, then the burst bytes divided by the transmission interval between the 512,000-byte bursts must not exceed the SR. The transmission interval (Ti) must adhere to a formula similar to the formula described in Section 6.2.1.1 for queues, namely:

   Ti = Ingress Queue * 8 / SR

For the example from the previous paragraph, the Ti between bursts must be greater than 82 milliseconds (512,000 bytes * 8 / 50,000,000 bps). This yields an average rate of 50 Mbps so that an Ingress Queue would not overflow 🌈

Test Metrics: the metrics defined in Section 4.1 (LP, OOS, PDV, SR, SBB, and SBI) SHALL be measured at the egress port and recorded 📊

Procedure:

  1. Configure the DUT shaper ingress QL and shaper egress rate parameters (SR, Bc, Be).
  2. Configure the tester to generate a stateless traffic burst equal to QL and an interval equal to Ti (QL in bits/BB).
  3. Generate bursts of QL traffic into the DUT, and measure the metrics defined in Section 4.1 (LP, OOS, PDV, SR, SBB, and SBI) at the egress port and across the entire Td (default 30-second duration) ⏱️

Reporting Format: the Shaper Stateless Traffic individual report MUST contain all results for each QL/SR test run. A recommended format is as follows:

***********************************************************

Test Configuration Summary: Tr, Td

DUT Configuration Summary: Ingress Burst Rate, QL, SR

The results table should contain entries for each test run,
as follows (Test #1 to Test #Tr):

-  LP, OOS, PDV, SR, SBB, and SBI

***********************************************************

6.3.1.2. Testing Shaper with Stateful Traffic ✨

Objective: okay so the goal here is to blast the shaper's ingress port with stateful traffic bursts and then check that the egress traffic comes out shaped to match the shaper traffic profile 🎀

Test Summary: stateful testing is also recommended for the shaper tests, bestie, because it's way more realistic and it lets you poke at queues in Layer 4 devices like firewalls 🔥 these tests also use the Network Delay Emulator (NDE) from the network setup in Section 1.2 🌸

the BDP of the TCP test traffic must be calculated the way Section 6.2.1.2 explains. to properly stress the network buffers and the traffic-shaping function, the TCP window size (the minimum of the TCP RWND and sender socket) should be greater than the BDP, which is what actually stresses the shaper 💪 BDP factors of 1.1 to 1.5 are recommended, but the values are up to the tester and should be documented 📝

the cumulative TCP window sizes* (RWND at the receiving end and CWND at the transmitting end) equals the TCP window size* for each connection, multiplied by the number of connections ✨

* As described in Section 3 of [RFC6349], the SSB MUST be large enough to fill the BDP.

for example, if the BDP is 256 KB and each connection is 64 KB, then you'd need four (4) connections to fill the BDP, and 5-6 connections (oversubscribing the BDP) to stress-test the traffic-shaping function 🦋

two types of TCP tests MUST be performed: the Bulk Transfer Test and the Micro Burst Test Pattern, both documented in Appendix B 🫶 the Bulk Transfer Test only bursts during the TCP Slow Start (or Congestion Avoidance) state, while the Micro Burst Test Pattern emulates application-layer bursting, which may occur any time during the TCP connection 💜

other types of tests SHOULD include simple web sites, complex web sites, business applications, email, and SMB/CIFS file copy (all also documented in Appendix B) 🍓

Test Metrics: the test results get recorded per the stateful metrics defined in Section 4.2 -- mainly the TCP Test Pattern Execution Time (TTPET), TCP Efficiency, and Buffer Delay 📊

Procedure:

  1. Configure the DUT shaper ingress QL and shaper egress rate parameters (SR, Bc, Be) 🎀
  2. Configure the test generator* with a profile of an emulated application traffic mixture 🧚
    • The application mixture MUST be defined in terms of percentage of the total bandwidth to be tested.
    • The rate of transmission for each application within the mixture MUST also be configurable.
    • * To ensure repeatable results, the test generator MUST be capable of generating precise TCP test patterns for each application specified.
  3. Generate application traffic between the ingress (client side) and egress (server side) ports of the DUT, then measure the metrics (TTPET, TCP Efficiency, and Buffer Delay) per application stream and at the ingress and egress ports, across the entire Td (default 30-second duration) 🔥

Reporting Format: the Shaper Stateful Traffic individual report MUST contain all results for each traffic scheduler and QL/SR test run 📋 a recommended format looks like this:

******************************************************************

Test Configuration Summary: Tr, Td DUT Configuration Summary: Ingress Burst Rate, QL, SR Application Mixture and Intensities: these are the percentages configured for each application type 🌈

the results table should have entries for each test run, with minimum, maximum, and average per application session, like this (Test #1 to Test #Tr):

  • Throughput (bps) and TTPET for each application session
  • Bytes In and Bytes Out for each application session
  • TCP Efficiency and Buffer Delay for each application session
******************************************************************

6.3.2. Shaper Capacity Tests ✨

Objective: these scalability tests check shaper performance in a scaled environment, with shapers active on multiple queues across multiple egress physical ports 👑 they benchmark the maximum number of shapers as specified by the device manufacturer 💅

the following sections give you the specific test scenarios, procedures, and reporting formats for each shaper capacity test 🫶

6.3.2.1. Single Queue Shaped, All Physical Ports Active

Test Summary: the first shaper capacity test is per-port shaping with all physical ports active 🌸 traffic from multiple ingress physical ports gets directed to the same egress physical port, which causes oversubscription on that egress port 😭 the same amount of traffic also goes to each egress physical port.

the benchmarking methodologies in Sections 6.3.1.1 (stateless) and 6.3.1.2 (stateful) (procedure, metrics, and reporting format) should be applied here. since this is a capacity test, the configuration and report results format (see Section 6.3.1) MUST also include:

Configuration:

  • The number of physical ingress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port
  • The shaped egress port shaper parameters (QL, SR, Bc, Be)

Report Results:

  • For each active egress port, the achieved throughput rate and shaper metrics for each ingress port traffic stream

Example:

  • Egress Port 1: throughput and metrics for ingress streams 1-n
  • Egress Port n: throughput and metrics for ingress streams 1-n

6.3.2.2. All Queues Shaped, Single Port Active

Test Summary: the second shaper capacity test runs with all queues actively shaping on a single physical port 🔥 the benchmarking methodology from the per-port shaping test (Section 6.3.2.1) is the foundation here. on top of that, each of the SP queues on the egress physical port gets its own shaper 💖 for the highest-priority queue, the max bandwidth is limited by the bandwidth of the shaper. for the lower-priority queues, the max bandwidth is limited by the bandwidth of the shaper and the traffic in higher-priority queues 👑

the benchmarking methodologies in Sections 6.3.1.1 (stateless) and 6.3.1.2 (stateful) (procedure, metrics, and reporting format) should be applied here. since this is a capacity test, the configuration and report results format (see Section 6.3.1) MUST also include:

Configuration:

  • The number of physical ingress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port
  • For the active egress port, each of the following shaper queue parameters: QL, SR, Bc, Be

Report Results:

  • For each queue of the active egress port, the achieved throughput rate and shaper metrics for each ingress port traffic stream

Example:

  • Egress Port High-Priority Queue: throughput and metrics for ingress streams 1-n
  • Egress Port Lower-Priority Queue: throughput and metrics for ingress streams 1-n

6.3.2.3. All Queues Shaped, All Ports Active

Test Summary: the third shaper capacity test is a combo of the tests in Sections 6.3.2.1 and 6.3.2.2 🧚 all queues are actively shaping and all physical ports are active 💅

the benchmarking methodologies in Sections 6.3.1.1 (stateless) and 6.3.1.2 (stateful) (procedure, metrics, and reporting format) should be applied here. since this is a capacity test, the configuration and report results format (see Section 6.3.1) MUST also include:

Configuration:

  • The number of physical ingress ports active during the test
  • The classification marking (DSCP, VLAN, etc.) for each physical ingress port
  • The traffic rate for stateful traffic and the traffic rate/mixture for stateful traffic for each physical ingress port
  • For each of the active egress ports: shaper port parameters and per-queue parameters (QL, SR, Bc, Be)

Report Results:

  • For each queue of each active egress port, the achieved throughput rate and shaper metrics for each ingress port traffic stream

Example:

  • Egress Port 1, High-Priority Queue: throughput and metrics for ingress streams 1-n
  • Egress Port 1, Lower-Priority Queue: throughput and metrics for ingress streams 1-n
  • ...
  • Egress Port n, High-Priority Queue: throughput and metrics for ingress streams 1-n
  • Egress Port n, Lower-Priority Queue: throughput and metrics for ingress streams 1-n

6.4. Concurrent Capacity Load Tests 🌈

as mentioned in Section 3 of this document, it is impossible to specify every permutation of concurrent traffic management functions to test on a device for capacity testing 😭 but these profiles might be useful for testing multiple configurations of traffic management functions:

  • Policers on ingress and queuing on egress
  • Policers on ingress and shapers on egress (not intended for a flow to be policed and then shaped; these would be two different flows tested at the same time)

the test procedures and reporting formats from Sections 6.1, 6.2, and 6.3 may be modified to accommodate the capacity test profile 🎀

7. Security Considerations 🔥

okay bestie, this part is serious 🙈 documents of this type do not directly affect the security of the Internet or of corporate networks, as long as benchmarking is not performed on devices or systems connected to production networks 🛡️

further, benchmarking is performed on a "black box" basis, relying solely on measurements observable external to the DUT/SUT 👀

special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes 💀 any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. no exceptions, periodt 💅

8. References 📚

the reference list is coming up next part 🧚

8.1. Normative References 📚✨

These are the references you really need to have read, bestie. 🫶

8.2. Informative References 🌸

These ones are extra reading if you want to go deeper, no pressure 💖

Appendix A. Open Source Tools for Traffic Management Testing 🧰💅

Okay so this framework says stateless and stateful behaviors SHOULD both get tested, and honestly that's the vibe 🙌 A few open source tools can help you run lots of the tests described here: iperf, netperf (with netperf-wrapper), "uperf", Tmix, TCP-incast-generator, and D-ITG (Distributed Internet Traffic Generator). 🎀

iperf can make UDP-based or TCP-based traffic. The client and the server have to run iperf in the same traffic mode 🤝 The server sits there listening, and then the client is the one controlling the test traffic. Both unidirectional and bidirectional concurrent testing are supported, so you can go either way ✨

UDP mode is your go-to for stateless traffic testing 🦋 You can control the target bandwidth, packet size, UDP port, and test duration, and the iperf receiver gives you a report of bytes transmitted, packets lost, and delay variation. Iconic. 😭

iperf (TCP mode), TCP-incast-generator, and D-ITG can all handle stateful traffic testing for bulk transfer traffic 🚚 You can control the TCP window size (which is actually the SSB), the number of connections, packet size, TCP port, and test duration. The iperf sender reports bytes transmitted and throughput achieved, while TCP-incast-generator and D-ITG give you even more statistics 📊

netperf is a software application for network bandwidth testing between two hosts on a network 🖥️ It supports UNIX domain sockets, TCP, SCTP, and UDP via BSD Sockets. It has a bunch of predefined tests, for example to measure bulk (unidirectional) data transfer or request/response performance (http://en.wikipedia.org/wiki/Netperf). netperf-wrapper is a Python script that runs multiple simultaneous netperf instances and aggregates the results. Main character energy 👑

uperf works from a description (or model) of an application mixture and generates load according to that model descriptor 🎭 It's more flexible than netperf when it comes to request/response application behavior inside a single TCP connection. The model descriptor can be based on empirical data, but at the time of this writing, importing packet captures is not directly supported 🙈

Tmix is another application traffic emulation tool, and it uses packet captures directly to build the traffic profile 🕵️‍♀️ The packet trace gets "reverse compiled" into a source-level characterization called a "connection vector" for each TCP connection in the trace. It's mostly used in ns2 simulation environments, but it also runs on Linux hosts 🐧

All of these open source tools make it way easier to emulate the TCP test patterns discussed in Appendix B 🧚💜

Appendix B. Stateful TCP Test Patterns ✨

Okay so bestie, this part is where the framework gets really concrete about the TCP traffic patterns it recommends you actually test with 🎀. The framework says at a minimum these are the ones you MUST NOT skip... wait, not quite, the original says it RECOMMENDS them at a minimum, so here they are! They're representative of real-world app traffic, and Section 5.2.1 has ways to cook up other application-based TCP patterns too 💅.

  • Bulk Transfer: Generate concurrent TCP connections whose aggregate number of in-flight data bytes would fill the BDP. The guidelines from [RFC6349] are used to create this traffic pattern 🌊.
  • Micro Burst: Generate precise burst patterns within a single TCP connection or multiple TCP connections. The idea is for TCP to establish equilibrium and then burst application bytes at defined sizes ⚡. The test tool must allow the burst size and burst time interval to be configurable 🔧.
  • Web Site Patterns: The HTTP traffic model shown in Table 4.1.3-1 of [3GPP2-C_R1002-A] shows a way to develop these TCP test patterns 🌸. The model has these parameters:
    • Main object size (Sm)
    • Embedded object size (Se)
    • Number of embedded objects per page (Nd)
    • Client processing time (Tcp)
    • Server processing time (Tsp)

Web site test patterns come in two flavours, illustrated with these examples:

  • Simple web site: Mimic the request/response and object download behavior of a basic web site (small company) 🏠.
  • Complex web site: Mimic the request/response and object download behavior of a complex web site (eCommerce site) 🛍️.

Referencing the HTTP traffic model parameters, the following table was derived (by analysis and experimentation) for simple and complex web site TCP test patterns 📊:

                         Simple         Complex
Parameter                Web Site       Web Site
-----------------------------------------------------
Main object              Ave. = 10KB    Ave. = 300KB
 size (Sm)               Min. = 100B    Min. = 50KB
                         Max. = 500KB   Max. = 2MB

Embedded object          Ave. = 7KB     Ave. = 10KB
 size (Se)               Min. = 50B     Min. = 100B
                         Max. = 350KB   Max. = 1MB

Number of embedded       Ave. = 5       Ave. = 25
 objects per page (Nd)   Min. = 2       Min. = 10
                         Max. = 10      Max. = 50

Client processing        Ave. = 3s      Ave. = 10s
 time (Tcp)*             Min. = 1s      Min. = 3s
                         Max. = 10s     Max. = 30s

Server processing        Ave. = 5s      Ave. = 8s
 time (Tsp)*             Min. = 1s      Min. = 2s
                         Max. = 15s     Max. = 30s
  • The client and server processing time is distributed across the transmission/receipt of all of the main and embedded objects.

To be clear, these parameters are reasonable guidelines for TCP test pattern traffic generation 🧁. The test tool can use fixed parameters for simpler tests and mathematical distributions for more complex tests 📐. However, the test pattern MUST be repeatable so the benchmark results can be reliably compared 🔁.

  • Interactive Patterns: While web site patterns are interactive to a degree, they mainly emulate downloading web sites of varying complexity 🌐. Interactive patterns are chattier, since there's a lot of user interaction with the servers 💬. Examples include business apps like PeopleSoft and Oracle, and consumer apps like Facebook and IM 📱. For the interactive patterns, packet capture was used to characterize some business applications and also the email application 📨.

In summary, an interactive application can be described by these parameters:

  • Client message size (Scm)
  • Number of client messages (Nc)
  • Server response size (Srs)
  • Number of server messages (Ns)
  • Client processing time (Tcp)
  • Server processing time (Tsp)
  • File size upload (Su)*
  • File size download (Sd)*
  • The file size parameters account for attachments uploaded or downloaded and may not be present in all interactive applications.

Again using packet capture to characterize, this table gives the guidelines for simple business apps, complex business apps, eCommerce, and email Send/Receive 💌:

                  Simple       Complex
                  Business     Business
Parameter         Application  Application  eCommerce*   Email
--------------------------------------------------------------------
Client message    Ave. = 450B  Ave. = 2KB   Ave. = 1KB   Ave. = 200B
 size (Scm)       Min. = 100B  Min. = 500B  Min. = 100B  Min. = 100B
                  Max. = 1.5KB Max. = 100KB Max. = 50KB  Max. = 1KB

Number of client  Ave. = 10    Ave. = 100   Ave. = 20    Ave. = 10
 messages (Nc)    Min. = 5     Min. = 50    Min. = 10    Min. = 5
                  Max. = 25    Max. = 250   Max. = 100   Max. = 25

Client processing Ave. = 10s   Ave. = 30s   Ave. = 15s   Ave. = 5s
 time (Tcp)**     Min. = 3s    Min. = 3s    Min. = 5s    Min. = 3s
                  Max. = 30s   Max. = 60s   Max. = 120s  Max. = 45s

Server response   Ave. = 2KB   Ave. = 5KB   Ave. = 8KB   Ave. = 200B
 size (Srs)       Min. = 500B  Min. = 1KB   Min. = 100B  Min. = 150B
                  Max. = 100KB Max. = 1MB   Max. = 50KB  Max. = 750B

Number of server  Ave. = 50    Ave. = 200   Ave. = 100   Ave. = 15
 messages (Ns)    Min. = 10    Min. = 25    Min. = 15    Min. = 5
                  Max. = 200   Max. = 1000  Max. = 500   Max. = 40

Server processing Ave. = 0.5s  Ave. = 1s    Ave. = 2s    Ave. = 4s
 time (Tsp)**     Min. = 0.1s  Min. = 0.5s  Min. = 1s    Min. = 0.5s
                  Max. = 5s    Max. = 20s   Max. = 10s   Max. = 15s

File size         Ave. = 50KB  Ave. = 100KB Ave. = N/A   Ave. = 100KB
 upload (Su)      Min. = 2KB   Min. = 10KB  Min. = N/A   Min. = 20KB
                  Max. = 200KB Max. = 2MB   Max. = N/A   Max. = 10MB

File size         Ave. = 50KB  Ave. = 100KB Ave. = N/A   Ave. = 100KB
 download (Sd)    Min. = 2KB   Min. = 10KB  Min. = N/A   Min. = 20KB
                  Max. = 200KB Max. = 2MB   Max. = N/A   Max. = 10MB
  • eCommerce used a combination of packet capture techniques and reference traffic flows as described in [SPECweb2009].

** The client and server processing time is distributed across the transmission/receipt of all of the messages. The client processing time consists mainly of the delay between user interactions (not machine processing).

Again, these parameters are the guidelines for TCP test pattern traffic generation 🎯. The test tool can use fixed parameters for simpler tests and mathematical distributions for more complex tests 🧮. However, the test pattern MUST be repeatable so the benchmark results can be reliably compared 🔁.

  • SMB/CIFS file copy: Mimic a network file copy, both read and write 📁. Unlike FTP, which is a bulk transfer and is only flow-controlled via TCP, SMB/CIFS chops a file into application blocks and uses application-level handshaking on top of TCP flow control 🤝.

In summary, an SMB/CIFS file copy can be described by these parameters:

  • Client message size (Scm)
  • Number of client messages (Nc)
  • Server response size (Srs)
  • Number of server messages (Ns)
  • Client processing time (Tcp)
  • Server processing time (Tsp)
  • Block size (Sb)

The client and server messages are SMB control messages 🗂️. The block size is the data portion of the file transfer 📦.

Again using packet capture to characterize, this table gives the guidelines for SMB/CIFS file copy 🧾:

                      SMB/CIFS
  Parameter           File Copy
  --------------------------------
  Client message      Ave. = 450B
   size (Scm)         Min. = 100B
                      Max. = 1.5KB

  Number of client    Ave. = 10
   messages (Nc)      Min. = 5
                      Max. = 25

  Client processing   Ave. = 1ms
   time (Tcp)         Min. = 0.5ms
                      Max. = 2

  Server response     Ave. = 2KB
   size (Srs)         Min. = 500B
                      Max. = 100KB

  Number of server    Ave. = 10
   messages (Ns)      Min. = 10
                      Max. = 200

  Server processing   Ave. = 1ms
   time (Tsp)         Min. = 0.5ms
                      Max. = 2ms

  Block               Ave. = N/A
   size (Sb)*         Min. = 16KB
                      Max. = 128KB
  • Depending upon the tested file size, the block size will be transferred "n" number of times to complete the example. An example would be a 10 MB file test and 64 KB block size. In this case, 160 blocks would be transferred after the control channel is opened between the client and server 🧩.

Acknowledgments 🫶

We would like to thank Al Morton for his continuous review and invaluable input to this document 💜. We would also like to thank Scott Bradner for providing guidance early in this document's conception, in the area of the benchmarking scope of traffic management functions 🌈. Additionally, we would like to thank Tim Copley for his original input, as well as David Taht, Gory Erg, and Toke Hoiland-Jorgensen for their review and input for the AQM group 🦋. Also, for the formal reviews of this document, we would like to thank Gilles Forget, Vijay Gurbani, Reinhard Schrage, and Bhuvaneswaran Vengainathan 👑.

Authors' Addresses 🎀

Barry Constantine JDSU, Test and Measurement Division Germantown, MD 20876-7100 United States

Phone: +1-240-404-2227 Email: barry.constantine@jdsu.com

Ram (Ramki) Krishnan Dell Inc. Santa Clara, CA 95054 United States

Phone: +1-408-406-7890 Email: ramkri123@gmail.com

📦 where this page comes from