RFC 8204
Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV)
RFC 8204: Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV) 💖
okay bestie, this is the title block and it's giving Informational energy 🎀. It's RFC 8204, published in September 2017 as part of the IETF stream. The authors are M. Tahhan and B. O'Mahony (both Intel) and A. Morton (AT&T Labs). The ISSN is 2070-1721 📚. There are no Obsoletes or Updates lines in the header, so this doc doesn't formally replace or amend any other RFC 🦋.
Abstract ✨
okay so this memo is about what the Open Platform for NFV (OPNFV) project brought to Virtual Switch Performance (VSPERF) 💅. The focus is on test setups and configuration parameters for the system under test. OPNFV built on the BMWG's existing work and cites existing literature too 📖. The BMWG has traditionally tested dedicated physical implementations of internetworking functions in a lab 🧪, so this memo covers the extra stuff you have to think about when virtual switches run on general-purpose hardware 💻. The expanded tests are also shaped by OPNFV's mission to virtualize the "telco" infrastructure 📞.
Status of This Memo 📢
this document is not an Internet Standards Track specification, it's published just for informational purposes 🌸.
it's a product of the IETF and represents the consensus of the IETF community 🫶. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). not every IESG-approved doc is a candidate for any level of Internet Standard, so check out Section 2 of RFC 7841 for the details 👀.
for the current status, any errata, and how to send feedback, go to https://www.rfc-editor.org/info/rfc8204 🔗.
Copyright Notice ©️
Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.
this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as in effect on the date of publication. please read these carefully because they spell out your rights and restrictions for this doc 📜. any Code Components pulled out of this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License ⚖️.
Table of Contents 🗂️
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
1.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4
2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 5
3.1. Comparison with Physical Network Functions . . . . . . . 5
3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 6
3.3. New Configuration Parameters . . . . . . . . . . . . . . 6
3.4. Flow Classification . . . . . . . . . . . . . . . . . . . 8
3.5. Benchmarks Using Baselines with Resource Isolation . . . 9
4. VSPERF Specification Summary . . . . . . . . . . . . . . . . 11
5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 18
5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 19
5.2. Accuracy of Activation . . . . . . . . . . . . . . . . . 19
5.3. Reliability of Activation . . . . . . . . . . . . . . . . 19
5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 19
5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 19
5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 19
5.7. Reliability of Operation . . . . . . . . . . . . . . . . 20
5.8. Scalability of Operation . . . . . . . . . . . . . . . . 20
5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 20
6. Security Considerations . . . . . . . . . . . . . . . . . . . 21
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 21
7.1. Normative References . . . . . . . . . . . . . . . . . . 21
7.2. Informative References . . . . . . . . . . . . . . . . . 22
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 23
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 241. Introduction ✨
so the BMWG has traditionally done lab characterization of dedicated physical boxes that do internetworking stuff 🔬. the black-box benchmarks for throughput, latency, forwarding rates, and more have served the industry for years 🙌. now Network Function Virtualization (NFV) wants to change how internetwork functions get implemented, so it's getting a LOT of attention 👀.
a virtual switch (vSwitch) is a huge piece of NFV infrastructure. it connects physical network functions and virtual network functions to each other 🔌. because of that, there are tons of vSwitch benchmarking efforts but not many specs to guide all the new test design choices 🤯. it's a complicated problem and still an industry-wide work in progress 🚧. down the road, several of BMWG's core specs will probably get updated as more testing experience builds consensus around new methodologies, and BMWG should keep working with every org that's chasing the same goal 🤝.
this memo covers what OPNFV contributed on VSPERF characterization through the Danube 3.0 (fourth) release [DanubeRel], all part of BMWG's chartered work (with stable references to their test descriptions) 💜. OPNFV extended the current and completed work of the BMWG IETF and cites existing literature along the way. the most-cited RFC is [RFC2544] (which depends on [RFC1242]), so the foundation of OPNFV's benchmarking work is solid and shared 🏛️. the recommended extensions are specifically about test setups and configuration parameters for the system under test 🎯.
see [VSPERFhome] for more background, and the OPNFV website for general info [OPNFV] 🌐.
the authors point out that OPNFV is different from other open source compute and networking projects because it focuses on existing "telco" services instead of cloud computing ☎️. telco requirements put different emphasis on performance dimensions than cloud computing does. for example, support for and transfer of isochronous media streams 🎬.
1.1. Requirements Language 🔑
here's the key word deal 💅: "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document get interpreted as described in BCP 14 [RFC2119] [RFC8174], but only when they show up in all capitals, like right here 🧐.
1.2. Abbreviations 📝
for the purposes of this document, these abbreviations apply, so bookmark this list 🔖:
ACK Acknowledge
ACPI Advanced Configuration and Power Interface
BIOS Basic Input Output System
BMWG Benchmarking Methodology Working Group
CPDP Control Plane Data Plane
CPU Central Processing Unit
DIMM Dual In-line Memory Module
DPDK Data Plane Development Kit
DUT Device Under Test
GRUB Grand Unified Bootloader
ID Identification
IMIX Internet Mix
IP Internet Protocol
IPPM IP Performance Metrics
LAN Local Area Network
LTD Level Test Design
NFV Network Functions Virtualization
NIC Network Interface Card
NUMA Non-uniform Memory Access
OPNFV Open Platform for NFV
OS Operating System
PCI Peripheral Component Interconnect
PDV Packet Delay Variation
SR/IOV Single Root / Input Output Virtualization
SUT System Under Test
TCP Transmission Control Protocol
TSO TCP Segment Offload
UDP User Datagram Protocol
VM Virtual Machine
VNF Virtualised Network Function
VSPERF OPNFV vSwitch Performance Project2. Scope 🔭
the main point of this memo is to lay out the key aspects of vSwitch benchmarking, especially test setups and configuration parameters for the system under test 🧩. it also extends the big pile of BMWG literature and experience 📚. early feedback says lots of these extensions could also apply beyond this memo's current scope, like to hardware switches in the NFV infrastructure and to virtual routers 🦋. the memo is also a vehicle for more detail and commentary from BMWG and other open source communities, all under BMWG's chartered work to characterize the NFV infrastructure 🛠️.
the benchmarking here should apply to many kinds of vSwitches and stay pretty vSwitch agnostic 💖. nobody tried to track and test every feature of any one specific vSwitch implementation 🙅.
3. Benchmarking Considerations 💻
this section highlights some specific considerations (from [RFC8172]) for benchmarks of virtual switches 🎀. OPNFV is sharing its current thinking on these areas as they build their specs in the Level Test Design (LTD) document, as defined by [IEEE829] 📐.
3.1. Comparison with Physical Network Functions ⚖️
to compare virtual designs and implementations with their physical counterparts, you need identical benchmarks 👯. BMWG has made specs for lots of physical network functions 🏭. BMWG recommended reusing existing benchmarks and methods in [RFC8172], and the OPNFV LTD expands on them as described here 🌱. a key config aspect for vSwitches is how many parallel CPU cores you need to hit performance comparable to a given physical device, or whether some scale limit gets hit before the vSwitch can reach that level 🧮.
it's unlikely the virtual switch will be the only app running on the SUT, so CPU utilization, cache utilization, and memory footprint should also be recorded for virtual implementations 📊. BUT internally measured metrics like these are not benchmarks 🚫. they can still be useful for the audience (e.g., operations) to know, and they may also help if a problem shows up during testing 🔍.
benchmark comparability between virtual and physical/hardware implementations of equivalent functions will likely place more detailed and exact requirements on the "testing systems" (in terms of stream generation, algorithms to search for maximum values, and their configurations) 🧪. this is another area for standards development to appreciate, but it's a topic for a future document 📅.
3.2. Continued Emphasis on Black-Box Benchmarks ⬛
external observations stay essential as the basis for benchmarks 👁️. internal observations, with a fixed specification and interpretation, will be provided in parallel to help develop operations procedures when the technology gets deployed 🚀.
3.3. New Configuration Parameters 🎀
Okay bestie, the biggest vibe check in any benchmark is making sure your results are consistent and repeatable 💅 With a vSwitch there are SO many things that can shift your numbers, and one of the big ones is matching the hardware and software details of the SUT 😭 This section lists some of the many new parameters this project thinks are critical to report so people can actually repeat the tests ✨
The whole goal of the project was repeatable results 🦋 so they made a big set of parameters they consider critical. It shows the benchmarking crowd how much more config complexity comes with this work 🙈 The parameter set below is assumed sufficient for the infrastructure the VSPERF project uses to get repeatable results from test to test 🫶
Hardware details (platform, processor, memory, and network) including:
- BIOS version, release date, and any configurations that were modified
- Power management at all levels (ACPI sleep states, processor package, OS, etc.)
- CPU microcode level
- Number of enabled cores
- Number of cores used for the test
- Memory information (type and size)
- Memory DIMM configurations (quad rank performance may not be the same as dual rank) in size, frequency, and slot locations
- Number of physical NICs and their details (manufacturer, versions, type, and the PCI slot they are plugged into)
- NIC interrupt configuration (and any special features in use)
- PCI configuration parameters (payload size, early ACK option, etc.)
Software details including:
- OS RunLevel
- OS version (for host and VNF)
- Kernel version (for host and VNF)
- GRUB boot parameters (for host and VNF)
- Hypervisor details (type and version)
- Selected vSwitch, version number, or commit ID used
- vSwitch launch command line if it has been parameterized
- Memory allocation to the vSwitch
- Which NUMA node it is using and how many memory channels
- DPDK or any other software dependency version number or commit ID used
- Memory allocation to a VM - if it's from Hugepages/elsewhere
- VM storage type - snapshot, independent persistent, independent non-persistent
- Number of VMs
- Number of virtual NICs (vNICs) - versions, type, and driver
- Number of virtual CPUs and their core affinity on the host
- Number of vNICs and their interrupt configurations
- Thread affinitization for the applications (including the vSwitch itself) on the host
- Details of resource isolation, such as CPUs designated for Host/ Kernel (
isolcpu) and CPUs designated for specific processes (taskset).
Test traffic information:
- Test duration
- Number of flows
- Traffic type - UDP, TCP, and others
- Frame Sizes - fixed or IMIX [RFC6985] (note that with [IEEE802.1ac], frames may be longer than 1500 bytes and up to 2000 bytes)
- Deployment Scenario - defines the communications path in the SUT
3.4. Flow Classification 🌸
Virtual switches group packets into flows by processing and matching particular packet or frame header information, or by matching packets based on the input ports. So a flow can be thought of as a sequence of packets that have the same set of header field values or have arrived on the same physical or logical port 🧚 Performance results can vary based on the parameters the vSwitch uses to match for a flow 👀
The recommended flow classification parameters for any vSwitch performance tests are: the input port (physical or logical), the source MAC address, the destination MAC address, the source IP address, the destination IP address, and the Ethernet protocol type field (although classification may take place on other fields, such as source and destination transport port numbers). 💜
It is essential to increase the flow timeout time on a vSwitch before conducting any performance tests that do not intend to measure the flow setup time (see Section 3 of [RFC2889]). Normally, the first packet of a particular stream will install the flow in the virtual switch, which introduces additional latency. Subsequent packets of the same flow are not subject to this latency if the flow is already installed on the vSwitch. ⭐
3.5. Benchmarks Using Baselines with Resource Isolation 🦋
This outline describes the measurement of baselines with isolated resources at a high level, which is the intended approach at this time 🍓
- Baselines:
- Optional: Benchmark platform forwarding capability without a vSwitch or VNF for at least 72 hours (serves as a means of platform validation and a means to obtain the base performance for the platform in terms of its maximum forwarding rate and latency).
__
+--------------------------------------------------+ |
| +------------------------------------------+ | |
| | | | |
| | Simple Forwarding App | | Host
| | | | |
| +------------------------------------------+ | |
| | NIC | | |
+---+------------------------------------------+---+ __|
^ :
| |
: v
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+Figure 1: Benchmark Platform Forwarding Capability
- Benchmark VNF forwarding capability with direct connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves as a means of VNF validation and a means to obtain the base performance for the VNF in terms of its maximum forwarding rate and latency). The metrics gathered from this test will serve as a key comparison point for vSwitch bypass technologies performance and vSwitch performance.
__
+--------------------------------------------------+ __ |
| +------------------------------------------+ | | |
| | | | Host/ |
| | VNF | | Guest |
| | | | | |
| +------------------------------------------+ | __| |
| | Passthrough/SR-IOV | | Host
| +------------------------------------------+ | |
| | NIC | | |
+---+------------------------------------------+---+ __|
^ :
| |
: v
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+Figure 2: Benchmark VNF Forwarding Capability
- Benchmarking with isolated resources alone and with other resources (both hardware and software) disabled; for example, vSwitch and VM are SUT.
- Benchmarking with isolated resources alone, thus leaving some resources unused.
- Benchmarking with isolated resources and all resources occupied.
- Next Steps:
- Limited sharing
- Production scenarios
- Stressful scenarios
4. VSPERF Specification Summary 🧚
The overall specification in preparation is called a Level Test Design (LTD) document, and it will contain a suite of performance tests 📋 The base performance tests in the LTD are built on the pre-existing specifications the BMWG developed to test the performance of physical switches. Those specifications include:
- Benchmarking Methodology for Network Interconnect Devices [RFC2544]
- Benchmarking Methodology for LAN Switching [RFC2889]
- Device Reset Characterization [RFC6201]
- Packet Delay Variation Applicability Statement [RFC5481]
The two most recent RFCs above ([RFC6201] and [RFC5481]) are being applied in benchmarking for the first time, and they're a real development challenge for test equipment developers 😤 Thankfully, lots of people in the testing system community have jumped in on the VSPERF project, including an open source test system 🙌
On top of that, the LTD also reuses the terminology defined by:
- Benchmarking Terminology for LAN Switching Devices [RFC2285]
It is recommended that these references be included in future benchmarking specifications:
As one might expect, the most fundamental internetworking characteristics of throughput and latency still matter when the switch is virtualized, so these benchmarks figure prominently in the specification 💖
When you're considering characteristics that matter to "telco" network functions, extra performance metrics are needed 📡 In this case, the project specifications reference metrics from the IETF IP Performance Metrics (IPPM) literature. That means the latency test described in [RFC2544] is replaced by measurement of a metric derived from IPPM's [RFC7679], where a set of statistical summaries will be provided (mean, max, min, and percentiles). More metrics planned for benchmarking include packet delay variation as defined by [RFC5481], reordering, burst behaviour, DUT availability, DUT capacity, and packet loss in long-term testing at the throughput level, where some low level of background loss may be present and characterized 🔥
Tests have been designed to collect the metrics below:
- Throughput tests are designed to measure the maximum forwarding rate (in frames per second, fps) and bit rate (in Mbps) for a constant load (as defined by [RFC1242]) without traffic loss.
- Packet and frame-delay distribution tests are designed to measure the average minimum and maximum packet (and/or frame) delay for constant loads.
- Packet delay tests are designed to understand latency distribution for different packet sizes and to uncover outliers over an extended test run.
- Scalability tests are designed to understand how the virtual switch performs with an increasing number of flows, number of active ports, configuration complexity of the forwarding logic, etc.
- Stream performance tests (with TCP or UDP) are designed to measure bulk data transfer performance, i.e., how fast systems can send and receive data through the switch.
- Control-path and data-path coupling tests 🧚 figure out how closely the data path and the control path are coupled, and how that coupling affects the DUT's performance (for example, the delay on the first packet of a flow) 👀
- CPU and memory consumption tests 💻 are all about the virtual switch's footprint on the system. They're run as auxiliary measurements alongside the benchmarks above and cover CPU utilization, cache utilization, and memory footprint 🫶
- "Soak" tests 🛁 run the selected test over a long period. The ideal duration is 24 hours, but you only need to go long enough to tell whether stability issues exist when they show up 💀 There's no requirement to keep going once a DUT shows instability over time. The key performance characteristics and benchmarks get pinned down first, using short duration tests, before any soak tests happen. The whole point of soak tests is to catch transient changes in performance, which might come from infrequent processes, memory leaks, or the super rare coincidence of two or more processes lining up 😭 Stability of the DUT is the paramount consideration, so performance must be evaluated periodically during continuous testing. That's why frame rate metrics [RFC2889] get used instead of throughput [RFC2544], which requires stopping traffic so everything can drain out of internal queues, for example 🎀
Okay, next up, the other test specs that should be developed 👑
Additional test specification development should include:
- Request/response performance tests (with TCP or UDP) 🍓, which measure the transaction rate through the switch.
- Noisy neighbor tests 📢, in order to understand the effects of resource sharing on the performance of a virtual switch.
- Tests derived from examination of ETSI NFV Draft GS IFA003 requirements [IFA003] on characterization of acceleration technologies applied to vSwitches ⚡
A vSwitch can be deployed pretty much anywhere in a network, so its performance has to be characterized across different deployment scenarios 🌈 The scenarios under consideration are shown in the following figures 👇
__
+--------------------------------------------------+ |
| +--------------------+ | |
| | | | |
| | v | | Host
| +--------------+ +--------------+ | |
| | PHY Port | vSwitch | PHY Port | | |
+---+--------------+------------+--------------+---+ __|
^ :
| |
: v
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+
Figure 3: Physical Port to Virtual Switch to Physical Port __
+---------------------------------------------------+ |
| | |
| +-------------------------------------------+ | |
| | Application | | |
| +-------------------------------------------+ | |
| ^ : | |
| | | | | Guest
| : v | |
| +---------------+ +---------------+ | |
| | Logical Port 0| | Logical Port 1| | |
+---+---------------+-----------+---------------+---+ __|
^ :
| |
: v __
+---+---------------+----------+---------------+---+ |
| | Logical Port 0| | Logical Port 1| | |
| +---------------+ +---------------+ | |
| ^ : | |
| | | | | Host
| : v | |
| +--------------+ +--------------+ | |
| | PHY Port | vSwitch | PHY Port | | |
+---+--------------+------------+--------------+---+ __|
^ :
| |
: v
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+
Figure 4: Physical Port to Virtual Switch to VNF to Virtual Switch to
Physical Port __
+----------------------+ +----------------------+ |
| Guest 1 | | Guest 2 | |
| +---------------+ | | +---------------+ | |
| | Application | | | | Application | | |
| +---------------+ | | +---------------+ | |
| ^ | | | ^ | | |
| | v | | | v | | Guests
| +---------------+ | | +---------------+ | |
| | Logical Ports | | | | Logical Ports | | |
| | 0 1 | | | | 0 1 | | |
+---+---------------+--+ +---+---------------+--+__|
^ : ^ :
| | | |
: v : v _
+---+---------------+---------+---------------+--+ |
| | 0 1 | | 3 4 | | |
| | Logical Ports | | Logical Ports | | |
| +---------------+ +---------------+ | |
| ^ | ^ | | | Host
| | \-----------------/ v | |
| +--------------+ +--------------+ | |
| | PHY Ports | vSwitch | PHY Ports | | |
+---+--------------+----------+--------------+---+_|
^ :
| |
: v
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+
Figure 5: Physical Port to Virtual Switch to VNF to Virtual Switch to
VNF to Virtual Switch to Physical Port __
+---------------------------------------------------+ |
| | |
| +-------------------------------------------+ | |
| | Application | | |
| +-------------------------------------------+ | |
| ^ | |
| | | | Guest
| : | |
| +---------------+ | |
| | Logical Port 0| | |
+---+---------------+-------------------------------+ __|
^
|
: __
+---+---------------+------------------------------+ |
| | Logical Port 0| | |
| +---------------+ | |
| ^ | |
| | | | Host
| : | |
| +--------------+ | |
| | PHY Port | vSwitch | |
+---+--------------+------------ -------------- ---+ __|
^
|
:
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+
Figure 6: Physical Port to Virtual Switch to VNF __
+---------------------------------------------------+ |
| | |
| +-------------------------------------------+ | |
| | Application | | |
| +-------------------------------------------+ | |
| : | |
| | | | Guest
| v | |
| +---------------+ | |
| | Logical Port | | |
+-------------------------------+---------------+---+ __|
:
|
v __
+------------------------------+---------------+---+ |
| | Logical Port | | |
| +---------------+ | |
| : | |
| | | | Host
| v | |
| +--------------+ | |
| vSwitch | PHY Port | | |
+-------------------------------+--------------+---+ __|
:
|
v
+--------------------------------------------------+
| |
| Traffic Generator |
| |
+--------------------------------------------------+
Figure 7: VNF to Virtual Switch to Physical Port __
+----------------------+ +----------------------+ |
| Guest 1 | | Guest 2 | |
| +---------------+ | | +---------------+ | |
| | Application | | | | Application | | |
| +---------------+ | | +---------------+ | |
| | | | ^ | |
| v | | | | | Guests
| +---------------+ | | +---------------+ | |
| | Logical Ports | | | | Logical Ports | | |
| | 0 | | | | 0 | | |
+---+---------------+--+ +---+---------------+--+__|
: ^
| |
v : _
+---+---------------+---------+---------------+--+ |
| | 1 | | 1 | | |
| | Logical Ports | | Logical Ports | | |
| +---------------+ +---------------+ | |
| | ^ | | Host
| \-----------------/ | |
| | |
| vSwitch | |
+------------------------------------------------+_|Figure 8: VNF to Virtual Switch to VNF
Here's a whole set of deployment scenario figures, if you want to see more examples, over on the VSPERF "Test Methodology" wiki page 🌸 [TestTopo].
5. 3x3 Matrix Coverage ✨
okay so this section sorts a huge pile of existing test specs into the "3x3" matrix 🧚 (the one introduced in [RFC8172]). The LTD spec IDs are super long, so we're doing a list for each cell that actually has stuff in it. Not every cell is filled in, and btw the matrix has grown to 3x4 so it can fit scale metrics when we show coverage for lots of metrics and benchmarks 📊. The current version of the LTD spec is out there, see [LTD] 💖.
The tests below all check the activation of paths in the data plane, not the control plane 🦋.
If you want the full list with short summaries, it's on the VSPERF "LTD Test Spec Overview" wiki page [LTDoverV] 💅.
5.1. Speed of Activation
Activation.RFC2889.AddressLearningRatePacketLatency.InitialPacketProcessingLatency
5.2. Accuracy of Activation
CPDP.Coupling.Flow.Addition
5.3. Reliability of Activation
5.4. Scale of Activation
Activation.RFC2889.AddressCachingCapacity
5.5. Speed of Operation
Throughput.RFC2544.PacketLossRateStress.RFC2544.0PacketLossThroughput.RFC2544.PacketLossRateFrameModificationThroughput.RFC2544.BackToBackFramesThroughput.RFC2889.MaxForwardingRateThroughput.RFC2889.ForwardPressureThroughput.RFC2889.BroadcastFrameForwardingThroughput.RFC2544.WorstN-BestNThroughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
5.6. Accuracy of Operation
5.7. Reliability of Operation
5.8. Scalability of Operation
Scalability.RFC2544.0PacketLossMemoryBandwidth.RFC2544.0PacketLoss.ScalabilityScalability.VNF.RFC2544.PacketLossProfileScalability.VNF.RFC2544.PacketLossRatio
5.9. Summary
Here's the big picture in one glance 👑:
|---------------------------------------------------------------------|
| | | | | |
| | SPEED | ACCURACY | RELIABILITY | SCALE |
| | | | | |
|---------------------------------------------------------------------|
| | | | | |
| Activation | X | X | X | X |
| | | | | |
|---------------------------------------------------------------------|
| | | | | |
| Operation | X | X | X | X |
| | | | | |
|---------------------------------------------------------------------|
| | | | | |
| De-activation| | | | |
| | | | | |
|---------------------------------------------------------------------|6. Security Considerations 🔒
bestie, this one is serious, so listen up 💀. Everything in this memo is about technology characterization of a Device Under Test/System Under Test (DUT/SUT). That means controlled stimuli, in a lab, using dedicated address space, and sticking to the constraints from the sections above 🧪.
The benchmarking network topology is supposed to be its own independent test setup. It MUST NOT be connected to any devices that could forward the test traffic into a production network or misroute traffic into the test management network. Not me getting a network outage because of a loose cable 😭.
Also, benchmarking is done "black-box" style, so it only relies on measurements you can observe from outside the DUT/SUT 🕵️♀️.
Special capabilities SHOULD NOT exist in the DUT/SUT just for benchmarking purposes. And any security implications coming from the DUT/SUT SHOULD be exactly the same in the lab as they are in production networks 🫶.
7. References 📚
7.1. Normative References
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.
- [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, DOI 10.17487/RFC2285, February 1998, <https://www.rfc-editor.org/info/rfc2285>.
- [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <https://www.rfc-editor.org/info/rfc2544>.
- [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN Switching Devices", RFC 2889, DOI 10.17487/RFC2889, August 2000, <https://www.rfc-editor.org/info/rfc2889>.
- [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October 2004, <https://www.rfc-editor.org/info/rfc3918>.
- [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., and J. Perser, "Packet Reordering Metrics", RFC 4737, DOI 10.17487/RFC4737, November 2006, <https://www.rfc-editor.org/info/rfc4737>.
- [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera, "Device Reset Characterization", RFC 6201, DOI 10.17487/RFC6201, March 2011, <https://www.rfc-editor.org/info/rfc6201>.
- [RFC6985] Morton, A., "IMIX Genome: Specification of Variable Packet Sizes for Additional Testing", RFC 6985, DOI 10.17487/RFC6985, July 2013, <https://www.rfc-editor.org/info/rfc6985>.
- [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A One-Way Delay Metric for IP Performance Metrics (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 2016, <https://www.rfc-editor.org/info/rfc7679>.
- [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
7.2. Informative References
- [BENCHMARK-METHOD] Huang, L., Ed., Rong, G., Ed., Mandeville, B., and B. Hickman, "Benchmarking Methodology for Virtualization Network Performance", Work in Progress, draft-huang-bmwg-virtual-network-performance-03, July 2017.
- [DanubeRel] OPNFV, "Danube", <https://wiki.opnfv.org/display/SWREL/Danube>.
- [IEEE802.1ac] IEEE, "IEEE Standard for Local and metropolitan area networks -- Media Access Control (MAC) Service Definition", IEEE 802.1AC-2016, DOI 10.1109/IEEESTD.2017.7875381, 2016, <https://standards.ieee.org/findstds/standard/802.1AC-2016.html>.
- [IEEE829] IEEE, "IEEE Standard for Software and System Test Documentation", IEEE 829-2008, DOI 10.1109/IEEESTD.2008.4578383, <http://ieeexplore.ieee.org/document/4578383/>.
- [IFA003] ETSI, "Network Functions Virtualisation (NFV); Acceleration Technologies; vSwitch Benchmarking and Acceleration Specification", ETSI GS NFV-IFA 003 V2.1.1, April 2016, <http://www.etsi.org/deliver/etsi_gs/NFV-IFA/001_099/003/02.01.01_60/gs_NFV-IFA003v020101p.pdf>.
- [LTD] Tahhan, M., "VSPERF Level Test Design (LTD)", <http://docs.opnfv.org/en/stable-danube/submodules/vswitchperf/docs/testing/developer/requirements/vswitchperf_ltd.html#>.
- [LTDoverV] Morton, A., "LTD Test Spec Overview", <https://wiki.opnfv.org/display/vsperf/LTD+Test+Spec+Overview>.
- [OPNFV] OPNFV, "OPNFV", <https://www.opnfv.org/>.
- [RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <https://www.rfc-editor.org/info/rfc1242>.
- [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, March 2009, <https://www.rfc-editor.org/info/rfc5481>.
- [RFC8172] Morton, A., "Considerations for Benchmarking Virtual Network Functions and Their Infrastructure", RFC 8172, DOI 10.17487/RFC8172, July 2017, <https://www.rfc-editor.org/info/rfc8172>.
- [TestTopo] Snyder, E., "Test Methodology", <https://wiki.opnfv.org/display/vsperf/Test+Methodology>.
- [VSPERFhome] Tahhan, M., "VSPERF Home", <https://wiki.opnfv.org/display/vsperf/VSperf+Home>.
Acknowledgements 💜
The authors gave a big shout-out to Scott Bradner, Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, Christian Trautman, Benoit Claise, and others for their reviews 🙌.
They also acknowledged the early work in [BENCHMARK-METHOD] and the useful discussions with its authors 🦋.
Authors' Addresses 🎀
Maryam Tahhan Intel
Email: maryam.tahhan@intel.com
Billy O'Mahony Intel
Email: billy.o.mahony@intel.com
Al Morton AT&T Labs 200 Laurel Avenue South Middletown, NJ 07748 United States of America
Phone: +1 732 420 1571 Fax: +1 732 368 1192 Email: acmorton@att.com
📦 where this page comes from
- ietf: rfc8204 2017-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms