RFC 8172

Considerations for Benchmarking Virtual Network Functions and Their Infrastructure

📅 published July 2017, from ietf

Abstract ✨

Okay bestie, let's start with the header facts. 🎀 This is RFC 8172, titled Considerations for Benchmarking Virtual Network Functions and Their Infrastructure. It was written by A. Morton from AT&T Labs, and it's dated July 2017. It's Informational, so it's not a Standards Track spec 🙈, and its ISSN is 2070-1721. It came from the Internet Engineering Task Force (IETF). The header has no Obsoletes or Updates line, so it doesn't replace or change any other RFC.

The Benchmarking Methodology Working Group (BMWG) has historically done lab testing on dedicated physical gear for internetworking functions. This memo looks at the extra things to think about when network functions get virtualized and run on general-purpose hardware. 💅

Status of This Memo

This document is not an Internet Standards Track specification. It's published for informational purposes. 📚

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841.

Info about the current status of this document, any errata, and how to give feedback on it can be found at http://www.rfc-editor.org/info/rfc8172. 🔗

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved. 🌈

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info), as in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Considerations for Hardware and Testing . . . . . . . . . . .   4
     3.1.  Hardware Components . . . . . . . . . . . . . . . . . . .   4
     3.2.  Configuration Parameters  . . . . . . . . . . . . . . . .   5
     3.3.  Testing Strategies  . . . . . . . . . . . . . . . . . . .   6
     3.4.  Attention to Shared Resources . . . . . . . . . . . . . .   7
   4.  Benchmarking Considerations . . . . . . . . . . . . . . . . .   8
     4.1.  Comparison with Physical Network Functions  . . . . . . .   8
     4.2.  Continued Emphasis on Black-Box Benchmarks  . . . . . . .   8
     4.3.  New Benchmarks and Related Metrics  . . . . . . . . . . .   9
     4.4.  Assessment of Benchmark Coverage  . . . . . . . . . . . .  10
     4.5.  Power Consumption . . . . . . . . . . . . . . . . . . . .  12
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .  12
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  13
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  13
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .  13
     7.2.  Informative References  . . . . . . . . . . . . . . . . .  14
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  15
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  15

1. Introduction ✨

BMWG has traditionally done lab characterization of dedicated physical implementations of internetworking functions (also called physical network functions, or PNFs). The black-box benchmarks for throughput, latency, forwarding rates, and more have served the industry for many years. [RFC1242] and [RFC2544] are the cornerstones of this work. 🍓

Service providers and vendors have a shared set of goals: cut costs, gain flexibility in network devices, and drastically reduce deployment time. Network Function Virtualization (NFV) promises to hit those goals, which is why it's gotten so much attention. It now seems certain that some network functions will be virtualized. Cloud computing and virtual desktops have been a huge success, as long as the network path has enough capacity, performance, and wide deployment, and many of the same techniques will help with NFV. 🦋

In the context of Virtual Network Functions (VNFs), the supporting Infrastructure needs general-purpose computing systems, storage systems, networking systems, virtualization support systems (such as hypervisors), and management systems for the virtual and physical resources. There will be lots of potential suppliers of Infrastructure systems, with a ton of flexibility in configuring them for the best performance. There are also lots of potential suppliers of VNFs, which adds even more combinations to the mix. 🧚 Splitting up hardware and software suppliers has a big impact on benchmarking. Much more of the internal configuration of the black-box Device Under Test (DUT) now has to be specified and reported with the results, so that testing can be repeated and compared later on. 👑

Here's a user story for extra background and motivation:

I'm designing and building my NFV Infrastructure platform. The first steps were easy because I had a small number of categories of VNFs to support and the VNF vendor gave hardware recommendations that I followed. Now I need to deploy more VNFs from new vendors, and there are different hardware recommendations. How well will the new VNFs perform on my existing hardware? Which among several new VNFs in a given category are most efficient in terms of capacity they deliver? And, when I operate multiple categories of VNFs (and PNFs) concurrently on a hardware platform such that they share resources, what are the new performance limits, and what are the software design choices I can make to optimize my chosen hardware platform? Conversely, what hardware platform upgrades should I pursue to increase the capacity of these concurrently operating VNFs?

Check out <http://www.etsi.org/technologies-clusters/technologies/nfv> for more background. The white papers there could be a useful starting point. 📝 The "NFV Performance & Portability Best Practices" document [NFV.PER001] is especially relevant to BMWG. There are also documents in the Approved ETSI NFV Specifications [Approved_ETSI_NFV], including ones that describe Infrastructure performance aspects and service quality metrics. The drafts in the ETSI NFV Open Area [Draft_ETSI_NFV] might also be relevant to benchmarking. 💜

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 💅

2. Scope 🌸

BMWG is currently looking at the new topic of Virtual Network Functions and related Infrastructure. The goal is to recognize common issues from the start. Background materials from other standards development organizations and Open Source development projects are being used for this, such as IETF, ETSI NFV, and the Open Platform for Network Function Virtualization (OPNFV). 🌈

This memo looks into the extra methodological considerations needed when benchmarking VNFs that are instantiated and hosted on general-purpose hardware. They might run on bare metal hypervisors [BareMetal] or in other isolation environments, like Linux containers. An essential consideration is benchmarking physical and Virtual Network Functions in the same way when possible, so you can compare them directly. Benchmarking combinations of physical and virtual devices and functions in a System Under Test (SUT) is another topic people care a lot about. 🔥

There's also a closely related goal: investigating benchmarks for how much a general-purpose platform can host a bunch of VNF instances. Existing networking technology benchmarks will be considered for adapting to NFV and closely associated technologies too. ⭐

Something that's a non-goal: overlapping with traditional computer benchmark development and its specific metrics (for example, SPECmark suites such as SPEC CPU). 🙅‍♀️

Also a continued non-goal: any form of architecture development related to NFV and associated technologies in BMWG. That's consistent with all chartered work since BMWG began in 1989. 🎀

3. Considerations for Hardware and Testing 🧚

This section lists the new considerations that must be addressed to benchmark VNF(s) and their supporting Infrastructure. The SUT is made up of the hardware platform components, the VNFs installed, and lots of other supporting systems. It's critical to document every aspect of the SUT so that testing can be repeated. 📋

3.1. Hardware Components

The following new hardware components will become part of the test setup: 🔧

  1. High-volume server platforms (general-purpose, possibly with virtual technology enhancements)
  2. Storage systems with large capacity, high speed, and high reliability
  3. Network interface ports specially designed for efficient service of many virtual Network Interface Cards (NICs)
  4. High-capacity Ethernet switches

The components above are the subjects for developing specialized benchmarks that focus on the special demands of network function deployment. 💖

Labs that compare different VNFs may be able to use the same hardware platform for many studies. That lasts until the steady march of innovation overtakes its capabilities, which is what happens with a lab's traffic generation and testing devices today. 🫶

3.2. Configuration Parameters

It's necessary to configure and document the settings for the entire general-purpose platform, so that testing is repeatable and future comparisons are possible. That includes, but is clearly not limited to, the following: 📝

  • number of server blades (shelf occupation)
  • CPUs
  • caches
  • memory
  • storage system
  • I/O

It also includes the configurations that support the devices hosting the VNF itself: 🏠

  • Hypervisor (or other forms of virtual function hosting)
  • Virtual Machine (VM)
  • Infrastructure virtual network (which interconnects virtual machines with physical network interfaces or with each other through virtual switches, for example)

And finally, the VNF itself, with items such as: 🦋

  • specific function being implemented in VNF
  • reserved resources for each function (e.g., CPU pinning and Non-Uniform Memory Access (NUMA) node assignment)
  • number of VNFs (or sub-VNF components, each with its own VM) in the service function chain (see Section 1.1 of [RFC7498] for a definition of service function chain)
  • number of physical interfaces and links transited in the service function chain

In the physical device benchmarking context, the vendor decided most of the corresponding Infrastructure configuration choices. Even though the platform itself is now one of the configuration variables, it's important to keep the emphasis on the networking benchmarks and capture the platform variables as input factors. 💜✨

3.3. Testing Strategies

The concept of characterizing performance at capacity limits may change. For example:

  1. It may be more representative of system capacity to characterize the case where the VMs hosting the VNFs are operating at 50% utilization and therefore sharing the "real" processing power across many VMs.
  2. Another important test case stems from the need to partition (or isolate) network functions. A noisy neighbor (VM hosting a VNF in an infinite loop) would ideally be isolated; the performance of other VMs would continue according to their specifications, and tests would evaluate the degree of isolation.
  3. System errors will likely occur as transients, implying a distribution of performance characteristics with a long tail (like latency) and leading to the need for longer-term tests of each set of configuration and test parameters.
  4. The desire for elasticity and flexibility among network functions will include tests where there is constant flux in the number of VM instances, the resources the VMs require, and the setup/ teardown of network paths that support VM connectivity. Requests for and instantiation of new VMs, along with releases for VMs hosting VNFs that are no longer needed, would be a normal operational condition. In other words, benchmarking should include scenarios with production life-cycle management of VMs and their VNFs and network connectivity in progress, including VNF scaling up/down operations, as well as static configurations.
  5. All physical things can fail, and benchmarking efforts can also examine recovery aided by the virtual architecture with different approaches to resiliency.
  6. The sheer number of test conditions and configuration combinations encourage increased efficiency, including automated testing arrangements, combination sub-sampling through an understanding of inter-relationships, and machine-readable test results.

3.4. Attention to Shared Resources

Since many components of the new NFV Infrastructure are virtual, test setup design must have prior knowledge of interactions/dependencies within the various resource domains in the SUT. For example, a virtual machine performing the role of a traditional tester function, such as generating and/or receiving traffic, should avoid sharing any SUT resources with the DUT. Otherwise, the results will have unexpected dependencies not encountered in physical device benchmarking.

Note that the term "tester" has traditionally referred to devices dedicated to testing in BMWG literature. In this new context, "tester" additionally refers to functions dedicated to testing, which may be either virtual or physical. "Tester" has never referred to the individuals performing the tests.

The possibility to use shared resources in test design while producing useful results remains one of the critical challenges to overcome. Benchmarking setups may designate isolated resources for the DUT and other critical support components (such as the host/ kernel) as the first baseline step and add other loading processes. The added complexity of each setup leads to shared-resource testing scenarios, where the characteristics of the competing load (in terms of memory, storage, and CPU utilization) will directly affect the benchmarking results (and variability of the results), but the results should reconcile with the baseline.

The physical test device remains a solid foundation to compare with results using combinations of physical and virtual test functions or results using only virtual testers when necessary to assess virtual interfaces and other virtual functions.

4. Benchmarking Considerations 🌸

This section discusses considerations related to benchmarks applicable to VNFs and their associated technologies.

4.1. Comparison with Physical Network Functions 🦋

In order to compare the performance of VNFs and system implementations with their physical counterparts, identical benchmarks must be used. Since BMWG has already developed specifications for many network functions, there will be re-use of existing benchmarks through references, while allowing for the possibility of benchmark curation during development of new methodologies. Consideration should be given to quantifying the number of parallel VNFs required to achieve comparable scale/capacity with a given physical device or whether some limit of scale was reached before the VNFs could achieve the comparable level. Again, implementation based on different hypervisors or other virtual function hosting remain as critical factors in performance assessment.

4.2. Continued Emphasis on Black-Box Benchmarks 🍓

When the network functions under test are based on open-source code, there may be a tendency to rely on internal measurements to some extent, especially when the externally observable phenomena only support an inference of internal events (such as routing protocol convergence observed in the data plane). Examples include CPU/Core utilization, network utilization, storage utilization, and memory committed/used. These "white-box" metrics provide one view of the resource footprint of a VNF. Note that the resource utilization metrics do not easily match the 3x4 Matrix, described in Section 4.4.

However, external observations remain essential as the basis for benchmarks. Internal observations with fixed specification and interpretation may be provided in parallel (as auxiliary metrics), to assist the development of operations procedures when the technology is deployed, for example. Internal metrics and measurements from open-source implementations may be the only direct source of performance results in a desired dimension, but corroborating external observations are still required to assure the integrity of measurement discipline was maintained for all reported results.

A related aspect of benchmark development is where the scope includes multiple approaches to a common function under the same benchmark. For example, there are many ways to arrange for activation of a network path between interface points, and the activation times can be compared if the start-to-stop activation interval has a generic and unambiguous definition. Thus, generic benchmark definitions are preferred over technology/protocol-specific definitions where possible.

There will be new classes of benchmarks needed for network design and assistance when developing operational practices (possibly automated management and orchestration of deployment scale). Examples follow in the paragraphs below, many of which are prompted by the goals of increased elasticity and flexibility of the network functions, along with reduced deployment times.

  • Time to deploy VNFs: In cases where the general-purpose hardware is already deployed and ready for service, it is valuable to know the response time when a management system is tasked with "standing up" 100s of virtual machines and the VNFs they will host.
  • Time to migrate VNFs: In cases where a rack or shelf of hardware must be removed from active service, it is valuable to know the response time when a management system is tasked with "migrating" some number of virtual machines and the VNFs they currently host to alternate hardware that will remain in service.
  • Time to create a virtual network in the general-purpose Infrastructure: This is a somewhat simplified version of existing benchmarks for convergence time, in that the process is initiated by a request from (centralized or distributed) control, rather than inferred from network events (link failure). The successful response time would remain dependent on data-plane observations to confirm that the network is ready to perform.
  • Effect of verification measurements on performance: A complete VNF, or something as simple as a new policy to implement in a VNF, is implemented. The action to verify instantiation of the VNF or policy could affect performance during normal operation.

Also, it appears to be valuable to measure traditional packet transfer performance metrics during the assessment of traditional and new benchmarks, including metrics that may be used to support service engineering such as the spatial composition metrics found in [RFC6049]. Examples include mean one-way delay in Section 4.1 of [RFC6049], Packet Delay Variation (PDV) in [RFC5481], and Packet Reordering [RFC4737] [RFC4689].

4.4. Assessment of Benchmark Coverage 🗂️

okay so benchmarks are way easier to keep track of when you sort them by life-cycle stage and by the performance criteria they were built to check 💅 The table below (based on [X3.102]) gives you a grid where you can literally see which combos are covered and which are giving empty ✨

|----------------------------------------------------------|
|               |             |            |               |
|               |   SPEED     |  ACCURACY  |  RELIABILITY  |
|               |             |            |               |
|----------------------------------------------------------|
|               |             |            |               |
|  Activation   |             |            |               |
|               |             |            |               |
|----------------------------------------------------------|
|               |             |            |               |
|  Operation    |             |            |               |
|               |             |            |               |
|----------------------------------------------------------|
|               |             |            |               |
| De-activation |             |            |               |
|               |             |            |               |
|----------------------------------------------------------|

So like, the "Time to deploy VNFs" benchmark from earlier lives at the crossing of Activation and Speed 🚀 But bestie, it's giving so many other things we could measure there, like the "percentage of unsuccessful VM/VNF stand-ups" across a set of 100 attempts 🙈 That's the real point: Activation and De-activation are the key stages for NFV and the Infrastructure under it, so we should totally go beyond the usual normal-operation benchmarks 🔥 Checking coverage with this table (people call it the 3x3 Matrix) is honestly a worthwhile exercise in BMWG 🫶

In one of the first times BMWG used the 3x3 Matrix [SDN-BENCHMARK], we hit a snag: metrics about measured size, capacity, or scale just don't fit neatly into any of the three columns 😭 After some discussion, we fixed it in two ways:

  • Add a column, Scale, for categorizing and assessing benchmark coverage (without measured results). You can see this in [OPNFV-BENCHMARK] (and a variation shows up in [SDN-BENCHMARK]). This is the 3x4 Matrix 💜
  • If you're using the matrix to report results in an organized way, keep size, capacity, and scale metrics separate from the 3x3 Matrix and fold them into the report along with the other qualifications of the results 🎀

Quick side note: resource utilization metrics (like CPU) don't fit in the matrix at all 💀 They aren't benchmarks, and leaving them out is exactly what shows they're auxiliary metrics. Resource assignments are configuration parameters, so they get reported separately 🌸

This whole approach encourages using the 3x3 Matrix to organize reports, where the capacity the metrics were measured at can go in the matrix title. If you have results at multiple capacities, you'd end up with separate 3x3 Matrices (assuming there are enough measurements to organize that way) 🦋

For example, results for each VM and VNF could go in the 3x3 Matrix, organized to show resource occupation (CPU Cores) on one physical computing system, like this 👑

                 VNF#1
             .-----------.
             |__|__|__|__|
   Core 1    |__|__|__|__|
             |__|__|__|__|
             |  |  |  |  |
             '-----------'
                 VNF#2
             .-----------.
             |__|__|__|__|
   Cores 2-5 |__|__|__|__|
             |__|__|__|__|
             |  |  |  |  |
             '-----------'
                 VNF#3             VNF#4             VNF#5
             .-----------.    .-----------.     .-----------.
             |__|__|__|__|    |__|__|__|__|     |__|__|__|__|
   Core 6    |__|__|__|__|    |__|__|__|__|     |__|__|__|__|
             |__|__|__|__|    |__|__|__|__|     |__|__|__|__|
             |  |  |  |  |    |  |  |  |  |     |  |  |  |  |
             '-----------'    '-----------'     '-----------'
                  VNF#6
             .-----------.
             |__|__|__|__|
   Core 7    |__|__|__|__|
             |__|__|__|__|
             |  |  |  |  |
             '-----------'

You can build these tables step by step: start with VNF#1 and one Core, then add VNFs based on which Cores they're supported by 🧚 X-Y plots of the critical benchmarks would also show how increased hardware utilization affects things ✨ All the VNFs could be the same type, or, to match a production environment, you could mix VNF types and categories 🌈 In this figure, VNFs #3-#5 are assumed to need small CPU resources, while VNF#2 needs four Cores to do its job 💖

4.5. Power Consumption ⚡

okay so there's still unfinished work on benchmarking physical network function power consumption in a meaningful way 😩 But wanting to measure the physical Infrastructure that supports the virtual functions only makes the need bigger 🍓 Both maximum power consumption and dynamic power consumption (with varying load) would be useful. The Intelligent Platform Management Interface (IPMI) standard [IPMI2.0] has been implemented by lots of manufacturers, and it supports measuring instantaneous energy consumption 🔋

To assess the instantaneous energy consumption of virtual resources, it may be possible to estimate the value with an overall metric based on utilization readings, following [NFVIaas-FRAMEWORK] 🫶

5. Security Considerations 🔐

Benchmarking activities described in this memo are limited to technology characterization of a DUT/SUT using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above 🧪

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. This is serious, bestie 😭🚨

Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT/SUT 🖤

Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. No shortcuts, no cap 🙅‍♀️💅

6. IANA Considerations 🌸

This document does not require any IANA actions. Nothing to register, periodt ✨

7. References 📚

7.1. Normative References

[NFV.PER001] ETSI, "Network Function Virtualization: Performance & Portability Best Practices", ETSI GS NFV-PER 001, V1.1.2, December 2014.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <http://www.rfc-editor.org/info/rfc2119>.

[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <http://www.rfc-editor.org/info/rfc2544>.

[RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, "Terminology for Benchmarking Network-layer Traffic Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689, October 2006, <http://www.rfc-editor.org/info/rfc4689>.

[RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, S., and J. Perser, "Packet Reordering Metrics", RFC 4737, DOI 10.17487/RFC4737, November 2006, <http://www.rfc-editor.org/info/rfc4737>.

[RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for Service Function Chaining", RFC 7498, DOI 10.17487/RFC7498, April 2015, <http://www.rfc-editor.org/info/rfc7498>.

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <http://www.rfc-editor.org/info/rfc8174>.

7.2. Informative References

[Approved_ETSI_NFV] ETSI, Network Functions Virtualisation Technical Committee, "ETSI NFV", <http://www.etsi.org/standards-search>.

[BareMetal] Popek, G. and R. Goldberg, "Formal requirements for virtualizable third generation architectures", Communications of the ACM, Volume 17, Issue 7, Pages 412-421, DOI 10.1145/361011.361073, July 1974.

[Draft_ETSI_NFV] ETSI, "Network Functions Virtualisation: Specifications", <http://www.etsi.org/technologies-clusters/technologies/nfv>.

[IPMI2.0] Intel Corporation, Hewlett-Packard Company, NEC Corporation, and Dell Inc., "Intelligent Platform Management Interface Specification v2.0 (with latest errata)", April 2015, <http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/ipmi-intelligent-platform-mgt-interface-spec-2nd-gen-v2-0-spec-update.pdf>.

[NFVIaas-FRAMEWORK] Krishnan, R., Figueira, N., Krishnaswamy, D., Lopez, D., Wright, S., Hinrichs, T., Krishnaswamy, R., and A. Yerra, "NFVIaaS Architectural Framework for Policy Based Resource Placement and Scheduling", Work in Progress, draft-krishnan-nfvrg-policy-based-rm-nfviaas-06, March 2016.

[OPNFV-BENCHMARK] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking Virtual Switches in OPNFV", Work in Progress, draft-ietf-bmwg-vswitch-opnfv-04, June 2017.

[RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <http://www.rfc-editor.org/info/rfc1242>.

[RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, DOI 10.17487/RFC5481, March 2009, <http://www.rfc-editor.org/info/rfc5481>.

[RFC6049] Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, <http://www.rfc-editor.org/info/rfc6049>.

[SDN-BENCHMARK] Vengainathan, B., Basil, A., Tassinari, M., Manral, V., and S. Banks, "Terminology for Benchmarking SDN Controller Performance", Work in Progress, draft-ietf-bmwg-sdn-controller-benchmark-term-04, June 2017.

[X3.102] ANSI, "Information Systems - Data Communication Systems and Services - User-Oriented Performance Parameters Communications Framework", ANSI X3.102, 1983.

Acknowledgements 🙏✨

Okay bestie, the author gives a huge shoutout to everyone who helped make this happen 🫶. Back in November 2013 they had an encouraging convo about this topic with Mukhtiar Shaikh and Ramki Krishnan 💜. Bhavani Parise and Ilya Varlashkin dropped useful suggestions that helped expand these considerations 🌸. Bhuvaneswaran Vengainathan already tried the 3x3 Matrix with the SDN controller document and was part of so many discussions 💅. Scott Bradner quickly spotted shared resource dependencies in an early vSwitch measurement proposal, so that topic got included here as a key consideration 👑. Barry Constantine's comments after the BMWG session at IETF 92 pushed things further, and ngl the session itself was an affirmation for this memo 🦋. Then there are the iconic contributions from Maryam Tahhan, Marius Georgescu, Jacob Rapp, Saurabh Chattopadhyay, and others 🧚.

Author's Address 💌

Al Morton
AT&T Labs
200 Laurel Avenue South
Middletown, NJ  07748
United States of America

Phone: +1 732 420 1571
Fax:   +1 732 368 1192
Email: acmorton@att.com
📦 where this page comes from