RFC 6815

RFC 6815: Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful 💖

Okay bestie, RFC 6815 is an Informational doc from the IETF, not a Standards Track spec, just published so people can read it 📚. It's dated November 2012 and has ISSN 2070-1721. It Updates RFC 2544 🎀 and its header lists no obsoletes. The authors are S. Bradner (Harvard University), K. Dubray (Juniper Networks), J. McQuaid (Turnip Video), and A. Morton (AT&T Labs). The whole vibe is "please stop using RFC 2544 on live networks" 💅

Abstract ✨

The Benchmarking Methodology Working Group (BMWG) has been building key performance metrics and lab test methods since 1990, and they're still at it ✨. The RFC 2544 methods are designed to blast traffic that overloads device resources so you can see how much capacity a box has 💥. But overloading shared resources would likely wreck user traffic on a production network, and there are more negative consequences too 🙈. This memo narrows the scope of RFC 2544 and other IETF BMWG benchmarking work to isolated test environments only, and it encourages new standards work on measurement methods that are safe outside that scope 🫶.

Status of This Memo 🌈

This document is not an Internet Standards Track specification; it is published for informational purposes.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 5741.

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc6815.

Copyright Notice 💜

Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents 📋

1. Introduction ....................................................3
   1.1. Requirements Language ......................................4
2. Scope and Goals .................................................4
3. The Concept of an Isolated Test Environment .....................4
4. Why the Methods of RFC 2544 Are Intended Only for ITE ...........4
   4.1. Experimental Control and Accuracy ..........................4
   4.2. Containing Damage ..........................................5
5. Advisory on RFC 2544 Methods in Production Networks .............5
6. Considering Performance Testing in Production Networks ..........6
7. Security Considerations .........................................7
8. Acknowledgements ................................................7
9. References ......................................................8
   9.1. Normative References .......................................8
   9.2. Informative References .....................................8
Appendix A. Example of RFC 2544 Method Failure in Production
            Network Measurement ....................................9

---

1. Introduction ✨

This memo clarifies the scope and use of IETF Benchmarking Methodology Working Group (BMWG) tests, including [RFC2544], which discusses and defines several tests that may be used to characterize the performance of a network interconnecting device. Everyone reading this memo must read and fully understand [RFC2544] 📖.

Benchmarking methodologies (starting with [RFC2544]) have always relied on test conditions that can only be produced and replicated reliably in the lab 🧪. These methodologies are not appropriate for inclusion in wider specifications such as:

1. Validation of telecommunication service configuration, such as the Committed Information Rate (CIR).
2. Validation of performance metrics in a telecommunication Service Level Agreement (SLA), such as frame loss and latency.
3. Telecommunication service activation testing, where traffic that shares network resources with the test might be adversely affected.

Quick translation: "telecommunication service" means a network service provider contracts with a customer to transfer information between specified interfaces at different geographic locations, which is different from the generic term "service" 💅. Below, "production" means networks carrying live user traffic 🌍. [RFC2544] used the term "real-world" for production networks, to tell them apart from test networks.

Here's the tea ☕: [RFC2544] has been held up as the standard reference for the testing listed above, but we think the actual methods people use differ from [RFC2544] in significant ways 👀. Since the only citation is to [RFC2544], those modifications are opaque to the standards community and to users in general 🙈.

Applying the test traffic and methods described in [RFC2544] on a production network risks overloading shared resources, so there's a direct risk of harming user traffic if the methods get misused like that 😭. That's why the IETF BMWG developed this Applicability Statement for [RFC2544] to address the situation directly 🫶.

1.1. Requirements Language 🎀

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119] 🧚.

2. Scope and Goals 🌸

This memo clarifies the scope of [RFC2544] so the industry has guidance on where it applies, which is limited to laboratory testing 🔬.

3. The Concept of an Isolated Test Environment 🦋

An Isolated Test Environment (ITE) used with the methods of [RFC2544] (as illustrated in Figures 1 through 3 of [RFC2544]) has the ability to:

• contain the test streams to paths within the desired setup
• prevent non-test traffic from traversing the test setup

These features allow unfettered experimentation, and at the same time they protect lab equipment management/control LANs and other production networks from the unwanted effects of the test traffic 🛡️✨.

4. Why the Methods of RFC 2544 Are Intended Only for ITE 🔥

The following sections explain why the [RFC2544] methods only apply to isolated lab use, and what happens when you apply these methods outside the lab 🙃.

4.1. Experimental Control and Accuracy 🍓

All of the tests described in RFC 2544 require that the tester and device under test are the only devices on the networks that are transmitting data. If other traffic shows up (unwanted on the ITE network), the specified test conditions haven't been achieved, and flawed results are a likely consequence 💀.

If any other traffic appears and its amount changes over time, the repeatability of any test result will likely depend, to some degree, on how much other traffic there is and how much it varies 📉.

Other traffic makes accurate, repeatable, and consistent measurements of the device under test's performance very unlikely, since the complete details of the test conditions won't be reported 🙈.

For example, the RFC 2544 Throughput Test tries to characterize a maximum reliable load, so there will be testing above the maximum that causes packet/frame loss 📦. Any other traffic on the network will cause packet loss at a tester data rate lower than the rate that would be achieved without that extra traffic 😤.

4.2. Containing Damage 💥

[RFC2544] methods, specifically to determine Throughput as defined in [RFC1242] and other benchmarks, may overload the resources of the device under test, and they may cause failure modes in the device under test 🫠. Since failures can become the root cause of wider failures, it's clearly desirable to contain all test traffic within the ITE 🔒.

On top of that, this kind of testing can hurt any traffic that shares resources with the test stream(s), because in most cases the traffic load will be close to the capacity of the network links 🚗💨.

Appendix C.2.2 of [RFC2544] (as adjusted by errata) gives the private IPv4 address range for testing:

"...The network addresses 198.18.0.0 through 198.19.255.255 have been
assigned to the BMWG by the IANA for this purpose.  This assignment
was made to minimize the chance of conflict in case a testing device
were to be accidentally connected to part of the Internet.  The
specific use of the addresses is detailed below."

So basically, devices on the Internet may be configured to discard any traffic they see in that address range, because it's meant for laboratory ITE use only 🚫. That means if testers using the assigned testing address ranges are connected to the Internet and test packets get forwarded across it, the packets will likely be discarded and the test won't work 😩.

We also note that a range of IPv6 addresses has been assigned to BMWG for laboratory test purposes, in [RFC5180] (as amended by errata) 🌐.

See the Security Considerations section below for further considerations on containing damage 🔐.

5. Advisory on RFC 2544 Methods in Production Networks ✨

Okay so the tests in [RFC2544] were built to measure how well network devices perform, not whole networks, and DEFINITELY not production networks carrying real user traffic on shared resources 😭 Running them outside the isolated test environment is going to cause some seriously undesirable consequences, bestie 💀

One big problem is that [RFC2544] methods treat frame loss as the sign that resources are maxed out. But in real life, link-layer and physical-layer errors mean production networks are never loss-free 🙈 So the [RFC2544] methods can't correctly assess Throughput when loss from uncontrolled sources is around. Frame loss at the SLA levels of some networks could mess up every single iteration of Throughput testing (any time a step sends enough packets to run into facility-related loss). Flawed results waste the time and resources of the testing service user and the service provider, who then have to argue about the measurement 😩 These are more examples of harm that following this advisory should help avoid. Check out Appendix A for an example 🎀

The methods in [RFC2544] are meant to generate traffic that overloads network device resources so their capacity can be measured 🔥 Overloading shared resources would probably wreck user traffic performance on a production network. These tests MUST NOT be used on production networks, for all the reasons above. They won't give a reliable or accurate benchmarking result on a production network 💅

Also, [RFC2544] methods have never been validated on a network path, even one that isn't part of a production network and carries no other traffic 👀 Nobody knows if the tests can measure valid and reliable performance across a multi-device, multi-network path. Some tests might work in some path scenarios, but that work hasn't been done or shared with the IETF community yet. So the BMWG says this kind of testing is contraindicated 🚫

6. Considering Performance Testing in Production Networks 🦋

The IETF handled production network performance measurement by chartering a different working group: IP Performance Metrics (IPPM) 🌸 IPPM has developed a set of standard metrics to assess the quality, performance, and reliability of Internet packet transfer services. Network operators, end users, or independent testing groups can all measure these metrics ✨ Heads up though: some IPPM metrics have names similar to [RFC2544] metrics but are actually different, so it's giving confusion if you skip the details 🙈

IPPM hasn't standardized methods for raw capacity measurement of Internet paths yet 🧚 Testing like that needs to properly account for the strong possibility of degrading any other traffic present because of congestion. There are no specific methods proposed for activating a packet transfer service in IPPM right now. So if you need to run capacity tests on production networks, you should actively join standards development so your methods get proper industry review and agreement, either in the IETF or in other standards development organizations 🫶

Other standards can help fill gaps in telecommunication service testing. For example, the IETF has lots of standards for network Operations, Administration, and Maintenance (OAM) 💜 And ITU-T Study Group 12 has a Recommendation on service activation test methodology [Y.1564] 🌈

The world will not spin off its axis while we wait for appropriate, standardized methods to come out of the consensus process 🌍✨

7. Security Considerations 💀🔐

This Applicability Statement is here to help preserve the security of the Internet 🛡️ It clarifies that the scope of [RFC2544] and other BMWG memos is limited to testing in a laboratory ITE. That way we avoid accidental Denial-of-Service attacks or congestion caused by high-volume test streams. This is SO important, bestie 😭

All benchmarking activities are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the other constraints in [RFC2544] 🎀

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network or misroute traffic to the test management network. No exceptions, no cap 🚫

Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the device under test/system under test (DUT/SUT) 🖤

Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks 🔒 Lab and prod should match, periodt 💅

8. Acknowledgements 🌷

Thanks to Matt Zekauskas, Bill Cerveny, Barry Constantine, Curtis Villamizar, David Newman, and Adrian Farrel for suggesting improvements to this memo 🫶

Specifically, Al Morton would like to thank his coauthors, who make up the complete set of Chairmen-Emeritus of the BMWG 👑 They came back from other pursuits to develop this statement and see it through to approval. This has been a rare privilege, one that likely will not be matched in the IETF again 🥹

Scott Bradner served as Chairman from 1990 to 1993 Jim McQuaid served as Chairman from 1993 to 1995 Kevin Dubray served as Chairman from 1995 to 2006

It's all about the band 🎸💖

9. References 📚

9.1. Normative References

[RFC1242]  Bradner, S., "Benchmarking terminology for network
           interconnection devices", RFC 1242, July 1991.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119, March 1997.

[RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
           Network Interconnect Devices", RFC 2544, March 1999.

[RFC5180]  Popoviciu, C., Hamza, A., Van de Velde, G., and D.
           Dugatkin, "IPv6 Benchmarking Methodology for Network
           Interconnect Devices", RFC 5180, May 2008.

9.2. Informative References

[Bryant]   Bonica, R. and S. Bryant, "RFC2544 Testing in Production
           Network", Work in Progress, October 2012.

[Y.1564]   ITU-T Recommendation Y.1564, "Ethernet Service Activation
           Test Methodology", March 2011.

Appendix A. Example of RFC 2544 Method Failure in Production Network Measurement 🧐

This Appendix gives an example of how [RFC2544] methods applied on production networks can easily cause harm through flawed and misleading results 💀

The [RFC2544] Throughput benchmarking method usually includes the following steps:

a. Set the offered traffic level, less than max of the ingress link(s).

b. Send the test traffic through the device under test (DUT) and count all frames successfully transferred.

c. If all frames are received, increment traffic level and repeat step b.

d. If one or more frames are lost, the level is in the DUT-overload region (step b may be repeated at a reduced traffic level to more exactly determine the maximum rate at which none of the frames are dropped by the DUT, defined as the Throughput [RFC1242]).

e. Report the Throughput values, the x-y of graph of frame size and Throughput, and other information in accordance with [RFC2544].

In this method, frame loss is the sole indicator of overload and therefore the determining factor in the measurement of Throughput using the [RFC2544] methodology (even though the results may not report frame loss per se) 🤔

Frame loss is subject to many factors besides operating above the Throughput traffic level 😬 These include optical interference (which may be due to dirty interfaces, crossover from other signals, fiber bend and temperature, etc.) and electrical interference (caused by local sources of radio signals, electrical spikes, solar particles, etc.). In the laboratory environment, many of these issues can be carefully controlled through cleaning and isolation ✨ Since [RFC2544] methodologies are primarily intended to test devices and not paths, the total length of path, the number of interfaces, and the compound risk of random frame loss can be kept to a minimum 🧼

In a production network, however, there will be many interfaces and many kilometers of path under test. This considerably increases the risk of random frame loss 📈

The risk of frame loss caused by outside effects is significantly higher in production networks, and significantly higher with long paths (both those with long physical path lengths and those with large numbers of interfaces in the path) 😱 So the risk of falsely low reported Throughput using an [RFC2544] methodology test is considerably increased in a production network 🙃

Therefore, to successfully conduct tests with similar objectives to those in [RFC2544] in a production network, it will be necessary to develop modifications to the methodologies defined in [RFC2544] and standards to describe them. See [Bryant] for an in-progress effort and [Y.1564] for an approved method adapted to production service activation 🫶

Authors' Addresses 📮

Scott Bradner
Harvard University
1350 Mass. Ave., Room 760
Cambridge, MA  02138
USA

Phone: +1 617 495 3864
EMail: sob@harvard.edu
URI:   http://www.sobco.com


Kevin Dubray
Juniper Networks


Jim McQuaid
Turnip Video
6 Cobbleridge Court
Durham, North Carolina  27713
USA

Phone: +1 919-619-3220
EMail: jim@turnipvideo.com
URI:   www.turnipvideo.com


Al Morton
AT&T Labs
200 Laurel Avenue South
Middletown,, NJ  07748
USA

Phone: +1 732 420 1571
Fax:   +1 732 368 1192
EMail: acmorton@att.com
URI:   http://home.comcast.net/~acmacm/