RFC 2544
Benchmarking Methodology for Network Interconnect Devices
RFC 2544 💖 Benchmarking Methodology for Network Interconnect Devices
Okay bestie, let's get the vibe check first 🎀. This one is from March 1999 and it's an Informational doc, so it's sharing info and it doesn't set an Internet standard. It came from the Network Working Group, written by S. Bradner (Harvard University) and J. McQuaid (NetScout Systems). It obsoletes RFC 1944 🍓 and the header doesn't list anything it updates. Iconic lineup, honestly 👑.
Status of This Memo 📜
This memo gives information to the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.
Copyright Notice ⭐
Copyright (C) The Internet Society (1999). All Rights Reserved.
IESG Note 🧚
This doc is a republication of RFC 1944 that fixes the values for the IP addresses assigned as the default addresses for networking test equipment. (See section C.2.2 ). This RFC replaces and obsoletes RFC 1944.
Abstract 🌸
This doc talks about and defines a bunch of tests that can be used to describe the performance characteristics of a network interconnecting device. Besides defining the tests, it also describes specific formats for reporting the results 📊. Appendix A lists the tests and conditions we think should be included for specific cases and gives extra info about testing practices. Appendix B is a reference list of maximum frame rates for specific frame sizes on various media, and Appendix C gives some examples of frame formats to use in testing 🦋.
1. Introduction ✨
Okay so vendors are notorious for "specsmanship" 💅, basically trying to make their products look better in the marketplace. This usually involves "smoke & mirrors" that confuse potential users. Not it 🙄.
This doc defines a specific set of tests vendors can use to measure and report the performance characteristics of network devices. The results give users comparable data from different vendors so they can evaluate these devices fairly 🫶.
A previous doc, "Benchmarking Terminology for Network Interconnect Devices" (RFC 1242), defined many of the terms used in this one. You should check out the terminology doc before trying to use this one, no cap 🤓.
2. Real world 🌍
When the authors wrote this, they kept in mind that the apparatus to perform the tests has to actually be built. We don't know of any "off the shelf" equipment that can run all of the tests, but in our opinion, that kind of equipment can be constructed 🔥.
3. Tests to be run 🏃♀️
There are a bunch of tests in this doc, and not all of them apply to all types of devices under test (DUTs). Vendors should perform all of the tests that can be supported by a specific type of product. The authors get that it will take a considerable amount of time to run all of the recommended tests under all of the recommended conditions, but ngl the results are worth the effort 💖. Appendix A lists some of the tests and conditions we think should be included for specific cases.
4. Evaluating the results 📈
Running all of the recommended tests gives you a LOT of data 😭. Much of it won't apply to evaluating the devices in every situation. For example, the rate at which a router forwards IPX frames is basically useless when you're picking a router for an environment that doesn't (and won't) support that protocol 💀. Evaluating even the data that is relevant to a particular network install takes experience that might not be easy to get. Also, picking which tests to run and evaluating the test data has to be done with an understanding of generally accepted testing practices about repeatability, variance, and statistical significance of small numbers of trials 🧠.
5. Requirements 🎀
In this doc, the words that define the significance of each requirement are capitalized. Here they are:
- "MUST" This word, or the words "REQUIRED" and "SHALL", mean that the item is an absolute requirement of the specification.
- "SHOULD" This word, or the adjective "RECOMMENDED", means there may be valid reasons in particular circumstances to ignore this item, but the full implications should be understood and the case carefully weighed before choosing a different course 🤔.
- "MAY" This word, or the adjective "OPTIONAL", means this item is truly optional. One vendor may choose to include the item because a particular marketplace requires it or because it enhances the product, for example. Another vendor may omit the same item 🌈.
So here's the tea ☕. An implementation is not compliant if it fails to satisfy one or more of the MUST requirements for the protocols it implements. An implementation that satisfies all the MUST and all the SHOULD requirements for its protocols is called "unconditionally compliant" 👑. One that satisfies all the MUST requirements but not all the SHOULD requirements for its protocols is called "conditionally compliant" 🦋.
6. Test set up 🧚♀️
The ideal way to run this series of tests is to use a tester with both transmitting and receiving ports 💜. Connections go from the sending ports of the tester to the receiving ports of the DUT, and from the sending ports of the DUT back to the tester (see Figure 1). Since the tester both sends the test traffic and gets it back, it can easily figure out, after the traffic has been forwarded by the DUT, whether all the transmitted packets were received and whether the correct packets showed up 😎. The same functionality can be achieved with separate transmitting and receiving devices (see Figure 2). Unless they're remotely controlled by some computer in a way that simulates the single tester, though, the labor needed to perform some tests accurately (especially the throughput test) can be prohibitive 😩.
+------------+
| |
+------------| tester |<-------------+
| | | |
| +------------+ |
| |
| +------------+ |
| | | |
+----------->| DUT |--------------+
| |
+------------+
Figure 1
+--------+ +------------+ +----------+
| | | | | |
| sender |-------->| DUT |--------->| receiver |
| | | | | |
+--------+ +------------+ +----------+
Figure 26.1 Test set up for multiple media types 🌸
There are two different setups you could use to test a DUT that's used in real-world networks to connect networks with different media types, like a local Ethernet to a backbone FDDI ring, for example. If the tester supports both media types, you'd use the set up shown in Figure 1 ✨.
The other test set up uses two identical DUTs (see Figure 3). In lots of cases this set up might simulate the real world more accurately, like connecting two LANs together with a WAN link or a high speed backbone 🌐. It wouldn't be as good at simulating a system where clients on an Ethernet LAN are interacting with a server on an FDDI backbone 🙃.
+-----------+
| |
+---------------------| tester |<---------------------+
| | | |
| +-----------+ |
| |
| +----------+ +----------+ |
| | | | | |
+------->| DUT 1 |-------------->| DUT 2 |---------+
| | | |
+----------+ +----------+
Figure 37. DUT set up 🎀
Before starting the tests, the DUT being tested MUST be configured following the instructions provided to the user. Specifically, all of the supported protocols are expected to be configured and enabled during this set up (See Appendix A). All of the tests are expected to run without changing the configuration or setup of the DUT in any way other than what's required for the specific test. For example, it's not acceptable to change the size of frame handling buffers between tests of frame handling rates, or to disable all but one transport protocol when testing the throughput of that protocol 🙅♀️. You do need to modify the configuration when starting a test to find out the effect of filters on throughput, but the only change MUST be to enable the specific filter 🔍. The DUT set up SHOULD include the normally recommended routing update intervals and keep alive frequency 💅. The specific version of the software and the exact DUT configuration used during the tests, including what functions are disabled, MUST be included as part of the report of the results 📝.
8. Frame formats 🦋
The formats of the test frames to use for TCP/IP over Ethernet are shown in Appendix C: Test Frame Formats. These exact frame formats SHOULD be used in the tests described in this doc for this protocol/media combination, and these frames will be used as a template for testing other protocol/media combinations 💖. The specific formats used to define the test frames for a particular test series MUST be included in the report of the results 📋.
9. Frame sizes 🌈
All of the described tests SHOULD be performed at a number of frame sizes. Specifically, the sizes SHOULD include the maximum and minimum legitimate sizes for the protocol under test on the media under test, plus enough sizes in between to fully characterize the DUT's performance 📏. Except where noted, at least five frame sizes SHOULD be tested for each test condition 🍓.
Theoretically, the minimum size UDP Echo request frame would consist of an IP header (minimum length 20 octets), a UDP header (8 octets), and whatever MAC level header the media in use requires 🧩. The theoretical maximum frame size is determined by the size of the length field in the IP header. In almost all cases, though, the actual maximum and minimum sizes are determined by the limitations of the media 🙌.
In theory it would be ideal to spread the frame sizes out so they evenly distribute the theoretical frame rates. These recommendations take that theory into account but specify frame sizes that are easy to understand and remember 🧠. Plus, many of the same frame sizes are specified on each media type so you can make easy performance comparisons 📊.
Note: Including an unrealistically small frame size on some of the media types (i.e. with little or no space for data) helps characterize the per-frame processing overhead of the DUT 🔬.
9.1 Frame sizes to be used on Ethernet 🎀
okay so for Ethernet, the frame sizes are:
64, 128, 256, 512, 1024, 1280, 1518these include the max and min frame sizes Ethernet allows, plus a selection of sizes in between. the smaller sizes are spaced more finely, since that's where the higher frame rates live 🦋
9.2 Frame sizes to be used on 4Mb and 16Mb token ring 🌸
54, 64, 128, 256, 1024, 1518, 2048, 4472so the token ring picks assume there's no RIF field in the frames of routed protocols. a RIF field would show up in any direct source route bridge performance test. the min frame size for UDP on token ring is 54 octets. the max is 4472 octets for 16Mb token ring, instead of the theoretical 17.9Kb, because a lot of token ring interfaces can't handle anything that big. the rest of the sizes are there so you can compare directly with other media 💅 if you want a higher data rate, you can also use an IP (not UDP) frame, and then the min frame size is 46 octets.
9.3 Frame sizes to be used on FDDI 🧚
54, 64, 128, 256, 1024, 1518, 2048, 4472the min frame size for UDP on FDDI is 53 octets, but 54 is recommended so you can compare directly with token ring performance. the max is 4472, not the theoretical 4500 octets, so the same kind of comparison still works 🔥 and if you want a higher data rate you can add an IP (not UDP) frame, where the min frame size is 45 octets.
9.4 Frame sizes in the presence of disparate MTUs 💜
when the interconnect DUT supports connecting links with disparate MTUs, the frame sizes for the link with the larger MTU SHOULD be used, up to the limit of the protocol being tested. if the interconnect DUT doesn't support fragmenting frames when there's an MTU mismatch, the forwarding rate for that frame size shall be reported as zero.
for example, testing IP forwarding through a bridge or router that joins FDDI and Ethernet should use FDDI frame sizes when going from the FDDI link to the Ethernet link. if the bridge doesn't support IP fragmentation, the forwarding rate for frames that are too big for Ethernet should be reported as zero 🙈
10. Verifying received frames ✨
the test equipment SHOULD discard any frames received during a test run that aren't actual forwarded test frames. for example, keep-alive and routing update frames SHOULD NOT be included in the count of received frames. either way, the test equipment SHOULD verify the length of the received frames and check that they match the expected length 📏
preferably, the test equipment SHOULD include sequence numbers in the transmitted frames and check those numbers on the received frames. if you do this, the reported results SHOULD include, in addition to the number of frames dropped, the number of frames received out of order, the number of duplicate frames received, and the number of gaps in the received frame numbering sequence. this functionality is required for some of the described tests 🫶
11. Modifiers 🌈
it could be really useful to know how the DUT performs under different conditions, so some of those conditions are listed below. the reported results SHOULD include as many of these conditions as the test equipment can generate. the suite of tests SHOULD be run first without any modifying conditions, then repeated under each condition separately. to keep these results comparable, any frames needed to create the modifying conditions (management queries, for example) get included in the same data stream as the normal test frames, taking the place of one of the test frames. they're not sent to the DUT on a separate network port 💖
11.1 Broadcast frames 📡
in most router designs, special processing is needed when frames addressed to the hardware broadcast address come in. in bridges (or bridge mode on routers), those broadcast frames have to be flooded out to a bunch of ports. the stream of test frames SHOULD be augmented with 1% frames addressed to the hardware broadcast address. the frames sent to the broadcast address should be a type that the router won't need to process. the goal of this test is to see whether there's any effect on the forwarding rate of the other data in the stream. the specific frames to use are included in the test frame format document. the broadcast frames SHOULD be evenly distributed throughout the data stream, for example, every 100th frame 🍓
the same test SHOULD be run on bridge-like DUTs, but in that case the broadcast packets will get processed and flooded to all outputs.
ngl, 1% broadcast is way higher than most networks ever see, but like drug toxicity evaluations, you need the higher level to be able to gauge an effect that would otherwise get lost in the normal variability of the system's performance 🧪 some test equipment can't generate a level of alternate frames this low. in those cases, the percentage SHOULD be as small as the equipment can provide, and the actual level SHOULD be described in the report of the test results. (the original wording here is "and that the actual level be described", so the same idea applies.)
11.2 Management frames 🧸
most data networks now use management protocols like SNMP. in a lot of environments, there can be several management stations sending queries to the same DUT at the same time.
the stream of test frames SHOULD be augmented with one management query as the first frame sent each second for the whole duration of the trial. the result of the query must fit into one response frame. the response frame SHOULD be verified by the test equipment. one example of the specific query frame to use is shown in Appendix C 💅
11.3 Routing update frames 🦋
processing dynamic routing protocol updates can have a big impact on how well a router forwards data frames. the stream of test frames SHOULD be augmented with one routing update frame, sent as the first frame of the trial 🔁
routing update frames SHOULD be sent at the rate given in Appendix C for the specific routing protocol being tested. Appendix C defines two routing update frames for the TCP/IP over Ethernet example. the routing frames are built to change the routing to a bunch of networks that aren't involved in forwarding the test data. the first frame sets the routing table state to "A", and the second one changes it to "B". the frames MUST be alternated during the trial 🌀
the test SHOULD verify that the DUT processed the routing update 👑
11.4 Filters 🔐
filters get added to routers and bridges so they can selectively block frames that would normally be forwarded. this is usually done to set up security controls on the data accepted between one area and another. different products have different filter capabilities 🙃
the DUT SHOULD first be configured with one filter condition, and the tests get run. this filter SHOULD permit the forwarding of the test data stream. on routers, this filter SHOULD be of the form:
forward input_protocol_address to output_protocol_addresson bridges, the filter SHOULD be of the form:
forward destination_hardware_addressthen the DUT SHOULD be reconfigured to have a total of 25 filters. the first 24 of these filters SHOULD be of the form:
block input_protocol_address to output_protocol_addressthe 24 input and output protocol addresses SHOULD not be any that show up in the test data stream. the last filter SHOULD permit the forwarding of the test data stream. when we say "first" and "last," we mean that in the second case, 25 conditions have to be checked before the data frames hit the conditions that allow them to be forwarded. of course, if the DUT reorders the filters, or doesn't use a linear scan of the filter rules, then the effect of the order the filters were entered in is lost 🤷♀️
the exact filter configuration command lines used SHOULD be included with the report of the results 📝
11.4.1 Filter Addresses 🗝️
two sets of filter addresses are required, one for the single filter case and one for the 25 filter case.
the single filter case should permit traffic from IP address 198.18.1.2 to IP address 198.19.65.2 and deny all other traffic.
the 25 filter case should follow this sequence:
deny aa.ba.1.1 to aa.ba.100.1
deny aa.ba.2.2 to aa.ba.101.2
deny aa.ba.3.3 to aa.ba.103.3
...
deny aa.ba.12.12 to aa.ba.112.12
allow aa.bc.1.2 to aa.bc.65.1
deny aa.ba.13.13 to aa.ba.113.13
deny aa.ba.14.14 to aa.ba.114.14
...
deny aa.ba.24.24 to aa.ba.124.24
deny all elseall previous filter conditions should be cleared from the router before this sequence is entered. the sequence is picked to check whether the router sorts the filter conditions or accepts them in the order they were entered. both of those approaches cost more performance than some form of hash coding would 🧠
12. Protocol addresses 🌍
it's way easier to run these tests with a single logical stream of data, one source protocol address and one destination protocol address. for some conditions, like the filters above, that's a practical requirement. real-world networks aren't limited to single streams of data, though 🙌 the test suite SHOULD first be run with a single source and destination address pair (or a single hardware pair for bridge tests). then the tests SHOULD be repeated using a random destination address. while testing routers, the addresses SHOULD be random and uniformly distributed over a range of 256 networks, and random and uniformly distributed over the full MAC range for bridges. the specific address ranges to use for IP are shown in Appendix C 🦄
13. Route Set Up 🛣️
it's not realistic to set up all the routing info needed to forward the test stream by hand, especially in the multiple address case 😩 at the start of each trial, a routing update MUST be sent to the DUT. this routing update MUST include all of the network addresses that will be required for the trial. all of the addresses SHOULD resolve to the same "next-hop." normally that'll be the address of the receiving side of the test equipment. this routing update will need to be repeated at the interval required by the routing protocol being used. an example of the format and repetition interval of the update frames is given in Appendix C ⏰
14. Bidirectional traffic ↔️
normal network activity isn't all in one direction. to test the bidirectional performance of a DUT, the test series SHOULD be run with the same data rate offered from each direction. the sum of the data rates should not exceed the theoretical limit for the media 💜
15. Single stream path 🛤️
the full suite of tests SHOULD be run, along with whatever modifier conditions are relevant, using a single input and output network port on the DUT. if the DUT's internal design has multiple distinct pathways (say, multiple interface cards, each with multiple network ports), then all possible types of pathways SHOULD be tested separately 👑✨
16. Multi-port 🌸
Okay so modern router and bridge boxes love packing a ton of network ports into one module 🧚 For these tests, the first half of the ports get labelled "input ports" and the other half become "output ports" 💅 The ports SHOULD be spread evenly across the DUT architecture. Like, if a DUT has two interface cards with four ports each, two ports on each card are inputs and two are outputs. The specified tests run at the same data rate being offered to every input port. The addresses in the input streams SHOULD be set so a frame goes to each output port in sequence, so every "output" port gets an even split of packets from that input. The same setup MAY be used for a bidirectional multi-stream test. In that case every port counts as both an input and an output, and each data stream MUST consist of frames addressed to all of the other ports. Main character energy 👑
Consider this 6 port DUT 🦋
--------------
---------| in A out X|--------
---------| in B out Y|--------
---------| in C out Z|--------
--------------The addressing of the data streams for each input SHOULD be:
stream sent to input A:
packet to out X, packet to out Y, packet to out Z
stream sent to input B:
packet to out X, packet to out Y, packet to out Z
stream sent to input C
packet to out X, packet to out Y, packet to out ZNote that these streams each follow the same sequence 🍓 so 3 packets arrive at output X at the same time, then 3 packets at Y, then 3 packets at Z. This is the whole point: it makes sure the DUT has to deal with multiple packets aimed at the same output at the same time, just like in the real world. Iconic stress test, honestly 😭
17. Multiple protocols ✨
This document doesn't tackle testing the effects of a mixed protocol environment 🙈 It only suggests that if you do want those tests, frames SHOULD be spread across all of the test protocols. The distribution MAY approximate the conditions on the network where the DUT would actually be used. Realistic vibes 🌈
18. Multiple frame sizes 💜
Same story here: this document doesn't cover testing a mixed frame size environment 💅 If you want those tests, frames SHOULD be distributed across all of the listed sizes for the protocol under test. The distribution MAY approximate the conditions on the network where the DUT would be used. The authors are honest that they have no idea how the results of such a test would be interpreted, other than to directly compare multiple DUTs in some very specific simulated network. Not gonna lie, that's a very specific vibe 💀
19. Testing performance beyond a single DUT. 🦋
Okay so for single DUT performance testing, the paradigm is basically: put some input into a DUT and watch the output 👀 The results can then be used as a basis for characterizing that device under those test conditions.
This model works well when the test input and output are homogenous, like 64-byte IP, 802.3 frames going in and 64 byte IP, 802.3 frames coming out. It also works when the test method can tell dissimilar input and output apart, like 1518 byte IP, 802.3 frames going in and 576 byte, fragmented IP, X.25 frames coming out.
Now here's the glow-up 🧚 By extending the single DUT model, you can collect reasonable benchmarks for multiple DUTs or heterogeneous environments. In this extension, the single DUT gets replaced by a system of interconnected network DUTs. This methodology supports benchmarking all kinds of device, media, service and protocol combinations. For example, a LAN-to-WAN-to-LAN test configuration might look like this:
(1) 802.3-> DUT 1 -> X.25 @ 64kbps -> DUT 2 -> 802.3Or a mixed LAN configuration might look like this:
(2) 802.3 -> DUT 1 -> FDDI -> DUT 2 -> FDDI -> DUT 3 -> 802.3In both examples 1 and 2, you can empirically ascertain end-to-end benchmarks of each system. Other behavior can be characterized by using intermediate devices. In example 2, the configuration can give an indication of the FDDI to FDDI capability exhibited by DUT 2. Big brain move 🧠
Heads up, bestie: because multiple DUTs get treated as a single system, there are limitations to this methodology. This methodology may yield an aggregate benchmark for the tested system. That benchmark alone, however, may not necessarily reflect asymmetries in behavior between the DUTs, latencies introduced by other apparatus (e.g., CSUs/DSUs, switches), etc.
Also, be careful when comparing benchmarks of different systems 🫶 Make sure the DUTs' features and configurations have the appropriate common denominators so the comparison actually makes sense.
20. Maximum frame rate 🌈
The maximum frame rates used when testing LAN connections SHOULD be the listed theoretical maximum rate for the frame size on the media.
The maximum frame rate used when testing WAN connections SHOULD be greater than the listed theoretical maximum rate for the frame size on that speed connection. Why the higher rate for WAN? It's to compensate for the fact that some vendors use various forms of header compression ✨
A list of maximum frame rates for LAN connections is included in Appendix B. Go check it out 💖
21. Bursty traffic 🌸
It's convenient to measure DUT performance under steady state load, but honestly that's an unrealistic way to gauge how a DUT is functioning 🙄 Real network traffic normally comes in bursts of frames. So some of the tests described below SHOULD be run with both steady state traffic and with traffic made up of repeated bursts of frames. The frames inside a burst are transmitted with the minimum legitimate inter-frame gap.
The goal here is to find the minimum interval between bursts that the DUT can process with no frame loss. During each test, the number of frames in each burst is held constant while the inter-burst interval varies. Tests SHOULD be run with burst sizes of 16, 64, 256 and 1024 frames. Burst sizes, babe 🎀
22. Frames per token 🍓
It's possible to configure some token ring and FDDI interfaces to transmit more than one frame each time they receive the token, but most network devices available right now only transmit one frame per token. So these tests SHOULD first be performed while transmitting only one frame per token.
Some current high-performance workstation servers do transmit more than one frame per token on FDDI to maximize throughput 🔥 Since this may become a common feature in future workstations and servers, interconnect devices with FDDI interfaces SHOULD be tested with 1, 4, 8, and 16 frames per token. The reported frame rate SHOULD be the average rate of frame transmission over the total trial period.
23. Trial description 🧚
A particular test is made up of multiple trials. Each trial returns one piece of information, like the loss rate at a particular input frame rate. Each trial has a number of phases, and they go like this:
- a) If the DUT is a router, send the routing update to the "input" port and pause two seconds to be sure that the routing has settled.
- b) Send the "learning frames" to the "output" port and wait 2 seconds to be sure that the learning has settled. Bridge learning frames are frames with source addresses that are the same as the destination addresses used by the test frames. Learning frames for other protocols are used to prime the address resolution tables in the DUT. The formats of the learning frame that should be used are shown in the Test Frame Formats document. Learning frames are basically the hype squad for the DUT 📣
- c) Run the test trial. Showtime 🎬
- d) Wait for two seconds for any residual frames to be received. Patience, bestie ⏳
- e) Wait for at least five seconds for the DUT to restabilize. Rest up, queen 😴
24. Trial duration 💜
The aim of these tests is to determine the rate the DUT can continuously support. The actual duration of the test trials has to be a compromise between this aim and the duration of the benchmarking test suite. The duration of the test portion of each trial SHOULD be at least 60 seconds. Tests that involve some form of "binary search", like the throughput test, can use a shorter trial duration to minimize the length of the search procedure, and they MAY do so. But the final determination SHOULD be made with full length trials. Speedrun the search, then do the real run ✨
25. Address resolution 🦋
The DUT SHOULD be able to respond to address resolution requests sent by the DUT wherever the protocol requires such a process. Basically, the DUT needs to show up when it's supposed to 💅
26. Benchmarking tests: 👑
Note: The notation "type of data stream" refers to the above modifications to a frame stream with a constant inter-frame gap, for example, the addition of traffic filters to the configuration of the DUT. Iykyk 🙈
26.1 Throughput 🌟
Objective: To determine the DUT throughput as defined in RFC 1242. Simple goal, big impact 💖
Procedure: Send a specific number of frames at a specific rate through the DUT, then count the frames that the DUT transmits. If the count of offered frames is equal to the count of received frames, the fewer frames are received than were transmitted, the rate of the offered stream is reduced and the test is rerun. Yes, that sentence is a little messy, but the idea is a binary-search-style rate hunt 🌀
The throughput is the fastest rate at which the count of test frames transmitted by the DUT equals the number of test frames sent to it by the test equipment. That's the number everyone's chasing ✨
Reporting format: The results of the throughput test SHOULD be reported in the form of a graph. If it is, the x coordinate SHOULD be the frame size and the y coordinate SHOULD be the frame rate. There SHOULD be at least two lines on the graph. One line SHOULD show the theoretical frame rate for the media at the various frame sizes. The second line SHOULD be the plot of the test results. Additional lines MAY be used on the graph to report the results for each type of data stream tested. Text accompanying the graph SHOULD indicate the protocol, data stream format, and type of media used in the tests. Receipts or it didn't happen 📊
We assume that if a single value is wanted for advertising purposes, the vendor will select the rate for the minimum frame size for the media. If this is done, the figure MUST be expressed in frames per second. The rate MAY also be expressed in bits (or bytes) per second if the vendor so desires 🧾 The statement of performance MUST include a/ the measured maximum frame rate, b/ the size of the frame used, c/ the theoretical limit of the media for that frame size, and d/ the type of protocol used in the test. Even if a single value shows up in the advertising copy, the full table of results SHOULD be included in the product data sheet. No hiding the tea 🍵
26.2 Latency ✨
Objective: okay so we want to figure out the latency, the way RFC 1242 defines it. 🕰️
Procedure: first, figure out the throughput for the DUT at each of the listed frame sizes. Then send a stream of frames at that frame size through the DUT, at the throughput rate you just found, to one specific destination. The stream SHOULD be at least 120 seconds long. 💅 Put an identifying tag in one frame after the 60 second mark. The type of tag is up to the implementation 🧚. Record the time that this frame is fully transmitted (timestamp A). The receiver logic in the test equipment MUST recognize the tag info in the frame stream and record the time it got the tagged frame (timestamp B). 🫶
Latency is timestamp B minus timestamp A, using whichever definition from RFC 1242 applies: latency for store and forward devices, or latency for bit forwarding devices. 🎀
The test MUST be repeated at least 20 times, and the reported value is the average of the recorded values. 📊 This test SHOULD use test frames addressed to the same destination as the rest of the data stream. It also SHOULD use a setup where each test frame goes to a new destination network. Main character energy 👑
Reporting format: the report MUST say which latency definition from RFC 1242 was used for this test. 📝 The latency results SHOULD be shown as a table with one row for each tested frame size. There SHOULD be columns for the frame size, the rate the latency test ran at for that frame size, the media types tested, and the resulting latency values for each type of data stream tested. 🌸
26.3 Frame loss rate 🦋
Objective: we want the frame loss rate (as defined in RFC 1242) of a DUT across the entire range of input data rates and frame sizes. 🌈
Procedure: send a specific number of frames at a specific rate through the DUT, then count how many frames the DUT actually transmits. Work out the frame loss rate at each point with this equation:
( ( input_count - output_count ) * 100 ) / input_countThe first trial SHOULD run at the frame rate that matches 100% of the maximum rate for that frame size on the input media. Then repeat for the rate that's 90% of the maximum rate, then 80% of that, and so on. 📉 This sequence SHOULD keep going in 10% steps until there are two trials in a row with no frames lost. The maximum granularity of the trials MUST be 10% of the maximum rate, but a finer granularity is encouraged. 🔥
Reporting format: if you graph the frame loss rate results, the results SHOULD be plotted as a graph. In that case, the X axis MUST be the input frame rate as a percent of the theoretical rate for the media at that frame size, and the Y axis MUST be the percent loss at that input rate. The left end of the X axis and the bottom of the Y axis MUST be 0 percent. The right end of the X axis and the top of the Y axis MUST be 100 percent. 📐 Multiple lines on the graph MAY be used to show the frame loss rate for different frame sizes, protocols, and types of data streams. 🌷
Note: see section 18 for the maximum frame rates that SHOULD be used. 🍓
26.4 Back-to-back frames 🍭
Objective: we want to see how well a DUT handles back-to-back frames, as defined in RFC 1242. 🧁
Procedure: send a burst of frames with the minimum inter-frame gaps to the DUT, then count how many frames the DUT forwards. If the number of frames transmitted matches the number forwarded, make the burst longer and rerun the test. ⬆️ If the number forwarded is less than the number transmitted, make the burst shorter and rerun the test. ⬇️
The back-to-back value is the number of frames in the longest burst the DUT can handle without losing any frames. 🎯 The trial length MUST be at least 2 seconds, and the test SHOULD be repeated at least 50 times. The reported value is the average of the recorded values. 💜
Reporting format: the back-to-back results SHOULD be shown as a table with one row for each tested frame size. There SHOULD be columns for the frame size and for the resulting average frame count for each type of data stream tested. 🌺 The standard deviation for each measurement MAY also be reported. 🤓
26.5 System recovery 🌊
Objective: we want to know how fast a DUT bounces back from an overload condition. ⚡
Procedure: first, figure out the throughput for a DUT at each of the listed frame sizes.
Then send a stream of frames at a rate of 110% of the throughput rate you recorded, or the maximum rate for the media, whichever is lower. Keep this going for at least 60 seconds. ⏱️ At timestamp A, drop the frame rate to 50% of the rate above, and record the time of the last frame lost (timestamp B). The system recovery time is timestamp A minus timestamp B. 🧮 The test SHOULD be repeated a number of times, and the average of the recorded values is what gets reported. 🫧
Reporting format: the system recovery results SHOULD be shown as a table with one row for each tested frame size. There SHOULD be columns for the frame size, the frame rate used as the throughput rate for each type of data stream tested, and the measured recovery time for each type of data stream tested. 💖
26.6 Reset 🔄
Objective: we want to know how fast a DUT recovers from a device or software reset. 🧯
Procedure: first, figure out the throughput for the DUT at the minimum frame size on the media used in testing.
Send a continuous stream of minimum sized frames at that throughput rate. Then cause a reset in the DUT. 💫 Watch the output until frames start getting forwarded again, and record the time the last frame of the initial stream (timestamp A) and the first frame of the new stream (timestamp B) are received. A power interruption reset test is done the same way, except the power to the DUT should be interrupted for 10 seconds instead of causing a reset. 🔌
This test SHOULD only be run using frames addressed to networks directly connected to the DUT. That way there's no need to wait around for a routing update. 🛣️
The reset value is timestamp B minus timestamp A. 🧠
Hardware and software resets, as well as a power interruption, SHOULD be tested. 🌟
Reporting format: the reset value SHOULD be reported in a simple set of statements, one for each reset type. 📣
27. Security Considerations 🚨
Security issues are not addressed in this document. 🙈 Not me getting a little dramatic about it, but security is a serious topic, so please go read the rest of the spec's context before you trust anything here. 💀
28. Editors' Addresses 💌
Scott Bradner
Harvard University
1350 Mass. Ave, room 813
Cambridge, MA 02138
Phone: +1 617 495-3864
Fax: +1 617 496-8500
EMail: sob@harvard.edu
Jim McQuaid
NetScout Systems
4 Westford Tech Park Drive
Westford, MA 01886
Phone: +1 978 614-4116
Fax: +1 978 614-4004
EMail: mcquaidj@netscout.comAppendix A: Testing Considerations 🧠
A.1 Scope Of This Appendix 🔍
This appendix covers some benchmarking methodology issues where experience or judgment might shape which tests you run or how you set up the test with a particular DUT. 🎀 So this appendix MUST not be read as an amendment to the methodology in the body of this document. It's more of a guide to testing practice. 🫶
- Typical testing practice has been to enable all protocols and run all testing with no extra configuration of protocols, even though a given set of trials might only exercise one protocol at a time. This keeps people from "tuning" a DUT for a single protocol. 🙅♀️
- The least common denominator of the available filter functions should be used. That gives a basis for comparison between vendors. Because of product differences, the people conducting and evaluating the tests have to make a judgment call on this one. ⚖️
- Architectural considerations may need to be looked at. For example, first run the tests with the stream going between ports on the same interface card, then repeat them with the stream going into a port on one interface card and out of a port on a second interface card. There will almost always be a best case and a worst case configuration for a given DUT architecture. 🏗️
- Testing with mixed protocol traffic streams hasn't shown much difference from testing with individual protocols. So if protocol A and protocol B give two different performance results, mixed protocol testing seems to give a result that's the average of the two. 🥤
- Wide Area Network (WAN) performance can be tested by setting up two identical devices connected by the appropriate short-haul versions of the WAN modems. Performance is then measured from a LAN interface on one DUT to a LAN interface on the other DUT. 🌐
Deciding the maximum frame rate for LAN-WAN-LAN configurations is a judgment call. It can be based on known characteristics of the overall system, including compression effects, fragmentation, and gross link speeds. Practice suggests the rate should be at least 110% of the slowest link speed. 🐢 Substantive issues about testing compression itself are beyond the scope of this document. 🙃
Appendix B: Maximum frame rates reference 📚
(Provided by Roger Beeman, Cisco Systems) 🧑💻
Size Ethernet 16Mb Token Ring FDDI
(bytes) (pps) (pps) (pps)
64 14880 24691 152439
128 8445 13793 85616
256 4528 7326 45620
512 2349 3780 23585
768 1586 2547 15903
1024 1197 1921 11996
1280 961 1542 9630
1518 812 1302 8138
Ethernet size
Preamble 64 bits
Frame 8 x N bits
Gap 96 bits
16Mb Token Ring size
SD 8 bits
AC 8 bits
FC 8 bits
DA 48 bits
SA 48 bits
RI 48 bits ( 06 30 00 12 00 30 )
SNAP
DSAP 8 bits
SSAP 8 bits
Control 8 bits
Vendor 24 bits
Type 16 bits
Data 8 x ( N - 18) bits
FCS 32 bits
ED 8 bits
FS 8 bits
Tokens or idles between packets are not included
FDDI size
Preamble 64 bits
SD 8 bits
FC 8 bits
DA 48 bits
SA 48 bits
SNAP
DSAP 8 bits
SSAP 8 bits
Control 8 bits
Vendor 24 bits
Type 16 bits
Data 8 x ( N - 18) bits
FCS 32 bits
ED 4 bits
FS 12 bitsAppendix C: Test Frame Formats 🎀
This appendix defines the frame formats that may be used with these tests. ✨ It also includes protocol specific parameters for TCP/IP over Ethernet, to be used with the tests as an example. 🧚
C.1. Introduction 💖
okay so the logic behind picking the parameters and designing the frame formats gets explained for each case inside the TCP/IP section 🎀 The same logic is used in the other sections too. We only leave comments when there's a protocol-specific feature that needs explaining. If you want parameters and frame formats for other protocols, you can totally work them out yourself using the same logic 🧚
C.2. TCP/IP Information ✨
this section is all about the TCP/IP protocol suite 🌈
C.2.1 Frame Type. 🦋
for the test data frame, the protocols that support it use an application-level datagram echo request 💅 Using a datagram protocol means a router is less likely to expect a specific session initialization sequence, which could happen with a reliable stream protocol 🙈 We also use a specifically defined protocol because some routers check the protocol field and refuse to forward protocols they don't recognize 💀
for TCP/IP, we use a UDP Echo Request 🫶
C.2.2 Protocol Addresses 🌸
two sets of addresses need to be defined. First, the addresses assigned to the router ports. Second, the addresses that go inside the frames themselves and into the routing updates 👑
the network addresses 192.18.0.0 through 198.19.255.255 were assigned to the BMWG by the IANA for this purpose 😭 The whole point is to lower the chance of a conflict in case a testing device ends up accidentally plugged into part of the Internet. Here's how the addresses get used 🍓
C.2.2.1 Router port protocol addresses 🔥
half the ports on a multi-port router are called "input" ports, and the other half are called "output" ports 🎀 Some tests use every port as both input and output though, so don't panic if that's the case. A contiguous series of IP Class C network addresses, from 198.18.1.0 to 198.18.64.0, is set aside for the "input" ports. A second series, from 198.19.1.0 to 198.19.64.0, is set aside for the "output" ports 💖 In every case the router port is node 1 on its network. For example, a two-port DUT would have 198.18.1.1 on one port and 198.19.1.1 on the other 🦋
some of the tests in the methodology memo use an SNMP management connection to the DUT. The management access address for the DUT is assumed to be the first of the "input" ports, which is 198.18.1.1 🧚
C.2.2.2 Frame addresses 🌈
some of the tests assume adjacent network routing. The reboot time test is one example 🙈 For these, the IP address in the test frame is node 2 on the right Class C network, like 198.19.1.2 for example ✨
if the test involves non-adjacent network routing, the phantom routers are located at node 10 of each of the right Class C networks 🫶 A series of Class C network addresses, from 198.18.65.0 to 198.18.254.0, has been assigned for the networks reachable through the phantom routers on the "input" side of the DUT. The series of Class C networks from 198.19.65.0 to 198.19.254.0 has been assigned for the networks visible through the phantom routers on the "output" side of the DUT 💅
C.2.3 Routing Update Frequency ⭐
the update interval for each routing protocol may have to be determined by the specs of that individual protocol 🌸 For IP RIP, Cisco IGRP, and OSPF, a routing update frame (or frames) should come 5 seconds before each stream of test frames 👑 That frequency is enough for trial durations of up to 60 seconds. If you pick longer trial periods, routing updates must be mixed in with the stream of test frames 😭 Use the table below for how often to send updates 🍓
IP-RIP 30 sec
IGRP 90 sec
OSPF 90 secC.2.4 Frame Formats - detailed discussion 🎀
C.2.4.1 Learning Frame 💖
in most protocols there's a procedure that works out the mapping between the protocol node address and the MAC address 🦋 For TCP/IP, that job goes to the Address Resolution Protocol (ARP) 🧚 XNS and IPX don't need this step, because the MAC address is already used as the protocol node address 🌈
ideally the tester would respond to ARP requests coming from the DUT ✨ When that's not possible, an ARP request should be sent to the router's "output" port 🫶 This request should look like it's coming from the immediate destination of the test frame stream (the phantom router in Figure 2, or the end node if adjacent network routing is being used) 💅 It's assumed that the router will cache the MAC address of the requesting device 🙈 The ARP request should go out 5 seconds before the test frame stream starts in each trial 🍓 If trials run longer than 50 seconds, the router might need an extended ARP timeout configured 😭
+--------+ +------------+
| | | phantom |------ P LAN
A
IN A------| DUT |------------| |------ P LAN
B
| | OUT A | router |------ P LAN
C
+--------+ +------------+
Figure 2in the case where full routing is being used 🔥
C.2.4.2 Routing Update Frame 👑
if the test doesn't involve adjacent net routing, the tester has to supply the proper routing information using a routing update 🌸 A single routing update gets sent before each trial on each "destination" port (see section C.24) 🎀 This update includes the network addresses that are reachable through a phantom router on the network attached to the port 🦋 For a full mesh test, the update has one destination network address for each of the "input" ports 💖 The test stream on each "input" port is a repeating sequence of frames, with one going to each of the "output" ports 🧚
C.2.4.3 Management Query Frame 🌈
the management overhead test uses SNMP to query a set of variables that should be present in all DUTs that support SNMP ✨ The NMS reads only the variables for a single interface, at the right intervals 🫶 Here's the list of variables to retrieve 💅
sysUpTime
ifInOctets
ifOutOctets
ifInUcastPkts
ifOutUcastPktsC.2.4.4 Test Frames 🙈
the test frame is a UDP Echo Request with enough data to fill out the required frame size 🍓 The data should not be all bits off or all bits on 😭 Those patterns can trigger a "bit stuffing" process, which keeps clocks synchronized on WAN links, and that makes the frame longer than you meant it to be 💀
the data pattern matters, bestie. All zeros or all ones can quietly turn your frame into something longer than intended 🔥
C.2.4.5 Frame Formats - TCP/IP on Ethernet 👑
each of the frames below is described for the 1st pair of DUT ports, meaning "input" port #1 and "output" port #1 🌸 If you want to use a frame on other ports, the addresses must be changed 🎀
C.2.6.1 Learning Frame 🦋
ARP Request on Ethernet
-- DATAGRAM HEADER
offset data (hex) description
00 FF FF FF FF FF FF dest MAC address send to
broadcast address
06 xx xx xx xx xx xx set to source MAC address
12 08 06 ARP type
14 00 01 hardware type Ethernet = 1
16 08 00 protocol type IP = 800
18 06 hardware address length 48 bits
on Ethernet
19 04 protocol address length 4 octets
for IP
20 00 01 opcode request = 1
22 xx xx xx xx xx xx source MAC address
28 xx xx xx xx source IP address
32 FF FF FF FF FF FF requesting DUT's MAC address
38 xx xx xx xx DUT's IP addressC.2.6.2 Routing Update Frame 💖
-- DATAGRAM HEADER
offset data (hex) description
00 FF FF FF FF FF FF dest MAC address is broadcast
06 xx xx xx xx xx xx source hardware address
12 08 00 type
-- IP HEADER
14 45 IP version - 4, header length (4
byte units) - 5
15 00 service field
16 00 EE total length
18 00 00 ID
20 40 00 flags (3 bits) 4 (do not
fragment),
fragment offset-0
22 0A TTL
23 11 protocol - 17 (UDP)
24 C4 8D header checksum
26 xx xx xx xx source IP address
30 xx xx xx destination IP address
33 FF host part = FF for broadcast
-- UDP HEADER
34 02 08 source port 208 = RIP
36 02 08 destination port 208 = RIP
38 00 DA UDP message length
40 00 00 UDP checksum
-- RIP packet
42 02 command = response
43 01 version = 1
44 00 00 0
-- net 1
46 00 02 family = IP
48 00 00 0
50 xx xx xx net 1 IP address
53 00 net not node
54 00 00 00 00 0
58 00 00 00 00 0
62 00 00 00 07 metric 7
-- net 2
66 00 02 family = IP
68 00 00 0
70 xx xx xx net 2 IP address
73 00 net not node
74 00 00 00 00 0
78 00 00 00 00 0
82 00 00 00 07 metric 7
-- net 3
86 00 02 family = IP
88 00 00 0
90 xx xx xx net 3 IP address
93 00 net not node
94 00 00 00 00 0
98 00 00 00 00 0
102 00 00 00 07 metric 7
-- net 4
106 00 02 family = IP
108 00 00 0
110 xx xx xx net 4 IP address
113 00 net not node
114 00 00 00 00 0
118 00 00 00 00 0
122 00 00 00 07 metric 7
-- net 5
126 00 02 family = IP
128 00 00 0
130 00 net 5 IP address
133 00 net not node
134 00 00 00 00 0
138 00 00 00 00 0
142 00 00 00 07 metric 7
-- net 6
146 00 02 family = IP
148 00 00 0
150 xx xx xx net 6 IP address
153 00 net not node
154 00 00 00 00 0
158 00 00 00 00 0
162 00 00 00 07 metric 7C.2.4.6 Management Query Frame 🌈
to be defined 🧚
C.2.6.4 Test Frames 🧪✨
Okay so the test frames are where the whole thing gets real 💅. This is a UDP echo request on Ethernet, and the layout below shows every byte, bestie 🎀
-- DATAGRAM HEADER
offset data (hex) description
00 xx xx xx xx xx xx set to dest MAC address
06 xx xx xx xx xx xx set to source MAC address
12 08 00 type
-- IP HEADER
14 45 IP version - 4 header length 5 4
byte units
15 00 TOS
16 00 2E total length*
18 00 00 ID
20 00 00 flags (3 bits) - 0 fragment
offset-0
22 0A TTL
23 11 protocol - 17 (UDP)
24 C4 8D header checksum*
26 xx xx xx xx set to source IP address**
30 xx xx xx xx set to destination IP address**
-- UDP HEADER
34 C0 20 source port
36 00 07 destination port 07 = Echo
38 00 1A UDP message length*
40 00 00 UDP checksum
-- UDP DATA
42 00 01 02 03 04 05 06 07 some data***
50 08 09 0A 0B 0C 0D 0E 0F
* - change for different length frames
** - change for different logical streams
*** - fill remainder of frame with incrementing octets,
repeated if required by frame lengthThe ones that change are the star-marked fields 🌟 The total length and UDP message length values depend on the frame size, so here's the cheat sheet 👇
Values to be used in Total Length and UDP message length fields:
frame size total length UDP message length
64 00 2E 00 1A
128 00 6E 00 5A
256 00 EE 00 9A
512 01 EE 01 9A
768 02 EE 02 9A
1024 03 EE 03 9A
1280 04 EE 04 9A
1518 05 DC 05 C8Full Copyright Statement 🎀
Okay, the legal bit, but it's still important 🫶 Here it is, word for word:
Copyright (C) The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 💜
📦 where this page comes from
- ietf: rfc2544 1999-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms