RFC 1242

Benchmarking Terminology for Network Interconnection Devices

📅 published July 1991, from ietf

RFC 1242: Benchmarking Terminology for Network Interconnection Devices 🎀

Okay bestie, here's the title block tea ☕ This one's from the Network Working Group, it's RFC 1242, and S. Bradner wrote it as Editor from Harvard University in July 1991. It's giving "for your information" energy, not a standard 📚 It doesn't list anything it obsoletes or updates, so it's standing on its own ✨

Status of This Memo

Okay so this memo just provides information for the Internet community 💅 It does not specify an Internet standard. Distribution of this memo is unlimited 🦋

Abstract

This memo is all about defining a bunch of terms we use when we talk about performance benchmarking tests and their results 🔥 The terms in here are going to be used by later memos, which will define the actual tests and how to write up the results. Iconic 👑 This memo is a product of the Benchmarking Methodology Working Group (BMWG) of the Internet Engineering Task Force (IETF) 🌈

1. Introduction ✨

Okay so vendors are literally always doing "specsmanship" 💀 to make their products look better than everyone else's in the marketplace. It's a lot of "smoke & mirrors" honestly, and it's designed to confuse the user 🙈 This memo and its follow-ups are trying to lock down a specific set of terminology and tests that vendors can use to measure and report how their network devices perform 🫶 The payoff is that users get comparable data from different vendors, so they can actually evaluate these devices fairly. Main character energy for the people reading the specs 💖

2. Definition format 🌸

Every term gets the same little template, so here it is:

Term to be defined. (e.g., Latency)

Definition:
        The specific definition for the term.

Discussion:
        A brief discussion about the term, it's application
        and any restrictions on measurement procedures.

Measurement units:
        The units used to report measurements of this
        term, if applicable.

Issues:
        List of issues or conditions that effect this term.

See Also:
        List of other terms that are relevant to the discussion
        of this term.

3. Term definitions 🧚

Okay here comes the good stuff 🍓 Each term below has a definition, a discussion, its units, and the issues and "see also" links, so you always know where it fits.

3.1. Back-to-back 💜

Definition: Fixed length frames sent at the minimum legal separation for a given medium, over a short to medium period of time, starting from an idle state. Basically, the frames are shoved out as fast as the rules allow 🏃‍♀️

Discussion: A growing number of devices on a network can produce bursts of back-to-back frames 😭 Remote disk servers using protocols like NFS, remote disk backup systems like rdump, and remote tape access systems can be set up so that one request returns a block of data as big as 64K octets. Over networks like ethernet, which have a fairly small MTU, that turns into lots of fragments. Fragment reassembly only gets attempted when every fragment has arrived, so losing even one fragment, because some middle device couldn't keep up with enough continuous frames, can cause an endless loop as the sender keeps trying to send its giant data block 💀

With the internet getting bigger, routing updates can span many frames, and modern routers can transmit super fast 🚀 Missing frames of routing information can make things look unreachable when they actually aren't 🙈 Tests of this parameter are meant to figure out how much data buffering the device has.

Measurement units: Number of N-octet frames in burst.

Issues:

See Also:

3.2. Bridge 🦋

Definition: A system that forwards data frames based on information in the data link layer.

Discussion:

Measurement units: n/a

Issues:

See Also:

  • bridge/router (3.3)
  • router (3.15)

3.3. bridge/router 🎀

Definition: A bridge/router is a network device that can selectively work as a router and/or a bridge, depending on the protocol of a specific frame. Two-in-one, iconic 👑

Discussion:

Measurement units: n/a

Issues:

See Also:

  • bridge (3.2)
  • router (3.15)

3.4. Constant Load 🌊

Definition: Fixed length frames at a fixed interval time.

Discussion: Okay so it's rare, to say the least, to find a steady state load on a network device in the real world 🙃 But measuring steady state performance can still be useful when you're comparing competing devices. The frame size is specified and constant. All device parameters are constant too. When there's a checksum in the frame, it must be verified 🔍

Measurement units: n/a

Issues: unidirectional vs. bidirectional

See Also:

Definition: The number of octets in the frame, from the first octet after the preamble to the end of the FCS (if there is one), or to the last octet of the data if there's no FCS.

Discussion: There is SO much confusion about how frame sizes get reported when people test network devices or measure networks 😭 Some authors include the checksum and some don't. This is a specific definition for use in this memo and the ones after it, so everyone's on the same page 💅

Measurement units: octets

Issues:

See Also:

3.6. Frame Loss Rate 🔥

Definition: The percentage of frames that a network device should have forwarded under steady state (constant) load, but didn't forward because it ran out of resources.

Discussion: This measurement is useful for reporting how a device performs when it's overloaded 😵 It can be a helpful hint about how a device would hold up under pathological network conditions, like broadcast storms 🌪️

Measurement units: Percentage of N-octet offered frames that are dropped. To be reported as a graph of offered load vs frame loss.

Issues:

See Also:

  • overhead behavior (3.11)
  • policy based filtering (3.13)
  • MTU mismatch behavior (3.10)

3.7. Inter Frame Gap ⭐

Definition: The delay from the end of a data link frame (as defined in section 3.5) to the start of the preamble of the next data link frame.

Discussion: People get really confused about how to report the between-frame time when testing network devices 🙈 This is a specific definition for use in this memo and the ones after it, so no more guessing 💖

Measurement units: Time with fine enough units to distinguish between 2 events.

Issues: Link data rate.

See Also:

3.8. Latency 💜

Definition: Okay, this one has two versions, so pay attention 👀

For store and forward devices: The time interval starting when the last bit of the input frame reaches the input port, and ending when the first bit of the output frame is seen on the output port.

For bit forwarding devices: The time interval starting when the end of the first bit of the input frame reaches the input port, and ending when the start of the first bit of the output frame is seen on the output port.

Discussion: Variability of latency can be a real problem 😩 Some protocols are timing dependent (e.g., LAT and IPX), and future applications are likely to be sensitive to network latency. Increased device delay can shrink the useful diameter of a network. The goal is to get rid of the effect of the data rate on the latency measurement, so it only reflects the actual within-device latency. Measurements should be taken across a spectrum of frame sizes without changing the device setup 🎯

Ideally, measurements for all devices would start from the first actual bit of the frame after the preamble. In theory, a vendor could build a device that normally counts as store and forward (a bridge, for example) but starts transmitting a frame before it's fully received. That's called a "cut through" device ✂️ The assumption is that the device would somehow invalidate the partially transmitted frame if, while receiving the rest of the input frame, something turned up showing the frame or this particular forwarding of it was in error, like a bad checksum. In that case the device would still be considered store and forward, and its latency would still be measured from last bit in to first bit out, even though the value would be negative 🤯 The intent is to treat the device as a unit, without caring about its internal structure.

Measurement units: Time with fine enough units to distinguish between 2 events.

Issues:

See Also:

  • link speed mismatch (3.9)
  • constant load (3.4)
  • back-to-back (3.1)
  • policy based filtering (3.13)
  • single frame behavior (3.16)

Definition: A speed mismatch between input and output data rates.

Discussion: This isn't about frame rate per se. It's about the actual data rate of the data path 🛣️ For example, you might have ethernet on one side and a 56KB serial link on the other. People also call this the "fire hose effect" 🚒 Networks that use serial links between local high speed networks will usually have a link speed mismatch at each end of those serial links.

Measurement units: Ratio of input and output data rates.

Issues:

See Also:

  • constant load (3.4)
  • back-to-back (3.1)

3.10. MTU-mismatch behavior 🧩

Definition: The network MTU (Maximum Transmission Unit) of the output network is smaller than the MTU of the input network, and this results in fragmentation.

Discussion: Fragmenting frames can seriously hurt the performance of network devices 😤

Measurement units: Description of behavior.

Issues:

See Also:

⏭️ Part 1 of 2 is done, bestie! Part 2 is coming up next 💖

3.11. Overhead behavior 🧹

Definition: Processing that a device does beyond handling normal data frames. ✨

Discussion: okay so network devices do SO much more than just forwarding frames 😭 they run internal hardware testing, process routing information, and respond to network management requests. It's useful to know how all of that affects device performance 💅. Like, if a router stopped forwarding or accepting frames while it chewed through a huge routing update for a complex protocol like OSPF, you'd want to know that ahead of time 🙈

Measurement units: Any quantitative understanding of this behavior comes from figuring out its effect on other measurements. 📏

Issues:

  • bridging and routing protocols
  • control processing
  • icmp
  • ip options processing
  • fragmentation
  • error processing
  • event logging/statistics collection
  • arp

See Also:

  • policy based filtering (3.13)

3.12. Overloaded behavior 🌊

Definition: What happens when demand exceeds the available system resources. 🚨

Discussion: a device that's overloaded will lose frames, and that's the messy part 💀 it might drop frames carrying routing or configuration information too. And here's the tea: a device counts as overloaded whenever there is any frame loss at all 🍓

Measurement units: A description of the device's behavior in any overloaded state, for both input and output overload conditions. 📝

Issues:

  • How well does the device recover from overloaded state?
  • How does source quench production effect device?
  • What does device do when its resources are exhausted?
  • What is response to system management in overloaded state?

See Also:

3.13. Policy based filtering 🚧

Definition: Filtering is when a device discards received frames by administrative decision, even though normal operation would be to forward them. 🙅‍♀️

Discussion: lots of network devices can be configured to throw frames away based on all kinds of criteria 🧚 the criteria can be as simple as source or destination addresses, or as detailed as checking specific fields inside the data frame itself. Configuring a device to do all this filtering does hit its throughput, so keep that in mind 🫶

Measurement units: n/a

Issues:

  • flexibility of filter options
  • number of filter conditions

See Also:

3.14. Restart behavior 🔄

Definition: Reinitializing the system in a way that causes data loss. 💔

Discussion: after a power up or reset, network devices don't accept and forward frames for a while 😩 how long that unavailability lasts is super useful when you're evaluating devices. Some devices also need a reset whenever certain setup variables change. If that reset takes forever, network managers might be too scared to touch those variables on production networks, and nobody wants that 🥺

Measurement units: A description of device behavior under various restart conditions. 📋

Issues:

  • Types:
    • power on
    • reload software image
    • flush port, reset buffers
    • restart current code image, without reconfuration
  • Under what conditions is a restart required?
  • Does the device know when restart needed (i.e., hung state timeout)?
  • Does the device recognize condition of too frequent auto-restart?
  • Does the device run diagnostics on all or some resets?
  • How may restart be initiated?
    • physical intervention
    • remote via terminal line or login over network

See Also:

3.15. Router 🗺️

Definition: A system that forwards data frames based on information in the network layer. 🦋

Discussion: this means the device has to be "running" the network-level protocol's routing algorithm and doing whatever actions that protocol requires 🧠 for example, decrementing the TTL field in the TCP/IP header. Very main character energy 👑

Measurement units: n/a

Issues:

See Also:

  • bridge (3.2)
  • bridge/router (3.3)

3.16. Single frame behavior 🎯

Definition: One frame received on the input to a device. 🍼

Discussion: a data "stream" that's just one single frame can make a network device do a LOT of work 😭 it has to figure out routes, perform ARPs, check permissions, and generally set up cache entries. Devices often take way longer to process one frame by itself than they would if that same frame were part of a steady stream. There's also a worry that some devices would even throw away a single frame during cache setup, because they assume it's only the first of many 🙈

Measurement units: A description of the behavior of the device. 📝

Issues:

See Also:

  • policy based filtering (3.13)

3.17. Throughput 🏎️

Definition: The maximum rate at which none of the offered frames are dropped by the device. 💯

Discussion: this is the number vendors can report as a single value, and it's proven really useful in the marketplace ✨ even losing one frame from a data stream can cause big delays while higher-level protocols wait to time out, so knowing the actual maximum data rate the device can handle is a must-know 🔥 measurements should be taken over a assortment of frame sizes. For devices that can support both, take separate measurements for routed and bridged data. And if there is a checksum in the received frame, full checksum processing must be done. no shortcuts bestie 🫡

Measurement units:

  • N-octet input frames per second
  • input bits per second

Issues:

  • single path vs. aggregate
  • load
  • unidirectional vs bidirectional
  • checksum processing required on some protocols

See Also:

  • frame loss rate (3.6)
  • constant load (3.4)
  • back-to-back (3.1)

4. Acknowledgements 🫶

This memo is a product of the IETF BMWG working group:

  • Chet Birger, Coral Networks
  • Scott Bradner, Harvard University (chair)
  • Steve Butterfield, independant consultant
  • Frank Chui, TRW
  • Phill Gross, CNRI
  • Stev Knowles, FTP Software, Inc.
  • Mat Lew, TRW
  • Gary Malkin, FTP Software, Inc.
  • K.K. Ramakrishnan, Digital Equipment Corp.
  • Mick Scully, Ungerman Bass
  • William M. Seifert, Wellfleet Communications Corp.
  • John Shriver, Proteon, Inc.
  • Dick Sterry, Microcom
  • Geof Stone, Network Systems Corp.
  • Geoff Thompson, SynOptics
  • Mary Youssef, IBM

Security Considerations 🔐

Security issues are not discussed in this memo. 🙈

Author's Address 💌

Scott Bradner
Harvard University
William James Hall 1232
33 Kirkland Street
Cambridge, MA 02138

Phone: (617) 495-3864

EMail: SOB@HARVARD.HARVARD.EDU
Or, send comments to: bmwg@harvisr.harvard.edu.
📦 where this page comes from