RFC 6645
IP Flow Information Accounting and Export Benchmarking Methodology
RFC 6645: IP Flow Information Accounting and Export Benchmarking Methodology 🎀
Okay bestie, so this is RFC 6645, titled IP Flow Information Accounting and Export Benchmarking Methodology 🦋. It was written by J. Novak at Cisco Systems, Inc., and it's Informational, so it's published for info and isn't a Standards Track spec 💅. It's dated July 2012 and has ISSN 2070-1721. The header doesn't list anything it obsoletes or updates, so no drama there 🧚.
Abstract ✨
This doc gives a methodology and framework for measuring how much monitoring IP flows slows down a network device, plus exporting that info to a Collector 📊. It defines the rate at which IP flows get created, expired, and successfully exported as a brand new performance metric, alongside regular throughput. The metric only applies to devices that follow RFC 5470, "Architecture for IP Flow Information Export" 💜. The methodology quantifies the impact of the IP flow monitoring process on the network equipment, and it's giving main character energy 👑.
Status of This Memo 📜
This doc is not an Internet Standards Track spec. It's published for informational purposes 🌸.
This doc is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, it went through public review, and the Internet Engineering Steering Group (IESG) approved it for publication 🫶. Not all IESG-approved docs are candidates for any level of Internet Standard, so see Section 2 of RFC 5741 for the details 🙈.
For info on the current status of this doc, any errata, and how to send feedback, check out http://www.rfc-editor.org/info/rfc6645 🔗.
Copyright Notice ©️
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved.
This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this doc 📚. Code Components extracted from this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 💖.
Table of Contents 🗂️
1. Introduction ....................................................4
2. Terminology .....................................................5
2.1. Existing Terminology .......................................5
2.2. New Terminology ............................................6
3. Flow Monitoring Performance Benchmark ...........................8
3.1. Definition .................................................8
3.2. Device Applicability .......................................8
3.3. Measurement Concept ........................................8
3.4. The Measurement Procedure Overview .........................9
4. Measurement Setup ..............................................11
4.1. Measurement Topology ......................................11
4.2. Baseline DUT Setup ........................................13
4.3. Flow Monitoring Configuration .............................13
4.4. Collector .................................................19
4.5. Sampling ..................................................19
4.6. Frame Formats .............................................19
4.7. Frame Sizes ...............................................20
4.8. Flow Export Data Packet Sizes .............................20
4.9. Illustrative Test Setup Examples ..........................20
5. Flow Monitoring Throughput Measurement Methodology .............22
5.1. Flow Monitoring Configuration .............................23
5.2. Traffic Configuration .....................................24
5.3. Cache Population ..........................................25
5.4. Measurement Time Interval .................................25
5.5. Flow Export Rate Measurement ..............................26
5.6. The Measurement Procedure .................................27
6. RFC 2544 Measurements ..........................................28
6.1. Flow Monitoring Configuration..............................28
6.2. Measurements with the Flow Monitoring Throughput Setup ....29
6.3. Measurements with Fixed Flow Export Rate...................29
7. Flow Monitoring Accuracy .......................................30
8. Evaluating Flow Monitoring Applicability .......................31
9. Acknowledgements ...............................................32
10. Security Considerations .......................................32
11. References ....................................................33
11.1. Normative References .....................................33
11.2. Informative References ...................................33
Appendix A. Recommended Report Format .............................35
Appendix B. Miscellaneous Tests ...................................36
B.1. DUT Under Traffic Load ...................................36
B.2. In-Band Flow Export ......................................36
B.3. Variable Packet Rate .....................................37
B.4. Bursty Traffic ...........................................37
B.5. Various Flow Monitoring Configurations ...................38
B.6. Tests with Bidirectional Traffic .........................38
B.7. Instantaneous Flow Export Rate ...........................391. Introduction ✨
Monitoring IP flows (Flow monitoring) is defined in the "Architecture for IP Flow Information Export" [RFC5470] and the related IPFIX docs specified in Section 1.2 of [RFC5470] 🧠. It analyzes traffic using predefined fields from the packet header as keys, and it stores the traffic and other internal info in the DUT (Device Under Test) memory. That cached flow info then gets formatted into records (see Section 2.1 for term definitions) and exported from the DUT to an external data collector for analysis 📦. More details on the measurement architecture are in Section 3.3.
Flow monitoring is widely deployed on network devices, and it has loads of uses in both service-provider and enterprise segments, as detailed in "Requirements for IP Flow Information Export (IPFIX)" [RFC3917] 💅. This doc gives a methodology for measuring Flow monitoring performance, so network operators have a framework to measure the impact on the network and network equipment 🔍.
The goal here is to provide a series of methodology specs for measuring Flow monitoring performance in a way that's comparable across different implementations, platforms, and vendor devices 🌈.
Flow monitoring is, in most cases, run on network devices that also forward packets 🚀. So this doc also provides the methodology for [RFC2544] measurements in the presence of Flow monitoring. It's applicable to IPv6 and MPLS traffic, with their specifics defined in [RFC5180] and [RFC5695], respectively 🦋.
This doc specifies a methodology to measure the maximum IP Flow Export Rate a network device can sustain without impacting the Forwarding Plane, without losing any IP flow information, and without compromising IP flow accuracy (see Section 7 for details) 💖.
[RFC2544], [RFC5180], and [RFC5695] specify benchmarking of network devices forwarding IPv4, IPv6, and MPLS [RFC3031] traffic, respectively. The methodology in this doc stays the same for any traffic type. The only restriction may be the DUT's lack of support for Flow monitoring of a particular traffic type 🙌.
A variety of different DUT architectures can do Flow monitoring and export. So this doc doesn't try to list the various white-box variables (e.g., CPU load, memory utilization, hardware resources utilization, etc.) that could be gathered, even though they always help in comparison evaluations 🧁. A more complete understanding of a particular device's stress points can be attained using this internal info, and the tester MAY choose to gather it during the measurement iterations 🫶.
2. Terminology 📖
The terminology used in this doc is based on what's defined in [RFC5470], [RFC2285], and [RFC1242], as summarized in Section 2.1 🎀. The only new terms needed for this methodology are defined in Section 2.2.
Additionally, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this doc are to be interpreted as described in RFC 2119 [RFC2119] 🌸.
2.1. Existing Terminology 🧚
Device Under Test (DUT) [RFC2285, Section 3.1.1]
Flow [RFC5101, Section 2]
Flow Key [RFC5101, Section 2]
Flow Record [RFC5101, Section 2]
Template Record [RFC5101, Section 2]
Observation Point [RFC5470, Section 2]
Metering Process [RFC5470, Section 2]
Exporting Process [RFC5470, Section 2]
Exporter [RFC5470, Section 2]
Collector [RFC5470, Section 2]
Control Information [RFC5470, Section 2]
Data Stream [RFC5470, Section 2]
Flow Expiration [RFC5470, Section 5.1.1]
Flow Export [RFC5470, Section 5.1.2]
Throughput [RFC1242, Section 3.17]2.2. New Terminology 🆕
2.2.1. Cache 💜
Definition: Memory area held and dedicated by the DUT to store Flow information prior to the Flow Expiration.
2.2.2. Cache Size 🍓
Definition: The size of the Cache in terms of how many entries the Cache can hold.
Discussion: This term is typically represented as a configurable option in the particular Flow monitoring implementation. Its highest value depends on the memory available in the network device 🔥.
Measurement units: Number of Cache entries
2.2.3. Active Timeout ⏰
Definition: For long-running Flows, the time interval after which the Metering Process expires a Cache entry to ensure Flow data is regularly updated.
Discussion: This term is typically presented as a configurable option in the particular Flow monitoring implementation. See Section 5.1.1 of [RFC5470] for a more detailed discussion 🧐.
Flows are considered long running when they last longer than several multiples of the Active Timeout. If the Active Timeout is zero, then Flows are considered long running if they contain many more packets (tens of packets) than usually observed in a single transaction 🙈.
Measurement units: Seconds
2.2.4. Idle Timeout 😴
Definition: The time interval used by the Metering Process to expire an entry from the Cache when no more packets belonging to that specific Cache entry have been observed during the interval.
Discussion: Idle Timeout is typically represented as a configurable option in the particular Flow monitoring implementation. See Section 5.1.1 of [RFC5470] for more detailed discussion. Note that some documents in the industry refer to "Idle Timeout" as "inactive timeout" 💅.
Measurement units: Seconds
2.2.5. Flow Export Rate 📤✨
Definition: okay so this one is the number of Cache entries that expire from the Cache (the Flow Expiration thing we just covered 💅) and get exported to the Collector within a measurement time interval. There SHOULD NOT be any export filtering, so that all the expired Cache entries are exported. If there is export filtering and it can't be disabled, this MUST be indicated in the measurement report. 🔍
The measured Flow Export Rate MUST include both the Data Stream and the Control Information, as defined in Section 2 of [RFC5470]. 🫶
Discussion: the Flow Export Rate gets measured using Flow Export data observed at the Collector, by counting the exported Flow Records during the measurement time interval (see Section 5.4) 🧮. The value you get is an average of the instantaneous export rates seen during that interval. Now, if you're trying to measure a nearly instantaneous export rate instead of an average on the DUT, the smallest possible measurement interval is limited by the export capabilities of the particular Flow monitoring implementation. (Physical-layer issues between the DUT and the Collector are excluded here, so don't blame the cable, bestie 🙈) 🔌
Measurement units: Number of Flow Records per second ⏱️
3. Flow Monitoring Performance Benchmark 🎀
3.1. Definition
Flow Monitoring Throughput 👑
Definition: the maximum Flow Export Rate the DUT can sustain without losing a single Cache entry. Plus, for packet forwarding devices, it's also the maximum Flow Export Rate the DUT can sustain without dropping packets in the Forwarding Plane (see Figure 1) 🚀.
Measurement units: Number of Flow Records per second
Discussion: losses of Cache entries, or forwarded packets under this definition, are assumed to happen because the DUT doesn't have enough resources to process any additional traffic information, or because it doesn't have enough resources to process Flow Export data 😩. Physical-layer issues, like insufficient bandwidth from the DUT to the Collector or a lack of Collector resources, MUST be excluded as detailed in Section 4. No drama, we just skip those 🙅♀️.
3.2. Device Applicability
The Flow monitoring performance metric applies to network devices that deploy the architecture described in [RFC5470] 🦋. These devices can be network packet forwarding devices, or appliances that analyze traffic but don't forward it (e.g., probes, sniffers, replicators) 🕵️♀️.
This document doesn't aim to measure Collector performance. It only requires sufficient Collector resources (as specified in Section 4.4) so we can measure the DUT characteristics 💖.
3.3. Measurement Concept
Okay, Figure 1 is the functional block diagram of the DUT 🗺️. The traffic in the figure is test traffic sent to the DUT and forwarded by the DUT, if possible. When you test devices that don't act as network packet forwarding devices (like probes, sniffers, and replicators), the Forwarding Plane is basically just an Observation Point, as defined in Section 2 of [RFC5470]. The Throughput of those devices will always be zero, so the only metric that applies is the Flow Monitoring Throughput. Netflow is specified by [RFC3954] ✨.
+------------------------- +
| IPFIX | NetFlow | Others |
+------------------------- +
| ^ |
| Flow Export |
| ^ |
| +-------------+ |
| | Monitoring | |
| | Plane | |
| +-------------+ |
| ^ |
| traffic information |
| ^ |
| +-------------+ |
| | | |
traffic ---|---->| Forwarding |------|---->
| | Plane | |
| +-------------+ |
| |
| DUT |
+------------------------- +Figure 1. The Functional Block Diagram of the DUT 🖼️
Flow monitoring is shown in Figure 1 as the Monitoring Plane, and it's enabled as specified in Section 4.3 🔧. It uses the traffic information from the Forwarding Plane plus the configured Flow Keys to create Cache entries that represent the traffic forwarded (or observed) by the DUT, and those live in the DUT Cache 🗃️. Cache entries get expired from the Cache depending on the Cache configuration (e.g., the Active and Idle Timeouts, the Cache Size), the number of Cache entries, and the traffic pattern. The Exporting Process uses the Cache entries to format the Flow Records, which then get exported from the DUT to the Collector (see Figure 2 in Section 4) 📦.
The Forwarding Plane and Monitoring Plane are two separate functional blocks, each with its own performance capability 🌗. The Forwarding Plane handles user data packets and is fully characterized by the metrics defined by [RFC1242] 💅.
The Monitoring Plane handles Flows that reflect the analyzed traffic. Its metric is the Flow Export Rate, and the benchmark is the Flow Monitoring Throughput 📊.
3.4. The Measurement Procedure Overview
The measurement procedure is fully specified in Sections 4, 5, and 6. This section just gives you an overview of the principles behind the measurements 🧠.
The basic measurement procedure for a DUT with Flow monitoring enabled is a conventional Throughput measurement. It uses a search algorithm to find the maximum packet rate at which none of the offered packets and corresponding Flow Records get dropped by the DUT, as described in [RFC1242] and Section 26.1 of [RFC2544] 🏃♀️.
The DUT with Flow monitoring enabled has two functional blocks that need to be measured using characteristics that apply to one or both blocks (see Figure 1). See Sections 3.4.1 and 3.4.2 for the deets 🔍.
On one hand, the Monitoring Plane and Forwarding Plane (see Figure 1) need to be looked at as two independent blocks, and each one's performance gets measured independently. On the other hand, when you're measuring the performance of one, the status and performance of the other MUST be known and benchmarked when both are present. Think of it like tag-teaming, but make it measurable 🤝💜.
3.4.1. Monitoring Plane Performance Measurement
The Flow Monitoring Throughput MUST be (and can only be) measured with one packet per Flow, as specified in Section 5. This traffic type is the most demanding from the Flow monitoring point of view, and it'll exercise the Monitoring Plane (see Figure 1) of the DUT the most 💪. In this scenario, every packet the DUT sees creates a new Cache entry, which forces the DUT to fill the Cache instead of just updating the packet and byte counters of an Cache entry that already exists 😤.
The exit criteria for the Flow Monitoring Throughput measurement are one of the following (e.g., if any of the conditions are reached):
a. The Flow Export Rate at which the DUT starts to lose Flow Information, or the Flow Information gets corrupted. 💀
b. The Flow Export Rate at which the Forwarding Plane starts to drop or corrupt packets (if the Forwarding Plane is present). 🫠
A corrupted packet here means packet header corruption (which results in a cyclic redundancy check failure on the transmission level and a consequent packet drop), or packet payload corruption, which leads to lost application-level data 🧩.
3.4.2. Forwarding Plane Performance Measurement
The Forwarding Plane (see Figure 1) performance metrics are fully specified by [RFC1242] and MUST be measured accordingly. A detailed traffic analysis (see below) related to Flow monitoring MUST be performed prior to any [RFC2544] measurements. Most importantly, the Flow Export Rate caused by the test traffic during an [RFC2544] measurement MUST be known and reported 📝🔥.
The required test traffic analysis mainly involves the following:
a. Which packet header parameters are incremented or changed during traffic generation. 🔢
b. Which Flow Keys the Flow monitoring configuration uses to generate Flow Records. 🔑
The performance metrics described in RFC 1242 can be measured in one of three modes:
a. As a baseline of forwarding performance without Flow monitoring. 🌱
b. At a certain level of Flow monitoring activity, specified by a Flow Export Rate lower than the Flow Monitoring Throughput. 🌤️
c. At the maximum level of Flow monitoring performance, e.g., using traffic conditions that represent a measurement of Flow Monitoring Throughput. ⚡
The measurement mode in point a. is an ordinary Throughput measurement, as specified in RFC 2544. The details of how to set up the measurements in points b. and c. are in Section 6 🎀.
4. Measurement Setup 🧚
This section concentrates on the setup of all the components needed to perform Flow monitoring performance measurement. The recommended reporting format can be found in Appendix A 📋💖.
4.1. Measurement Topology ✨
okay so this section is only for measurements with packet forwarding network devices 🌸 Traffic monitoring appliances (see Section 3.2) come in SO many architectures and implementations that this document can't cover them all 😭 Instead of the Forwarding Plane, these appliances usually get some kind of feed (like an optical splitter, an interface sniffing traffic on a shared media, or an internal channel on the DUT that gives a copy of the traffic) that provides the traffic info needed for Flow monitoring analysis. So the measurement topology has to be adjusted to the appliance's architecture, and it MUST be part of the measurement report 📝
The setup is identical to the one used by [RFC2544], with one addition: a Collector that analyzes the Flow Export 💅
In the unidirectional traffic topology, the traffic goes from the sender to the receiver through the DUT. The received traffic gets analyzed to check that it's identical to the generated traffic 🦋
The ideal way to do this is with a single device that acts as both the sender and receiver, with one sending port and one receiving port. That makes it super easy to check whether all the traffic the sender sent was re-transmitted by the DUT and received at the receiver 👑
+-----------+
| |
| Collector |
| |
|Flow Record|
| analysis |
| |
+-----------+
^
| Flow Export
|
| Export Interface
+--------+ +-------------+ +----------+
| | | | | traffic |
| traffic| (*)| | | receiver |
| sender |-------->| DUT |--------->| |
| | | | | traffic |
| | | | | analysis |
+--------+ +-------------+ +----------+Figure 2. Measurement Topology with Unidirectional Traffic
okay this part is IMPORTANT 🚨 The DUT's export interface (the one connecting to the Collector) MUST NOT be used for forwarding test traffic. It's only for the Flow Export data that carries the Flow Records 🙈 In every measurement, the export interface MUST have enough bandwidth to transmit Flow Export data without congestion. In other words, the export interface MUST NOT be a bottleneck during the measurement 💀
The traffic receiver MUST have sufficient resources to measure all the test traffic that the DUT transferred successfully. You can check this by measuring with and without the DUT 🫶
Also, more complex topologies might be needed 🌈 For example, if you care about the effects of enabling Flow monitoring on several interfaces, or the maximum media transmission speed is lower than the DUT Throughput, the topology can be expanded with several input and output ports. Just know that the topology MUST be clearly written in the measurement report 📝✨
4.2. Baseline DUT Setup
the baseline DUT setup, and how that setup gets reported in the measurement results, is fully specified in Section 7 of [RFC2544] 🎀
The baseline DUT configuration might include other features, like packet filters or quality of service on the input and/or output interfaces, if you need to study Flow monitoring alongside those features 🧚 The Flow monitoring measurement procedures don't change in that case. But when you evaluate results, you have to think about how those features might change the packet rates offered to the DUT, and how that affects Flow monitoring after the features are applied to the configuration 🤔 Any such feature configuration MUST be part of the measurement report.
The DUT export interface (see Figure 2) SHOULD be configured with enough output buffers so the Flow Export data doesn't get dropped just because the interface hardware is out of resources 🍓 The applied configuration MUST be part of the measurement report.
The test designer is free to run tests in multiple configurations 💜 So it's possible to run both non-production and real deployment configurations in the lab, depending on what the tester needs. All configurations MUST be part of the measurement report.
4.3. Flow Monitoring Configuration
this section covers every aspect of the Flow monitoring configuration needed on the DUT to run the Flow monitoring performance measurement 💅 The necessary configuration has a few components (see [RFC5470]): Observation Points, the Metering Process, and the Exporting Process. They're detailed below 👑
The DUT MUST support the Flow monitoring architecture as specified by [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] so the results can be meaningfully compared, since it's a standardized export protocol 🌸
The DUT configuration, any existing Cache, and Cache entries MUST be erased before any new configuration is applied for the measurement that's currently running 🧹
4.3.1. Observation Points
The Observation Points specify the interfaces and direction where the Flow monitoring traffic analysis happens ✨
The (*) in Figure 2 marks the Observation Points in the default configuration. Other DUT Observation Points might be configured depending on what the measurement needs, like this:
- a. ingress port/ports only
- b. egress port/ports only
- c. both ingress and egress
This test topology only corresponds to unidirectional traffic, with traffic analysis performed on the input and/or output interface. Testing with bidirectional traffic is discussed in Appendix B 🦋
Generally, where the Observation Points go depends on where the DUT sits in the deployed network and what Flow monitoring is for. See [RFC3917] for a detailed discussion 📚 Otherwise, the measurement procedures are the same for all of these configurations.
If ingress AND egress Flow monitoring are both enabled on one DUT, the analysis should account for the fact that each Flow will be represented in the DUT Cache by two Flow Records (one for each direction). So the Flow Export will also contain those two Flow Records 🔥
If more than one Observation Point for one direction is defined on the DUT, the traffic going through each of the Observation Points MUST be configured so that it creates Flows and Flow Records that do NOT overlap. Each packet (or set of packets if you're measuring more than one packet per Flow, see Section 6.3.1) sent to the DUT on different ports still creates one unique Flow Record 🫶
The specific Observation Points and their monitoring direction MUST be included in the measurement report 📝
4.3.2. Metering Process
The Metering Process MUST be enabled so the Cache gets created in the DUT, and so you can configure the Cache-related parameters 🌈
The Cache Size available to the DUT MUST be known and taken into account when designing the measurement, as specified in Section 5. Usually the Cache Size shows up in the "show" commands of the Flow monitoring process, either in the actual configuration or in the DUT vendor's product docs 💜 The Cache Size MUST have a fixed value for the entire duration of the measurement. This method doesn't apply to benchmarking any Flow monitoring applications that dynamically change their Cache Size 🙈
The configuration of the Metering Process MUST be included in the measurement report. For example, if a Flow monitoring implementation uses timeouts to expire entries from the Cache, the Cache's Idle and Active Timeouts MUST be known and taken into account when designing the measurement, as specified in Section 5. If the implementation only allows timeouts equal to zero (for example, an immediate timeout or a non-existent Cache), then the measurement conditions in Section 5 are inherently fulfilled without any extra configuration. The DUT just exports info about every packet immediately, subject to the Flow Export Rate definition in Section 2.2.5 ⚡
If the Flow monitoring implementation lets you configure multiple Metering Processes on a single DUT, the exact configuration of each process MUST be included in the measurement report. Only measurements with the same number of Metering Processes can be compared 🎀
The Cache Size and the Idle and Active Timeouts MUST be included in the measurement report 💖
4.3.3. Exporting Process
The Exporting Process MUST be configured so the Flow Record data gets exported to the Collector 📤
The Exporting Process MUST be configured so that all Flow Records from all configured Observation Points are exported towards the Collector, after the expiration policy. That policy is made up of the Idle and Active Timeouts and the Cache Size 🦋
The Exporting Process SHOULD be configured with IPFIX [RFC5101] as the protocol for formatting the Flow Export data. If the Flow monitoring implementation doesn't support IPFIX, proprietary protocols MAY be used. Only measurements with the same export protocol SHOULD be compared, since protocols can differ in export efficiency 🌸 Export efficiency can also be influenced by the Template Record used and the order of the individual export fields within the template 🧠
The Template Records used by the tested implementations SHOULD be analyzed and documented as part of the measurement report. Ideally, only tests with the same Template Records should be compared 🫶
Different Flow monitoring implementations might use different default values for exporting Control Information [RFC5470]. So the Flow Export corresponding to Control Information SHOULD be analyzed and reported as a separate item in the measurement report 📝 The export of Control Information SHOULD always be configured consistently across all testing, and set to the minimal possible value. Ideally, just one set of Control Information should be exported during each measurement. And yes, Control Information includes options and Template Records [RFC5470] 🤓
Section 10 of [RFC5101] and Section 8.1 of [RFC5470] talk about the various transport-layer protocols that can deliver Flow Export data from the DUT to the Collector 🚚 The selected protocol MUST be included in the measurement report. Only benchmarks with the same transport-layer protocol SHOULD be compared. If the Flow monitoring implementation lets you use multiple transport-layer protocols, each one SHOULD be measured in a separate measurement run, with the results reported independently in the measurement report 💜
If a reliable transport protocol is used to transmit the Flow Export data from the DUT, the configuration of the Transport session MUST allow for non-blocking data transmission. Parameters like the TCP window size and maximum segment size (MSS) are examples of what to look at 👀 The most substantial transport-layer parameters should be included in the measurement report 🌈✨
4.3.4. Flow Records ✨
okay so a Flow Record is basically the tea on one specific Flow that got spotted at an Observation Point 👀. It holds the measured stuff (like the total bytes across every packet in the Flow) and usually some characteristic stuff that describes the Flow (like the source IP address) 💅
What goes in the Flow Record is totally up to the implementation 🧚. Some tools only give you a fixed record built from the usual IP header fields in IPv4 or IPv6, like source and destination IP addresses, IP protocol numbers, or transport-level port numbers. Other tools let you build your own custom Flow Record to watch the traffic your way 🎀. For the measurements in this doc, the only real requirement is having a big number of Cache entries in the Cache, and the Flow Keys you'd use to get there are typically source and destination IP addresses plus transport-level port numbers 🌸
Here's the recommended full IPv4, IPv6, or MPLS Flow Record. The IP address part means IPv4 or IPv6, depending on what traffic you're testing 🦋. Since the Flow Record setup depends on the specific Flow monitoring implementation, these examples can't give you the exact spec for every entry in each record. Picking the best key fields is up to the test designer, using whatever the specific implementation can do 🫶
Flow Keys:
Source IP address
Destination IP address
MPLS label (for MPLS traffic type only)
Transport-layer source port
Transport-layer destination port
IP protocol number (IPv6 next header)
IP type of service (IPv6 traffic class)
Other fields:
Packet counter
Byte counter
Table 1: Recommended ConfigurationIf the Flow monitoring lets you make user-defined Flow Records, these are the minimal setups that still allow large numbers of Cache entries, for example:
Flow Keys:
Source IP address
Destination IP address
Other fields:
Packet counter
or:
Flow Keys:
Transport-layer source port
Transport-layer destination port
Other fields:
Packet counter
Table 2: User-Defined ConfigurationOkay, this part is important, bestie 🚨 The Flow Record configuration MUST be clearly noted in the measurement report. And Flow Monitoring Throughput measurements on different DUTs, or on different Flow monitoring implementations, MUST only be compared when the Flow Record configuration is exactly the same 🔥
4.3.5. Flow Monitoring with Multiple Configurations
The Flow monitoring architecture in [RFC5470] lets you do fancier setups with multiple Metering and Exporting Processes on one DUT 🧩 Depending on the implementation, that can change the DUT performance you measure, so the measurement report should contain info about how many Metering and Exporting Processes were configured on the DUT for the chosen Observation Points 📋
Some examples of these setups are:
a. Several Observation Points with a single Metering Process and a single Exporting Process.
b. Several Observation Points, each with one Metering Process, but all using just one instance of Exporting Process.
c. Several Observation Points with a per-Observation-Point Metering Process and Exporting Process.
4.3.6. MPLS Measurement Specifics
For MPLS encapsulated traffic, the Flow Record configuration SHOULD contain the MPLS label 💜 In this doc, "MPLS Label" means the entire 4 byte MPLS header. The label you care about is usually at the top of the label stack, but that depends on how your MPLS test setup is built 🌈
The tester SHOULD ensure that the data received by the Collector contains the expected MPLS labels 🫶
The MPLS forwarding performance document [RFC5695] lists a bunch of MPLS label operations you can test 🧁 The Observation Points MUST be placed on all the DUT test interfaces where that particular MPLS label operation happens 🔍 The performance measurements SHOULD be done with only one MPLS label operation at a time 🎯
The DUT MUST be configured so that all the traffic is subject to the measured MPLS label operation 💅
4.4. Collector
The Collector is what captures the Flow Export data, and that's how the Flow Monitoring Throughput gets measured 📥
The Collector can be used purely as a capture device, just giving you the hexadecimal format of the Flow Export data 🍓 In that case it doesn't need any extra Flow Export decoding abilities, and all the decoding happens offline 🌙
BUT if the Collector is also decoding the Flow Export data, it SHOULD support IPFIX [RFC5101] so your results analysis actually makes sense 📊 If proprietary Flow Export is being used, the Collector MUST support it, otherwise analyzing the Flow Export data is not possible 😭
The Collector MUST be able to capture the export packets sent from the DUT at full rate without losing any of them 🚀 When reliable transport protocols are used to send Flow Export data (see also Section 4.3.3), the Collector MUST have enough resources to guarantee non-blocking data transmission on the transport-layer session 💪
During analysis, the Flow Export data needs to be decoded and the received Flow Records counted 🧮
The capture buffer MUST be cleared at the beginning of each measurement 🧹
4.5. Sampling
Packet sampling and flow sampling are out of scope for this document 🙅♀️ This doc only applies to situations with no packet, flow, or export sampling 🌸
4.6. Frame Formats
Flow monitoring itself doesn't depend in any way on the media used on the input and output ports 🦋 You can use any media the DUT and the test equipment support. This applies to both the data forwarding interfaces and the export interface (see Figure 2) 🎀
As of when this was written, the most common transmission media and frame formats (like Ethernet and Packet over SONET) for IPv4, IPv6, and MPLS traffic are specified in [RFC2544], [RFC5180], and [RFC5695] 📚
The presented frame formats MUST be recorded in the measurement report 📝
4.7. Frame Sizes
The frame sizes of traffic analyzed by the DUT are specified in Section 9 of [RFC2544] for Ethernet type interfaces (64, 128, 256, 1024, 1280, 1518 bytes) and in Section 5 of [RFC5180] for Packet over SONET interfaces (47, 64, 128, 256, 1024, 1280, 1518, 2048, 4096 bytes) 📏
Heads up 🚨 When you measure with large frame sizes, be careful to avoid packet fragmentation on the DUT interfaces, since that could hurt the performance values you measure.
The presented frame sizes MUST be recorded in the measurement report 📝
4.8. Flow Export Data Packet Sizes
Flow monitoring performance is affected by the packet size the implementation uses to send Flow Export data to the Collector 📦 The packet size used MUST be part of the measurement report, and only measurements with the same packet sizes SHOULD be compared 💕
The DUT export interface (see Figure 2) maximum transmission unit (MTU) SHOULD be set to the largest available value for the media 🌟 The Flow Export MTU MUST be recorded in the measurement report 📝
4.9. Illustrative Test Setup Examples
These examples are a hypothetical test setup to make the Flow monitoring parameters and configuration make sense, along with the traffic parameters for testing Flow monitoring 🧚 The actual benchmarking specs are in Sections 5 and 6 🎀
4.9.1. Example 1 - Idle Timeout Flow Expiration
The traffic generator sends 1000 packets per second in 10000 defined streams, and each stream is identified by a unique destination IP address 🌈 So each stream has a packet rate of 0.1 packets per second 🐢
The packets go out round-robin (stream 1 to 10000) while the destination IP address increments for each packet sent 🔁 After a packet for stream 10000 goes out, the next packet's destination IP address goes back to stream 1's address 💫
The configured Cache Size is 20000 Flow Records. The configured Active Timeout is 100 seconds, and the Idle Timeout is 5 seconds ⏱️
Flow monitoring on the DUT uses the destination IP address as the Flow Key 🔑
A packet with destination IP address equal to A is sent every 10 seconds, so its Cache entry gets refreshed every 10 seconds 🔄 But the Idle Timeout is 5 seconds, so Cache entries expire due to the Idle Timeout, and when a new packet with the same IP address A shows up, it creates a new entry in the Cache 💀 How this behaves depends on the design and efficiency of the Cache ager, and any multi-packet flows spotted during this test should be noted 📝
The measured Flow Export Rate in this case will be 1000 Flow Records per second, since every single sent packet always creates a new Cache entry and 1000 packets per second are sent 🎯
The expected number of Cache entries in the Cache during the whole measurement is around 5000 🧮 That lines up with the Idle Timeout being 5 seconds, since 5000 entries get created in those five seconds ✨ This expectation might change in real measurement setups with large Cache Sizes and a high packet rate, where the DUT's actual export rate might be limited and lower than the Flow Expiration activity caused by the traffic offered to the DUT 🙈 This behavior is entirely implementation-specific 💜
4.9.2. Example 2 - Active Timeout Flow Expiration 🦋
okay so the traffic generator sends 1000 packets per second spread across 100 defined streams, and each stream gets its own unique destination IP address 🫶 each stream runs at 10 packets per second. the packets go round-robin (stream 1 all the way to stream 100) and the destination IP address goes up by one with every packet sent. after the packet for stream 100 goes out, the next packet goes right back to stream 1's address 🔁
the configured Cache Size is 1000 Flow Records, the configured Active Timeout is 100 seconds, and the Idle Timeout is 10 seconds. bestie the numbers are doing a lot here 💅
flow monitoring on the DUT uses the destination IP address as the Flow Key 🎀
after the first 100 packets go out, 100 Cache entries have been created in the Flow monitoring Cache. every packet after that gets counted against those entries that already exist, because the destination IP address (the Flow Key) has already been seen by the DUT. (that's only true as long as the Cache entries haven't expired yet, which is the whole plot 👀)
a packet with the destination IP address equal to A shows up every 0.1 second, so the Cache entry keeps getting refreshed every 0.1 second. the Idle Timeout is 10 seconds, so it never gets a chance to expire on idle. the entries only go once the Active Timeout kicks in, meaning they expire every 100 seconds and then get created all over again ✨
if the test measurement time is 50 seconds from when the traffic generator starts, the measured Flow Export Rate is 0, because nothing expired from the Cache during that window 🙈
if the test measurement time is 100 seconds from when the traffic generator starts, the measured Flow Export Rate is 1 Flow Record per second 🔥
if the test measurement time is 290 seconds from when the traffic generator starts, the measured Flow Export Rate is 2/3 of a Flow Record per second. the Cache expired the same number of Flows twice (100) during that 290-second period, so the math checks out 🧮💖
5. Flow Monitoring Throughput Measurement Methodology 👑
Objective:
To measure the Flow monitoring performance in a way that can be compared between different Flow monitoring implementations 🌈
Metric definition:
Flow Monitoring Throughput - see Section 3. 📏
Discussion:
different Flow monitoring implementations might handle Flow Export from a partially empty Cache differently than when the Cache is fully occupied. software-based and hardware-based DUTs can also deal with the same situation differently 🤖 the whole point of the benchmark in this section is to define one measurement procedure that covers all of those possible behaviors. it's giving fairness 🫶
the only criterion is to measure as defined here until Flow Record or packet losses are seen. whether to dig deeper into the conditions behind those packet losses is up to the tester 🕵️♀️
5.1. Flow Monitoring Configuration 🎀
Cache Size
okay the Cache Size config depends on where the DUT sits in the network and on the Flow Keys you picked for the Flow Record. the number of unique sets of Flow Keys that the traffic generator (sender) provides should be multiple times larger than the Cache Size. this makes sure existing Cache entries never get updated by a packet from the sender before that particular Flow Expiration and Flow Export happens 💫 it's easy to hit this with linearly incremented Flow Keys (like IP addresses or transport-layer ports), as long as the range of values is larger than the Cache Size. if randomized traffic generation is in use, the generator must make sure the same Flow Keys don't repeat within a range of randomly generated values 🎲
the Cache Size MUST be known so you can define the measurement circumstances properly. usually you'll find it with the "show" commands of the Flow monitoring implementation in the actual configuration, or in the vendor's product documentation 📚
Idle Timeout
Idle Timeout is set (if it's configurable) to the minimum possible value on the DUT. this makes the Cache entries expire as soon as possible so they get exported out of the DUT Cache ⏩ it MUST be known to define the measurement circumstances completely and equally across implementations 🍓
Active Timeout
Active Timeout is set (if it's configurable) to a value equal to or higher than the Idle Timeout. it MUST be known to define the measurement circumstances completely and equally across implementations 💜
Flow Keys Definition:
the test needs a huge number of unique Cache entries, so it creates them by incrementing the values of one or several Flow Keys. the number of unique combinations of Flow Keys values SHOULD be several times larger than the DUT Cache Size. that way, any incoming packet will never refresh a Cache entry that already exists 🧚
the availability of Cache Size, Idle Timeout, and Active Timeout as configuration parameters is implementation-specific. if the Flow monitoring implementation doesn't support these parameters, the test possibilities specified by this document get restricted. some testing might still be viable if the implementation follows the guidance in the [IPFIX-CONFIG] document, and that gets judged case by case 🧐
5.2. Traffic Configuration 🚚
Traffic Generation
the traffic generator needs to increment the Flow Keys values with each sent packet. that way, each packet represents one Cache entry in the DUT Cache 🩷
a particular Flow monitoring implementation might use a hashing mechanism to match incoming data packets to a certain Flow. in that case, how the traffic is built and the hashing together might affect the DUT's Flow monitoring performance 🧠 for example, if IP addresses are the Flow Keys, there could be a performance difference between linearly incremented addresses (ascending or descending) and IP addresses randomized within a certain range. if randomized IP address sequences are used, the traffic generator needs to be able to reproduce that randomization (for example, sending the same set of IP addresses in the same order in different test runs) so you can compare different DUTs and Flow monitoring implementations fairly 🤝
if the test traffic rate is below the maximum media rate for that packet size, the traffic generator MUST send the packets at equidistant time intervals. traffic generators that don't meet this condition MUST NOT, and cannot, be used for the Flow Monitoring Throughput measurement 🚫 for example, say the test traffic rate is half the media rate. a generator that sends packets for the first half of every second at full media rate and sends nothing for the second half is not okay, because then it would be impossible to tell whether the DUT failed to handle the Flows because of a shortage of input buffers during the burst, or because of the limits in Flow monitoring performance 😭
Measurement Duration
the measurement duration (how long the test traffic is sent to the DUT) MUST be at least two times longer than the Idle Timeout. otherwise, no Flow Export would be seen 🙃 the measurement duration SHOULD guarantee that the number of Cache entries created during the measurement exceeds the available Cache Size 📈
5.3. Cache Population 🪴
the product of the Idle Timeout and the packet rate offered to the DUT (the Cache population) during one measurement decides the total number of Cache entries in the DUT Cache during that measurement. you also have to leave some margin for dynamic behavior during high DUT loads while it processes the Flows 🎢
the Flow monitoring implementation might behave differently depending on how the Cache population compares to the available Cache Size during the measurement. this behavior is totally implementation-specific, and it's also influenced by whether the DUT architecture is software based or hardware based 💻
the Cache population during a particular benchmark measurement (whether it's lower or higher than the available Cache Size) SHOULD be noted, and mainly only measurements with the same Cache population SHOULD be compared 📝✨
5.4. Measurement Time Interval ⏱️
the measurement time interval is the time value used to calculate the measured Flow Export Rate from the captured Flow Export data. here's how you get it 💅
RFC 2544 specifies, down to the precision of the packet beginning and ending, the time intervals used to measure DUT time characteristics. for a Flow Monitoring Throughput measurement, the start and stop time need to be clearly defined, but the granularity can be limited to just marking the start and stop with the start and stop of the traffic generator. this assumes the traffic generator and DUT are collocated, and that the variance in transmission delay from the generator to the DUT is negligible compared to the total traffic generation time 🫶
The measurement start time: the time when the traffic generator is started 🟢
The measurement stop time: the time when the traffic generator is stopped 🔴
the measurement time interval is then calculated as the difference between the stop time and the start time, minus the Idle Timeout: (stop time) - (start time) - (Idle Timeout) 🧮
this supposes the Cache Size is large enough that the time needed to fill it with Cache entries is longer than the Idle Timeout. otherwise, the time needed to fill the Cache has to be used in place of the Idle Timeout when calculating the measurement time interval 🫧
instead of measuring the absolute values of the stop and start times, you can set up the traffic generator to send traffic for a certain predefined time interval. that interval then gets used in the definition above instead of the difference (stop time) - (start time) 🧸
The Collector MUST stop collecting the Flow Export data at the measurement stop time. no cap, this one is non-negotiable 🛑
the Idle Timeout (or the time needed to fill the Cache) delays the Flow Export data behind the test traffic that the DUT analyzes. for example, if the traffic starts at time point X, Flow Export doesn't start until time point X + Idle Timeout (or X + time to fill the Cache) 🕰️ and since Flow Export capture has to stop along with the traffic (that's when the DUT stops processing the Flows at the given rate), the time window in which the DUT kept exporting data ends up shorter by the Idle Timeout than the window in which test traffic was sent from the traffic generator to the DUT. so the math is literally giving delay 😩🌸
5.5. Flow Export Rate Measurement
okay so the Flow Export Rate has to be measured in two steps, bestie 💅 Step one (point a. below) gets us the actual number. Step two (point b. below) is the double-check that no Flow Records got dropped during the measurement 🔍
a. In the first step, the captured Flow Export data MUST be analyzed only for the capturing interval (measurement time interval) from Section 5.4. During that window the DUT is forced to process Cache entries at the rate the packets are sent 🏃♀️ Once traffic generation finishes, what happens when the Cache empties is completely implementation-specific, so the Flow Export data from that period can't be used for benchmarking 🙅♀️
b. In the second step, ALL the Flow Export data from the DUT MUST be captured so we can work out the Flow Record losses 📦 Heads up: with big Cache Sizes (think hundreds of thousands of entries and up) the Flow Export can take many multiples of Idle Timeout to empty the Cache after the measurement. That's also completely implementation-specific 🫠
If the Collector can redirect the Flow Export data after the measurement time interval into a different capture buffer (or timestamp the received Flow Export data after that point), you can do it all in one step ✨ If it can't, every Flow Monitoring Throughput measurement at a given packet rate has to run twice: once to capture just the measurement time interval (to find the actual Flow Export Rate), and once more to capture all the Flow Export data (to find the Flow Record losses at that packet rate) 🔁
At the end of the measurement time interval, the DUT might still be processing Cache entries for Flows that expired from the Cache before the interval ended 👀 Those Flow Records can show up in an export packet that gets sent only after the interval is over. This imprecision can be softened by using a large number of Flow Records during the measurement (so the few stragglers in one export packet don't matter much) or by using timestamps exported with the Flow Records 🕰️
5.6. The Measurement Procedure
The measurement procedure is the same as the Throughput measurement in Section 26.1 of [RFC2544] on the traffic sending side 📤 The DUT output analysis happens on the traffic generator's receiving side for the test traffic, same as in RFC 2544 measurements 🙌
There's also an extra analysis, using the data captured by the Collector 🕵️♀️ Its job is to pin down the Flow Export Rate during the current measurement step and to verify that no Flow Records were dropped. The method for the Flow Export Rate is in Section 5.5 💖
Here's the tea 🍵: Flow Export performance can be seriously affected by how the Flow monitoring implementation formats Flow Records into Flow Export packets. The ordering and frequency of exported Control Information, and the number of Flow Records in one Flow Export packet, are all things we care about. The worst case is just one Flow Record per Flow Export packet 😭
Flow Export data should be sanity checked during the benchmark measurement for:
a. the number of Flow Records per packet. Just divide the exported Flow Records by the number of Flow Export packets captured during the measurement (this count should be available as a counter on the Collector capture buffer) 🧮
b. the number of Flow Records carrying Control Information per Flow Export packet. This is the total number of such Flow Records in the Flow Export data divided by the number of Flow Export packets 📊
6. RFC 2544 Measurements
RFC 2544 measurements can be run under two Flow monitoring setups (see also Section 3.4.2) 🎀 This section covers both and explains how to build the test traffic so RFC 2544 measurements happen in a controlled environment, from the Flow monitoring point of view 🧁 "Controlled" here means the tester always knows what Flow monitoring activity (Flow Export Rate) the offered traffic causes. No surprises, bestie 🙅♀️
This section mainly applies to the Throughput (RFC 2544, Section 26.1) and latency (RFC 2544, Section 26.2 ) measurements. It could also be used for frame loss rate (RFC 2544, Section 26.3) and back-to-back frames (RFC 2544, Section 26.4). Flow Export needs DUT resources to be generated and transmitted, so in most cases the Throughput will be much lower with Flow monitoring enabled on the DUT than without it 📉
Objective:
Provide RFC 2544 network device characteristics in the presence of Flow monitoring on the DUT. RFC 2544 studies lots of characteristics of network devices. The DUT forwarding and time characteristics without Flow monitoring can vary a lot once Flow monitoring is deployed on the device 🌪️
Metric definition:
Metric as specified in [RFC2544] 📏
The measured Throughput MUST NOT include the packet rate corresponding to the Flow Export data, because that isn't user traffic forwarded by the DUT 🚫 The DUT generates it as a result of enabling Flow monitoring, and it doesn't count toward the test traffic the DUT can handle 💅 Again: Flow Export needs DUT resources to be generated and transmitted, so the Throughput in most cases will be much lower when Flow monitoring is enabled than when it isn't 📉
6.1. Flow Monitoring Configuration
The Flow monitoring configuration (from Section 4.3) gets applied the same way as in Section 5, with one exception: the Active Timeout configuration 🧐
The Active Timeout SHOULD be set to several times the measurement time interval (see Section 5.4) 🕒 That way, if measurements with two traffic components are run (see Section 6.3.2), there's no Flow monitoring activity from the second traffic component 🙈
The Flow monitoring configuration doesn't change in any other way for the measurement in this section. What changes the outcome is the traffic configuration, covered in the sections below 🎀
6.2. Measurements with the Flow Monitoring Throughput Setup
To run a measurement with the Flow Monitoring Throughput setup, the big requirement is that traffic and Flow monitoring are configured so that each sent packet creates one entry in the DUT Cache 🧚 That limits the possible setups to the measurement with two traffic components from Section 6.3.2 💖
6.3. Measurements with a Fixed Flow Export Rate
This section covers measurements where the RFC 2544 metrics need to be measured with Flow monitoring enabled, but at a certain Flow Export Rate that's lower than the Flow Monitoring Throughput 🦋
The tester has both options, covered in Sections 6.3.1 and 6.3.2 🫶
6.3.1. Measurements with a Single Traffic Component
Section 12 of [RFC2544] talks about using protocol source and destination addresses for defined measurements. To run all the RFC 2544 type measurements with Flow monitoring enabled, the defined Flow Keys SHOULD contain an IP source and destination address 🌸 Then the RFC 2544 type measurements with Flow monitoring enabled can run under these extra conditions:
a. the test traffic is not limited to a single, unique pair of source and destination addresses 🌈
b. the traffic generator defines the test traffic like this: it has a parameter to send N packets (where N is an integer starting at 1 and incremented in small steps) with source IP address A and destination IP address B, before changing both IP addresses to the next value 🔢
This test traffic definition lets you run Flow monitoring measurements at a fixed Flow Export Rate while measuring the DUT's RFC 2544 characteristics ✨ This is the better option, because it best simulates live network traffic, where Flows contain more than one packet 🔥
The initial packet rate at N equal to 1 defines the Flow Export Rate for the whole measurement procedure. Increasing N later won't change the Flow Export Rate, because the time and Cache characteristics of the test traffic stay the same 🧠 This setup works for measurements with Flow Export Rates below the Flow Monitoring Throughput 🫧
6.3.2 Measurements with Two Traffic Components
The test traffic setup from Section 6.3.1 might be hard to pull off with commercial traffic generators, or the granularity of traffic rates defined by the initial packet rate at N equal to 1 might not suit the measurement you need 😩 The alternative is to define two traffic components in the test traffic: one to populate the Flow monitoring Cache, and one to run the RFC 2544 measurements 🎀
a. Flow monitoring test traffic component -- the exact traffic definition from Section 5.2 💜
b. RFC 2544 Test Traffic Component -- test traffic as specified by RFC 2544 MUST create just one entry in the DUT Cache 🎯 In this setup, that means a traffic stream with just one pair of unique source and destination IP addresses. (It could be avoided if the Flow Keys were, for example, UDP/TCP source and destination ports, and the Flow Keys didn't include the addresses 🙌)
The Flow monitoring traffic component exercises the DUT in terms of Flow activity, while the second traffic component measures the RFC 2544 characteristics 🌟
The measured Throughput is the sum of the packet rates of both traffic components 🧮 The definition of the other RFC 1242 metrics stays unchanged 💅
7. Flow Monitoring Accuracy
The pure Flow Monitoring Throughput measurement from Section 5 lets you verify Flow monitoring accuracy in terms of the exported Flow Record data 🔍 Since every Cache entry is populated by just one packet, the full set of data captured on the Collector can be parsed (for example, getting the values of all Flow Keys and other Flow Record fields, not only the overall Flow Record count in the exported data) 🤓 Then each set of parameters from each Flow Record can be checked against the parameters configured on the traffic generator and set in the packets sent to the DUT 💖 An exported Flow Record counts as accurate if:
a. all the Flow Record fields are present in each exported Flow Record ✅
b. all the Flow Record field values match the value ranges set by the traffic generator (for example, an IP address falls within the range of IP address increments on the traffic generator) 🎯
c. every possible Flow Record field value defined at the traffic generator has been found in the captured export data on the Collector 🔎 This check has to be weighed against any packet losses detected at the DUT during the measurement 📉
For a DUT with packet forwarding, Flow monitoring accuracy also involves data checks on the received traffic, as already discussed in Section 4 🫶
8. Evaluating Flow Monitoring Applicability ✨
okay so this section is basically the "will this box survive real life" check 🙈 The measurement results from this document, for certain DUTs, let you do a preliminary analysis of a Flow monitoring deployment using the traffic analysis data from providers' networks. An example of that kind of analysis on the Internet is [CAIDA], and I'll explain how you can use it below 💅 To estimate whether a network device can handle live traffic with Flow monitoring turned on, you need this data:
Average packet size: 350 bytes
Number of packets per IP flow: 20
Expected data rate on the network device: 1 Gbit/sYou also need the average number of Flows created per second on the network device. You work it out like this 🧮
Expected packet rate
Flows per second = --------------------
Packet per flowWith those example values, the network device is required to process 18000 Flows per second 😭 If you run the benchmarking as specified in this document, you can find a platform that can handle that load for that part of the user network. Main character energy, honestly 👑
Keep in mind that this is a very rough, averaged Flow activity estimate. It can't account for traffic anomalies. For example, a huge pile of DNS request packets (small packets coming from lots of different sources, mostly just one packet per Flow) can totally throw it off 💀
9. Acknowledgements
This work happened thanks to the patience and support of the Cisco Systems NetFlow development team: Paul Aitken, Paul Atkins, and Andrew Johnson 🫶 Huge thanks to Benoit Claise for so many detailed reviews and presentations of the document, and to Aamer Akhter for kicking this whole thing off 🌸 A special shoutout to the entire BMWG working group, especially chair Al Morton for all the support and work on this document, and to Paul Aitken for a super detailed technical review 💜
10. Security Considerations 🔥
okay this one is serious, so listen up 👀 Documents of this type don't directly affect the security of the Internet or corporate networks, as long as benchmarking is not performed on devices or systems connected to operating networks.
Benchmarking, as described in this memo, is limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above 🧪
The benchmarking network topology is supposed to be an independent test setup. It MUST NOT be connected to devices that might forward test traffic into a production network, or misroute traffic to the test management network. Absolutely not, bestie 🙅♀️
Benchmarking is also done on a "black-box" basis, relying only on measurements you can observe from outside the DUT 🦋
Special capabilities SHOULD NOT exist in the DUT just for benchmarking purposes. Any security implications arising from the DUT SHOULD be identical in the lab and in production networks 🔐
11. References
11.1. Normative References
[RFC2119]Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 📚[RFC2544]Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999. 📚
11.2. Informative References
[RFC1242]Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991. 🌷[RFC2285]Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, February 1998. 🌷[RFC3031]Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, January 2001. 🌷[RFC3917]Quittek, J., Zseby, T., Claise, B., and S. Zander, "Requirements for IP Flow Information Export (IPFIX)", RFC 3917, October 2004. 🌷[RFC3954]Claise, B., Ed., "Cisco Systems NetFlow Services Export Version 9", RFC 3954, October 2004. 🌷[RFC5101]Claise, B., Ed., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", RFC 5101, January 2008. 🌷[RFC5180]Popoviciu, C., Hamza, A., Van de Velde, G., and D. Dugatkin, "IPv6 Benchmarking Methodology for Network Interconnect Devices", RFC 5180, May 2008. 🌷[RFC5470]Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", RFC 5470, March 2009. 🌷[RFC5695]Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding Benchmarking Methodology for IP Flows", RFC 5695, November 2009. 🌷[CAIDA]Claffy, K., "The nature of the beast: recent traffic measurements from an Internet backbone",http://www.caida.org/publications/papers/1998/Inet98/Inet98.html🌐[IPFIX-CONFIG]Muenz, G., Muenchen, TU, Claise, B., and P. Aitken, "Configuration Data Model for IPFIX and PSAMP", Work in Progress, July 2011. 🚧[PSAMP-MIB]Dietz, T., Claise, B., and J. Quittek, "Definitions of Managed Objects for Packet Sampling", Work in Progress, October 2011. 🚧[IPFIX-MIB]Dietz, T., Kobayashi, A., Claise, B., and G. Muenz, "Definitions of Managed Objects for IP Flow Information Export", Work in Progress, March 2012. 🚧
Appendix A. (Informative) Recommended Report Format 📋
Here's the report template, so grab your clipboard 💖
Parameter Units
----------------------------------- ------------------------------------
Test Case test case name (Sections 5 and 6)
Test Topology Figure 2, other
Traffic Type IPv4, IPv6, MPLS, other
Test Results
Flow Monitoring Throughput Flow Records per second or Not
Applicable
Flow Export Rate Flow Records per second or Not
Applicable
Control Information Export Rate Flow Records per second
Throughput packets per second
(Other RFC 1242 Metrics) (as appropriate)
General Parameters
DUT Interface Type Ethernet, POS, ATM, other
DUT Interface Bandwidth MegaBits per second
Traffic Specifications
Number of Traffic Components (see Sections 6.3.1 and 6.3.2)
For each traffic component:
Packet Size bytes
Traffic Packet Rate packets per second
Traffic Bit Rate MegaBits per second
Number of Packets Sent number of entries
Incremented Packet Header Fields list of fields
Number of Unique Header Values number of entries
Number of Packets per Flow number of entries
Traffic Generation linearly incremented or
randomized
Flow monitoring Specifications
Direction ingress, egress, both
Observation Points DUT interface names
Cache Size number of entries
Active Timeout seconds
Idle Timeout seconds
Flow Keys list of fields
Flow Record Fields total number of fields
Number of Flows Created number of entries
Flow Export Transport Protocol UDP, TCP, SCTP, other
Flow Export Protocol IPFIX, NetFlow, other
Flow Export data packet size bytes
Flow Export MTU bytes
Parameter Units (continued)
----------------------------------- ------------------------------------
MPLS Specifications (for traffic type MPLS only)
Tested Label Operation imposition, swap, dispositionThe format of the report in this appendix is informative, but the entries in its contents are required as specified in the corresponding sections of this document 📝
Lots of the configuration parameters the measurement report needs can be pulled from the [IPFIX-MIB] and [PSAMP-MIB] MIB modules, and from the [IPFIX-CONFIG] YANG module or other general MIBs. So querying those modules from the DUT is super useful. First, it helps fill in the required entries of the report. Second, it documents all the other configuration parameters from the DUT too 🧚
Appendix B. (Informative) Miscellaneous Tests 🎀
This section lists tests that could help you check that Flow monitoring is working properly under different operational or stress conditions. They aren't considered suitable for benchmarking, for various reasons 🙈
B.1. DUT Under Traffic Load
The Flow Monitoring Throughput should be measured under different levels of static traffic load through the DUT. You can only do this by using two traffic components, as discussed in Section 6.3.2 🌈 One traffic component exercises the Flow Monitoring Plane. The second traffic component loads only the Forwarding Plane without affecting Flow monitoring. (In other words, it just creates a certain amount of permanent Cache entries.)
You should note the variance in Flow Monitoring Throughput as a function of traffic load, so you can compare two DUTs of similar architecture and capability 📊
B.2. In-Band Flow Export
The test topology in Section 4.1 requires a separate Flow Export interface, so the Flow Export data from the DUT doesn't mix with the test traffic from the traffic generator. That's needed to get clear, reproducible test conditions for the benchmark measurement 💅
Real network deployments might not have that luxury, for example on a very geographically large network 🌍
In that case, Flow Export uses an ordinary traffic forwarding interface, which is called in-band Flow Export.
You should verify Flow monitoring operation with the in-band Flow Export configuration by following these test steps:
a. Perform the benchmark test as specified in Section 5. One of the results is how much bandwidth Flow Export used on the dedicated Flow Export interface. b. Change the Flow Export configuration to use the test interface. c. Repeat the benchmark test while the receiver filters the Flow Export data out of the analysis.
The expected result is that the Throughput achieved in step a. is the same as the Throughput achieved in step c., provided that the bandwidth of the output DUT interface isn't the bottleneck. In other words, it must have enough capacity to forward both the test traffic and the Flow Export traffic 🔥
B.3. Variable Packet Size
It would be interesting to repeat the Flow monitoring measurements specified in this document with variable packet sizes within one particular test, for example test traffic containing mixed packet sizes 🌸 The packet forwarding tests specified mainly in [RFC2544] don't recommend doing this. But Flow monitoring doesn't depend on packet size, so this kind of test could run during the Flow Monitoring Throughput measurement, and checking its value doesn't depend on the offered traffic packet sizes ✨ Just be careful to design these tests so you don't get measurement errors from physical bandwidth limits or from changes in base forwarding performance with packet size 🧐
B.4. Bursty Traffic 💥
okay so bestie, RFC 2544 Section 21 is where bursty traffic gets discussed and defined 🎀. this test uses it to gauge how the DUT handles short-term overload when it's doing Flow monitoring 🔥. the twist is that we are NOT measuring the Flow Export Rate the DUT can sustain here. the number we care about is the absolute count of Flow Records the DUT can process without dropping any single one 💅 ngl, that's a totally different vibe.
the traffic setup looks like this:
- a. each sent packet creates a new Cache entry ✨
- b. the packet rate is set to the maximum transmission speed of the DUT interface used for the test 🚀
B.5. Various Flow Monitoring Configurations 🌈
this section is basically a translator 🧚. it takes the terminology from the IPFIX documents ([RFC5470], [RFC5101], and others) and maps it onto the terms used in this document. heads up: Section B.5.2 proposes another measurement that you literally cannot verify with a black box test 🙈
B.5.1. Throughput without the Metering Process 🍓
if the Metering Process isn't defined on the DUT, then there's no Flow monitoring Cache and no Flow analysis happening at all 💀. so the performance measurement in that case is just pure [RFC2544] measurement. easy peasy 🫶.
B.5.2. Throughput with the Metering Process 🦋
if only the Metering Process is enabled, Flow analysis is on and running, but no Flow Export happens. this one is a super useful test of the DUT's capabilities 👑. it matches the case where a network operator uses Flow monitoring for things like manually spotting denial-of-service attacks, and doesn't want to use Flow Export.
you can run the performance testing on this DUT as discussed in this document, but it's not possible to verify the operation and results without interrogating the DUT. so you gotta ask the DUT directly 😭.
B.5.3. Throughput with the Metering and Exporting Processes 💜
this is the performance testing as discussed in Section 6. the full package 🎁.
B.6. Tests With Bidirectional Traffic 🔁
bidirectional traffic is not part of the normative benchmarking tests. this came from discussion with, and a recommendation from, the Benchmarking working group. the experienced participants said this kind of traffic didn't produce reproducible results 🙅 (not me being dramatic, that's literally what they said lol).
the test topology in Figure 2 can be expanded to verify Flow monitoring functionality with bidirectional traffic. just use the interfaces in full duplex mode, e.g., sending and receiving at the same time on each of them 🌸.
the same rules should be applied for Flow creation in the DUT Cache (as per Sections 4.1 and 4.3.1). traffic passing through each Observation Point should always create a new Cache entry. e.g., the same traffic should not just get looped back on the receiving interfaces to fake bidirectional traffic 🙃. no cheating, bestie.
B.7. Instantaneous Flow Export Rate ⏱️
extra useful info when you're analyzing Flow Export data is the time distribution of the instantaneous Flow Export Rate ✨. you can derive it during the measurements in two ways:
- a. the Collector might provide the capability to decode Flow Export during capturing, and at the same time count the Flow Records and give you the instantaneous Flow Export Rate (or simply an average over a shorter time interval than the one specified in Section 5.4) 🧮
- b. the Flow Export protocol (like IPFIX [RFC5101]) can provide time stamps in the Flow Export packets. that lets you do time-based analysis and calculate the Flow Export Rate as an average over a much shorter time interval than Section 5.4 specifies ⏰
the accuracy and the shortest time average will always be limited by the precision of the time stamps (1 second for IPFIX), or by the capabilities of the DUT and the Collector 🌈
Author's Address 🖋️
Jan Novak (editor)
Cisco Systems
Edinburgh
United Kingdom
EMail: janovak@cisco.com📦 where this page comes from
- ietf: rfc6645 2012-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms