RFC 9693

RFC 9693: Benchmarking Methodology for Stateful NATxy Gateways 💖

(Part 1 of 7, more parts coming bestie 🫶)

Okay so the title block is giving very important-document energy ✨. This is RFC 9693, titled Benchmarking Methodology for Stateful NATxy Gateways. It was written by G. Lencse (Széchenyi István University) and K. Shima (SoftBank Corp.). It's an Informational document, which means it's published for information and is not an Internet Standards Track specification. The date is January 2025 and the ISSN is 2070-1721. Its header doesn't list anything it obsoletes or updates, so it's kind of a solo main character 🦋.

Abstract 🌸

Okay so RFC 2544 is the OG benchmarking methodology for network interconnect devices. RFC 5180 handles IPv6 specificities and gives us a technology update, but it leaves out IPv6 transition technologies. RFC 8219 covers IPv6 transition technologies, including stateful NAT64. BUT none of them talk about how to use the pseudorandom port numbers from RFC 4814 with any stateful NATxy technologies (like NAT44, NAT64, and NAT66). 😭

This document explains why using pseudorandom port numbers with stateful NATxy gateways is a seriously difficult problem. It recommends a fix that limits the port number ranges and uses two test phases (phase 1 and phase 2). It also shows how the classic performance measurement procedures (e.g., throughput, frame loss rate, latency, etc.) can be carried out. And it defines new performance metrics and measurement procedures for measuring the maximum connection establishment rate, the connection tear-down rate, and the connection tracking table capacity. Iconic, honestly 👑.

Status of This Memo 📜

This document is not an Internet Standards Track specification; it is published for informational purposes.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841.

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc9693.

Copyright Notice 🔒

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.

Table of Contents 🗺️

1.  Introduction
  1.1.  Requirements Language
2.  Pseudorandom Port Numbers and Stateful Translation
3.  Test Setup and Terminology
  3.1.  When Testing with a Single IP Address Pair
  3.2.  When Testing with Multiple IP Addresses
4.  Recommended Benchmarking Method
  4.1.  Restricted Number of Network Flows
  4.2.  Test Phase 1
  4.3.  Consideration of the Cases of Stateful Operation
  4.4.  Control of the Connection Tracking Table Entries
  4.5.  Measurement of the Maximum Connection Establishment Rate
  4.6.  Validation of Connection Establishment
  4.7.  Test Phase 2
  4.8.  Measurement of the Connection Tear-Down Rate
  4.9.  Measurement of the Connection Tracking Table Capacity
  4.10. Writing and Reading Order of the State Table
5.  Scalability Measurements
  5.1.  Scalability Against the Number of Network Flows
  5.2.  Scalability Against the Number of CPU Cores
6.  Reporting Format
7.  Implementation and Experience
8.  Limitations of Using UDP as a Transport Layer Protocol
9.  IANA Considerations
10. Security Considerations
11. References
  11.1.  Normative References
  11.2.  Informative References
Acknowledgements
Authors' Addresses

1. Introduction ✨

Okay so [RFC2544] defines a comprehensive benchmarking methodology for network interconnect devices, and people are still using it today 💅. It's mainly independent of IP version, but it uses IPv4 in its examples. [RFC5180] addresses IPv6 specificities and also adds technology updates, but it declares IPv6 transition technologies are out of its scope. [RFC8219] addresses the IPv6 transition technologies, including stateful NAT64. It reuses several benchmarking procedures from [RFC2544] (e.g., throughput, frame loss rate), and it redefines the latency measurement and adds further ones (e.g., the Packet Delay Variation (PDV) measurement).

However, none of them discuss how to apply pseudorandom port numbers from [RFC4814] when benchmarking stateful NATxy gateways (such as NAT44 [RFC3022], NAT64 [RFC6146], and NAT66). (It should be noted that stateful NAT66 is not an IETF specification but refers to an IPv6 version of the stateful NAT44 specification.) The authors are not aware of any other RFCs that address this question. 🙈

First, this document discusses why using pseudorandom port numbers with stateful NATxy gateways is a difficult problem. Then, a solution is recommended. 🧚

1.1. Requirements Language 🔑

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

2. Pseudorandom Port Numbers and Stateful Translation 🎀

In its appendix, [RFC2544] defines a frame format for test frames, including specific source and destination port numbers. [RFC4814] recommends using pseudorandom and uniformly distributed values for both source and destination port numbers. However, stateful NATxy (such as NAT44, NAT64, and NAT66) solutions use the port numbers to identify connections. The usage of pseudorandom port numbers causes different problems depending on the direction:

• For the client-to-server direction, pseudorandom source and destination port numbers could be used; however, this approach would be a denial-of-service attack against the stateful NATxy gateway, because it would exhaust its connection tracking table capacity. To that end, let us see some calculations using the recommendations of [RFC4814]:
  • The recommended source port range is 1024-65535; thus, its size is 64512.
  • The recommended destination port range is 1-49151; thus, its size is 49151.
  • The number of source and destination port number combinations is 3,170,829,312.

It should be noted that the usage of different source and destination IP addresses further increases the number of connection tracking table entries.

• For the server-to-client direction, the stateful Device Under Test (DUT) would drop any packets that do not belong to an existing connection; therefore, the direct usage of pseudorandom port numbers from the ranges mentioned above is not feasible. 💀

3. Test Setup and Terminology 🧚

Section 12 of [RFC2544] requires testing using a single protocol source and destination address pair first and then also using multiple protocol addresses. The same approach is followed: first, a single source and destination IP address pair is used, and then it is explained how to use multiple IP addresses. 🦋

---

That's the first chunk of RFC 9693 done bestie! 🌈 Next up we're getting into the good stuff 💜

3.1. When Testing with a Single IP Address Pair ✨

okay so this methodology works with ANY IP version for benchmarking stateful NATxy gateways, where x and y are in {4, 6} 🦋 To keep things easy to follow, we're using two typical examples: stateful NAT44 and stateful NAT64 💅

The setup for the well-known stateful NAT44 (also called Network Address and Port Translation, or NAPT) is in Figure 1 🎀

Quick heads up: the private IPs from [RFC1918] are only used so the example is easy to read 🙈 Using the IP addresses reserved for benchmarking is totally legit too 💖

              +--------------------------------------+
     10.0.0.2 |Initiator                    Responder| 198.19.0.2
+-------------|                Tester                |<------------+
| private IPv4|                         [state table]| public IPv4 |
|             +--------------------------------------+             |
|                                                                  |
|             +--------------------------------------+             |
|    10.0.0.1 |                 DUT:                 | 198.19.0.1  |
+------------>|        Stateful NAT44 gateway        |-------------+
  private IPv4|     [connection tracking table]      | public IPv4
              +--------------------------------------+

Figure 1: Test Setup for Benchmarking Stateful NAT44 Gateways

The setup for the stateful NAT64 solution [RFC6146] is also super popular, and it's in Figure 2 🌈

              +--------------------------------------+
    2001:2::2 |Initiator                    Responder| 198.19.0.2
+-------------|                Tester                |<------------+
| IPv6 address|                         [state table]| IPv4 address|
|             +--------------------------------------+             |
|                                                                  |
|             +--------------------------------------+             |
|   2001:2::1 |                 DUT:                 | 198.19.0.1  |
+------------>|        Stateful NAT64 gateway        |-------------+
  IPv6 address|     [connection tracking table]      | IPv4 address
              +--------------------------------------+

Figure 2: Test Setup for Benchmarking Stateful NAT64 Gateways

For the transport layer, [RFC2544] recommended testing with UDP, and [RFC8219] kept that going 🧚 UDP is still the general recommendation too, so any port numbers in the text below are UDP port numbers. The rationale and limitations of this approach are covered in Section 8 💜

Here are the most important pieces of the proposed benchmarking system:

• Connection: ngl this one's a little plot twist 🙃 UDP is connectionless, but stateful NATxy gateways still track their translation mappings as a "connection," and for UDP they use the same kind of entries they use for TCP.
• Connection tracking table: the stateful NATxy gateway uses this to do the stateful translation in the server-to-client direction. Its size, policy, and content are unknown to the Tester 🔍
• Four tuple: the four numbers that identify a connection: source IP address, source port number, destination IP address, and destination port number.
• State table: the Responder of the Tester pulls the four tuple out of each received test frame and stores it in its state table 📝 There's a recommendation on how to write and read this table in Section 4.10.
• Initiator: the port of the Tester that may start a connection through the stateful DUT in the client-to-server direction. Theoretically it can use any source and destination port numbers from the ranges recommended by [RFC4814]. If the four tuple doesn't belong to an existing connection, the DUT registers a new connection in its connection tracking table ✨
• Responder: the port of the Tester that may not start a connection through the stateful DUT in the server-to-client direction. It may only send frames that belong to an existing connection. To do that, it uses four tuples it previously pulled from received test frames and stored in its state table 🫶
• Test phase 1: test frames are sent only by the Initiator to the Responder through the DUT, filling up both the DUT's connection tracking table and the Responder's state table 🌸 This is a brand-new operation phase for stateful NATxy benchmarking, and the reason it exists is explained in Section 4.2.
• Test phase 2: the measurement procedures defined by [RFC8219] (e.g., throughput, latency, etc.) run in this phase, after test phase 1 is done 🔥 Test frames get sent as required (e.g., a bidirectional or unidirectional test in either direction).

One more definition is used in this document:

• Black box testing: the Tester doesn't know the internal structure and operation of the DUT. It can send input to the DUT and observe the DUT's output 👀

---

3.2. When Testing with Multiple IP Addresses ✨

This section looks at how many IP addresses are needed and how many are actually available 🧮

In Figure 1, the single 198.19.0.1 IPv4 address sits on the WAN side port of the stateful NAT44 gateway. In practice though, it's not really one IP address. It's an IP address range assigned to the WAN side port of stateful NAT44 gateways 🌍 How big that range needs to be depends on the number of client nodes and on the type of stateful NAT44 algorithm. (The traditional algorithm always replaces the source port number when a new connection is established, so it needs a bigger range than the extended algorithm, which only replaces the source port number when it's necessary. Please refer to Tables 1 and 2 of [LEN2015].)

When router testing is done, Section 12 of [RFC2544] wants testing with a single source and destination IP address pair first, and then with destination IP addresses from 256 different networks 🍓 The 16-23 bits of the 198.18.0.0/24 and 198.19.0.0/24 addresses can be used to express those 256 networks. This document doesn't deal with router testing, so no multiple destination networks are needed. That means these bits are free to express multiple IP addresses that belong to the same "/16" network 💅 Plus, both the 198.18.0.0/16 and the 198.19.0.0/16 networks can be used on the right side of the test setup, since the left side uses private IP addresses from the 10.0.0.0/16 network.

  10.0.0.2/16 - 10.0.255.254/16      198.19.0.0/15 - 198.19.255.254/15
              \  +--------------------------------------+  /
               \ |Initiator                    Responder| /
   +-------------|                Tester                |<------------+
   | private IPv4|                         [state table]| public IPv4 |
   |             +--------------------------------------+             |
   |                                                                  |
   |             +--------------------------------------+             |
   | 10.0.0.1/16 |                 DUT:                 | public IPv4 |
   +------------>|        Stateful NAT44 gateway        |-------------+
     private IPv4|     [connection tracking table]      | \
                 +--------------------------------------+  \
                                      198.18.0.1/15 - 198.18.255.255/15

Figure 3: Test Setup for Benchmarking Stateful NAT44 Gateways Using Multiple IPv4 Addresses

A possible solution for assigning multiple IPv4 addresses is shown in Figure 3 🌸 On the left side, the private IP address range is abundantly large. (The 16-31 bits were used to generate nearly 64k potential different source addresses, but the 8-15 bits are also available if needed.) On the right side, the 198.18.0.0./15 network is used, and it's been cut into two equal parts. (An asymmetric division is also possible, if needed.)

It should be noted that these are the potential address ranges. The actual address ranges to be used are discussed in Section 4.1 🦋

In the case of stateful NAT64, a single "/64" IPv6 prefix has plenty of bits to express different IPv6 addresses 🧚 Figure 4 shows an example where bits 96-111 are used for that purpose.

  2001:2::[0000-ffff]:0002/64       198.19.0.0/15 - 198.19.255.254/15
              \  +--------------------------------------+  /
     IPv6      \ |Initiator                    Responder| /
   +-------------|                Tester                |<------------+
   | addresses   |                         [state table]| public IPv4 |
   |             +--------------------------------------+             |
   |                                                                  |
   |             +--------------------------------------+             |
   | 2001:2::1/64|                 DUT:                 | public IPv4 |
   +------------>|        Stateful NAT64 gateway        |-------------+
    IPv6 address |     [connection tracking table]      | \
                 +--------------------------------------+  \
                                      198.18.0.1/15 - 198.18.255.255/15

Figure 4: Test Setup for Benchmarking Stateful NAT64 Gateways Using Multiple IPv6 and IPv4 Addresses

---

4. Recommended Benchmarking Method ✨

4.1. Restricted Number of Network Flows ✨

okay so when you're testing with a single IP address pair, the number of network flows is decided by the combos of source and destination port numbers 🎀

The Initiator SHOULD use restricted ranges for source and destination port numbers so it doesn't exhaust the connection tracking table capacity of the DUT, which is explained in Section 2 💅 If it's possible, the source port number range SHOULD be bigger (like a few tens of thousands), and the destination port number range SHOULD be smaller (anywhere from a few to several hundreds or thousands, depending on what you need) 🦋 The reason is that the source and destination ports you see in real Internet traffic are not symmetrical. Source ports can be random, but a few destination ports are super popular (443 or 80, see [IIR2020]) and the rest barely show up at all. It's also giving asymmetry in the Linux kernel routing hash function [LEN2020] 🧚

However, in some special cases the source port range is limited 🙈 For example, when you benchmark the Customer Edge (CE) and Border Relay (BR) of a Mapping of Address and Port using Translation (MAP-T) system [RFC7599] together (as a compound system doing stateful NAT44), the source port range is limited to the number of source port numbers assigned to each subscriber. (It could be as low as 2048 ports.)

When multiple IP addresses are used, the port number ranges should be even more restricted, because the number of potential network flows is the product of the size of:

• the source IP address range,
• the source port number range,
• the destination IP address range, and
• the destination port number range.

Also, the recommended method requires enumerating all their possible combinations in test phase 1, as described in Section 4.4 🌸

The number of network flows can be used as a parameter. The performance of the stateful NATxy gateway MAY be examined as a function of this parameter, as described in Section 5.1 💜

4.2. Test Phase 1 🎀

Test phase 1 has two jobs, bestie 👑

1. The connection tracking table of the DUT gets filled. This matters because its maximum connection establishment rate may be lower than its maximum frame forwarding rate (that is, its throughput).
2. The state table of the Responder gets filled with valid four tuples. This is a precondition so the Responder can transmit frames that belong to connections existing in the connection tracking table of the DUT.

Those two things are always needed before test phase 2, but test phase 1 can be used without test phase 2. That happens when the maximum connection establishment rate is measured (as described in Section 4.5) ✨

Test phase 1 MUST be performed before all tests are performed in test phase 2. Here's what happens in test phase 1 🦋

1. The Initiator sends test frames to the Responder through the DUT at a specific frame rate.
2. The DUT performs the stateful translation of the test frames, and it also stores the new connections in its connection tracking table.
3. The Responder receives the translated test frames and updates its state table with the received four tuples. The Responder transmits no test frames during test phase 1.

When test phase 1 is done to prep for test phase 2, the applied frame rate SHOULD be safely lower than the maximum connection establishment rate. (That implies the maximum connection establishment rate measurement MUST be performed first 🫶) Check Section 4.4 for further conditions about timeout and enumerating all possible four tuples 💖

4.3. Consideration of the Cases of Stateful Operation 💅

The authors look at the most important events that can happen while a stateful NATxy gateway is running, plus the Actions the gateway takes in response 🔥

1. EVENT: A packet not belonging to an existing connection arrives in the client-to-server direction.
   • ACTION: A new connection is registered into the connection tracking table, and the packet is translated and forwarded.
2. EVENT: A packet not belonging to an existing connection arrives in the server-to-client direction.
   • ACTION: The packet is discarded.
3. EVENT: A packet belonging to an existing connection arrives (in any direction).
   • ACTION: The packet is translated and forwarded, and the timeout counter of the corresponding connection tracking table entry is reset.
4. EVENT: A connection tracking table entry times out.
   • ACTION: The entry is deleted from the connection tracking table.

Because of "black box" testing, the Tester can't directly examine (or delete) the entries of the connection tracking table 🙈 Still, the entries can and MUST be controlled by setting an appropriate timeout value and carefully picking the port numbers of the packets (as described in Section 4.4), so you get meaningful and repeatable measurement results 🍓

This document aims to support measuring these performance characteristics of a stateful NATxy gateway:

• maximum connection establishment rate
• all "classic" performance metrics like throughput, frame loss rate, latency, etc.
• connection tear-down rate
• connection tracking table capacity

4.4. Control of the Connection Tracking Table Entries 🌈

It's necessary to control the connection tracking table entries of the DUT so the measurements have clear conditions. You can get two extreme situations pretty simply 🧚

1. All frames create a new entry in the connection tracking table of the DUT, and no old entries get deleted during the test. This is required for measuring the maximum connection establishment rate.
2. No new entries get created in the connection tracking table of the DUT, and no old ones get deleted during the test. This is ideal for the measurements done in phase 2, like throughput, latency, etc.

From here, the following two assumptions are used:

1. The connection tracking table of the stateful NATxy is large enough to store all connections defined by the different four tuples.
2. Each experiment starts with an empty connection tracking table. (You can make sure of this by deleting its content before the experiment.)

The first extreme situation can be achieved by:

• using different four tuples for every single test frame in test phase 1, and
• setting the UDP timeout of the NATxy gateway to a value higher than the length of test phase 1.

The second extreme situation can be achieved by:

• enumerating all possible four tuples in test phase 1, and
• setting the UDP timeout of the NATxy gateway to a value higher than the length of test phase 1 plus the gap between the two phases plus the length of test phase 2.

As described in [RFC4814], pseudorandom port numbers are REQUIRED, which the authors believe is a good approximation of the distribution of the source port numbers a NATxy gateway on the Internet may face 🦋

Although enumerating all possible four tuples is not a requirement for the first extreme situation, and using different four tuples in test phase 1 is not a requirement for the second extreme situation, pseudorandom enumeration of all possible four tuples in test phase 1 is a good solution in both cases ✨ You can generate pseudorandom enumeration of all possible four tuples in a computationally efficient way by using Durstenfeld's random shuffle algorithm [DUST1964] to prepare a random permutation of the previously enumerated four tuples 💜

Enumerating the four tuples in increasing or decreasing order (or in any other specific order) MAY be used as an additional measurement 🍓

4.5. Measurement of the Maximum Connection Establishment Rate 👑

The maximum connection establishment rate is an important characteristic of the stateful NATxy gateway, and figuring it out is necessary for the safe execution of test phase 1 (without frame loss) before test phase 2 🔥

The measurement procedure is very similar to the throughput measurement procedure defined in [RFC2544], so if you know that one, you're already halfway there 🫶

The procedure goes like this:

• The Initiator sends a specific number of test frames, using all different four tuples, at a specific rate through the DUT.
• The Responder counts the frames that are successfully translated by the DUT.
• If the count of offered frames is equal to the count of received frames, the rate of the offered stream is raised and the test is rerun.
• If fewer frames are received than were transmitted, the rate of the offered stream is reduced and the test is rerun.

The maximum connection establishment rate is the fastest rate at which the count of test frames successfully translated by the DUT is equal to the number of test frames sent to it by the Initiator 💅

Note: In practice, the usage of binary search is RECOMMENDED. 🧚

4.6. Validation of Connection Establishment 🦋

Because of "black box" testing, the entries of the connection tracking table of the DUT may not be directly examined 🙈 But you can still check whether the connections are there by sending frames from the Responder to the Initiator in test phase 2, using all four tuples stored in the state table of the Tester (at a low enough frame rate). If all the test frames arrive, the connections are indeed present 🎀

The procedure goes like this:

When all the desired N number of test frames are sent by the Initiator to the Receiver at frame rate R in test phase 1 for the maximum connection establishment rate measurement, and the Receiver has successfully received all N frames, the establishment of the connections is checked in test phase 2 as follows:

• The Responder sends test frames to the Initiator at frame rate r=R*alpha for the duration of N/r, using a different four tuple from its state table for each test frame.
• The Initiator counts the received frames. If all N frames have arrived, the R frame rate of the maximum connection establishment rate measurement (performed in test phase 1) is raised for the next iteration; otherwise, it is lowered (and it's lowered as well if test frames were missing in the preliminary test phase) 💖

Notes:

• The alpha is a kind of "safety factor" 🫶 It aims to make sure the frame rate used for the validation isn't too high, and the test may fail only if at least one connection is not present in the connection tracking table of the DUT. (So alpha should typically be less than 1, e.g., 0.8 or 0.5.)
• The duration of N/r and the frame rate of r means that N frames are sent for validation.
• The order of four tuple selection is arbitrary, provided that all four tuples MUST be used.
• Refer to Section 4.9 for a short analysis of how the measurement operates and what problems may occur 🌸

4.7. Test Phase 2 ✨

okay so, there are three possible traffic direction cases during test phase 2 🦋

1. Bidirectional traffic: The Initiator sends test frames to the Responder, and the Responder sends test frames to the Initiator.
2. Unidirectional traffic from the Initiator to the Responder: The Initiator sends test frames to the Responder, but the Responder does not send test frames to the Initiator.
3. Unidirectional traffic from the Responder to the Initiator: The Responder sends test frames to the Initiator, but the Initiator does not send test frames to the Responder.

bestie, if the Initiator is sending test frames, it uses pseudorandom source port numbers and destination port numbers from the restricted port number ranges 🎀 (and if it uses multiple source and/or destination IP addresses, those ranges are limited too, so no sneaky extras 💅).

The Responder receives the test frames, updates its state table, and processes them the way the measurement procedure says to 🧚 (for example, it only counts them for the throughput test, or it handles timestamps for latency or PDV tests, etc.).

if the Responder is the one sending test frames, it uses the four tuples from its state table 🌈. The order it reads that state table in can follow different policies, which we talk about in Section 4.10. Then the Initiator receives those frames and processes them as the measurement procedure requires ✨.

as for the actual measurement procedures, using the updated ones from Section 7 of [RFC8219] is RECOMMENDED 💜. Iconic, honestly.

4.8. Measurement of the Connection Tear-Down Rate 💀

okay, connection tear-down can cause serious load for the NATxy gateway 😭. Here's how you measure the connection tear-down performance 🍓:

1. Load a certain number of connections (N) into the connection tracking table of the DUT (same way you did for the maximum connection establishment rate 🫶).
2. Record TimestampA.
3. Delete the content of the connection tracking table of the DUT.
4. Record TimestampB.

and then you compute the connection tear-down rate like this 🔥:

     connection tear-down rate = N / ( TimestampB - TimestampA)

the connection tear-down rate SHOULD be measured for various values of N 🦋. Do not skip this, it's giving thorough 👑.

⚠️ It's assumed that the content of the connection tracking table can be deleted by an out-of-band control mechanism that's specific to the given NATxy gateway implementation. For example, you could remove the appropriate kernel module under Linux 🙈.

and lowkey, the performance of removing the entire table content in one go may be different from removing all the entries one by one 🌸. Literally two different vibes, so don't assume they match ✨.

4.9. Measurement of the Connection Tracking Table Capacity ✨

okay so the connection tracking table capacity is a huge metric for stateful NATxy gateways 💅 but measuring it is not easy bestie 😭 because an elementary step of a validated maximum connection establishment rate measurement (defined in Section 4.6) may only have a few distinct observable outcomes, and some of those outcomes can have totally different root causes 🙈

During test phase 1, the number of test frames the Responder receives is less than the number the Initiator sent. That can happen for a couple of reasons:

• The R frame sending rate was higher than the maximum connection establishment rate. (Note that the maximum connection establishment rate is now considered unknown, because you can't measure it without assumption 1 in Section 4.4 🫶) This one can be fixed by lowering the R rate and re-running the test. (You may need to do this multiple times while R>0.)
• The connection tracking table of the DUT is full. (The DUT might refuse to delete connections, or deleting them makes it slower. We won't dig into this case in test phase 1 🌸)

During test phase 1, the number of test frames the Responder receives equals the number the Initiator sent. Yay! In this case, the connections get validated in test phase 2 🎀 The validation can give two kinds of results:

1. The number of validation frames the Initiator receives equals the number the Responder sent. 🎉 This proves the connection tracking table of the DUT has enough room, and that both R and r were picked properly.
2. The number of validation frames the Initiator receives is less than the number the Responder sent. Honestly this one is messy, because it can have several root causes 💀
   • The connection tracking table of the DUT is full. (It doesn't matter whether old connections get discarded to make room for new ones, or the new ones get discarded. Some connections are lost either way, and that makes the validation fail.)
   • The R frame sending rate the Initiator used in test phase 1 was too high. So some connections never got established, even though all the test frames arrived at the Responder. This can be fixed by lowering the R rate and re-running the test. (You may need to do this multiple times while R>0.)
   • The r frame sending rate the Responder used in test phase 2 was too high. So some test frames never reached the Initiator, even though all the connections were sitting in the connection tracking table of the DUT. This can be fixed by lowering the r rate and re-running the test. (You may need to do this multiple times while r>0.)

This is the problem 😩 because all three of those root causes look exactly the same from the outside, so it's not easy to tell whether R or r should be lowered.

Experience shows that the DUT may collapse if its memory runs out. That makes measuring the connection tracking table capacity kinda inconvenient 🥲 This possibility is included in the recommended measurement procedure, but how to detect and fix that situation isn't covered here (for example, how the algorithm could reset the DUT).

For measuring the size of the connection tracking table, you first need a safe number, which is C0 🌈 It's a precondition that C0 connections can definitely be stored in the connection tracking table of the DUT. Using C0, you can determine the maximum connection establishment rate for C0 connections. You do this with a binary search using validation, and the result is R0. The values C0 and R0 become your "safe" starting values for the next two searches ✨

First, an exponential search finds the order of magnitude of the connection tracking table capacity. The search stops if the DUT collapses OR if the maximum connection establishment rate severely drops (for example, to one tenth of its value) because the number of connections got doubled 🍓

Then the result of the exponential search gives you the order of magnitude of the connection tracking table size. Before getting into the possible algorithms for finding the exact size, three possible replacement policies for the NATxy gateway get considered 👑

1. The gateway doesn't delete any live connections until their timeout expires.
2. The gateway replaces live connections according to the Least Recently Used (LRU) policy.
3. The gateway does a garbage collection when its connection tracking table is full and a frame with a new four tuple arrives. During the garbage collection, it deletes the K LRU connections, where K is greater than 1.

Now let's look at what happens, and how many validation frames arrive, in those three cases 🦋 Let the size of the connection tracking table be S, and the number of preliminary frames be N, where S is less than N.

1. The connections defined by the first S test frames get registered into the connection tracking table of the DUT, and the last N-S connections are lost. (Whether the last N-S test frames get translated and forwarded in test phase 1, or just dropped, is a different question 🙈) During validation, the validation frames with four tuples matching the first S test frames arrive at the Initiator, and the other N-S validation frames are lost.
2. All connections get registered into the connection tracking table of the DUT, but the first N-S connections get replaced (and so they're lost). During validation, the validation frames with four tuples matching the last S test frames arrive at the Initiator, and the other N-S validation frames are lost.
3. Depending on the values of K, S, and N, maybe fewer than S connections survive. In the worst case, only S-K+1 validation frames arrive, even though the size of the connection tracking table is S 😭

If you know the stateful NATxy gateway uses the first or second replacement policy, and you also know that both the R and r rates are low enough, then the final step of finding the size of the connection tracking table is simple 💖 If the Responder sent N validation frames and the Initiator received N' of them, then the size of the connection tracking table is N'.

In the general case, a binary search finds the exact value of the connection tracking table capacity within E error. The search picks the lower half of the interval if the DUT collapses OR if the maximum connection establishment rate severely drops (for example, to its half). Otherwise, it picks the higher half. The search stops when the size of the interval is less than the E error 🧚

The algorithms for the general case are defined using C-like pseudocode in Figure 5. In practice, this algorithm can be made more efficient: the binary search for the maximum connection establishment rate stops if an elementary test fails at a rate under RS*beta or RS*gamma, during the external search or the final binary search for the capacity of the connection tracking table, respectively. (This saves a lot of execution time, because it skips those long-lasting tests at low rates 🔥)

// The binarySearchForMaximumConnectionCstablishmentRate(c,r)
// function performs a binary search for the maximum connection
// establishment rate in the [0, r] interval using c number of
// connections.

// This is an exponential search for finding the order of magnitude
// of the connection tracking table capacity
// Variables:
//   C0 and R0 are beginning safe values for the connection
//     tracking table size and connection establishment rate,
//     respectively
//   CS and RS are their currently used safe values
//   CT and RT are their values for the current examination
//   beta is a factor expressing an unacceptable drop in R (e.g.,
//     beta=0.1)
//   maxrate is the maximum frame rate for the media
R0=binarySearchForMaximumConnectionCstablishmentRate(C0,maxrate);
for ( CS=C0, RS=R0; 1; CS=CT, RS=RT )
{
  CT=2*CS;
  RT=binarySearchForMaximumConnectionCstablishmentRate(CT,RS);
  if ( DUT_collapsed || RT < RS*beta )
    break;
}
// At this point, the size of the connection tracking table is
// between CS and CT.

// This is the final binary search for finding the connection
// tracking table capacity within E error
// Variables:
//   CS and RS are the safe values for connection tracking table size
//     and connection establishment rate, respectively
//   C and R are the values for the current examination
//   gamma is a factor expressing an unacceptable drop in R
//     (e.g., gamma=0.5)
for ( D=CT-CS;  D>E; D=CT-CS )
{
  C=(CS+CT)/2;
  R=binarySearchForMaximumConnectionCstablishmentRate(C,RS);
  if ( DUT_collapsed || R < RS*gamma )
    CT=C; // take the lower half of the interval
  else
    CS=C,RS=R; // take the upper half of the interval
}
// At this point, the size of the connection tracking table is
// CS within E error.

Figure 5: Measurement of the Connection Tracking Table Capacity

---

4.10. Writing and Reading Order of the State Table 🌸

For the writing policy of the Responder's state table, round robin is RECOMMENDED 🫶 because it makes sure its entries stay automatically fresh and consistent with the connection tracking table of the DUT.

The Responder can read its state table in various orders, for example:

• pseudorandom
• round robin

Pseudorandom is RECOMMENDED to follow the approach of [RFC4814] ✨ Round robin can be used as a computationally cheaper alternative 💅

5. Scalability Measurements 📈

As for scalability measurements, no new types of performance metrics are defined. But it is RECOMMENDED to run measurement series where the value of one or more parameter(s) gets changed, so you can discover how the different values of those parameter(s) affect the performance of the DUT 🦋

5.1. Scalability Against the Number of Network Flows ✨

okay so this section is all about figuring out how gracefully a stateful NATxy gateway slays (or doesn't 💀) as the number of network flows goes up. bestie, the whole point is to measure the degradation, like how much performance drops as the pile of flows grows 📉🎀

for the actual numbers of network flows you test with, it is RECOMMENDED to pick some representative values from the range the DUT could realistically be handling in its intended usage. so think about what the box will actually be doing in real life, not just random big numbers 🌸

how you generate those flows matters SO much 💅 the sizes of the source and destination IP address ranges and the port number ranges are essential parameters, and you have to report them together with the results. see Section 6 for the reporting format, obviously 🫶

if you're using a single IP address pair, then it is RECOMMENDED to use:

• a fixed, larger source port number range (e.g., a few times 10,000) and
• a variable-size destination port number range (e.g., 10, 100, 1,000, etc.), where its expedient granularity depends on the purpose.

5.2. Scalability Against the Number of CPU Cores 🧚

stateful NATxy gateways are often software running on regular commodity servers, not special hardware, which is so iconic and also means comparing them can be kinda tricky 🙈 it can be useful to figure out:

• the performance of the various implementations using a single core of a well-known CPU and
• the scale-up of the performance of the various implementations with the number of CPU cores.

if the number of available CPU cores is a power of two, then it is RECOMMENDED to run the tests with 1, 2, 4, 8, 16, etc. active CPU cores on the DUT. the doubling pattern just makes the scale-up curve way easier to read, periodt 📈💜

6. Reporting Format 📝

okay this part is the reporting rules, so listen up bestie 👑

measurements MUST be executed multiple times. how many repetitions you need to get statistically reliable results depends on whether your results are consistent or all over the place 🌈 the report MUST contain the number of repetitions you actually ran. the median is RECOMMENDED as the summary function for your results, paired with the first percentile and the 99th percentile as the dispersion indices. the average and standard deviation MAY also be reported, totally your call ✨

all parameters and settings that may influence the performance of the DUT MUST be reported 💅 some are specific to the gateway implementation, like the hashsize (hash table size) and nf_conntrack_max (number of connection tracking table entries) values for iptables, or the limit of the number of states for OpenBSD PF (set by the set limit states number command in the pf.conf file).

here's the example table for iptables, which is giving "how to lay out your results" energy 🦋

   +----------------------------+--------+--------+--------+--------+
   | number of sessions (req.)  |   0.4M |     4M |    40M |   400M |
   +----------------------------+--------+--------+--------+--------+
   | source port numbers (req.) | 40,000 | 40,000 | 40,000 | 40,000 |
   +----------------------------+--------+--------+--------+--------+
   | destination port numbers   |     10 |    100 |  1,000 | 10,000 |
   | (req.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+
   | "hashsize" (i.s.)          |   2^17 |   2^20 |   2^23 |   2^27 |
   +----------------------------+--------+--------+--------+--------+
   | "nf_conntrack_max" (i.s.)  |   2^20 |   2^23 |   2^26 |   2^30 |
   +----------------------------+--------+--------+--------+--------+
   | num. sessions / "hashsize" |   3.05 |   3.81 |   4.77 |   2.98 |
   | (i.s.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+
   | number of experiments      |     10 |     10 |     10 |     10 |
   | (req.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+
   | error of binary search     |  1,000 |  1,000 |  1,000 |  1,000 |
   | (req.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+
   | connections/s median       |        |        |        |        |
   | (req.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+
   | connections/s 1st perc.    |        |        |        |        |
   | (req.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+
   | connections/s 99th perc.   |        |        |        |        |
   | (req.)                     |        |        |        |        |
   +----------------------------+--------+--------+--------+--------+

     Table 1: Example Table of the Maximum Connection Establishment
            Rate of Iptables Against the Number of Sessions

so Table 1 shows example headings for reporting the scalability of the iptables stateful NAT44 implementation against the number of sessions 📊 the table marks the required fields (req.) and the implementation-specific ones (i.s.). row 6 has a computed value too, the number of sessions per hashsize ratio, which helps you read the achieved maximum connection establishment rate 🧠 (a lower value means shorter linked lists hanging off the hash table entries, so higher performance, lowkey obvious once you see it). the ratio jumps around because the number of sessions is always a power of 10, while the hash table size is a power of 2 🙃

to show how accurate the results are, the table includes the "error" of the binary search, which is the stopping criterion. the binary search stops when the difference between the "higher limit" and "lower limit" is less than or equal to the "error" ✨

the table MUST be complemented with the relevant parameters of the DUT 💖 if the DUT is a general-purpose computer running some software NATxy gateway, then the hardware description SHOULD include the following:

• computer type
• CPU type
• number of active CPU cores
• memory type, size, and speed
• network interface card type (also reflecting the speed)
• the fact that direct cable connections were used, or the type of switch used for interconnecting the Tester and the DUT

the operating system type and version, kernel version, and version of the NATxy gateway implementation (including the last commit date and number if applicable) SHOULD also be given 🫶

7. Implementation and Experience 🌈

the stateful extension of siitperf [SIITPERF] is an implementation of this concept 🎀 its first version, which only supports multiple port numbers, is documented in this (open access) paper: [LEN2022]. its extended version, which also supports multiple IP addresses, is documented in this (open access) paper: [LEN2024b].

the proposed benchmarking methodology was validated by running benchmarking measurements with three radically different stateful NAT64 implementations (Jool, tayga+iptables, and OpenBSD PF) in this (open access) paper: [LEN2023] 🔥

further experience with using siitperf to measure the scalability of the iptables stateful NAT44 and Jool stateful NAT64 implementations is described in [SCALABILITY] 🦋

this methodology was successfully applied to benchmark various IPv4-as-a-Service (IPv4aas) technologies without technology-specific Testers, by reducing the aggregate of their Customer Edge (CE) and Provider Edge (PE) devices to a stateful NAT44 gateway. that's documented in this (open access) paper: [LEN2024a] 👑

8. Limitations of Using UDP as a Transport Layer Protocol 🙈

the test frame format defined in [RFC2544] exclusively uses UDP (and not TCP) as the transport layer protocol 🍓 UDP was kept in both [RFC5180] and [RFC8219] for the standard benchmarking procedures (throughput, latency, frame loss rate, etc.), so this document follows that long-established tradition too 💜

the rationale: the standard benchmarking procedures require sending frames at arbitrary constant frame rates, which would violate the flow control and congestion control algorithms of TCP. and TCP connection setup (the three-way handshake) would make testing even more complicated 😭

further transport layer protocols, like the Datagram Congestion Control Protocol (DCCP) [RFC4340] and the Stream Control Transmission Protocol (SCTP) [RFC9260], are outside the scope of this document, because the widely used stateful NAT44 and stateful NAT64 implementations don't support them 💅 QUIC [RFC9000] is also considered a transport layer protocol, but QUIC packets are carried in UDP datagrams, so QUIC doesn't need special handling ✨

some stateful NATxy solutions handle TCP and UDP differently. for example, iptables uses a 30s timeout for UDP and a 60s timeout for TCP 🕰️ so benchmarking results produced using UDP don't necessarily characterize the performance of a NATxy gateway well enough when it's forwarding Internet traffic. as for this example, the timeout values of the DUT may be adjusted, but that requires extra consideration 🧐

other differences in handling UDP or TCP are also possible, so the authors recommend that further investigations should be performed in this field 🔍

as a mitigation for this problem, this document recommends that testing with protocols using TCP (like HTTP and HTTPS up to version 2) can be performed as described in [RFC9411] 🌸 this approach also solves the potential problem of protocol helpers that may be present in the stateful DUT, bestie, which is such a relief 🥹

as for HTTP/3, it uses QUIC, which uses UDP as stated above. it should be noted that QUIC is treated as any other UDP payload 🫧 the proposed measurement method does not aim to measure the performance of QUIC; it aims to measure the performance of the stateful NATxy gateway 🎯

9. IANA Considerations 🎀

this document has no IANA actions 💅

10. Security Considerations 🔒

this document has no further security considerations beyond that of [RFC8219] 🚨 they should be cited here so that they can be applied not only for the benchmarking of IPv6 transition technologies but also for the benchmarking of any stateful NATxy gateways (allowing for x=y, too) 😭

11. References ✨

11.1. Normative References ✨

okay bestie, these are the ones you actually need to read before anything else 📚 they're basically the foundation the whole spec sits on 🧱

[RFC1918]  Rekhter, Y., Moskowitz, B., Karrenberg, D., de Groot, G.
           J., and E. Lear, "Address Allocation for Private
           Internets", BCP 5, RFC 1918, DOI 10.17487/RFC1918,
           February 1996, <https://www.rfc-editor.org/info/rfc1918>.

[RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
           Requirement Levels", BCP 14, RFC 2119,
           DOI 10.17487/RFC2119, March 1997,
           <https://www.rfc-editor.org/info/rfc2119>.

[RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
           Network Interconnect Devices", RFC 2544,
           DOI 10.17487/RFC2544, March 1999,
           <https://www.rfc-editor.org/info/rfc2544>.

[RFC3022]  Srisuresh, P. and K. Egevang, "Traditional IP Network
           Address Translator (Traditional NAT)", RFC 3022,
           DOI 10.17487/RFC3022, January 2001,
           <https://www.rfc-editor.org/info/rfc3022>.

[RFC4340]  Kohler, E., Handley, M., and S. Floyd, "Datagram
           Congestion Control Protocol (DCCP)", RFC 4340,
           DOI 10.17487/RFC4340, March 2006,
           <https://www.rfc-editor.org/info/rfc4340>.

[RFC4814]  Newman, D. and T. Player, "Hash and Stuffing: Overlooked
           Factors in Network Device Benchmarking", RFC 4814,
           DOI 10.17487/RFC4814, March 2007,
           <https://www.rfc-editor.org/info/rfc4814>.

[RFC5180]  Popoviciu, C., Hamza, A., Van de Velde, G., and D.
           Dugatkin, "IPv6 Benchmarking Methodology for Network
           Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May
           2008, <https://www.rfc-editor.org/info/rfc5180>.

[RFC6146]  Bagnulo, M., Matthews, P., and I. van Beijnum, "Stateful
           NAT64: Network Address and Protocol Translation from IPv6
           Clients to IPv4 Servers", RFC 6146, DOI 10.17487/RFC6146,
           April 2011, <https://www.rfc-editor.org/info/rfc6146>.

[RFC7599]  Li, X., Bao, C., Dec, W., Ed., Troan, O., Matsushima, S.,
           and T. Murakami, "Mapping of Address and Port using
           Translation (MAP-T)", RFC 7599, DOI 10.17487/RFC7599, July
           2015, <https://www.rfc-editor.org/info/rfc7599>.

[RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
           2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
           May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8219]  Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking
           Methodology for IPv6 Transition Technologies", RFC 8219,
           DOI 10.17487/RFC8219, August 2017,
           <https://www.rfc-editor.org/info/rfc8219>.

[RFC9000]  Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
           Multiplexed and Secure Transport", RFC 9000,
           DOI 10.17487/RFC9000, May 2021,
           <https://www.rfc-editor.org/info/rfc9000>.

[RFC9260]  Stewart, R., Tüxen, M., and K. Nielsen, "Stream Control
           Transmission Protocol", RFC 9260, DOI 10.17487/RFC9260,
           June 2022, <https://www.rfc-editor.org/info/rfc9260>.

[RFC9411]  Balarajah, B., Rossenhoevel, C., and B. Monkman,
           "Benchmarking Methodology for Network Security Device
           Performance", RFC 9411, DOI 10.17487/RFC9411, March 2023,
           <https://www.rfc-editor.org/info/rfc9411>.

---

11.2. Informative References 🌸

these ones are the extra reading 🤓 super helpful for the backstory and the research behind everything, but you don't need them to get the core spec 💅

[DUST1964] Durstenfeld, R., "Algorithm 235: Random permutation",
           Communications of the ACM, vol. 7, no. 7, p. 420,
           DOI 10.1145/364520.364540, July 1964,
           <https://dl.acm.org/doi/pdf/10.1145/364520.364540>.

[IIR2020]  Kurahashi, T., Matsuzaki, Y., Sasaki, T., Saito, T., and
           F. Tsutsuji, "Periodic Observation Report: Internet Trends
           as Seen from IIJ Infrastructure - 2020", Internet
           Initiative Japan Inc., Internet Infrastructure Review,
           vol. 49, December 2020,
           <https://www.iij.ad.jp/en/dev/iir/pdf/
           iir_vol49_report_EN.pdf>.

[LEN2015]  Lencse, G., "Estimation of the Port Number Consumption of
           Web Browsing", IEICE Transactions on Communications, vol.
           E98-B, no. 8. pp. 1580-1588,
           DOI 10.1587/transcom.E98.B.1580, August 2015,
           <https://www.hit.bme.hu/~lencse/publications/
           e98-b_8_1580.pdf>.

[LEN2020]  Lencse, G., "Adding RFC 4814 Random Port Feature to
           Siitperf: Design, Implementation and Performance
           Estimation", International Journal of Advances in
           Telecommunications, Electrotechnics, Signals and Systems,
           vol 9, no 3, pp. 18-26., DOI 10.11601/ijates.v9i3.291,
           November 2020,
           <http://ijates.org/index.php/ijates/article/view/291>.

[LEN2022]  Lencse, G., "Design and Implementation of a Software
           Tester for Benchmarking Stateful NAT64xy Gateways: Theory
           and Practice of Extending Siitperf for Stateful Tests",
           Computer Communications, vol. 192, pp. 75-88,
           DOI 10.1016/j.comcom.2022.05.028, August 2022,
           <https://www.sciencedirect.com/science/article/pii/
           S0140366422001803>.

[LEN2023]  Lencse, G., Shima, K., and K. Cho, "Benchmarking
           methodology for stateful NAT64 gateways", Computer
           Communications, vol. 210, pp. 256-272,
           DOI 10.1016/j.comcom.2023.08.009, October 2023,
           <https://www.sciencedirect.com/science/article/pii/
           S0140366423002931>.

[LEN2024a] Lencse, G. and Á. Bazsó, "Benchmarking methodology for
           IPv4aaS technologies: Comparison of the scalability of the
           Jool implementation of 464XLAT and MAP-T", Computer
           Communications, vol. 219, pp. 243-258,
           DOI 10.1016/j.comcom.2024.03.007, April 2024,
           <https://www.sciencedirect.com/science/article/pii/
           S0140366424000999>.

[LEN2024b] Lencse, G., "Making stateless and stateful network
           performance measurements unbiased", Computer
           Communications, vol. 225, pp. 141-155,
           DOI 10.1016/j.comcom.2024.05.018, September 2024,
           <https://www.sciencedirect.com/science/article/abs/pii/
           S0140366424001993>.

[SCALABILITY]
           Lencse, G., "Scalability of IPv6 Transition Technologies
           for IPv4aaS", Work in Progress, Internet-Draft, draft-
           lencse-v6ops-transition-scalability-05, 14 October 2023,
           <https://datatracker.ietf.org/doc/html/draft-lencse-v6ops-
           transition-scalability-05>.

[SIITPERF] "Siitperf: An RFC 8219 compliant SIIT and stateful NAT64/
           NAT44 tester", commit 165cb7f, September 2023,
           <https://github.com/lencsegabor/siitperf>.

---

Acknowledgements 🫶

the authors gave a big shoutout to Al Morton, Sarah Banks, Edwin Cordeiro, Lukasz Bromirski, Sándor Répás, Tamás Hetényi, Timothy Winters, Eduard Vasilenko, Minh Ngoc Tran, Paolo Volpato, Zeqi Lai, and Bertalan Kovács for their comments 💜

they also thanked Warren Kumari, Michael Scharf, Alexey Melnikov, Robert Sparks, David Dong, Roman Danyliw, Erik Kline, Murray Kucherawy, Zaheduzzaman Sarker, and Éric Vyncke for their reviews and comments 🎀 the whole squad showed up 😭✨

and funding credit goes to the Japan Trust International Research Cooperation Program of the National Institute of Information and Communications Technology (NICT), Japan 🇯🇵🌈

---

Authors' Addresses 🦋

Gábor Lencse
Széchenyi István University
Győr
Egyetem tér 1.
H-9026
Hungary
Email: lencse@sze.hu


Keiichi Shima
SoftBank Corp.
1-7-1 Kaigan, Minato-ku, Tokyo
105-7529
Japan
Email: shima@wide.ad.jp
URI:   https://softbank.co.jp/