RFC 3511

Benchmarking Methodology for Firewall Performance

๐Ÿ“… published April 2003, from ietf

RFC 3511: Benchmarking Methodology for Firewall Performance ๐Ÿ›ก๏ธโœจ

okay bestie, this is RFC 3511, and it's all about how to benchmark firewall performance ๐Ÿ’… it's Informational, which means it shares info with the internet community and does not set an internet standard ๐ŸŒธ it's from the Network Working Group, dated April 2003, and the header lists four authors: B. Hickman (Spirent Communications), D. Newman (Network Test), S. Tadjudin (Spirent Communications), and T. Martin (GVNW Consulting Inc) ๐Ÿซถ the header doesn't list anything it obsoletes or updates, so it stands on its own ๐Ÿฆ‹

This is part 1 of 6, so more is coming, bestie ๐Ÿ’œ

Status of this Memo โœจ

This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited.

Copyright (C) The Internet Society (2003). All Rights Reserved.

Abstract ๐ŸŒˆ

This document talks about and defines a bunch of tests you can use to describe how well firewalls perform ๐Ÿ”ฅ on top of the tests themselves, it also explains exact formats for reporting the results ๐Ÿ“

This document comes from the Benchmarking Methodology Working Group (BMWG) of the Internet Engineering Task Force (IETF) ๐Ÿงš

Table of Contents ๐Ÿ“

   1. Introduction . . . . . . . . . . . . . . . . . . . . . . . .  2
   2. Requirements . . . . . . . . . . . . . . . . . . . . . . . .  2
   3. Scope  . . . . . . . . . . . . . . . . . . . . . . . . . . .  3
   4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . .  3
      4.1 Test Considerations. . . . . . . . . . . . . . . . . . .  4
      4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . .  4
      4.3 Test Traffic Requirements. . . . . . . . . . . . . . . .  5
      4.4 DUT/SUT Traffic Flows. . . . . . . . . . . . . . . . . .  5
      4.5 Multiple Client/Server Testing . . . . . . . . . . . . .  5
      4.6 Network Address Translation (NAT). . . . . . . . . . . .  6
      4.7 Rule Sets. . . . . . . . . . . . . . . . . . . . . . . .  6
      4.8 Web Caching. . . . . . . . . . . . . . . . . . . . . . .  6
      4.9 Authentication . . . . . . . . . . . . . . . . . . . . .  7
      4.10 TCP Stack Considerations. . . . . . . . . . . . . . . .  7
   5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . .  7
      5.1 IP throughput. . . . . . . . . . . . . . . . . . . . . .  7
      5.2 Concurrent TCP Connection Capacity . . . . . . . . . . .  9
      5.3 Maximum TCP Connection Establishment Rate. . . . . . . . 12
      5.4 Maximum TCP Connection Tear Down Rate. . . . . . . . . . 14
      5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 16
      5.6 HTTP Transfer Rate . . . . . . . . . . . . . . . . . . . 18
      5.7 Maximum HTTP Transaction Rate. . . . . . . . . . . . . . 21
      5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 23
      5.9 IP Fragmentation Handling. . . . . . . . . . . . . . . . 24
      5.10 Latency . . . . . . . . . . . . . . . . . . . . . . . . 26
   6. References . . . . . . . . . . . . . . . . . . . . . . . . . 29
      6.1 Normative References . . . . . . . . . . . . . . . . . . 29
      6.2 Informative References . . . . . . . . . . . . . . . . . 30
   7. Security Consideration . . . . . . . . . . . . . . . . . . . 30
   Appendix A - HyperText Transfer Protocol (HTTP) . . . . . . . . 31
   Appendix B - Connection Establishment Time Measurements . . . . 31
   Appendix C - Connection Tear Down Time Measurements . . . . . . 32
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . 33
   Full Copyright Statement  . . . . . . . . . . . . . . . . . . . 34

1. Introduction ๐ŸŒธ

This document gives you methodologies for benchmarking firewall performance ๐Ÿ’– it covers four areas: forwarding, connection, latency and filtering ๐Ÿ”ฅ it also describes exact formats for reporting test results ๐Ÿ“Š

There's a previous document, "Benchmarking Terminology for Firewall Performance" [1], that defines a lot of the terms used here ๐Ÿ“š you SHOULD consult the terminology document before you try to use this one ๐Ÿค“

2. Requirements ๐Ÿ’œ

In this document, the words that define how important each requirement is get capitalized ๐ŸŽ€ they are:

  • "MUST" This word, or the words "REQUIRED" and "SHALL" mean that the item is an absolute requirement of the specification.
  • "SHOULD" This word or the adjective "RECOMMENDED" means that there may exist valid reasons in particular circumstances to ignore this item, but the full implications should be understood and the case carefully weighed before choosing a different course.
  • "MAY" This word or the adjective "OPTIONAL" means that this item is truly optional. One vendor may choose to include the item because a particular marketplace requires it or because it enhances the product, for example; another vendor may omit the same item.

An implementation is not compliant if it fails to satisfy one or more of the MUST requirements ๐Ÿ™…โ€โ™€๏ธ an implementation that satisfies all the MUST and all the SHOULD requirements is said to be "unconditionally compliant" โœจ one that satisfies all the MUST requirements but not all the SHOULD requirements is said to be "conditionally compliant" ๐ŸŒˆ

3. Scope ๐Ÿงš

Firewalls can control access between networks. Usually, a firewall protects a private network from public or shared network(s) it's connected to ๐Ÿฐ a firewall can be as simple as one device that filters packets, or as complex as a group of devices that combine packet filtering, application-level proxy and network translation services ๐Ÿ”ฅ this document focuses on benchmarking firewall performance, wherever possible, independent of implementation ๐Ÿ’…

4. Test Setup ๐ŸŽ€

The test configurations in this document will be confined to dual-homed and tri-homed, as shown in figure 1 and figure 2 respectively ๐Ÿฆ‹

Firewalls using dual-homed configurations connect two networks ๐ŸŒ one interface of the firewall is attached to the unprotected network [1], which is typically the public network (Internet) ๐ŸŒ the other interface connects to the protected network [1], which is typically the internal LAN ๐Ÿ 

In the case of dual-homed configurations, servers that are made accessible to the public (Unprotected) network are attached to the private (Protected) network ๐Ÿ™ˆ

   +----------+                                       +----------+
   |          |    |       +----------+        |      |          |
   | Servers/ |----|       |          |        |------| Servers/ |
   | Clients  |    |       |          |        |      | Clients  |
   |          |    |-------|  DUT/SUT |--------|      |          |
   +----------+    |       |          |        |      +----------+
        Protected  |       +----------+        | Unprotected
         Network   |                           |   Network
                       Figure 1 (Dual-Homed)

Tri-homed [1] configurations use a third segment called a Demilitarized Zone (DMZ) ๐Ÿ›ก๏ธ with tri-homed configurations, servers accessible to the public network are attached to the DMZ ๐Ÿงธ tri-homed configurations offer extra security by separating the server(s) that are accessible to the public network from internal hosts ๐Ÿ’œ

   +----------+                                       +----------+
   |          |    |       +----------+        |      |          |
   | Clients  |----|       |          |        |------| Servers/ |
   |          |    |       |          |        |      | Clients  |
   +----------+    |-------|  DUT/SUT |--------|      |          |
                   |       |          |        |      +----------+
                   |       +----------+        |
         Protected |            |              | Unprotected
          Network               |                   Network
                                |
                          -----------------
                                    |    DMZ
                                    |
                                    |
                             +-----------+
                             |           |
                             | Servers   |
                             |           |
                             +-----------+

                          Figure 2 (Tri-Homed)

4.1 Test Considerations ๐ŸŒท

This section is empty in the original document, so there's nothing to add here yet ๐Ÿซง

4.2 Virtual Clients/Servers ๐Ÿ‘ฏโ€โ™€๏ธ

Since firewall testing may involve data sources that emulate multiple users or hosts, the methodology uses the terms virtual clients/servers ๐Ÿ’– for these firewall tests, virtual clients/servers are application layer entities that may not be tied to a unique physical interface ๐Ÿฆ‹ for example, four virtual clients may come from the same data source [1] ๐Ÿคฏ the test report MUST indicate the number of virtual clients and virtual servers participating in the test ๐Ÿ“

4.3 Test Traffic Requirements ๐Ÿงพ

While a firewall's job is to enforce access control policies, the criteria used to define those policies vary depending on the implementation ๐Ÿ” firewalls may use network layer, transport layer or, in many cases, application-layer criteria to make access-control decisions ๐Ÿง 

For benchmarking firewall performance, this document uses HTTP 1.1 or higher as the application layer entity ๐ŸŒ the methodologies MAY be used as a template for benchmarking with other applications โœจ since testing may involve proxy-based DUT/SUTs, HTTP version considerations are discussed in appendix A ๐Ÿ“–

4.4 DUT/SUT Traffic Flows ๐ŸŒŠ

Since the number of interfaces is not fixed, the traffic flows depend on the configuration used when benchmarking the DUT/SUT ๐Ÿ”„ note that the term "traffic flows" is associated with client-to-server requests ๐Ÿ’…

For Dual-Homed configurations, there are two unique traffic flows:

   Client         Server
   ------         ------
   Protected   -> Unprotected
   Unprotected -> Protected

For Tri-Homed configurations, there are three unique traffic flows:

   Client         Server
   ------         ------
   Protected ->   Unprotected
   Protected ->   DMZ
   Unprotected -> DMZ

4.5 Multiple Client/Server Testing ๐ŸŽฏ

One or more clients may target multiple servers for a given application ๐ŸŽ€ each virtual client MUST initiate connections in a round-robin fashion ๐Ÿ” for example, if the test has six virtual clients targeting three servers, the pattern looks like this ๐Ÿ‘‘

   Client          Target Server (In order of request)
   #1              1     2     3     1...
   #2              2     3     1     2...
   #3              3     1     2     3...
   #4              1     2     3     1...
   #5              2     3     1     2...
   #6              3     1     2     3...

4.6 Network Address Translation (NAT) ๐Ÿ”€

Many firewalls implement network address translation (NAT) [1], a function that translates private internet addresses to public internet addresses ๐ŸŒ this adds extra processing for the DUT/SUT and may impact performance ๐Ÿ˜ฌ therefore, tests SHOULD be ran with NAT disabled and with NAT enabled, so you can figure out the performance differential, if any ๐Ÿ“Š the test report MUST indicate whether NAT was enabled or disabled ๐Ÿ’œ


Part 1 done, bestie ๐ŸŽ‰ next up in part 2: Section 4.7, Rule Sets ๐Ÿงšโœจ

4.7 Rule Sets

Okay bestie, rule sets are ๐Ÿ’… basically a collection of access control policies that decide which packets the DUT/SUT will forward and which it will reject [1]. The criteria you can use to write these policies depend on what the DUT/SUT can actually do, so this section only gives guidelines for setting up rule sets when you're benchmarking performance ๐ŸŽ€

It is RECOMMENDED that a rule be entered for each host (Virtual client). Testing SHOULD also be performed with different size rule sets to see how they affect the performance of the DUT/SUT ๐Ÿ“ˆ Rule sets MUST be configured so that rules tied to actual test traffic sit at the end of the set, not at the beginning ๐Ÿซถ

The DUT/SUT SHOULD be configured to deny access to all traffic that wasn't defined in the rule set beforehand ๐Ÿ”’ The test report SHOULD include the DUT/SUT configured rule set(s) โœจ

4.8 Web Caching

Some firewalls include caching agents to reduce network load ๐ŸŒธ When a request goes through a caching agent, the agent tries to answer from its own internal memory. The cache saves the responses it receives, like responses to HTTP GET requests ๐Ÿ’พ

Testing SHOULD be performed with any caching agents on the DUT/SUT disabled ๐Ÿ™…

4.9 Authentication

Access control may involve authentication processes like user, client or session authentication ๐Ÿ” Authentication is usually handled by devices outside the firewall itself, like authentication server(s), and it may add to the latency of the system ๐Ÿข

Any authentication processes MUST be included as part of the connection setup process ๐Ÿซถ

4.10 TCP Stack Considerations

Some test instruments let you configure one or more TCP stack parameters, which can influence the traffic flows that get offered and change the performance measurements ๐Ÿงš This document doesn't try to specify which TCP parameters should be configurable, but any such TCP parameter(s) MUST be noted in the test report ๐Ÿ“ Also, when comparing multiple DUT/SUTs, the same TCP parameters MUST be used ๐Ÿ‘‘


5. Benchmarking Tests

5.1 IP Throughput

5.1.1 Objective

Okay, this is the big one ๐Ÿ”ฅ The goal is to find the throughput of network-layer data going through the DUT/SUT, as defined in RFC 1242 [3]. Quick heads up: RFC 1242 talks about frames, which is a link-layer word, but this procedure says packets because we're working at the network layer ๐Ÿฆ‹

5.1.2 Setup Parameters

The following parameters MUST be defined:

  • Packet size - Number of bytes in the IP packet, exclusive of any link layer header or checksums.
  • Test Duration - Duration of the test, expressed in seconds.

5.1.3 Procedure

The test instrument MUST offer unicast IP packets to the DUT/SUT at a constant rate ๐ŸŽฏ The test MAY consist of either bi-directional or unidirectional traffic. For example, an emulated client may offer a unicast stream of packets to an emulated server, or the test instrument may simulate a client/server exchange by offering bidirectional traffic ๐Ÿ”

This test will employ an iterative search algorithm ๐Ÿ” Each iteration will involve the test instrument varying the intended load until the maximum rate, at which no packet loss occurs, is found. Backpressure mechanisms may be employed, which means the intended load and offered load can end up being different, so the test SHOULD be performed in either a packet based or time based manner as described in RFC 2889 [5]. As with RFC 1242, the term packet is used in place of frame. The duration of the test portion of each trial MUST be at least 30 seconds โฑ๏ธ

It is RECOMMENDED to perform the throughput measurements with different packet sizes ๐Ÿ“ When testing with different packet sizes, the DUT/SUT configuration MUST remain the same ๐Ÿ”’

5.1.4 Measurement

5.1.4.1 Network Layer

Throughput: Maximum offered load, expressed in either bits per second or packets per second, at which no packet loss is detected. The bits to be counted are in the IP packet (header plus payload). Other fields, such as link-layer headers and trailers, MUST NOT be included in the measurement ๐Ÿ™ˆ

Forwarding Rate: Forwarding rate, expressed in either bits per second or packets per second, is the rate the device is observed to successfully forward to the correct destination interface in response to a specified offered load. The bits to be counted are in the IP packet (header plus payload). Other fields, such as link-layer headers and trailers, MUST NOT be included in the measurement ๐Ÿ’–

5.1.5 Reporting Format

The test report MUST note the packet size(s), test duration, throughput and forwarding rate. The test report MUST also conform to the reporting requirements set in section 4, Test Setup ๐Ÿ“‹ If the test involved offering packets which target more than one segment (Protected, Unprotected or DMZ), the report MUST identify the results as an aggregate throughput measurement ๐ŸŒˆ

The throughput results SHOULD be reported in the format of a table with a row for each of the tested packet sizes. There SHOULD be columns for the packet size, the intended load, the offered load, resultant throughput and forwarding rate for each test ๐Ÿ“Š

The intermediate results of the search algorithm MAY be saved in a log file, which includes the packet size, test duration and, for each iteration:

  • Step Iteration
  • Pass/Fail Status
  • Total packets offered
  • Total packets forwarded
  • Intended load
  • Offered load (If applicable)
  • Forwarding rate

5.2 Concurrent TCP Connection Capacity

5.2.1 Objective

Now we're talking about connection tables ๐Ÿงš The goal here is to find the maximum number of concurrent TCP connections supported through or with the DUT/SUT, as defined in RFC 2647 [1]. This test is basically hunting for the maximum number of entries the DUT/SUT can store in its connection table ๐Ÿ‘‘

5.2.2 Setup Parameters

The following parameters MUST be defined for all tests:

5.2.2.1 Transport-Layer Setup Parameters

  • Connection Attempt Rate - The aggregate rate, expressed in connections per second, at which TCP connection requests are attempted. The rate SHOULD be set at or lower than the maximum rate at which the DUT/SUT can accept connection requests ๐Ÿšฆ
  • Aging Time - The time, expressed in seconds, the DUT/SUT will keep a connection in its connection table after receiving a TCP FIN or RST packet โณ

5.2.2.2 Application-Layer Setup Parameters

  • Validation Method - HTTP 1.1 or higher MUST be used for this test for both clients and servers. The client and server MUST use the same HTTP version ๐Ÿค
  • Object Size - Defines the number of bytes, excluding any bytes associated with the HTTP header, to be transferred in response to an HTTP 1.1 or higher GET request ๐Ÿ“ฆ

5.2.3 Procedure

This test will employ an iterative search algorithm to determine the maximum number of concurrent TCP connections supported through or with the DUT/SUT ๐Ÿ”

For each iteration, the aggregate number of concurrent TCP connections attempted by the virtual client(s) will be varied. The destination address will be that of the server or that of the NAT proxy. The aggregate rate will be defined by connection attempt rate, and will be attempted in a round-robin fashion (See 4.5) ๐Ÿ”„

To validate all connections, the virtual client(s) MUST request an object using an HTTP 1.1 or higher GET request. The requests MUST be initiated on each connection after all of the TCP connections have been established โœ…

When testing proxy-based DUT/SUTs, the virtual client(s) MUST request two objects using HTTP 1.1 or higher GET requests. The first GET request is required for connection time establishment [1] measurements as specified in appendix B. The second request is used for validation as previously mentioned. When comparing proxy and non-proxy based DUT/SUTs, the test MUST be performed in the same manner ๐Ÿชž

Between each iteration, it is RECOMMENDED that the test instrument issue a TCP RST referencing each connection attempted for the previous iteration, regardless of whether or not the connection attempt was successful ๐Ÿงน The test instrument will wait for aging time before continuing to the next iteration โŒ›

5.2.4 Measurements

5.2.4.1 Application-Layer Measurements

  • Number of objects requested
  • Number of objects returned

5.2.4.2 Transport-Layer Measurements

  • Maximum concurrent connections - Total number of TCP connections open for the last successful iteration performed in the search algorithm ๐Ÿ†
  • Minimum connection establishment time - Lowest TCP connection establishment time measured, as defined in appendix B ๐Ÿฃ
  • Maximum connection establishment time - Highest TCP connection establishment time measured, as defined in appendix B ๐Ÿข
  • Average connection establishment time - The mean of all measurements of connection establishment times ๐Ÿงฎ
  • Aggregate connection establishment time - The total of all measurements of connection establishment times โž•

5.2.5 Reporting Format

The test report MUST conform to the reporting requirements set in section 4, Test Setup ๐Ÿ“‹

5.2.5.1 Application-Layer Reporting

The test report MUST note the object size, number of completed requests and number of completed responses ๐Ÿ“

The intermediate results of the search algorithm MAY be reported in a tabular format with a column for each iteration. There SHOULD be rows for the number of requests attempted, number and percentage requests completed, number of responses attempted, and number and percentage of responses completed. The table MAY be combined with the transport-layer reporting, provided that the table identifies this as an application layer measurement ๐Ÿ—‚๏ธ

Version information: The test report MUST note the version of HTTP client(s) and server(s) ๐Ÿ”–

5.2.5.2 Transport-Layer Reporting

The test report MUST note the connection attempt rate, aging time, minimum TCP connection establishment time, maximum TCP connection establishment time, average connection establishment time, aggregate connection establishment time and maximum concurrent connections measured ๐Ÿ“Š

The intermediate results of the search algorithm MAY be reported in the format of a table with a column for each iteration. There SHOULD be rows for the total number of TCP connections attempted, number and percentage of TCP connections completed, minimum TCP connection establishment time, maximum TCP connection establishment time, average connection establishment time and the aggregate connection establishment time ๐ŸŒธ

5.3 Maximum TCP Connection Establishment Rate

5.3.1 Objective

This one's the speed test ๐ŸŽ๏ธ The goal is to find the maximum TCP connection establishment rate through or with the DUT/SUT, as defined by RFC 2647 [1]. This test is intended to find the maximum rate the DUT/SUT can update its connection table ๐Ÿ’จ

5.3.2 Setup Parameters

The following parameters MUST be defined for all tests:

5.3.2.1 Transport-Layer Setup Parameters

  • Number of Connections - Defines the aggregate number of TCP connections that must be established ๐Ÿ”ข
  • Aging Time - The time, expressed in seconds, the DUT/SUT will keep a connection in it's state table after receiving a TCP FIN or RST packet โณ

5.3.2.2 Application-Layer Setup Parameters

Validation Method: okay so for this one, HTTP 1.1 or higher MUST be used for this test, for both clients and servers ๐Ÿ’–. The client and server MUST use the same HTTP version, no mixing and matching ๐Ÿซถ.

Object Size: this is the number of bytes sent back in response to an HTTP 1.1 or higher GET request. Any bytes that belong to the HTTP header don't count toward it ๐ŸŽ€.

5.3.3 Procedure

Okay, so this test runs an iterative search algorithm to figure out the maximum rate at which the DUT/SUT can accept TCP connection requests โœจ. For each iteration, the aggregate rate of TCP connection requests from the virtual client(s) gets varied ๐Ÿฆ‹. The destination address is either the server or the NAT proxy ๐Ÿงš. The aggregate number of connections, which is defined by number of connections, gets attempted in a round-robin fashion (see 4.5) ๐Ÿ”.

The same application-layer object transfers that are needed for validation and establishment time measurements, as described in the concurrent TCP connection capacity test, MUST be performed ๐Ÿ’….

Between each iteration, it is RECOMMENDED that the test instrument send a TCP RST referencing each connection attempted in the previous iteration, regardless of whether that connection attempt worked ๐Ÿ™ˆ. Then it waits for aging time before moving on to the next iteration โณ.

5.3.4 Measurements

5.3.4.1 Application-Layer measurements

  • Number of objects requested
  • Number of objects returned

5.3.4.2 Transport-Layer measurements

  • Highest connection rate: the highest rate, in connections per second, for which all connections successfully opened in the search algorithm ๐Ÿ”ฅ.
  • Minimum connection establishment time: the lowest TCP connection establishment time measured, as defined in appendix B ๐ŸŒธ.
  • Maximum connection establishment time: the highest TCP connection establishment time measured, as defined in appendix B ๐ŸŒˆ.
  • Average connection establishment time: the mean of all measurements of connection establishment times ๐Ÿ’œ.
  • Aggregate connection establishment time: the total of all measurements of connection establishment times ๐Ÿงฎ.

5.3.5 Reporting Format

The test report MUST conform to the reporting requirements set in section 4, Test Setup ๐Ÿ“‹.

5.3.5.1 Application-Layer Reporting:

The test report MUST note the object size(s), the number of completed requests, and the number of completed responses ๐Ÿ“.

The intermediate results of the search algorithm MAY be reported in a tabular format with a column for each iteration. There SHOULD be rows for the number of requests attempted, the number and percentage of requests completed, the number of responses attempted, and the number and percentage of responses completed ๐Ÿ“Š. The table MAY be combined with the transport-layer reporting, provided that the table identifies this as an application layer measurement ๐Ÿซถ.

  • Version information: the test report MUST note the version of the HTTP client(s) and server(s) ๐Ÿท๏ธ.

5.3.5.2 Transport-Layer Reporting:

The test report MUST note the number of connections, aging time, minimum TCP connection establishment time, maximum TCP connection establishment time, average connection establishment time, aggregate connection establishment time, and highest connection rate measured ๐Ÿ’….

The intermediate results of the search algorithm MAY be reported in the format of a table with a column for each iteration. There SHOULD be rows for the connection attempt rate, total number of TCP connections attempted, total number of TCP connections completed, minimum TCP connection establishment time, maximum TCP connection establishment time, average connection establishment time, and aggregate connection establishment time ๐ŸŽ€.

5.4 Maximum TCP Connection Tear Down Rate

5.4.1 Objective

To determine the maximum TCP connection tear down rate through or with the DUT/SUT, as defined by RFC 2647 [1] ๐Ÿฆ‹.

5.4.2 Setup Parameters

  • Number of Connections: defines the number of TCP connections that will be attempted to be torn down ๐ŸŒท.
  • Aging Time: the time, expressed in seconds, that the DUT/SUT will keep a connection in its state table after receiving a TCP FIN or RST packet โณ.
  • Close Method: defines the method for closing TCP connections. The test MUST be performed with either a three-way or four-way handshake. In a four-way handshake, each side sends separate FIN and ACK messages. In a three-way handshake, one side sends a combined FIN/ACK message upon receipt of a FIN ๐Ÿค.
  • Close Direction: defines whether closing of connections is to be initiated from the client or from the server ๐Ÿงญ.

5.4.3 Procedure

This test runs an iterative search algorithm to find the maximum TCP connection tear down rate supported by the DUT/SUT โœจ. It goes through different TCP connection tear down rates with a fixed number of TCP connections ๐Ÿ”.

For proxy based DUT/SUTs, the DUT/SUT will itself receive the ACK in response to issuing a FIN packet to close its side of the TCP connection ๐Ÿงš. For validation purposes, the virtual client or server, whichever is applicable, MAY verify that the DUT/SUT received the final ACK by re-transmitting the final ACK. A TCP RST should be received in response to the retransmitted ACK ๐Ÿ’….

Between each iteration, it is RECOMMENDED that the virtual client(s) or server(s), whichever is applicable, issue a TCP RST referencing each connection that was attempted to be torn down, regardless of whether that tear down attempt worked ๐Ÿ™ˆ. The test waits for aging time before continuing to the next iteration โณ.

5.4.4 Measurements

  • Highest connection tear down rate: the highest rate, in connections per second, for which all TCP connections were successfully torn down in the search algorithm ๐Ÿ”ฅ.

The following tear down time [1] measurements MUST only include connections for which both sides of the connection were successfully torn down. For example, tear down times for connections which are left in a FINWAIT-2 [8] state should not be included ๐Ÿ™…โ€โ™€๏ธ:

  • Minimum connection tear down time: the lowest TCP connection tear down time measured, as defined in appendix C ๐ŸŒธ.
  • Maximum connection tear down time: the highest TCP connection tear down time measured, as defined in appendix C ๐ŸŒˆ.
  • Average connection tear down time: the mean of all measurements of connection tear down times ๐Ÿ’œ.
  • Aggregate connection tear down time: the total of all measurements of connection tear down times ๐Ÿงฎ.

5.4.5 Reporting Format

The test report MUST note the number of connections, aging time, close method, close direction, minimum TCP connection tear down time, maximum TCP connection tear down time, average TCP connection tear down time, aggregate TCP connection tear down time, and highest connection tear down rate measured. In addition, the test report MUST conform to the reporting requirements set in section 4, Test Setup ๐Ÿ“‹.

The intermediate results of the search algorithm MAY be reported in the format of a table with a column for each iteration. There SHOULD be rows for the number of TCP tear downs attempted, the number and percentage of TCP connection tear downs completed, minimum TCP connection tear down time, maximum TCP connection tear down time, average TCP connection tear down time, aggregate TCP connection tear down time, and validation failures, if required ๐Ÿ“Š.

5.5 Denial Of Service Handling

5.5.1 Objective

To determine the effect of a denial of service attack on a DUT/SUT TCP connection establishment and/or HTTP transfer rates ๐Ÿ˜ญ. The denial of service handling test MUST be run after obtaining baseline measurements from sections 5.3 and/or 5.6 ๐ŸŽฏ.

The TCP SYN flood attack exploits TCP's three-way handshake mechanism by having an attacking source host generate TCP SYN packets with random source addresses towards a victim host, thereby consuming that host's resources ๐Ÿ’€.

5.5.2 Setup Parameters

Use the same setup parameters as defined in section 5.3.2 or 5.6.2, depending on whether testing against the baseline TCP connection establishment rate test or HTTP transfer rate test, respectfully ๐Ÿซถ.

In addition, the following setup parameters MUST be defined:

  • SYN attack rate: the rate, expressed in packets per second, at which the server(s) or NAT proxy address is targeted with TCP SYN packets ๐Ÿ”ฅ.

5.5.3 Procedure

Use the same procedure as defined in section 5.3.3 or 5.6.3, depending on whether testing against the baseline TCP connection establishment rate or HTTP transfer rate test, respectfully. In addition, the test instrument will generate TCP SYN packets targeting the server(s) IP address or NAT proxy address at a rate defined by SYN attack rate ๐Ÿงš.

The test instrument originating the TCP SYN attack MUST be attached to the unprotected network. In addition, the test instrument MUST not respond to the SYN/ACK packets sent by the target server or NAT proxy in response to the SYN packet ๐Ÿ™…โ€โ™€๏ธ.

Some firewalls employ mechanisms to guard against SYN attacks. If such mechanisms exist on the DUT/SUT, tests SHOULD be run with these mechanisms enabled and disabled, to determine how well the DUT/SUT can maintain, under such attacks, the baseline connection establishment rates and HTTP transfer rates determined in section 5.3 and section 5.6, respectively ๐Ÿ›ก๏ธ.

5.5.4 Measurements

Perform the same measurements as defined in section 5.3.4 or 5.6.4, depending on whether testing against the baseline TCP connection establishment rate test or HTTP transfer rate, respectfully ๐Ÿ“.

In addition, the test instrument SHOULD track TCP SYN packets associated with the SYN attack which the DUT/SUT forwards on the protected or DMZ interface(s) ๐Ÿ‘€.

5.5.5 Reporting Format

The test SHOULD use the same reporting format as described in section 5.3.5 or 5.6.5, depending on whether testing against the baseline TCP connection establishment rate test or HTTP transfer rate, respectfully ๐Ÿ“‹.

In addition, the report MUST indicate a denial of service handling test, the SYN attack rate, the number of TCP SYN attack packets transmitted, and the number of TCP SYN attack packets forwarded by the DUT/SUT. The report MUST indicate whether or not the DUT has any SYN attack mechanisms enabled ๐Ÿšจ.

5.6 HTTP Transfer Rate

5.6.1 Objective

To determine the transfer rate of HTTP requested objects traversing the DUT/SUT ๐Ÿฆ‹.

5.6.2 Setup Parameters

The following parameters MUST be defined for all tests:

5.6.2.1 Transport-Layer Setup Parameters

  • Number of connections: defines the aggregate number of connections attempted. The number SHOULD be a multiple of the number of virtual clients participating in the test ๐ŸŒธ.
  • Close Method: defines the method for closing TCP connections. The test MUST be performed with either a three-way or four-way handshake. In a four-way handshake, each side sends separate FIN and ACK messages. In a three-way handshake, one side sends a combined FIN/ACK message upon receipt of a FIN ๐Ÿค.
  • Close Direction: defines whether closing of connections is to be initiated from the client or from the server ๐Ÿงญ.

5.6.2.2 Application-Layer Setup Parameters โœจ

Session Type: okay bestie, the virtual clients/servers MUST use HTTP 1.1 or higher ๐ŸŒธ and the client and server MUST use the same HTTP version. no mixing versions, that's a whole vibe clash ๐Ÿ’…

GET requests per connection: how many HTTP 1.1 or higher GET requests get attempted on each connection ๐Ÿฆ‹

Object Size: the number of bytes sent back in response to an HTTP 1.1 or higher GET request. the HTTP header bytes are NOT counted here, so only the object itself ๐ŸŽ€

5.6.3 Procedure ๐Ÿงš

each HTTP 1.1 or higher virtual client is gonna request one or more objects from an HTTP 1.1 or higher server, using one or more HTTP GET requests on each connection ๐Ÿ’– the total number of connections attempted (that's the number of connections parameter) MUST be split evenly across all of the participating virtual clients. fair is fair ๐Ÿ‘‘

if the virtual client(s) make multiple HTTP GET requests on the same connection, they MUST ask for the same object size for each GET request ๐Ÿซถ you can run this test more than once with objects of different sizes, though ๐ŸŒˆ

5.6.4 Measurements โญ

5.6.4.1 Application-Layer measurements ๐Ÿ”ฅ

Average Transfer Rate : the average transfer rate of the DUT/SUT MUST be measured, and it's tied to the requested object(s) ๐Ÿง the clock starts when the first bit of the first requested object gets sent and stops when the last bit of the last requested object gets sent. the average transfer rate (in bits per second) is worked out with this formula:

                             OBJECTS * OBJECTSIZE * 8
   TRANSFER RATE (bit/s) =  --------------------------
                                    DURATION

here's what the letters mean ๐Ÿ’œ

OBJECTS    - Total number of objects successfully transferred across
             all connections.

OBJECTSIZE - Object size in bytes

DURATION   - Aggregate transfer time based on aforementioned time
             references.

5.6.4.2 Measurements at or below the Transport-Layer ๐Ÿฆ‹

the following measurements SHOULD be performed for each connection-oriented protocol ๐ŸŒธ

Goodput [1]: goodput is the one defined in section 3.17 of RFC 2647 ๐ŸŽ€ the measurements MUST only count the protocol payload and skip the protocol header. the test instrument MUST also leave out any bits tied to connection establishment, connection tear down, security associations [1] or connection maintenance [1]. no sneaking those in ๐Ÿ™ˆ

since connection-oriented protocols need data to be acknowledged, the offered load [4] is gonna keep changing ๐Ÿ“ˆ so the test instrument should measure the average forwarding rate across the whole test. the measurement should start when the first bit of the payload of the first datagram gets sent, and end when the last bit of the payload of the last datagram gets sent. โœจ

Number of bytes transferred: total payload bytes transferred ๐Ÿ’…

Number of Timeouts: total number of timeout events โฐ

Retransmitted bytes: total number of retransmitted bytes ๐Ÿ”

5.6.5 Reporting Format ๐Ÿ“

the test report MUST follow the reporting requirements from section 4, Test Setup ๐Ÿ’–

5.6.5.1 Application-Layer reporting ๐ŸŒˆ

the test report MUST note the number of GET requests per connection and the object size(s) ๐ŸŽ€

the transfer rate results SHOULD be shown in a table with a column for each object size tested. there SHOULD also be a row for the number and percentage of completed requests, the number and percentage of completed responses, and the resultant transfer rate for each iteration of the test. very organised, very iconic ๐Ÿซถ

Failure analysis: the test report SHOULD show the number and percentage of HTTP GET requests and responses that failed to complete ๐Ÿ’€

Version information: the test report MUST note the version of the HTTP client(s) and server(s) ๐Ÿ”ฅ

5.6.5.2 Transport-Layer and below reporting ๐Ÿงš

the test report MUST note the number of connections, the close method, the close direction, and the protocol the measurement was made for ๐Ÿ‘‘

the results SHOULD be shown in a table for each HTTP object size tested. there SHOULD be a row for total bytes transferred, total timeouts, total retransmitted bytes, and the resultant goodput ๐Ÿฆ‹ quick reminder: total bytes means total datagram payload bytes transferred ๐Ÿ“ the table MAY be combined with the application-layer reporting, as long as it clearly says which protocol the measurement was made for ๐ŸŒธ

Failure analysis: the test report SHOULD show the number and percentage of connection establishment failures, plus the number and percentage of TCP tear down failures ๐Ÿ˜ญ

it is RECOMMENDED that the report include a graph plotting the distribution of both connection establishment failures and connection tear down failures ๐Ÿ“Š the x coordinate SHOULD be the elapsed test time, and the y coordinate SHOULD be the number of failures for a given sampling period. the graph SHOULD have two lines, one for connection failures and one for tear down failures. and the graph MUST note the sampling period ๐ŸŒˆโœจ


5.7 Maximum HTTP Transaction Rate ๐Ÿš€

5.7.1 Objective ๐Ÿ’–

figure out the maximum transaction rate the DUT/SUT can sustain ๐Ÿ† this test is all about finding the highest rate at which users can access objects. main character energy ๐Ÿ‘‘

5.7.2 Setup Parameters ๐ŸŽ€

5.7.2.1 Transport-Layer Setup Parameters ๐Ÿฆ‹

Close Method: this defines how TCP connections get closed. the test MUST be run with either a three-way or a four-way handshake ๐Ÿซถ in a four-way handshake, each side sends its own separate FIN and ACK messages. in a three-way handshake, one side sends a combined FIN/ACK message once it gets a FIN ๐Ÿง

Close Direction: whether closing the connections gets started by the client or by the server ๐ŸŒธ

5.7.2.2 Application-Layer Setup Parameters ๐Ÿ’…

Session Type: HTTP 1.1 or higher MUST be used for this test ๐Ÿ”ฅ and the client and server MUST use the same HTTP version ๐Ÿ’–

Test Duration: the amount of time, in seconds, that the virtual client(s) keep up the attempted GET request rate โฐ it is RECOMMENDED that the duration be at least 30 seconds ๐ŸŒˆ

Requests per connection: the number of object requests per connection ๐Ÿ“

Object Size: the number of bytes sent back in response to an HTTP 1.1 or higher GET request, excluding any bytes tied to the HTTP header ๐ŸŽ€

5.7.3 Procedure ๐Ÿงš

this test uses an iterative search algorithm to find the maximum transaction rate that the DUT/SUT can sustain ๐Ÿ”

on each iteration, the HTTP 1.1 or higher virtual client(s) change the aggregate GET request rate they offer to the HTTP 1.1 or higher server(s) โœจ and they keep that offered request rate going for the whole defined test duration ๐Ÿ’œ

if the virtual client(s) make multiple HTTP GET requests on the same connection, they MUST ask for the same object size for each GET request ๐Ÿซถ multiple tests MAY be run with different object sizes ๐ŸŒธ

5.7.4 Measurements ๐Ÿ“Š

Maximum Transaction Rate: the highest rate at which all transactions (aka all request/response cycles) get completed ๐Ÿ†

Transaction Time: the test instrument SHOULD measure the minimum, maximum and average transaction times โฑ๏ธ the clock starts when the virtual client issues the GET request, and stops when the requesting virtual client receives the last bit of the requested object ๐Ÿฆ‹

5.7.5 Reporting Format ๐Ÿ“

the test report MUST follow the reporting requirements from section 4, Test Setup ๐Ÿ’–

5.7.5.1 Application-Layer reporting ๐ŸŒˆ

the test report MUST note the test duration, object size, requests per connection, minimum transaction time, maximum transaction time, average transaction time, and the maximum transaction rate measured ๐ŸŽ€

the intermediate results of the search algorithm MAY be shown in a table with a column for each iteration. there SHOULD be rows for the GET request attempt rate, the number of requests attempted, the number and percentage of requests completed, the number of responses attempted, the number and percentage of responses completed, the minimum transaction time, the average transaction time, and the maximum transaction time ๐Ÿซถ

Version information: the test report MUST note the version of the HTTP client(s) and server(s) ๐Ÿ”ฅ

5.7.5.2 Transport-Layer ๐Ÿงš

the test report MUST note the close method, the close direction, the number of connections established, and the number of connections torn down ๐Ÿ‘‘

the intermediate results of the search algorithm MAY be shown in a table with a column for each iteration. there SHOULD be rows for the number of connections attempted, the number and percentage of connections completed, and the number and percentage of connection tear downs completed ๐Ÿฆ‹ the table MAY be combined with the application-layer reporting, as long as the table clearly labels it as a transport-layer measurement ๐ŸŒธ


5.8 Illegal Traffic Handling ๐Ÿ™ˆ

5.8.1 Objective ๐Ÿ’–

this one characterises how the DUT/SUT behaves when it's hit with a mix of legal and Illegal [1] traffic ๐ŸŽญ quick clarification bestie: Illegal traffic is NOT an attack. it's just traffic that's been explicitly defined by a rule(s) to drop ๐Ÿšซโœจ

5.8.2 Setup Parameters ๐ŸŽ€

the setup parameters are the same ones from the HTTP transfer rate test (Section 5.6.2) ๐Ÿฆ‹ plus, the following setup parameter MUST be defined:

Illegal traffic percentage: the percentage of HTTP 1.1 or higher connections that have been explicitly defined in a rule(s) to drop ๐Ÿ’€

5.8.3 Procedure ๐Ÿงš

each HTTP 1.1 or higher client is gonna request one or more objects from an HTTP 1.1 or higher server, using one or more HTTP GET requests on each connection ๐Ÿซถ the total number of connections attempted (the number of connections parameter) MUST be split evenly across all of the participating virtual clients โš–๏ธ

the virtual client(s) MUST offer the connection requests, both legal and illegal, in an evenly distributed manner ๐ŸŒˆ many firewalls can filter on different traffic criteria (IP addresses, port numbers, etc.) ๐Ÿ”ฅ multiple iterations of this test MAY be run with the DUT/SUT configured to filter on different traffic criteria ๐Ÿ’œ

5.8.4 Measurements ๐Ÿ“Š

the same measurements from the HTTP transfer rate test (Section 5.6.4) SHOULD be performed ๐Ÿ’… any forwarding rate measurements MUST only include bits that are associated with legal traffic ๐Ÿซถ

5.8.5 Reporting Format ๐Ÿ“

the reporting format SHOULD be the same as the HTTP transfer rate test (Section 5.6.5) ๐Ÿฆ‹

the report MUST also note the percentage of illegal HTTP connections ๐Ÿ“

Failure analysis: the test report MUST note the number and percentage of illegal connections that were allowed by the DUT/SUT ๐Ÿ˜ญ


5.9 IP Fragmentation Handling ๐Ÿงฉ

5.9.1 Objective ๐Ÿ’–

this test is about the performance hit when the DUT/SUT gets hit with IP fragmented traffic ๐Ÿงฉ IP packets get fragmented when they cross a network with a smaller MTU (Maximum Transmission Unit) than the actual IP packet, and the firewall may need to reassemble them before the rule set gets applied ๐Ÿ”ง

IP fragmentation is a common form of attack, either on the firewall itself or on internal hosts ๐Ÿ™…โ€โ™€๏ธ but this test focuses on how much the extra processing for reassembling the packets affects the DUT/SUT's forwarding rate ๐Ÿ“‰ and RFC 1858 covers some fragmentation attacks that sneak past IP filtering in routers and hosts ๐Ÿ›ก๏ธ

5.9.2 Setup Parameters ๐ŸŽ€

the following parameters MUST be defined. โœจ

5.9.2.1 Non-Fragmented Traffic Parameters โœจ

Okay, good news, bestie: the setup parameters here are exactly the same as the HTTP transfer rate test, so you already know them ๐Ÿ’– Just go back to Sections 5.6.2.1 and 5.6.2.2.

5.9.2.2 Fragmented Traffic Parameters ๐Ÿงฉ

Now for the fragmented part, which is where it gets spicy ๐ŸŒถ๏ธ

Packet size: The number of bytes in the IP/UDP packet, not counting link-layer headers and checksums, measured before fragmentation happens. ๐Ÿ“ฆ

MTU: The Maximum Transmission Unit, in bytes. For testing purposes, this MAY be configured to values smaller than the MTU that the link layer supports. ๐ŸŽ€

Intended Load: The intended load, written as a percentage of media utilization. ๐Ÿ“Š

5.9.3 Procedure ๐Ÿ’…

Each HTTP 1.1 or higher client will ask for one or more objects from an HTTP 1.1 or higher server using one or more HTTP GET requests over each connection. The aggregate number of connections attempted, which is defined by the number of connections, MUST be split evenly across all of the participating virtual clients. If a virtual client makes multiple HTTP GET requests per connection, it MUST request the same object size for each GET request. ๐Ÿฆ‹

A test instrument hooked up to the unprotected side of the network will offer a one-way stream of unicast fragmented IP/UDP traffic, aimed at a server attached to either the protected or DMZ segment. The test instrument MUST offer that unidirectional stream for the whole duration of the test, meaning the same duration the HTTP traffic is being offered. โฑ๏ธ

Baseline measurements SHOULD be taken with IP filtering deny rule(s) that filter out the fragmented traffic. If the DUT/SUT has logging capability, the log SHOULD be checked to see whether it has the correct information about the fragmented traffic. ๐Ÿ”

The test SHOULD be repeated after changing the DUT/SUT rule set so that it lets the fragmented traffic through. When you run multiple iterations of the test, it is RECOMMENDED to vary the MTU while keeping all the other parameters constant. ๐ŸŒˆ

After that, set the DUT/SUT to whatever policy or rule set the manufacturer says is required to protect against fragmentation attacks, then repeat the measurements from the baseline procedure. ๐Ÿ›ก๏ธ

5.9.4 Measurements ๐Ÿ“

The test instrument SHOULD perform the same measurements defined in the HTTP test (Section 5.6.4). โœจ

Transmitted UDP/IP Packets: The number of UDP packets the client transmitted. ๐Ÿ“ค

Received UDP/IP Packets: The number of UDP/IP packets the server received. ๐Ÿ“ฅ

5.9.5 Reporting Format ๐Ÿ“

5.9.5.1 Non-Fragmented Traffic ๐ŸŒธ

The test report SHOULD be the same as the one described in Section 5.6.5. Heads up: any forwarding rate measurements for the HTTP traffic exclude any bits that belong to fragmented traffic the DUT/SUT might forward. ๐Ÿ™ˆ

5.9.5.2 Fragmented Traffic ๐Ÿ”ฅ

The test report MUST note the packet size, the MTU size, the intended load, the number of UDP/IP packets transmitted, and the number of UDP/IP packets forwarded. The test report SHOULD also note whether or not the DUT/SUT forwarded the offered UDP/IP traffic in fragmented form. ๐Ÿงš


5.10 Latency โณ

5.10.1 Objective ๐ŸŽฏ

The goal here is to measure the latency of network-layer or application-layer data as it crosses the DUT/SUT. RFC 1242 [3] defines latency. ๐Ÿ’œ

5.10.2 Setup Parameters ๐Ÿง

The following parameters MUST be defined:

5.10.2.1 Network-layer Measurements ๐Ÿ›ฐ๏ธ

Packet size, written as the number of bytes in the IP packet, not counting link-layer headers and checksums. ๐Ÿ“ฆ

Intended load, written as a percentage of media utilization. ๐Ÿ“Š

Test duration, written in seconds. โฑ๏ธ

The test instruments MUST generate packets with unique timestamp signatures. โญ

5.10.2.2 Application-layer Measurements ๐Ÿ“

Object Size: The number of bytes, not counting any bytes that belong to the HTTP header, to be transferred in response to an HTTP 1.1 or higher GET request. The minimum object size supported by the media SHOULD be used, but other object sizes MAY be used as well. ๐ŸŽ€

Connection type: The test instrument MUST use one HTTP 1.1 or higher connection for latency measurements. ๐Ÿ”—

Number of objects requested. ๐Ÿ“จ

Number of objects transferred. ๐Ÿ“ฌ

Test duration, expressed in seconds. โฑ๏ธ

Test instruments MUST generate packets with unique timestamp signatures. โญ

5.10.3 Network-layer procedure ๐Ÿš€

A client will offer a one-way stream of unicast packets to a server. The packets MUST use a connectionless protocol like IP or UDP/IP. ๐Ÿ“ก

The test instrument MUST offer packets in a steady state. As the latency discussion in RFC 2544 [2] notes, latency measurements MUST be taken at the throughput level, meaning the highest offered load with zero packet loss. Only measurements taken at the throughput level can legitimately be called latency. ๐Ÿ‘‘

It is RECOMMENDED that implementers use offered loads not only at the throughput level, but also at load levels below and above the throughput level. To avoid confusion with existing terminology, measurements from such tests MUST be labeled as delay rather than latency. ๐Ÿซถ

It is RECOMMENDED to run the latency measurements with different packet sizes. When testing with different packet sizes, the DUT/SUT configuration MUST stay the same. ๐Ÿ”

If you want, a step test MAY be used, where offered loads go up or down through a range of load levels. ๐Ÿชœ

The duration of the test portion of each trial MUST be at least 30 seconds. ๐Ÿ•’

5.10.4 Application layer procedure ๐ŸŒ

An HTTP 1.1 or higher client will ask for one or more objects from an HTTP 1.1 or higher server using one or more HTTP GET requests. If the test instrument makes multiple HTTP GET requests, it MUST request the same-sized object each time. Multiple iterations of this test may be performed with objects of different sizes. ๐Ÿงฉ

Implementers MAY configure the test instrument to run for a fixed duration. In that case, the test instrument MUST report the number of objects requested and returned for the duration of the test. For fixed-duration tests, it is RECOMMENDED that the duration be at least 30 seconds. ๐Ÿ’ซ

5.10.5 Measurements ๐Ÿ“Š

Minimum delay: The smallest delay that data incurs while traversing the DUT/SUT, at the network layer or application layer as appropriate. ๐Ÿฃ

Maximum delay: The largest delay that data incurs while traversing the DUT/SUT, at the network layer or application layer as appropriate. ๐Ÿ˜

Average delay: The mean of all measured delays that data incurs while traversing the DUT/SUT, at the network layer or application layer as appropriate. ๐Ÿงฎ

Delay distribution: A set of histograms covering all delay measurements observed for data traversing the DUT/SUT, at the network layer or application layer as appropriate. ๐Ÿ“ˆ

5.10.6 Network-layer reporting format ๐Ÿ—‚๏ธ

The test report MUST note the packet size(s), offered load(s), and test duration used. On top of that, the test report MUST conform to the reporting requirements set in section 4, Test Setup. โœ…

The latency results SHOULD be reported in a table with one row for each tested packet size. There SHOULD be columns for the packet size, the intended rate, the offered rate, and the resulting latency or delay values for each test. ๐Ÿ’–

5.10.7 Application-layer reporting format ๐Ÿ’Œ

The test report MUST note the object size(s) and the number of requests and responses completed. If applicable, the report MUST note the test duration when a fixed duration was used. On top of that, the test report MUST conform to the reporting requirements set in section 4, Test Setup. ๐Ÿ“‹

The latency results SHOULD be reported in a table with one row for each object size. There SHOULD be columns for the object size, the number of completed requests, the number of completed responses, and the resulting latency or delay values for each test. ๐ŸŒท

Failure analysis: The test report SHOULD indicate the number and percentage of HTTP GET requests or responses that failed to complete within the test duration. ๐Ÿ˜ฌ

Version information: The test report MUST note the version of the HTTP client and server. ๐Ÿท๏ธ

6. References ๐Ÿ“š

6.1 Normative References ๐Ÿ”–

[1] Newman, D., "Benchmarking Terminology for Firewall Devices", RFC 2647, August 1999.

[2] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999.

[3] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991.

[4] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, February 1998.

[5] Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN Switching Devices", RFC 2889, August 2000.

6.2 Informative References ๐ŸŒŸ

[6] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol - HTTP/1.1", RFC 2616, June 1999.

[7] Clark, D., "IP Datagram Reassembly Algorithm", RFC 815, July 1982.

[8] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981.

7. Security Considerations ๐Ÿ›ก๏ธ

The main goal of this document is to provide methodologies for benchmarking firewall performance. There's some overlap between performance and security issues, but assessing firewall security is outside the scope of this document. ๐Ÿ”’

APPENDIX A: HTTP (HyperText Transfer Protocol) ๐ŸŒ

The most common versions of HTTP in use today are HTTP/1.0 and HTTP/1.1, and the big difference between them is persistent connections. HTTP 1.0, by default, does not support persistent connections. A separate TCP connection is opened for each GET request the client wants to start, and it gets closed once the requested object has been transferred. Some HTTP/1.0 implementations support persistence through a keep-alive, but there is no official specification for how keep-alive works. Also, HTTP 1.0 proxies do support persistent connections, since they don't recognize the connection header. ๐Ÿคทโ€โ™€๏ธ

HTTP/1.1, by default, does support persistent connections, which is why it's the version this methodology references. Proxy-based DUT/SUTs may monitor the TCP connection and, after a timeout, close the connection if no activity is detected. The duration of this timeout is not defined in the HTTP/1.1 specification and will vary between DUT/SUTs. If the DUT/SUT closes inactive connections, the aging timer on the DUT SHOULD be configured for a duration that exceeds the test time. โฐ

This document can't predict future changes to HTTP and how they might affect the methodologies defined here, but those changes should be accommodated so that newer versions of HTTP can be used when benchmarking firewall performance. ๐Ÿš€โœจ

APPENDIX B: Connection Establishment Time Measurements ๐ŸŽ€

okay so some connection oriented protocols, like TCP, need an odd number of messages to set up a connection ๐Ÿงš that's the vibe we're dealing with here. when there's a proxy based DUT/SUT, the DUT/SUT terminates the connection and opens a separate one to the server. the test instrument doesn't own both sides, so measurements get made two different ways ๐Ÿ’… the TCP examples below are just the illustration, but the method can be used with any other connection oriented protocol that has an odd number of messages โœจ

when testing non-proxy based DUT/SUTs, the establishment time shall be directly measured. it counts from the moment the first bit of the first SYN packet leaves the client, until the moment the last bit of the final ACK in the three-way handshake reaches the target server ๐Ÿฆ‹

if the DUT/SUT is proxy based, it's a bit more plot twist ๐Ÿ“ the connection establishment time runs from the first bit of the first SYN packet going out of the client, to the moment the client transmits the first bit of the first acknowledged TCP datagram. that's t4-t0 in the timeline below ๐Ÿ‘‘

   t0: Client sends a SYN.
   t1: Proxy sends a SYN/ACK.
   t2: Client sends the final ACK.
   t3: Proxy establishes separate connection with server.
   t4: Client sends TCP datagram to server.
   *t5: Proxy sends ACK of the datagram to client.

about that starred t5 ๐Ÿ’€ it's not part of the TCP connection establishment, BUT the acknowledgement of t4 must be received for the connection to be considered successful. so t5 isn't counted in the timing, yet it still matters for whether the whole thing counts as a success ๐Ÿ™ˆ

APPENDIX C: Connection Tear Down Time Measurements ๐ŸŒธ

so TCP connections are full duplex, right? but tearing them down happens in a simplex fashion. each host or device on each side of the TCP connection sends its own FIN segment ๐ŸŒˆ it's giving two separate hangups, one per direction ๐Ÿซถ

when we measure connection tear down times, we measure from the perspective of the entity that starts the teardown, meaning the virtual client or server that initiates the request โœจ the measurement is done the same way no matter whether the DUT/SUT is proxy-based or not, which keeps things consistent and so clean ๐Ÿ’œ the tear down time is the interval from the transmission of the first bit of the first TCP FIN packet sent by the virtual client or server (whichever applies) requesting the tear down, to the receipt of the last bit of the corresponding ACK packet on that same virtual client/server interface ๐Ÿ”ฅ

Authors' Addresses ๐Ÿงธ

Brooks Hickman Spirent Communications 26750 Agoura Road Calabasas, CA 91302 USA

Phone: + 1 818 676 2412 EMail: brooks.hickman@spirentcom.com

David Newman Network Test Inc. 31324 Via Colinas, Suite 113 Westlake Village, CA 91362-6761 USA

Phone: + 1 818 889-0011 EMail: dnewman@networktest.com

Saldju Tadjudin Spirent Communications 26750 Agoura Road Calabasas, CA 91302 USA

Phone: + 1 818 676 2468 EMail: Saldju.Tadjudin@spirentcom.com

Terry Martin GVNW Consulting Inc. 8050 SW Warm Springs Road Tualatin Or. 97062 USA

Phone: + 1 503 612 4422 EMail: tmartin@gvnw.com

the legal bits stay exactly as written, no vibe changes here ๐Ÿ™

Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement ๐Ÿซถ

funding for the RFC Editor function is currently provided by the Internet Society ๐Ÿ’–

๐Ÿ“ฆ where this page comes from