RFC 9347

Aggregation and Fragmentation Mode for Encapsulating Security Payload (ESP) and Its Use for IP Traffic Flow Security (IP-TFS)

📅 published January 2023, from ietf

Aggregation and Fragmentation Mode for Encapsulating Security Payload (ESP) and Its Use for IP Traffic Flow Security (IP-TFS) 🎀

Okay bestie, the title block is giving everything 💅. This is an IETF document, RFC 9347, written by C. Hopps at LabN Consulting, L.L.C. It's on the Standards Track and dated January 2023, with ISSN 2070-1721. The title block doesn't list anything it obsoletes or updates, so nothing gets replaced or amended here 🌸.

Abstract ✨

Okay so this document describes a way to aggregate and fragment IP packets when they get wrapped up in Encapsulating Security Payload (ESP) 🎁. This new payload type can do a bunch of things, like cutting down the encapsulation overhead on small IP packets. But the main goal here is to level up IP Traffic Flow Security (IP-TFS) by adding Traffic Flow Confidentiality (TFC) to encrypted, IP-encapsulated traffic 🔐. TFC works by hiding the size and frequency of IP traffic, using a fixed-size, constant-send-rate IPsec tunnel. The solution also supports congestion control and nonconstant send-rate usage, so it's not stuck in one lane 🦋.

Status of This Memo 📋

This is an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). If you want more info on Internet Standards, check out Section 2 of RFC 7841 🧚.

If you want the current status of this document, any errata, or a way to send feedback, head to https://www.rfc-editor.org/info/rfc9347 🔗.

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the date this document is published. Please read these carefully, because they spell out your rights and restrictions for this document 📜. Code Components pulled out of this document must include the Revised BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Revised BSD License ⚖️.

Table of Contents 🧚

   1.  Introduction
     1.1.  Terminology & Concepts
   2.  The AGGFRAG Tunnel
     2.1.  Tunnel Content
     2.2.  Payload Content
       2.2.1.  DataBlocks
       2.2.2.  End Padding
       2.2.3.  Fragmentation, Sequence Numbers, and All-Pad Payloads
       2.2.4.  Empty Payload
       2.2.5.  IP Header Value Mapping
       2.2.6.  IPv4 Time To Live (TTL), IPv6 Hop Limit, and ICMP
               Messages
       2.2.7.  Effective MTU of the Tunnel
     2.3.  Exclusive SA Use
     2.4.  Modes of Operation
       2.4.1.  Non-Congestion-Controlled Mode
       2.4.2.  Congestion-Controlled Mode
     2.5.  Summary of Receiver Processing
   3.  Congestion Information
     3.1.  ECN Support
   4.  Configuration of AGGFRAG Tunnels for IP-TFS
     4.1.  Bandwidth
     4.2.  Fixed Packet Size
     4.3.  Congestion Control
   5.  IKEv2
     5.1.  USE_AGGFRAG Notification Message
   6.  Packet and Data Formats
     6.1.  AGGFRAG_PAYLOAD Payload
       6.1.1.  Non-Congestion-Control AGGFRAG_PAYLOAD Payload Format
       6.1.2.  Congestion Control AGGFRAG_PAYLOAD Payload Format
       6.1.3.  Data Blocks
       6.1.4.  IKEv2 USE_AGGFRAG Notification Message
   7.  IANA Considerations
     7.1.  ESP Next Header Value
     7.2.  AGGFRAG_PAYLOAD Sub-Types
     7.3.  USE_AGGFRAG Notify Message Status Type
   8.  Security Considerations
   9.  References
     9.1.  Normative References
     9.2.  Informative References
   Appendix A.  Example of an Encapsulated IP Packet Flow
   Appendix B.  A Send and Loss Event Rate Calculation
   Appendix C.  Comparisons of IP-TFS
     C.1.  Comparing Overhead
       C.1.1.  IP-TFS Overhead
       C.1.2.  ESP with Padding Overhead
     C.2.  Overhead Comparison
     C.3.  Comparing Available Bandwidth
       C.3.1.  Ethernet
   Acknowledgements
   Contributors
   Author's Address

1. Introduction 💖

Okay so traffic analysis [RFC4301] [AppCrypt] is when someone extracts info about the data being sent across a network 👀. Even if you encrypt the data directly [RFC4303], the patterns in the message traffic can still leak stuff because of variations in its shape and timing [RFC8546] [AppCrypt]. Hiding the size and frequency of traffic is called Traffic Flow Confidentiality (TFC), per [RFC4303] 🙈.

[RFC4303] handles TFC by letting you add padding to encrypted IP packets, and by letting you send all-pad packets (marked with protocol 59). The catch is that this method can seriously underuse the available bandwidth 😭.

This document defines an aggregation and fragmentation (AGGFRAG) mode for ESP, plus ESP's use for IP Traffic Flow Security (IP-TFS) 🦋. This solution gives you full TFC without that bandwidth problem. It does this with a constant-send-rate IPsec [RFC4303] tunnel that uses fixed-size encapsulating packets. Those fixed-size packets can hold partial, whole, or multiple IP packets, so the tunnel's bandwidth gets used to the max 💅. A nonconstant send rate is allowed too, but the confidentiality properties of using one are outside the scope of this document 🫶.

If you want to compare the overhead of IP-TFS with the TFC solution prescribed in [RFC4303], check out Appendix C ✨.

IP-TFS also lets you operate fairly within congested networks [RFC2914] 👑. That matters when the IP-TFS user doesn't fully control the domain the IP-TFS tunnel path goes through 🌈.

The mechanisms defined in this document, like the AGGFRAG mode, are generic on purpose so they can support non-TFS uses. Those uses are outside the scope of this document though 🧚.

1.1. Terminology & Concepts 🌸

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here 📚.

This document assumes you're already familiar with IP security concepts, including TFC, as described in [RFC4301] 🔐.

2. The AGGFRAG Tunnel 🦋

As I mentioned in Section 1, the AGGFRAG mode uses an IPsec [RFC4303] tunnel as its transport 🚇. For IP-TFS, fixed-size encapsulating packets get sent at a constant rate on the AGGFRAG tunnel 💫.

The main input to the tunnel algorithm is the requested bandwidth the tunnel is supposed to use. From that, two values are needed to deliver that bandwidth: the fixed size of the encapsulating packets, and the rate at which to send them 📦.

The fixed packet size MAY be specified manually, or it MAY be figured out through other methods, like Packetization Layer MTU Discovery (PLMTUD) [RFC4821] [RFC8899] or Path MTU Discovery (PMTUD) [RFC1191] [RFC8201]. PMTUD has known issues, so PLMTUD is considered the more robust option 🛠️. For PLMTUD, congestion control payloads can be used as in-band probes (see Section 6.1.2 and [RFC8899]) 🔍.

Given the size of the encapsulating packet and the requested bandwidth, you can work out the matching packet send rate. The packet send rate is the requested bandwidth divided by the size of the encapsulating packet 🧮.

The egress (receiving) side of the AGGFRAG tunnel MUST allow for, and expect, the ingress (sending) side of the AGGFRAG tunnel to vary the size and rate of the encapsulating packets it sends, unless other policy constrains it 🫶.

2.1. Tunnel Content 🎀

As I said before, one issue with the TFC padding solution in [RFC4303] is all the wasted bandwidth, since only one IP packet can go in each encapsulating packet 😩. To max out the bandwidth, IP-TFS breaks that one-to-one relationship by introducing an AGGFRAG mode for ESP 🔥.

The AGGFRAG mode aggregates and fragments the inner IP traffic flow into encapsulating IPsec tunnel packets. For IP-TFS, those IPsec encapsulating tunnel packets are a fixed size. Padding only gets added to the tunnel packets if there's no data ready to send when a tunnel packet is transmitted, or if the receiver has disabled fragmentation 🧸.

All of this happens using a new Encapsulating Security Payload (ESP) [RFC4303] Next Header field value, AGGFRAG_PAYLOAD (Section 6.1) 💜.

People have suggested other non-IP-TFS uses for this AGGFRAG mode too, like better performance through packet aggregation, or handling MTU issues with fragmentation 🌟. Those uses aren't defined here, but this document doesn't restrict them either 🦋.

2.2. Payload Content 💅

The AGGFRAG_PAYLOAD payload content defined in this document is a 4- or 24-octet header, followed by either a partial data block, a full data block, or multiple partial or full data blocks 🎁. The diagram below shows where this payload sits inside the ESP packet. For the exact formats of the AGGFRAG_PAYLOAD payload, see Section 6.1 📐.

    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    . Outer Encapsulating Header ...                                .
    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
    . ESP Header...                                                 .
    +---------------------------------------------------------------+
    |   [AGGFRAG sub-type/flags]   :           BlockOffset          |
    +---------------------------------------------------------------+
    :                  [Optional Congestion Info]                   :
    +---------------------------------------------------------------+
    |       DataBlocks ...                                          ~
    ~                                                               ~
    ~                                                               |
    +---------------------------------------------------------------|
    . ESP Trailer...                                                .
    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 1: Layout of an AGGFRAG Mode IPsec Packet 🖼️

The BlockOffset value is either zero or some offset into, or past the end of, the DataBlocks data 🧭.

If BlockOffset is zero, the DataBlocks data begins with a new data block 🌱.

On the flip side, if BlockOffset is non-zero, it points to the start of the new data block, and the initial DataBlocks data belongs to a data block that's still being reassembled 🧩.

If BlockOffset points past the end of the DataBlocks data, the next data block shows up in a later encapsulating packet 🚀.

Always having BlockOffset point at the next available data block lets you recover the next inner packet even when outer encapsulating packets get lost 🙌. That's a big deal ngl 💖.

You can find an example AGGFRAG mode packet flow in Appendix A 🎀.

2.2.1. DataBlocks 🌈

    +---------------------------------------------------------------+
    | Type  | rest of IPv4, IPv6, or pad...
    +--------

Figure 2: Layout of a Data Block 🧃

A data block is made of a 4-bit type code, followed by the data block data 🔢. The type values were picked carefully so they line up with the IPv4/IPv6 version field values. That way, you don't need any per-data-block type overhead to encapsulate an IP packet 🪄. Also, the length of the data block comes from the encapsulated IPv4's Total Length field or IPv6's Payload Length field 📏.

2.2.2. End Padding

okay so bestie, padding is only needed in one situation 🎀 — when there's literally no data to wrap up. Since a data block's type is spelled out in its first 4 bits ✨, the only time you need padding is that moment. For this end padding, we use a Pad Data Block 💖.

2.2.3. Fragmentation, Sequence Numbers, and All-Pad Payloads

okay this section is giving sequence number drama 😭🫶. So a receiver has to be able to put fragmented inner packets back together, and for that the sender MUST send the inner packet fragments back to back in the logical outer packet stream (basically, using consecutive ESP sequence numbers) 🦋. BUT, the sender is allowed to sneak in "all-pad" payloads (payloads with a BlockOffset of zero and a single pad data block) in between the packets carrying the inner packet fragment payloads 🧚. Why? Because it lets the sender always send a tunnel packet, no matter how much work the encapsulation takes 💅.

When a receiver is reassembling an inner packet and gets an "all-pad" payload, it increments the expected sequence number that the next inner packet fragment is expected to arrive in 🌸.

Now, the receiver is going to have to deal with outer ESP packets arriving out of order before it does any reassembly 🙈. ESP already has an optional replay attack detection thing, which usually uses a window method 💜. A similar sequence-number-based sliding window can fix up reordering of the outer packet stream too ✨. Receiving a larger (newer) sequence number packet moves the window forward, and if older ESP packets show up whose sequence numbers the window has already passed, those packets get dropped 💀. What's a good window size? It really depends on how much misordering the user is experiencing 🍓, but a value of 3 has been suggested as a default when there's no more informed choice 🌈.

Since the amount of misordering is hard to predict, the window size SHOULD be configurable by the user 🎀. Implementations MAY also dynamically adjust the reordering window based on the actual misordering they see in arriving packets 👑.

Quick heads-up 👀: when IP-TFS is sending a continuous stream of packets, there's no requirement for an explicit lost packet timer, but using a lost packet timer is RECOMMENDED 🔥. If an implementation doesn't use one and only decides an outer packet is lost when the reorder window moves past it, the inner traffic can get delayed by up to the reorder window size times the per-packet send rate 😭. That delay could be seriously significant for slower send rates or when bigger reorder window sizes are in use 💀. Since the lost packet timer affects how late inner packets get delivered, an implementation or user could choose to set it proportionate to the tunnel rate 🦋.

Here's the tea ☕: ESP guarantees an increasing sequence number with each packet sent after it, but it doesn't actually require the numbers to be generated consecutively. Like, sending only even-numbered sequence numbers would be allowed, as long as they keep increasing 🌸. BUT gaps in the sequence numbers will not work for this document, so the sequence number stream MUST increase monotonically by 1 for each subsequent packet 🙅‍♀️💖.

When you're using AGGFRAG_PAYLOAD together with replay detection, the window size for both MAY be reduced to the smaller of the two window sizes 🧚. The reason is that packets outside the smaller window but inside the larger one would still be dropped by the mechanism with the smaller window anyway 💅. Still, there's no requirement to make these values the same ✨. In some cases, like slow tunnels where a very small or zero reorder window size is appropriate, the user may still want a big replay detection window so they can log replayed packets 📝. Also, large replay windows can be implemented with very little overhead, compared to large reorder windows 🫶.

Last thing in this section 🥺: sequence numbers reset when you switch Security Associations (SAs), like when rekeying a Child SA, so senders MUST NOT send initial fragments of an inner packet using one SA and the subsequent fragments in a different SA 🚫💅.

🎀 A note on BlockOffset values: Senders MUST encode the BlockOffset consistently with the immediately preceding non-all-pad payload packet. Specifically, if the immediately preceding non-all-pad payload packet ended with a Pad Data Block, this BlockOffset MUST be zero, since Pad Data Blocks are never fragmented. The BlockOffset MUST be consistent with the remaining size implied by the length field from the fragmented inner packet. 💖

2.2.3.1. Optional Extra Padding

When the tunnel bandwidth isn't being fully used, a sender MAY pad out the current encapsulating packet so it can deliver an inner packet unfragmented in the following outer packet 🦋. The benefit is avoiding inner packet fragmentation when the offered load is bursty (non-bursty traffic won't fragment naturally anyway) 🌸. Senders MAY also choose to allow a minimum fragment size to be configured (for example, as a percentage of the AGGFRAG_PAYLOAD payload size) to avoid fragmentation, but that comes at the cost of tunnel bandwidth 💸. The costs of these methods are complexity and added delay for inner traffic 😵‍💫. The main advantage of avoiding fragmentation is minimizing inner packet loss when outer packet loss happens 🫶. When this is actually worth it (how much loss and what type of loss, given different inner traffic shapes and utilization) and what values to use for the allowable or added delay might be worth researching, but that's outside the scope of this document 🔍.

Padding to avoid fragmentation doesn't affect interoperability ✨, BUT if padding gets used inappropriately, it can reduce the effective throughput of a tunnel 😩. Senders implementing either of the above approaches need to take care not to reduce the effective capacity, and overall utility, of the tunnel by overusing padding 🙏💅.

2.2.4. Empty Payload

So to support reporting congestion control information (more on that later) using a non-AGGFRAG_PAYLOAD-enabled SA, it's allowed to send an AGGFRAG_PAYLOAD payload with no data blocks 🧚. In other words, the ESP payload length is equal to the AGGFRAG_PAYLOAD header length 💖. This special payload is called an empty payload 🌈.

Right now this only applies in use cases without Internet Key Exchange Protocol Version 2 (IKEv2) 🎀.

2.2.5. IP Header Value Mapping

[RFC4301] gives some direction on when and how to map various values from an inner IP header to the outer encapsulating header 📖. Those values are the Don't Fragment (DF) bit [RFC0791], the Differentiated Services (DS) field [RFC2474], and the Explicit Congestion Notification (ECN) field [RFC3168] 🔥. Unlike in [RFC4301], the AGGFRAG mode may, and often will, encapsulate more than one IP packet per ESP packet 🫶. So to deal with that, these mappings are restricted further 💅.

2.2.5.1. DF Bit

The AGGFRAG mode never maps the inner DF bit 🙅‍♀️, because it's unrelated to the AGGFRAG tunnel functionality ✨. The AGGFRAG mode never needs to IP fragment the inner packets, and the inner packets won't affect the fragmentation of the outer encapsulation packets 🦋.

2.2.5.2. ECN Value

The ECN value doesn't need to be mapped 🌸, since any congestion related to the constant-send-rate IP-TFS tunnel is unrelated (by design) to the inner traffic flow 💜. The sender MAY still set the ECN value of inner packets based on the normal ECN specification [RFC3168] [RFC4301] [RFC6040] 🎀.

2.2.5.3. DS Field

By default, the DS field SHOULD NOT be copied 🙅‍♀️, although a sender MAY choose to allow configuration to override this behavior 🔧. A sender SHOULD also allow the DS value to be set by configuration ⭐.

2.2.6. IPv4 Time To Live (TTL), IPv6 Hop Limit, and ICMP Messages

How to modify the inner packet IPv4 TTL [RFC0791] or IPv6 Hop Limit [RFC8200] is specified in [RFC4301] 📖.

[RFC4301] specifies how to apply policy to authenticated and unauthenticated ICMP error packets (e.g., Destination Unreachable) arriving at or being forwarded through the endpoint 🛡️. In particular, it covers whether to process, ignore, or forward said packets 🔍. With the one exception that this document does not change the handling of these packets, they should be handled as specified in [RFC4301] 💅.

The one way an AGGFRAG tunnel differs in ICMP error packet mechanics is with PMTU 🌈. When fragmentation is enabled on the AGGFRAG tunnel, no ICMP "Too Big" errors need to be generated for arriving ingress traffic, because the arriving inner packets will be naturally fragmented by the AGGFRAG encapsulation 🧚✨.

Otherwise, when fragmentation has been disabled on the AGGFRAG tunnel, the treatment of arriving inner traffic exactly maps to that of a non-AGGFRAG ESP tunnel 🔁. Explicitly, IPv4 with DF set and IPv6 packets that can't fit in their own outer packet payload will generate the appropriate ICMP "Too Big" error, as described in [RFC4301] 🚨. IPv4 packets without DF set will be IP fragmented, as described in [RFC4301] 🧩.

Packets egressing the tunnel continue to be handled as specified in [RFC4301] 📦.

All other aspects of PMTU and the handling of ICMP "Too Big" messages (i.e., with regards to the outer AGGFRAG/ESP tunnel packet size) also remain unchanged from [RFC4301] 💖.

2.2.7. Effective MTU of the Tunnel

Unlike in [RFC4301], there is normally no effective MTU (EMTU) on an AGGFRAG tunnel 🦋, because all IP packet sizes are properly transmitted without requiring IP fragmentation prior to tunnel ingress 🌸. That said, a sender MAY allow explicitly configuring an MTU for the tunnel 🎀.

If fragmentation has been disabled on the AGGFRAG tunnel, then the tunnel's EMTU and behaviors are the same as normal IPsec tunnels [RFC4301] 🔁.

2.3. Exclusive SA Use

This document does not specify mixed use of an AGGFRAG_PAYLOAD-enabled SA 🙅‍♀️. A sender MUST only send AGGFRAG_PAYLOAD payloads over an SA configured for AGGFRAG mode 💅.

2.4. Modes of Operation

Just like with normal IPsec/ESP SAs, AGGFRAG SAs are unidirectional ➡️. Bidirectional IP-TFS functionality is achieved by setting up 2 AGGFRAG SAs, one in either direction 🔄✨.

An AGGFRAG tunnel used for IP-TFS can operate in 2 modes: a non-congestion-controlled mode and a congestion-controlled mode 👑🌸.

2.4.1. Non-Congestion-Controlled Mode 💖

okay so in the non-congestion-controlled mode, IP-TFS sends fixed-size packets over an AGGFRAG tunnel at a constant rate 💅 the send rate is locked and doesn't get auto-adjusted, no matter what network congestion is happening (like packet loss, for example) 😬

For similar reasons as given in [RFC7510], the non-congestion-controlled mode MUST only be used where the user has full administrative control over any path the tunnel will take and MUST NOT be used if this is not the case 🚨 this is required so the user can guarantee the bandwidth and also be sure they're not negatively affecting network congestion [RFC2914]. in this case, packet loss should be reported to the administrator (e.g., via syslog, YANG notification, SNMP traps, etc.) so that any failures due to a lack of bandwidth can be corrected 🛠️ the use of circuit breakers is also RECOMMENDED (Section 2.4.2.1) 🎀

users who choose this mode need to understand that it will send packets at a constant rate, using a constant, fixed bandwidth, and won't adjust for congestion 🙈 so if they don't guarantee the bandwidth the tunnel needs, the tunnel's operation, plus the rest of their network, may get negatively impacted 💀

one expected use case for the non-congestion-controlled mode is to guarantee the full tunnel bandwidth is available and preferred over other non-tunnel traffic 👑 in fact, a typical site-to-site use case might have all of the user traffic going through the IP-TFS tunnel ✨

the non-congestion-controlled mode is also appropriate if ESP over TCP is in use [RFC9329]. BUT the use of TCP is considered a fallback-only solution for IPsec, and it is highly not preferred 🙅‍♀️ this is also one of the reasons TCP wasn't chosen as the encapsulation for IP-TFS instead of AGGFRAG 🦋

2.4.2. Congestion-Controlled Mode 🌊

with the congestion-controlled mode, IP-TFS adapts to network congestion by lowering the packet send rate to accommodate the congestion, and raising the rate again when congestion subsides 📈 since overhead is per packet, allowing for maximal fixed-size packets and varying the send rate means transport overhead gets minimized 🧚

the output of the congestion control algorithm adjusts the rate at which the ingress sends packets. this document doesn't REQUIRE a specific congestion control algorithm, but best current practice RECOMMENDS that the algorithm conform to [RFC5348] 📚 congestion control principles are documented in [RFC2914] as well. there's an example in [RFC4342] of the algorithm in [RFC5348] that matches the requirements of IP-TFS (i.e., designed for fixed-size packets and a send rate that varies based on congestion) 💜

the required inputs for the TCP-friendly rate control algorithm described in [RFC5348] are the receiver's loss event rate and the sender's estimated round-trip time (RTT) 🧠 these values are provided by IP-TFS using the congestion information header fields described in Section 3. and honestly, these values are sufficient to implement the algorithm described in [RFC5348] 🫶

at a minimum, the congestion information MUST be sent, from the receiver and from the sender, at least once per RTT. before an RTT is established, the information SHOULD be sent constantly from the sender and the receiver so an RTT estimate can get established 🌸 not receiving this information over multiple consecutive RTT intervals should be considered a congestion event that causes the sender to adjust its sending rate lower ⚠️ for example, this is called the "no feedback timeout" in [RFC4342], and it's equal to 4 RTT intervals. when a "no feedback timeout" has occurred, the sending rate is halved, as per [RFC4342] 😭

an implementation MAY choose to always include the congestion information in its AGGFRAG payload header if it's sending it on an IP-TFS-enabled SA. since IP-TFS normally operates with a large packet size, the congestion information should represent a small portion of the available tunnel bandwidth 🍓 an implementation choosing to always send the data MAY also choose to only update the LossEventRate and RTT header field values it sends every RTT through 🎀

when choosing a congestion control algorithm (or a selection of algorithms), note that IP-TFS is not providing reliable delivery of IP traffic, so per-packet acknowledgements (ACKs) are not required and are not provided 🙅‍♀️

it's worth noting that the variable send rate of a congestion-controlled AGGFRAG tunnel is not private 👀 however, this send rate is being driven by network congestion. as long as the encapsulated (inner) traffic flow shape and timing aren't directly affecting the (outer) network congestion, the variations in the tunnel rate won't weaken the provided inner traffic flow confidentiality 🔐

2.4.2.1. Circuit Breakers ⚡

in addition to congestion control, implementations that support the non-congestion-control mode SHOULD implement circuit breakers [RFC8084] as a recovery method of last resort 🧯 when circuit breakers are enabled, an implementation SHOULD also enable congestion control reports so the circuit breakers have information to act on 📊

the pseudowire congestion considerations [RFC7893] are equally applicable to the mechanisms defined in this document, notably the text on inelastic traffic 🦋

one example of a simple, slow-trip circuit breaker that an implementation may provide would use 2 values: the amount of persistent loss rate required to trip the circuit breaker, and the required length of time this persistent loss rate must be seen to trip it 🕰️ these 2 values are required configurations from the user. when the circuit breaker is tripped, the tunnel traffic is disabled and an appropriate log message or other management-type alarm is triggered, indicating operator intervention is required 🚨

2.5. Summary of Receiver Processing 📬

an AGGFRAG-enabled SA receiver has a few tasks to perform, bestie 🎀

the receiver MAY process incoming AGGFRAG_PAYLOAD payloads as soon as they arrive, as much as it can. i.e., if the incoming AGGFRAG_PAYLOAD packet contains complete inner packet(s), the receiver should extract and transmit them immediately 🏃‍♀️ for partial packets, the receiver needs to keep them in memory until they fall out of the reordering window or until the missing parts are received, in which case it will reassemble and transmit them 🧩 if the AGGFRAG_PAYLOAD payload contains multiple packets, they SHOULD be sent out in the order they are in the AGGFRAG_PAYLOAD (i.e., keep the original order they were received in on the other end) 💅 the cost of this method is that an amplification of out-of-order delivery of inner packets can occur due to inner packet aggregation 😩

instead of the method in the previous paragraph, the receiver MAY reorder out-of-order AGGFRAG_PAYLOAD payloads into in-sequence-order AGGFRAG_PAYLOAD payloads (Section 2.2.3), and only after it has an in-order AGGFRAG_PAYLOAD payload stream would the receiver transmit the inner packets 📋 using this method ensures the inner packets are sent in order ✨ the cost is that a lost packet causes a delay of up to the lost packet timer interval (or the full reorder window if no lost packet timer is used) ⏳ there can also be extra burstiness in the output stream. this burstiness can happen when a lost packet is dropped from the reorder window, and the remaining outer packets in the reorder window are immediately processed and sent out back to back 🌪️

additionally, if congestion control is enabled, the receiver sends congestion control data (Section 6.1.2) back to the sender, as described in Sections 2.4.2 and 3 💌

finally, a note on receiving incorrect BlockOffset values 🙈 to account for misbehaving senders, a receiver SHOULD gracefully handle the case where the BlockOffset of consecutive packets, and/or the inner packet they share, do not agree. it MAY drop the inner packet or one or both of the outer packets 🫧

3. Congestion Information 📊

in order to support the congestion-controlled mode, the sender needs to know the loss event rate and to approximate the RTT [RFC5348] 🧠 to get these values, the receiver sends congestion control information on its SA back to the sender. so to support congestion control, the receiver MUST have a paired SA back to the sender (this is always the case when the tunnel was created using IKEv2) 🔁 if the SA back to the sender is a non-AGGFRAG_PAYLOAD-enabled SA, then an AGGFRAG_PAYLOAD empty payload (i.e., header only) is used to convey the information 🎀

to calculate a loss event rate compatible with [RFC5348], the receiver needs to have an RTT estimate. so the sender communicates this estimate in the RTT header field. on startup this value will be zero, since no RTT estimate is known yet 🌱

for the sender to estimate its RTT value, the sender places a timestamp value in the TVal header field 🕒 on first receipt of this TVal, the receiver records the new TVal value, along with the time it arrived locally. subsequent receipt of the same TVal MUST NOT update the recorded time 🙅‍♀️

when the receiver sends its congestion control header, it places this latest recorded TVal in the TEcho header field, along with 2 delay values: Echo Delay and Transmit Delay 💫 the Echo Delay value is the time delta from the recorded arrival time of TVal to the current clock, in microseconds. the second value, Transmit Delay, is the receiver's current transmission delay on the tunnel (i.e., the average time between sending packets on its half of the AGGFRAG tunnel) 🚀

when the sender gets its TVal back in the TEcho header field, it calculates 2 RTT estimates 🧮 the first is the actual delay, found by subtracting the TEcho value from its current clock and then subtracting the Echo Delay as well. the second RTT estimate is found by adding the received Transmit Delay header value to the sender's own transmission delay (i.e., the average time between sending packets on its half of the AGGFRAG tunnel). the larger of these 2 RTT estimates SHOULD be used as the RTT value 👑

the two RTT estimates are needed to handle different combinations of faster or slower tunnel packet paths with faster or slower fixed tunnel rates 🌈 choosing the larger of the two values guarantees the RTT is never considered faster than the aggregate transmission delay based on the IP-TFS send rate (the second estimate), and never considered faster than the actual RTT along the tunnel packet path (the first estimate) 💖

the receiver also calculates, and communicates in the LossEventRate header field, the loss event rate for use by the sender. this is slightly different from [RFC4342], which periodically sends all the loss interval data back to the sender so it can do the calculation 🧁 see Appendix B for a suggested way to calculate the loss event rate value. initially, this value will be zero (indicating no loss) until enough data has been collected by the receiver to update it 🥰

3.1. ECN Support

okay so AGGFRAG mode isn't just watching for packet loss 👀 it can also use the ECN bits in the encapsulating IP header [RFC3168] to spot congestion 🦋 if ECN is turned on and a packet shows up at the egress (receiving) side with the Congestion Experienced (CE) value set, the receiver counts that packet as dropped, even though it doesn't actually drop it 🙈 and the receiver MUST set the E bit in any AGGFRAG_PAYLOAD payload header that holds a LossEventRate value derived from a CE value it's counting.

the outer ECN stuff is where it gets spicy 🌶️ [RFC6040] updates [RFC3168] and [RFC4301] and defines how the outer ECN field gets marked based on the inner packet's ECN field. the catch is that AGGFRAG can cram several inner packets into one outer packet, and there's no obvious correct way to squish those multiple values into one outer ECN value 😭 so the tunnel ingress endpoint SHOULD operate in the "compatibility" mode instead of the "default" mode from [RFC6040]. basically, the sending endpoint of the tunnel always sets the freshly built outer encapsulating packet header's ECN field to Not-ECT [RFC6040] 💅

4. Configuration of AGGFRAG Tunnels for IP-TFS ✨

okay bestie, IP-TFS is designed to be super easy to set up with minimal configuration 🎀 all the IP-TFS-specific config should be specified at the unidirectional tunnel ingress (sending) side. it's meant to work without IKEv2 too, at least with local static configuration 🧚

YANG and MIB documents for IP-TFS have been defined in [RFC9348] and [RFC9349] 📄

4.1. Bandwidth

bandwidth is a local config option 💖 for the non-congestion-controlled mode, the bandwidth SHOULD be configured. for the congestion-controlled mode, you can configure it, or the congestion control algorithm can figure out and use the maximum bandwidth that's available on its own 🌈 no standardized configuration method is required.

4.2. Fixed Packet Size

the fixed packet size for the tunnel encapsulation packets MAY be configured manually, or it can be figured out automatically using other methods, like PLMTUD [RFC4821] [RFC8899] or PMTUD [RFC1191] [RFC8201] 🍓 PMTUD is known to have issues, so PLMTUD is considered the more robust option 🫶 no standardized configuration method is required.

4.3. Congestion Control

congestion control is a local config option 🌸 no standardized configuration method is required.

5. IKEv2 👑

5.1. USE_AGGFRAG Notification Message

as mentioned earlier, AGGFRAG tunnels use ESP payloads of type AGGFRAG_PAYLOAD ✨

when you're using IKEv2, a new USE_AGGFRAG notification message turns on the AGGFRAG_PAYLOAD payload on a Child SA pair 💜 it works kind of like how USE_TRANSPORT_MODE gets negotiated, as described in [RFC7296] 🦋

to ask for AGGFRAG_PAYLOAD on the Child SA pair, the initiator includes the USE_AGGFRAG notification in an SA payload that requests a new Child SA. this can happen during the initial IKE_AUTH or during CREATE_CHILD_SA exchanges 🎀 if the request is accepted, the response MUST also include a notification of type USE_AGGFRAG. if the responder says no, the Child SA still gets established, just without AGGFRAG_PAYLOAD enabled. and if that's unacceptable to the initiator, the initiator MUST delete the Child SA 😭

heads up, this one's a hard no 🙅 AGGFRAG_PAYLOAD use is currently only defined for non-transport-mode tunnels, so the USE_AGGFRAG notification MUST NOT be combined with the USE_TRANSPORT notification.

the USE_AGGFRAG notification carries a 1-octet payload of flags that specify requirements from the sender of the notification 🧚 if any requirement flags aren't understood or can't be supported by the receiver, the receiver SHOULD NOT enable use of AGGFRAG_PAYLOAD. it can either skip responding with the USE_AGGFRAG notification or, if it's the initiator, delete the Child SA when the now-established non-AGGFRAG_PAYLOAD SA is unacceptable 💅

the notification type and payload flag values are defined in Section 6.1.4 ✨

6. Packet and Data Formats 🌈

the packet and data formats below are generic, meant to allow for non-IP-TFS uses, but those uses are outside the scope of this document 🍓

6.1. AGGFRAG_PAYLOAD Payload

ESP Next Header value: 144 🎀

an AGGFRAG payload is identified by the ESP Next Header value AGGFRAG_PAYLOAD, which has the value 144, and that value has been reserved in the IP protocol numbers space 🔥 the first octet of the payload tells you the format of the remaining payload data 💖

  0 1 2 3 4 5 6 7
 +-+-+-+-+-+-+-+-+-+-+-
 |   Sub-type    | ...
 +-+-+-+-+-+-+-+-+-+-+-

Figure 3: AGGFRAG_PAYLOAD Payload Format

Sub-type: An 8-bit value indicating the payload format.

this document defines 2 payload sub-types 🦋 those payload formats are defined in the following sections.

6.1.1. Non-Congestion-Control AGGFRAG_PAYLOAD Payload Format

the non-congestion-control AGGFRAG_PAYLOAD payload is a 4-octet header followed by a variable amount of DataBlocks data, like this 🧁

                     1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |  Sub-Type (0) |   Reserved    |          BlockOffset          |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |       DataBlocks ...
 +-+-+-+-+-+-+-+-+-+-+-

Figure 4: Non-Congestion-Control Payload Format

Sub-type: An octet indicating the payload format. For this non-congestion-control format, the value is 0.

Reserved: An octet set to 0 on generation and ignored on receipt.

BlockOffset: A 16-bit unsigned integer counting the number of octets of DataBlocks data before the start of a new data block. If the start of a new data block occurs in a subsequent payload, the BlockOffset will point past the end of the DataBlocks data. In this case, all the DataBlocks data belongs to the current data block being assembled. When the BlockOffset extends into subsequent payloads, it continues to only count DataBlocks data (i.e., it does not count subsequent packets of the non-DataBlocks data, such as header octets).

DataBlocks: Variable number of octets that begins with the start of a data block or the continuation of a previous data block, followed by zero or more additional data blocks.

6.1.2. Congestion Control AGGFRAG_PAYLOAD Payload Format

the congestion control AGGFRAG_PAYLOAD payload is a 24-octet header followed by a variable amount of DataBlocks data, like so 🌸

                     1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |  Sub-type (1) |  Reserved |P|E|          BlockOffset          |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                          LossEventRate                        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                      RTT                  |   Echo Delay ...
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      ... Echo Delay   |           Transmit Delay                |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                              TVal                             |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                             TEcho                             |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |       DataBlocks ...
 +-+-+-+-+-+-+-+-+-+-+-

Figure 5: Congestion Control Payload Format

Sub-type: An octet indicating the payload format. For this congestion control format, the value is 1.

Reserved: A 6-bit field set to 0 on generation and ignored on receipt.

P: A 1-bit value that, if set, indicates that PLMTUD probing is in progress. This information can be used to avoid treating missing packets as loss events by the congestion control algorithm when running the PLMTUD probe algorithm.

E: A 1-bit value that, if set, indicates that Congestion Experienced (CE) ECN bits were received and used in deriving the reported LossEventRate.

BlockOffset: The same value as the non-congestion-controlled payload format value.

LossEventRate: A 32-bit value specifying the inverse of the current loss event rate, as calculated by the receiver. A value of zero indicates no loss. Otherwise, the loss event rate is 1/LossEventRate.

RTT: A 22-bit value specifying the sender's current RTT estimate in microseconds. The value MAY be zero prior to the sender having calculated an RTT estimate. The value SHOULD be set to zero on non-AGGFRAG_PAYLOAD-enabled SAs. If the RTT is equal to or larger than 0x3FFFFF, the value MUST be set to 0x3FFFFF.

Echo Delay: A 21-bit value specifying the delay in microseconds incurred between the receiver first receiving the TVal value, which it is sending back in TEcho. If the delay is equal to or larger than 0x1FFFFF, the value MUST be set to 0x1FFFFF.

Transmit Delay: A 21-bit value specifying the transmission delay in microseconds. This is the fixed (or average) delay on the receiver between it sending packets on the IP-TFS tunnel. If the delay is equal to or larger than 0x1FFFFF, the value MUST be set to 0x1FFFFF.

TVal: An opaque, 32-bit value that will be echoed back by the receiver in later packets in the TEcho field, along with an Echo Delay value of how long that echo took.

TEcho: The opaque, 32-bit value from a received packet's TVal field. The received TVal is placed in TEcho, along with an Echo Delay value indicating how long it has been since receiving the TVal value.

DataBlocks: Variable number of octets that begins with the start of a data block or the continuation of a previous data block, followed by zero or more additional data blocks. For the special case of sending congestion control information on a non-IP-TFS-enabled SA, this field MUST be empty (i.e., be zero octets long). 🫶

6.1.3. Data Blocks

here's how a data block is laid out 🧩

                     1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 | Type  | IPv4, IPv6, or pad...
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

Figure 6: Data Block Format

Type: A 4-bit field where 0x0 identifies a Pad Data Block, 0x4 indicates an IPv4 data block, and 0x6 indicates an IPv6 data block. 🦋

6.1.3.1. IPv4 Data Block 🎀

Okay so the IPv4 Data Block is the one for when the inner packet is IPv4. Here's how it's laid out 🦋

                         1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  0x4  |  IHL  |  TypeOfService  |         TotalLength         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Rest of the inner packet ...
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                      Figure 7: IPv4 Data Block Format

These values are the actual values from the encapsulated IPv4 header, so no faking 🙈 The start of this data block is literally the start of the encapsulated IP packet. Iconic.

Type: A 4-bit value of 0x4 indicating IPv4 (i.e., first nibble of the IPv4 packet). This is how you know it's IPv4 at a glance ✨

TotalLength: The 16-bit unsigned integer "Total Length" field of the IPv4 inner packet. 💅

6.1.3.2. IPv6 Data Block 🌈

Now the IPv6 version, which is the same vibe but with its own fields 💜

                         1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  0x6  | TrafficClass  |               FlowLabel               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |         PayloadLength         | Rest of the inner packet ...
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-

                      Figure 8: IPv6 Data Block Format

Same deal here: these values are the actual values within the encapsulated IPv6 header, and the start of this data block is the start of the encapsulated IP packet 🧚

Type: A 4-bit value of 0x6 indicating IPv6 (i.e., first nibble of the IPv6 packet). Main character energy 👑

PayloadLength: The 16-bit unsigned integer "Payload Length" field of the inner IPv6 inner packet. (Yes, the inner inner. Lol 💀)

6.1.3.3. Pad Data Block 🍓

And then there's the Pad Data Block, which is basically filler that keeps things lined up 🫶

                         1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  0x0  | Padding ...
    +-+-+-+-+-+-+-+-+-+-+-

                      Figure 9: Pad Data Block Format

Type: A 4-bit value of 0x0 indicating a padding data block. Zero means "padding, don't panic" 🙃

Padding: Extends to end of the encapsulating packet. It just keeps going until the outer packet ends ✨

6.1.4. IKEv2 USE_AGGFRAG Notification Message 🦋

As we talked about in Section 5.1, a notification message USE_AGGFRAG is used to negotiate use of the ESP AGGFRAG_PAYLOAD Next Header value. This is the handshake where both sides agree to play 🤝

The USE_AGGFRAG Notification Message State Type is 16442. Write that number down, bestie 📝

The notification payload contains 1 octet of requirement flags. There are currently 2 requirement flags defined. This may be revised by later specifications, so keep an eye out 👀

    +-+-+-+-+-+-+-+-+
    |0|0|0|0|0|0|C|D|
    +-+-+-+-+-+-+-+-+

                  Figure 10: USE_AGGFRAG Requirement Flags

Here's what each flag bit does 💖

0: 6 bits - Reserved MUST be zero on send, unless defined by later specifications. So the unused bits stay at zero, no cap 🙅

C: Congestion Control bit. If set, then the sender is requiring that congestion control information MUST be returned to it periodically, as defined in Section 3. The sender is basically saying "you better send me that congestion info, periodically" 😤🔥

D: Don't Fragment bit. If set, it indicates the sender of the notify message does not support receiving packet fragments (i.e., inner packets MUST be sent using a single Data Block). This value only applies to what the sender is capable of receiving; the sender MAY still send packet fragments unless similarly restricted by the receiver in its USE_AGGFRAG notification. So D is about what you can receive, not what you're allowed to send. Iykyk 🌸

Quick aside: the D bit only talks about what the notify sender can receive. Whether fragments are okay to send is a separate question, and the receiver's own USE_AGGFRAG notification is what decides that. Don't mix them up, bestie 🚨

7. IANA Considerations ✨

7.1. ESP Next Header Value 💜

IANA has allocated an IP protocol number from the "Protocol Numbers - Assigned Internet Protocol Numbers" registry as follows.

Decimal:  144
Keyword:  AGGFRAG
Protocol:  AGGFRAG encapsulation payload for ESP
Reference:  RFC 9347

7.2. AGGFRAG_PAYLOAD Sub-Types 🌈

IANA has created a registry called "AGGFRAG_PAYLOAD Sub-Types" under a new category named "ESP AGGFRAG_PAYLOAD". The registration policy for this registry is "Expert Review" [RFC8126] [RFC7120]. Very official, very serious 🧐

Name:  AGGFRAG_PAYLOAD Sub-Types
Description:  AGGFRAG_PAYLOAD Payload Formats
Reference:  RFC 9347

This initial content for this registry is as follows:

      +==========+===============================+===========+
      | Sub-Type | Name                          | Reference |
      +==========+===============================+===========+
      | 0        | Non-Congestion-Control Format | RFC 9347  |
      +----------+-------------------------------+-----------+
      | 1        | Congestion Control Format     | RFC 9347  |
      +----------+-------------------------------+-----------+
      | 3-255    | Reserved                      |           |
      +----------+-------------------------------+-----------+

                 Table 1: AGGFRAG_PAYLOAD Sub-Types

7.3. USE_AGGFRAG Notify Message Status Type 🦋

IANA has allocated a status type USE_AGGFRAG from the "IKEv2 Notify Message Types - Status Types" registry. Another one for the collection 💅

Decimal:  16442
Name:  USE_AGGFRAG
Reference:  RFC 9347

8. Security Considerations 💀

Okay, this section is serious, so buckle up 🚨 This document describes an aggregation and fragmentation mechanism to efficiently implement TFC for IP traffic. This approach is expected to reduce the efficacy of traffic analysis on IPsec communication. Other than the additional security afforded by using this mechanism, IP-TFS utilizes the security protocols [RFC4303] and [RFC7296], and so their security considerations apply to IP-TFS as well. So the protocols underneath have their own security considerations, and those apply here too.

As noted in Section 3.1, the ECN bits are not protected by IPsec and thus may constitute a covert channel. For this reason, ECN use SHOULD NOT be enabled by default. Please don't turn it on just because it's there 🙅‍♀️

As noted previously in Section 2.4.2, for TFC to be maintained, the encapsulated traffic flow should not be affecting network congestion in a predictable way, and if it would be, then non-congestion-controlled mode use should be considered instead. So if your traffic would make congestion predictable, you should be looking at the non-congestion-controlled mode instead 😭

9. References 📚

9.1. Normative References 🎀

9.2. Informative References ✨

[AppCrypt] Schneier, B., "Applied Cryptography: Protocols, Algorithms, and Source Code in C", 1996. 📚

Appendix A. Example of an Encapsulated IP Packet Flow

Okay bestie, this appendix is a little movie 🎬🍓 It shows an example inner IP packet flow sitting inside the encapsulating tunnel packet stream. Notice how the encapsulated IP packets can start and end anywhere, and there can be more than one or less than one of them in a single encapsulating packet. 🧚

  Offset: 0        Offset: 100    Offset: 2000    Offset: 600
 [ ESP1  (1404) ][ ESP2  (1404) ][ ESP3  (1404) ][ ESP4  (1404) ]
 [--750--][--750--][60][-240-][--3000----------------------][pad]

Figure 11: Inner and Outer Packet Flow

Each outer encapsulating ESP space is a fixed size of 1404 octets, and its first 4 octets hold the AGGFRAG header. The encapsulated IP packet flow is a 750-octet packet, a 750-octet packet, a 60-octet packet, a 240-octet packet, and a 3000-octet packet. ✨ The lengths include the IP header and payload, so no cheating 🙈

The BlockOffset values in the 4 AGGFRAG payload headers for this packet flow would therefore be 0, 100, 2000, and 600, in that order. The first encapsulating packet (ESP1) has a zero BlockOffset, which points at the IP data block right after the AGGFRAG header. The next packet (ESP2) has a BlockOffset that points inward 100 octets to the start of the 60-octet data block. The third encapsulating packet (ESP3) carries the middle portion of the 3000-octet data block, so its offset points past its own end and into the fourth encapsulating packet. The fourth packet (ESP4) has an offset of 600, pointing at the padding that comes after the continued 3000-octet packet. 💅

Appendix B. A Send and Loss Event Rate Calculation

Let's talk about the math 🤓🎀 The current best practice indicates that congestion control MUST be done in a TCP-friendly way. Hmm, wait, that's not quite what it says, so here it is exactly: the current best practice indicates that congestion control SHOULD be done in a TCP-friendly way. A TCP-friendly congestion control algorithm is described in [RFC5348]. For this IP-TFS use case (as with [RFC4342]), the (fixed) packet size is used as the segment size for the algorithm. The main formula for the send rate in that algorithm looks like this:

                              1
   X = -----------------------------------------------
       R * (sqrt(2*p/3) + 12*sqrt(3*p/8)*p*(1+32*p^2))

X is the send rate in packets per second, R is the RTT estimate, and p is the loss event rate (the inverse of which is provided by the receiver).

Also, the algorithm in [RFC5348] uses an X_recv value too (the receiver's receive rate). For IP-TFS, one MAY set this value according to the sender's current tunnel send rate (X). 🦋

The IP-TFS receiver already has the RTT estimate from the sender, so it can use the same method described in [RFC5348] and [RFC4342] to collect the loss intervals and calculate the loss event rate value using the weighted average as indicated. The receiver then sends the inverse of this value back to the sender in the AGGFRAG_PAYLOAD payload header field LossEventRate. 💜

The IP-TFS sender now has both the R and p values, so it can calculate the correct sending rate. If following [RFC5348], the sender should also use the slow start mechanism described therein when the IP-TFS SA is first established. Iconic behaviour, honestly 👑

Appendix C. Comparisons of IP-TFS

C.1. Comparing Overhead

Time to compare overhead, and yes, it's giving spreadsheet energy 📊💅 To compare overhead, you have to work out the ESP overhead for both normal and AGGFRAG tunnel packets, so an algorithm for encryption and authentication must be chosen. For the data below, AES-GCM-256 was selected. That leads to an IP+ESP overhead of 54.

54 = 20 (IP) + 8 (ESPH) + 2 (ESPF) + 8 (IV) + 16 (ICV)

Additionally, for IP-TFS, non-congestion-control AGGFRAG_PAYLOAD headers were chosen, which adds 4 octets, for a total overhead of 58. ✨

C.1.1. IP-TFS Overhead

For comparison, the overhead of an AGGFRAG payload is 58 octets per outer packet. So the octet overhead per inner packet is 58 divided by the number of outer packets required (fractions allowed). The overhead as a percentage of inner packet size is a constant based on the Outer MTU size. 🍓

   OH = 58 / Outer Payload Size / Inner Packet Size
   OH % of Inner Packet Size = 100 * OH / Inner Packet Size
   OH % of Inner Packet Size = 5800 / Outer Payload Size
               +=======+========+========+========+
               | Type  | IP-TFS | IP-TFS | IP-TFS |
               +=======+========+========+========+
               | MTU   | 576    | 1500   | 9000   |
               +=======+========+========+========+
               | PSize | 518    | 1442   | 8942   |
               +=======+========+========+========+
               | 40    | 11.20% | 4.02%  | 0.65%  |
               +-------+--------+--------+--------+
               | 576   | 11.20% | 4.02%  | 0.65%  |
               +-------+--------+--------+--------+
               | 1500  | 11.20% | 4.02%  | 0.65%  |
               +-------+--------+--------+--------+
               | 9000  | 11.20% | 4.02%  | 0.65%  |
               +-------+--------+--------+--------+

                   Table 2: IP-TFS Overhead as
                 Percentage of Inner Packet Size

C.1.2. ESP with Padding Overhead

Now the padded version 🧁 The overhead per inner packet for constant-send-rate-padded ESP (i.e., original IPsec TFC) is 36 octets plus any padding, unless fragmentation is required.

When the inner packet has to be fragmented to fit in the outer IPsec packet, the overhead is the number of outer packets required to carry the fragmented inner packet, times both the inner IP Overhead (20) and the outer packet overhead (54), minus the initial inner IP Overhead, plus any required tail padding in the last encapsulation packet. The required tail padding is the number of required packets times the difference between the Outer Payload Size and the IP Overhead, minus the Inner Payload Size. So, here's the maths 🙈

  Inner Payload Size = IP Packet Size - IP Overhead
  Outer Payload Size = MTU - IPsec Overhead

                  Inner Payload Size
  NF0 = ----------------------------------
         Outer Payload Size - IP Overhead

  NF = CEILING(NF0)

  OH = NF * (IP Overhead + IPsec Overhead)
       - IP Overhead
       + NF * (Outer Payload Size - IP Overhead)
       - Inner Payload Size

  OH = NF * (IPsec Overhead + Outer Payload Size)
       - (IP Overhead + Inner Payload Size)

  OH = NF * (IPsec Overhead + Outer Payload Size)
       - Inner Packet Size

C.2. Overhead Comparison 💅

okay bestie, these tables are the main event 🎀 they line up the overhead for a few common L3 MTU sizes so we can compare them side by side. The first table shows how many octets of overhead you get for each L3 MTU-sized packet. The second shows that same overhead as a percentage of the MTU-sized packet. 🧚

    +========+=========+=========+=========+========+========+========+
    | Type   | ESP+Pad | ESP+Pad | ESP+Pad | IP-TFS | IP-TFS | IP-TFS |
    +========+=========+=========+=========+========+========+========+
    | L3 MTU | 576     | 1500    | 9000    | 576    | 1500   | 9000   |
    +========+=========+=========+=========+========+========+========+
    | PSize  | 522     | 1446    | 8946    | 518    | 1442   | 8942   |
    +========+=========+=========+=========+========+========+========+
    | 40     | 482     | 1406    | 8906    | 4.5    | 1.6    | 0.3    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 128    | 394     | 1318    | 8818    | 14.3   | 5.1    | 0.8    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 256    | 266     | 1190    | 8690    | 28.7   | 10.3   | 1.7    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 518    | 4       | 928     | 8428    | 58.0   | 20.8   | 3.4    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 576    | 576     | 870     | 8370    | 64.5   | 23.2   | 3.7    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 1442   | 286     | 4       | 7504    | 161.5  | 58.0   | 9.4    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 1500   | 228     | 1500    | 7446    | 168.0  | 60.3   | 9.7    |
    +--------+---------+---------+---------+--------+--------+--------+
    | 8942   | 1426    | 1558    | 4       | 1001.2 | 359.7  | 58.0   |
    +--------+---------+---------+---------+--------+--------+--------+
    | 9000   | 1368    | 1500    | 9000    | 1007.7 | 362.0  | 58.4   |
    +--------+---------+---------+---------+--------+--------+--------+

Table 3: Overhead Comparison in Octets

    +=======+=========+=========+==========+========+========+========+
    | Type  | ESP+Pad | ESP+Pad | ESP+Pad  | IP-TFS | IP-TFS | IP-TFS |
    +=======+=========+=========+==========+========+========+========+
    | MTU   | 576     | 1500    | 9000     | 576    | 1500   | 9000   |
    +=======+=========+=========+==========+========+========+========+
    | PSize | 522     | 1446    | 8946     | 518    | 1442   | 8942   |
    +=======+=========+=========+==========+========+========+========+
    | 40    | 1205.0% | 3515.0% | 22265.0% | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 128   | 307.8%  | 1029.7% | 6889.1%  | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 256   | 103.9%  | 464.8%  | 3394.5%  | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 518   | 0.8%    | 179.2%  | 1627.0%  | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 576   | 100.0%  | 151.0%  | 1453.1%  | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 1442  | 19.8%   | 0.3%    | 520.4%   | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 1500  | 15.2%   | 100.0%  | 496.4%   | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 8942  | 15.9%   | 17.4%   | 0.0%     | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+
    | 9000  | 15.2%   | 16.7%   | 100.0%   | 11.20% | 4.02%  | 0.65%  |
    +-------+---------+---------+----------+--------+--------+--------+

Table 4: Overhead as Percentage of Inner Packet Size

C.3. Comparing Available Bandwidth 🦋

ok so another way to compare the two solutions is to look at how much available bandwidth each one leaves you 🌈 the next sections check out and compare the percentage of available bandwidth. for a solid baseline, we also include normal (unencrypted) Ethernet and normal ESP values so you can see exactly where everything sits 💜✨

C.3.1. Ethernet ⚡

okay so, to figure out the available bandwidth, we first need the per-packet overhead 💅 Ethernet's total overhead is 14+4 octets of header and Cyclic Redundancy Check (CRC) plus an extra 20 octets of framing (preamble, start, and inter-packet gap), so 38 octets total ✨ Also the minimum payload is 46 octets 🎀

   +====+=======+=======+=======+=======+=======+=======+======+======+
   |Size| E + P | E + P | E + P | IPTFS | IPTFS | IPTFS | Enet | ESP  |
   +====+=======+=======+=======+=======+=======+=======+======+======+
   |MTU | 590   | 1514  | 9014  | 590   | 1514  | 9014  | any  | any  |
   +====+=======+=======+=======+=======+=======+=======+======+======+
   |OH  | 92    | 92    | 92    | 96    | 96    | 96    | 38   | 74   |
   +====+=======+=======+=======+=======+=======+=======+======+======+
   |40  | 614   | 1538  | 9038  | 47    | 42    | 40    | 84   | 114  |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |128 | 614   | 1538  | 9038  | 151   | 136   | 129   | 166  | 202  |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |256 | 614   | 1538  | 9038  | 303   | 273   | 258   | 294  | 330  |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |518 | 614   | 1538  | 9038  | 614   | 552   | 523   | 574  | 610  |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |576 | 1228  | 1538  | 9038  | 682   | 614   | 582   | 614  | 650  |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |1442| 1842  | 1538  | 9038  | 1709  | 1538  | 1457  | 1498 | 1534 |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |1500| 1842  | 3076  | 9038  | 1777  | 1599  | 1516  | 1538 | 1574 |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |8942| 11052 | 10766 | 9038  | 10599 | 9537  | 9038  | 8998 | 9034 |
   +----+-------+-------+-------+-------+-------+-------+------+------+
   |9000| 11052 | 10766 | 18076 | 10667 | 9599  | 9096  | 9038 | 9074 |
   +----+-------+-------+-------+-------+-------+-------+------+------+

                      Table 5: L2 Octets Per Packet

   +====+=======+=======+======+=======+=======+=======+=======+=======+
   |Size| E + P | E +   | E +  | IPTFS | IPTFS | IPTFS | Enet  | ESP   |
   |    |       | P     | P    |       |       |       |       |       |
   +====+=======+=======+======+=======+=======+=======+=======+=======+
   |MTU | 590   | 1514  | 9014 | 590   | 1514  | 9014  | any   | any   |
   +====+=======+=======+======+=======+=======+=======+=======+=======+
   |OH  | 92    | 92    | 92   | 96    | 96    | 96    | 38    | 74    |
   +====+=======+=======+======+=======+=======+=======+=======+=======+
   |40  | 2.0M  | 0.8M  | 0.1M | 26.4M | 29.3M | 30.9M | 14.9M | 11.0M |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |128 | 2.0M  | 0.8M  | 0.1M | 8.2M  | 9.2M  | 9.7M  | 7.5M  | 6.2M  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |256 | 2.0M  | 0.8M  | 0.1M | 4.1M  | 4.6M  | 4.8M  | 4.3M  | 3.8M  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |518 | 2.0M  | 0.8M  | 0.1M | 2.0M  | 2.3M  | 2.4M  | 2.2M  | 2.1M  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |576 | 1.0M  | 0.8M  | 0.1M | 1.8M  | 2.0M  | 2.1M  | 2.0M  | 1.9M  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |1442| 678K  | 812K  | 138K | 731K  | 812K  | 857K  | 844K  | 824K  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |1500| 678K  | 406K  | 138K | 703K  | 781K  | 824K  | 812K  | 794K  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |8942| 113K  | 116K  | 138K | 117K  | 131K  | 138K  | 139K  | 138K  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+
   |9000| 113K  | 116K  | 69K  | 117K  | 130K  | 137K  | 138K  | 137K  |
   +----+-------+-------+------+-------+-------+-------+-------+-------+

                Table 6: Packets Per Second on 10G Ethernet

   +====+======+======+======+======+======+========+========+========+
   |Size|E + P |E + P |E + P |IP-TFS|IP-TFS| IP-TFS | Enet   | ESP    |
   +====+======+======+======+======+======+========+========+========+
   |MTU |590   |1514  |9014  |590   |1514  | 9014   | any    | any    |
   +====+======+======+======+======+======+========+========+========+
   |OH  |92    |92    |92    |96    |96    | 96     | 38     | 74     |
   +====+======+======+======+======+======+========+========+========+
   |40  |6.51% |2.60% |0.44% |84.36%|93.76%| 98.94% | 47.62% | 35.09% |
   +----+------+------+------+------+------+--------+--------+--------+
   |128 |20.85%|8.32% |1.42% |84.36%|93.76%| 98.94% | 77.11% | 63.37% |
   +----+------+------+------+------+------+--------+--------+--------+
   |256 |41.69%|16.64%|2.83% |84.36%|93.76%| 98.94% | 87.07% | 77.58% |
   +----+------+------+------+------+------+--------+--------+--------+
   |518 |84.36%|33.68%|5.73% |84.36%|93.76%| 98.94% | 93.17% | 87.50% |
   +----+------+------+------+------+------+--------+--------+--------+
   |576 |46.91%|37.45%|6.37% |84.36%|93.76%| 98.94% | 93.81% | 88.62% |
   +----+------+------+------+------+------+--------+--------+--------+
   |1442|78.28%|93.76%|15.95%|84.36%|93.76%| 98.94% | 97.43% | 95.12% |
   +----+------+------+------+------+------+--------+--------+--------+
   |1500|81.43%|48.76%|16.60%|84.36%|93.76%| 98.94% | 97.53% | 95.30% |
   +----+------+------+------+------+------+--------+--------+--------+
   |8942|80.91%|83.06%|98.94%|84.36%|93.76%| 98.94% | 99.58% | 99.18% |
   +----+------+------+------+------+------+--------+--------+--------+
   |9000|81.43%|83.60%|49.79%|84.36%|93.76%| 98.94% | 99.58% | 99.18% |
   +----+------+------+------+------+------+--------+--------+--------+

             Table 7: Percentage of Bandwidth on 10G Ethernet

ngl this is where it gets spicy 🌶️ An AGGFRAG tunnel (or any packet aggregating tunnel) can have a plot twist 😭 For small- to medium-sized packets, the available bandwidth can actually end up greater than plain Ethernet 🦋 That's because the Ethernet framing overhead gets reduced. But obviously nothing is free bestie 💀 This extra bandwidth comes with more latency, which is the time it takes to send the unrelated octets in the outer tunnel frame 🧚 Here's the latency for some common values on a 10G Ethernet link. The table also includes the latency added by padding if you're using ESP with padding 🍓

             +======+=========+=========+=========+=========+
             | Size | ESP+Pad | ESP+Pad | IP-TFS  | IP-TFS  |
             +======+=========+=========+=========+=========+
             | MTU  | 1500    | 9000    | 1500    | 9000    |
             +======+=========+=========+=========+=========+
             | 40   | 1.12 us | 7.12 us | 1.17 us | 7.17 us |
             +------+---------+---------+---------+---------+
             | 128  | 1.05 us | 7.05 us | 1.10 us | 7.10 us |
             +------+---------+---------+---------+---------+
             | 256  | 0.95 us | 6.95 us | 1.00 us | 7.00 us |
             +------+---------+---------+---------+---------+
             | 518  | 0.74 us | 6.74 us | 0.79 us | 6.79 us |
             +------+---------+---------+---------+---------+
             | 576  | 0.70 us | 6.70 us | 0.74 us | 6.74 us |
             +------+---------+---------+---------+---------+
             | 1442 | 0.00 us | 6.00 us | 0.05 us | 6.05 us |
             +------+---------+---------+---------+---------+
             | 1500 | 1.20 us | 5.96 us | 0.00 us | 6.00 us |
             +------+---------+---------+---------+---------+

                          Table 8: Added Latency

Look at that, the latency numbers are super similar between the two solutions 👑 But here's the main difference: IP-TFS gives you constant high bandwidth, and in some cases it even beats plain Ethernet ✨ ESP with padding, on the other hand, often greatly reduces the available bandwidth 💔

Acknowledgements 🫶

We would like to thank Don Fedyk for help in reviewing and editing this work 💖 We'd also like to thank Michael Richardson, Sean Turner, Valery Smyslov, and Tero Kivinen for reviews and lots of suggestions for improvements 🌸 And a big thank you to Joseph Touch for the transport area review and suggested improvements 🙈

Contributors 💜

The following person made significant contributions to this document.

Lou Berger LabN Consulting, L.L.C. Email: lberger@labn.net

Author's Address 🎀

Christian Hopps LabN Consulting, L.L.C. Email: chopps@chopps.org

📦 where this page comes from