RFC 5879
Heuristics for Detecting ESP-NULL Packets
RFC 5879: Heuristics for Detecting ESP-NULL Packets 🎀
bestie, this is RFC 5879 💖 It came out in May 2010 and is an Informational doc, so it's not a Standards Track spec. It's from the IETF, and the authors are T. Kivinen (AuthenTec, Inc.) and D. McDonald (Oracle Corporation). The ISSN is 2070-1721. The title block doesn't list anything it obsoletes or updates ✨
Abstract
This document describes a set of heuristics for distinguishing IPsec ESP-NULL (Encapsulating Security Payload without encryption) packets from encrypted ESP packets. These heuristics can be used on intermediate devices, like traffic analyzers and deep-inspection engines, to quickly decide whether or not a given packet flow is encrypted, meaning whether or not it can be inspected 🔍 Using these heuristics doesn't require any changes to existing IPsec hosts that are compliant with RFC 4303 🌸
Status of This Memo
okay this part is the legal-ish stuff, but it matters 🧐 This document is not an Internet Standards Track specification. It's published for informational purposes.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It went through public review and got approved for publication by the Internet Engineering Steering Group (IESG). Not every document the IESG approves is a candidate for any level of Internet Standard, so see Section 2 of RFC 5741 for that 💅
Info about the current status of this document, any errata, and how to send feedback is at http://www.rfc-editor.org/info/rfc5879 🦋
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved 🎀
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as in effect on the date of publication. Please read these carefully, because they describe your rights and restrictions for this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License ⚖️
Table of Contents
1. Introduction ....................................................3
1.1. Applicability: Heuristic Traffic Inspection and
Wrapped ESP ................................................4
1.2. Terminology ................................................4
2. Other Options ...................................................5
2.1. AH .........................................................5
2.2. Mandating by Policy ........................................6
2.3. Modifying ESP ..............................................6
3. Description of Heuristics .......................................6
4. IPsec Flows .....................................................7
5. Deep-Inspection Engine ..........................................9
6. Special and Error Cases .........................................9
7. UDP Encapsulation ..............................................10
8. Heuristic Checks ...............................................10
8.1. ESP-NULL Format ...........................................11
8.2. Self Describing Padding Check .............................12
8.3. Protocol Checks ...........................................14
8.3.1. TCP Checks .........................................15
8.3.2. UDP Checks .........................................16
8.3.3. ICMP Checks ........................................16
8.3.4. SCTP Checks ........................................17
8.3.5. IPv4 and IPv6 Tunnel Checks ........................17
9. Security Considerations ........................................17
10. References ....................................................18
10.1. Normative References .....................................18
10.2. Informative References ...................................18
Appendix A. Example Pseudocode ...................................20
A.1. Fastpath ..................................................20
A.2. Slowpath ..................................................231. Introduction ✨
okay so ESP (Encapsulating Security Payload [RFC4303]) can be used with NULL encryption [RFC2410] to get authentication, integrity protection, and optionally replay detection, but no confidentiality 🙈 ESP without encryption (called ESP-NULL) gives you similar properties to IPsec's AH (Authentication Header [RFC4302]). One reason to pick ESP-NULL over AH is that AH can't be used if there are NAT (Network Address Translation) devices on the path 💀 With AH, it'd be easy to spot packets that only have authentication and integrity protection, since AH has its own protocol number and a deterministic packet length. With ESP-NULL, that detection is nondeterministic, even though the base ESP packet format is fixed 😤
Sometimes intermediate devices want to spot ESP-NULL packets so they can do deep inspection or enforce access control. That kind of deep inspection covers virus detection, spam filtering, and intrusion detection 🛡️ Since end nodes could dodge those checks by using encrypted ESP instead of ESP-NULL, these scenarios also need very specific policies to forbid that kind of workaround. Not it 🙅♀️
These policy needs usually mean the whole network has to be controlled, meaning it's under the same administrative domain. Setups like this are usually limited to the inside of one enterprise or organization, and encryption isn't used because the network is considered safe enough from eavesdroppers 🥰
Because the traffic being inspected is usually host-to-host traffic inside one organization, transport mode IPsec is typically what's used. Just so you know, most current uses of IPsec are not host-to-host traffic inside one organization. But for the intended use cases of these heuristics, that will most likely be the case 🧚 Also, tunnel mode is way easier to deal with than transport mode, since it's much easier to detect the IP header inside the ESP-NULL packet 💅
Even if new protocol changes make ESP-NULL easier to detect in the future, this document will still help with the transition for older end-systems. That way, a solution can be deployed right away instead of after 5-10 years of upgrades and rollout 🚀 Even with protocol changes on end nodes, intermediate devices will still need heuristics until they can assume those changes exist on every end device. To make sure any solution doesn't break later on, it's best to document these heuristics. In other words, publish an RFC for what to do now, even if a new protocol shows up later that solves the same problem in a better way 🫶
1.1. Applicability: Heuristic Traffic Inspection and Wrapped ESP
There are two ways for intermediate security devices to tell encrypted and unencrypted ESP traffic apart 👀
- The heuristics approach has the intermediate node inspect the unchanged ESP traffic, to figure out with extremely high probability whether or not the traffic stream is encrypted.
- The Wrapped ESP (WESP) approach [RFC5840], in contrast, requires the ESP endpoints to be modified to support the new protocol. WESP lets the intermediate node tell encrypted and unencrypted traffic apart deterministically, using a simpler implementation for the intermediate node ✨
Both approaches are being documented at the same time by the IPsecME Working Group. WESP is going on the Standards Track, while the heuristics approach is being published as an Informational RFC. While endpoints are being modified to adopt WESP, both approaches will likely coexist for years, because the heuristic approach is needed to inspect traffic where at least one endpoint hasn't been modified 🥹 In other words, intermediate nodes are expected to support both approaches to get good security and performance during the transition period 🌈
1.2. Terminology
This document uses the following terminology 📖
Flow
A TCP/UDP or IPsec flow is a stream of packets that are part of the same TCP/UDP or IPsec stream. In other words, a TCP or UDP flow is a stream of packets having the same 5 tuple (source and destination IP and port, plus the TCP/UDP protocol). Note that this kind of flow is also called a microflow in some documents 🦋
Flow Cache
Deep-inspection engines and similar devices use a cache of the flows going through the device, and that cache keeps state for all flows going through the device 💜
IPsec Flow
An IPsec flow is a stream of packets sharing the same source IP, destination IP, protocol (ESP/AH), and Security Parameter Index (SPI). Strictly speaking, the source IP doesn't need to be part of the flow identification, but it can be. For this reason, it's safer to assume the source IP is always part of the flow identification 🔥
2. Other Options 🌸
This document goes into the heuristic approach for detecting ESP-NULL packets. There are some other options you could use, and this section briefly talks about them 🍓
2.1. AH
The most logical approach would use the protocol that's already defined to offer authentication and integrity protection without confidentiality, which is AH. AH traffic is clearly marked as not encrypted, so intermediate devices can always inspect it 👑
Using AH has two problems. First, since it also protects the IP headers, it protects against NATs on the path too. That means it won't work if there's a NAT between the end nodes. In some environments that might not be a problem, but other environments, including heavy NAT use even inside an enterprise or organization's internal network, do run into it 😭 NAT-Traversal (NAT-T, [RFC3948]) could have been extended to support AH too. Early versions of the NAT-T proposals did include that, but it was left out because it wasn't seen as necessary 🙄
The second problem is that in the new IPsec Architecture [RFC4301], AH support is now optional, which means not every implementation supports it. ESP-NULL was made mandatory to implement by "Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH)" [RFC4835] 🧠
AH also has quite complex processing rules for calculating the Integrity Check Value (ICV) compared to ESP. Those include things like zeroing out mutable fields. Plus, since AH isn't as widely used as ESP, AH support hasn't been tested as thoroughly in interoperability events 🥴
part 1 done bestie 🎀 part 2 is coming next, so stay tuned 💖
2.2. Mandating by Policy 📜✨
okay so the easiest way to fix this one is to mandate ESP-NULL with common parameters across a whole organization 🏢💅 Then the heuristics either become unnecessary (if no ESP-encrypted traffic is allowed at all) or way simpler (only one set of parameters needs inspecting). For example, everybody in the org using ESP-NULL has to use HMAC-SHA-1-96 as their integrity algorithm 🔐 This works unless one of the communicating machines is outside the same administrative domain as the deep-inspection engine 🙈 IPsec Security Associations (SAs) have to be satisfactory to all communicating parties, so only one peer needs a sufficiently narrow policy for it to break down 💀 Also, this kind of solution might need some centralized policy management so that everybody in an administrative domain uses the same policy, and so that changes to that single policy can be coordinated across the whole domain 🫶
2.3. Modifying ESP 🦋
several documents talk about changing ESP so intermediate devices can tell when an ESP packet is using NULL encryption 👀 The methods that have been discussed are: adding an IP-option, adding a new IP-protocol number plus an extra header [RFC5840], adding new IP-protocol numbers that signal the ESP-NULL parameters [AUTH-ONLY-ESP], reserving an SPI range for ESP-NULL [ESP-NULL], and using UDP encapsulation with a different format and ports 🌸
lowkey the problem is that all of these docs require changing ESP itself, which means every end node has to be updated before intermediate devices can assume the new ESP format is in use 😭 Updating end nodes takes a LOT of time 💀 Look at Internet Key Exchange Protocol version 2 (IKEv2) as an example of how slow end-node deployment can be. If you wanted an implementation that needs both IKEv2 and a new ESP format, it would take several years, possibly as long as a decade, before it was widely deployed 🕰️
3. Description of Heuristics 🔍💜
the heuristics for detecting ESP-NULL packets only need changes on the intermediate devices that do deep inspection or other operations requiring ESP-NULL detection 🛡️ Those nodes need changes no matter which ESP-NULL method you pick, so updating intermediate nodes is unavoidable anyway ✨ The heuristics don't require updates or modifications to any other devices on the rest of the network, especially end nodes 🙌
in this document, we assume the affected intermediate node acts as a stateful interception device 🧚 That means it keeps state on the IPsec flows going through it, where a flow is defined by the ESP SPI and the IP addresses forming an IPsec SA 🎀 The heuristics can also run without storing any state, but performance gets worse, since heuristic checks have to happen for every packet instead of once per flow 🥲 This also affects how reliable the heuristics are 💅
usually, an intermediate node only runs heuristics on the first few packets of a new flow (aka the new IPsec SA) ⭐ After those first few packets, the node has detected the parameters of the IPsec flow, so it skips detection heuristics and can do direct packet-inspecting actions based on its own policy 💖 Once detected, ESP-NULL packets will never be detected as encrypted ESP packets, which means valid ESP-NULL packets will never bypass deep inspection 🙅♀️
the only failure mode of these heuristics is assuming encrypted ESP packets are ESP-NULL packets, which means completely random packet data gets deeply inspected 😵💫 An attacker can easily send random-looking ESP-NULL packets that make the heuristics detect them as encrypted ESP, but that's no worse than sending non-ESP fuzz through an intermediate node 🫠 The only way an ESP-NULL flow can get mistaken for an encrypted ESP flow is if the ESP-NULL flow uses an authentication algorithm the packet inspector has no knowledge of 🤷♀️
for hardware implementations, all the flow lookups based on the ESP next header number (50), source address, destination address, and SPI can be done by the hardware (there's usually similar functionality there already, for TCP/UDP flows) 🔧 The heuristics can be implemented in hardware too, but using software lets you update faster when new protocol modifications come out or when new protocols need support 🚀
as described in Section 7, UDP-encapsulated ESP traffic may also have Network Address Port Translation (NAPT) applied to it, so there's already a 5-tuple state in the stateful inspection gateway 🌈
4. IPsec Flows 🎀🦋
ESP is a stateful protocol, which means there's state stored in both end nodes of the ESP IPsec SA, and that state is identified by the pair of destination IP and SPI 💜 End nodes also often fix the source IP address in an SA unless the destination is a multicast group 🫶 Most flows an intermediate device cares about are unicast, so it's safer to assume the receiving node also uses a source address, and the intermediate device should do the same ✅ Sometimes this creates extra cached ESP IPsec SA flows, but by using the source address, two distinct flows will never get mixed up 💅 For sites that use multicast a lot, that traffic is deterministically identifiable (224.0.0.0/4 for IPv4 and ff00::0/8 for IPv6), and an implementation can save the space of multiple cache entries for a multicast flow by checking the destination address first 🧠
when the intermediate device sees a new ESP IPsec flow, meaning a new flow of ESP packets where the source address, destination address, and SPI number form a triplet that hasn't been cached, it starts the heuristics to figure out whether this flow is ESP-NULL 🕵️♀️ These heuristics are covered in Section 8 ✨
when the heuristics finish, they label the flow as either encrypted (which means its packets are encrypted and cannot be ESP-NULL packets) or as ESP-NULL 🏷️ That label, along with the ESP-NULL parameters the heuristics detected, gets stored in a flow cache, which is used later when processing packets of the same flow 💾
both encrypted ESP and ESP-NULL flows are processed based on local policy 🌸 In normal operation, encrypted ESP flows are passed through or dropped according to local policy, and ESP-NULL flows are passed to the deep-inspection engine 🔍 Local policy also decides other packet-processing parameters 💅 Local policy issues are clearly marked in this document to make implementation easier 📌
sometimes the heuristics can't determine the flow type from a single packet, so they might need multiple packets before they can finish 🥺 In those cases, the heuristics return "unsure" status, and the packet is processed based on local policy and the flow cache is updated with "unsure" status ⏳ Local policy for "unsure" packets can range from dropping (which encourages end-node retransmission) to queuing (which may preserve delivery, at the cost of artificially inflating round-trip times if they're measured) 🎢 When the next packet for the flow arrives, it gets heuristically processed again, and the cached flow may stay "unsure", be marked as ESP, or be marked as an ESP-NULL flow 🔁
there are several reasons a single packet might not be enough to detect the flow type 🫠 One is that the next header number was unknown, meaning that if the heuristics don't know about the packet's protocol, they can't verify they've properly detected the ESP-NULL parameters, even when the packet otherwise looks like ESP-NULL 🤔 If the packet doesn't look like ESP-NULL at all, the encrypted ESP status can be returned quickly ⚡ Since ESP-NULL heuristics need to know the same protocols as a deep-inspection device, an ESP-NULL instance of an unknown protocol can be handled the same way as a cleartext instance of that same unknown protocol 🧚
5. Deep-Inspection Engine 🕵️💅
a deep-inspection engine running on an intermediate node usually checks deep into the packet and makes policy decisions based on the packet's contents 🔎 The deep-inspection engine should be able to tell the difference between success, failure, and garbage 💖 Success means the packet was successfully checked, passed the checks, and is allowed to be forwarded ✅ Failure means the packet was successfully checked, but the checks indicated it should be dropped, like if it contained a virus, was a known attack, or something similar 🚫
garbage means the packet's protocol headers or other portions couldn't be parsed 🗑️ For the heuristics, it would be super useful if the deep-inspection engine could tell garbage and failure apart, since garbage cases can be used to detect certain errors (like where the ESP-NULL parameters are wrong, or the flow is really an encrypted ESP flow and not an ESP-NULL flow) 🧐
if the deep-inspection engine only returns failure for all garbage packets, on top of the real failure cases, then a system implementing the ESP-NULL heuristics can't recover from error situations quickly 😭
6. Special and Error Cases 🚨🌈
there's a small probability that an encrypted ESP packet (which looks like it's full of completely random bytes) has plausible bytes in the expected locations, so the heuristics detect it as an ESP-NULL packet instead of as encrypted ESP 🎲 The actual probabilities get computed later in this document 🧮 A packet like that won't cause problems, since the deep-inspection engine will most likely reject it and return that it's garbage 🙌 If the deep-inspection engine is rejecting a high number of packets as garbage, it might mean the original ESP-NULL detection for the flow was wrong (an encrypted ESP flow got improperly detected as ESP-NULL) 🫣 In that case, the cached flow should be invalidated and discovery should happen again 🔄
each ESP-NULL flow should also keep statistics on how many packets were detected as garbage by deep inspection, how many passed checks, and how many failed checks with policy violations (meaning they failed because of actual inspection policy failures, not because the packet looked like garbage) 📊 If the number of garbage packets suddenly goes up (like most packets starting to look like garbage to the deep-inspection engine), the old ESP-NULL SA may have been replaced by an encrypted ESP SA with an identical SPI 😱 If both ends use random SPI generation, this is a very unlikely situation (1 in 2^32), but some nodes do reuse SPI numbers (for example, a 32-bit memory address of the SA descriptor), so this situation needs to be handled 🫶
actual limits for cache invalidation are local policy decisions 💅 Sample invalidation policies include: 50% of packets marked as garbage within a second, or, if a deep-inspection engine can't tell garbage from failure, failing more than 95% of packets in the last 10 seconds 📉 For implementations that don't distinguish between garbage and failure, failures shouldn't be treated too quickly as an indication of SA reuse 🥹 Often, single packets cause state-related errors that block otherwise normal packets from passing 💀
7. UDP Encapsulation ✨
okay bestie, so the flow lookup code also has to catch UDP packets going to or coming from port 4500, not just the ESP ones 🦋 Once it skips past the UDP header, it does pretty much the same processing it does for ESP. 💅
Here's the tea. NAT port translation loves to rewrite 4500 into some other number 🙈 That means every unique port pair is its own separate IPsec flow. So UDP-encapsulated IPsec flows get identified by the source and destination IP, the source and destination port numbers, and the SPI number. Literally that's the whole ID card 🪪
Also, devices might be using IKEv2 Mobility and Multihoming (MOBIKE) ([RFC4555]) 🌸 which means the flow cache should be shared between UDP-encapsulated IPsec flows and non-encapsulated ones. Like, they're all one big happy family 🫶 And as we said before, telling garbage apart from actual policy failures is going to help detection immensely 🔥
Now the sneaky part 👀 The checks run for packets that have just source port 4500, or just destination port 4500. So they might end up running on non-ESP traffic too 💀 Some traffic can randomly use port 4500 for other reasons, especially when a port-translating NAT is in the mix. The UDP encapsulation processing should be aware of that possibility, obviously. 🧚
8. Heuristic Checks 💜
okay this part is so satisfying 😭 Normally, HMAC-SHA1-96 or HMAC-MD5-96 gives 1 out of 2^96 probability that a random packet will pass the Hashed Message Authentication Code (HMAC) test. That means an end node has a 99.999999999999999999999999998% probability of correctly spotting a random packet as invalid. 🎀 So it should be enough for an intermediate device to check around 96 bits from the input packet.
By comparing those bits against known values for the packet, a deep-inspection engine ends up with more or less the same probability the end node is using ✨ That gives us an upper limit on how many bits heuristics need to check. Checking much more than that is pointless, because that same probability is already acceptable for the end node 🙅♀️ In most cases, though, the intermediate device doesn't need probability that high. Something around 32-64 bits is usually enough 🍓
ESP has a really well-understood packet layout, honestly iconic 👑 But its variable-length fields mean pure algorithmic matching isn't enough anymore, so we need heuristics and probabilities instead 💖
8.1. ESP-NULL Format 🧚
The ESP-NULL format looks like this, bestie:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Security Parameter Index (SPI) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IV (optional) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Payload Data (variable) |
~ ~
| |
+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Padding (0-255 bytes) |
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Pad Length | Next Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Integrity Check Value (variable) |
~ ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1The output of the heuristics should tell you whether the packet is encrypted ESP or ESP-NULL 🔍 If it's ESP-NULL, the heuristics should also give you the Integrity Check Value (ICV) field length and the Initialization Vector (IV) length. ✨
The ESP authentication algorithms defined so far have 4 different ICV lengths, which is a whole vibe 🌈 Here's the breakdown:
Different ICV lengths for different algorithm:
Algorithm ICV Length
------------------------------- ----------
AUTH_HMAC_MD5_96 96
AUTH_HMAC_SHA1_96 96
AUTH_AES_XCBC_96 96
AUTH_AES_CMAC_96 96
AUTH_HMAC_SHA2_256_128 128
AUTH_HMAC_SHA2_384_192 192
AUTH_HMAC_SHA2_512_256 256
Figure 2Besides those ESP authentication algorithms, there's also the encryption algorithm ENCR_NULL_AUTH_AES_GMAC 🎀 It doesn't give you confidentiality, but it does give you authentication, just like ESP-NULL. Its ICV Length is 128 bits, and it also needs 8 bytes of IV.
On top of the ICV length, there are two possible IV lengths: 0 bytes (the default) and 8 bytes (for ENCR_NULL_AUTH_AES_GMAC) 💅 Figuring out the IV length means understanding the payload, meaning the actual protocol data (TCP, UDP, and so on). That's needed so you can tell the optional IV apart from the real protocol data. How easy that is depends on how the IV was generated 🤔 If the IV is generated with a method that makes random-looking data (like an encrypted counter), then telling it apart from protocol data is pretty easy. If the IV is a counter or some other non-random value, there's a lot more room for mistakes 🙈
Now, if the protocol (aka the "next header") of the packet isn't one the heuristics support, then figuring out the IV length is impossible 💀 So the heuristics can't finish. In that situation they return "unsure" and ask for more packets.
This document does not cover RSA authentication in ESP ([RFC4359]), since that's considered beyond its scope. 🚫
8.2. Self Describing Padding Check 🫧
Before we can get to the next header field, we have to measure the ICV length first 📏 Four different ICV lengths mean there are four possible places where the pad length and padding could be. Implementations must be careful when they try the larger ICV sizes, so the bytes they inspect don't end up being data that isn't payload data 👀 For example, a 10-byte ICMP echo request has zero-length padding, but checks for 256-bit ICVs would be looking at sequence number or SPI data if the packet actually has a 96-bit or 128-bit ICV.
ICV lengths should always be checked from shortest to longest 🍭 It's way more likely to find valid-looking padding bytes in the cleartext part of the payload than in the ICV field of a longer ICV than the one currently being inspected. For instance, if a packet has a 96-bit ICV and the implementation checks for a 256-bit ICV first, the cleartext part of the payload might contain valid-looking bytes 😳 If it goes the other way around, a packet with a 256-bit ICV that gets checked for a 96-bit ICV first means the bytes being inspected are part of the longer ICV field, so they should look indistinguishable from random noise ✨
Every ESP packet always has between 0-255 bytes of padding, and the payload, pad length, and next header are always right-aligned within a 4-byte boundary 🧁 Normally, implementations use the minimum amount of padding. But the heuristics method would be even more reliable if some extra padding got added. The actual padding data starts at 01 and goes up to the pad length. So the exact padding and pad length bytes for 4 bytes of padding would be 01 02 03 04 04 🌸
There are two cases of ESP-NULL padding: matched bytes (like the 04 04 shown above), or the 0-byte padding case 🎯 When there's one or more bytes of padding, a node can run a super simple and fast test, which is just checking for a sequence of N N in any of those four locations. With four 2-byte locations (assuming the packet size lets all four possible ICV lengths fit), the upper-bound probability of finding a random encrypted packet that shows non-zero length ESP-NULL properties is:
1 - (1 - 255 / 65536) ^ 4 == 0.015 == 1.5%When there are 0 bytes of padding, a random encrypted ESP packet has:
1 - (1 - 1 / 256) ^ 4 == 0.016 == 1.6%.Put together, both cases give a 3.1% upper-bound chance of misclassifying an encrypted packet as an ESP-NULL packet 😬
In the matched bytes case, you can go further and inspect things (counting the pad bytes backward and downward from the pad-length match) to cut down the number of misclassified packets even more 🔎 A padding length of 255 means a specific 256^254 sequence of bytes has to show up, which basically eliminates pairs of FF FF as viable ESP-NULL padding. Every one of the 255 pairs for padding length N only has a 1 / 256^N probability of being correct ESP-NULL padding. That shrinks the 1.5% of matched pairs we talked about earlier down to basically nothing 🫠
At this point, a maximum of 1.6% of possible byte values is left, so the next header number gets inspected 💖 If the next header number is known (and supported), the packet can be inspected based on that number. If it's unknown (meaning it's not one of the ones with protocol checking support), the packet gets marked "unsure", because there's no way to figure out the IV length without inspecting the inner protocol payload 🤷♀️
There are six next header values in common use: TCP (6), UDP (17), ICMP (1), Stream Control Transmission Protocol (SCTP) (132), IPv4 (4), and IPv6 (41) 🌟 And if IPv6 is in heavy use, that goes up to nine, adding Fragment (44), ICMPv6 (58), and IPv6 options (60) 🚀 To make sure no packet gets misread as an encrypted ESP packet when it's actually an ESP-NULL packet, a packet can't be marked as a failure just because its next header number is one that isn't known and supported. In those cases, the packets get marked as "unsure" instead 🫶
An intermediate node's policy can still help catch an ESP-NULL flow even when the protocol isn't a common one 🕵️♀️ The node counts how many "unsure" results the heuristics have returned, and after it sees a consistent, but unknown, next header number in the same location (likely with the same ICV length), it can conclude the flow has a high probability of being ESP-NULL. That's because it's really unlikely that so many packets would pass the integrity check at the destination unless they're legit 🔥 The flow can then be classified as ESP-NULL with a known ICV length but an unknown IV length.
Thankfully, in unknown protocol cases the IV length doesn't matter 🙌 If the protocol is unknown to the heuristics, it'll most likely be unknown to the deep-inspection engine too. So it's really important that the heuristics support at least the same protocols the deep-inspection engine does 💅 Once any inner next header number that's known by the heuristics (and the deep-inspection engine) shows up, the heuristics can detect the IV length properly ✨
8.3. Protocol Checks ✨
Okay so generic protocol checking is SO much easier when the device already has state 💅🌸. Say many TCP/UDP flows are set up over one IPsec SA. A rekey makes a brand-new SA that needs heuristics to figure out its parameters, and those heuristics totally benefit from the TCP/UDP flows that were already running in the previous IPsec SA 🦋. So all you have to check is whether the new IPsec SA is carrying packets that belong to flows of some other IPsec SA (the previous one, before the rekey), and whether the deep-inspection engine already knows those flows. If it does, that's a strong indication that the new SA is really ESP-NULL 🎀✨.
The worst case is when an end node starts talking all on its own, i.e., it has no previous flows through the device 😭. The heuristics will run on the first few packets it sends. The later subsections mainly cover these start-up cases, because they're the hardest 💀.
Protocol checks come in two flavors, bestie 💖. The first is packet validity: certain spots must contain specific values. For example, the inner IPv4 header of an IPv4 tunnel packet must have its 4-bit version number set to 4 🧚. If it doesn't, the packet isn't valid and can be marked as a failure. Other positions, which depend on ICV and IV lengths, must also be checked, and if all of them are failures, then the packet is a failure. If any of the checks come back "unsure," the packet is marked as such 🤷♀️.
The second type is for variable but easy-to-parse values 🍓. Take the 4-bit header length field of an inner IPv4 packet. It has a fixed value (5) as long as there are no inner IPv4 options. If the header length has that specific value, the number of known "good" bits goes up. If it has some other value, the known "good" bit count stays the same. A local policy might include reaching a bit count over a threshold (for example, 96 bits), which causes a packet to be marked as valid 🏆.
8.3.1. TCP Checks
When the first TCP packet hits the heuristics, it's most likely going to be the SYN packet of the new connection, so it has less useful info than later packets might 🥺. The best valid packet checks are checking that the header length and flags have valid values, and checking source and destination port numbers. Those port numbers can be used for heuristics in some cases, but in general they can't be reliably told apart from random numbers, apart from some well-known ports like 25/80/110/143 🌈.
The most obvious field, the TCP checksum, might not be usable 😬. The packet may have already gone through a NAT box that changed the IP addresses, but that box assumed any ESP payload was encrypted and didn't fix the transport checksums for the new IP addresses. So the IP numbers used in the checksum are wrong; thus, the checksum is wrong. If the checksum is correct, it can again increase the valid bit count. Verifying checksums is a costly operation, though, so skipping that check might be best unless there's hardware to help with the calculation 🧮. Window size, urgent pointer, sequence number, and acknowledgment numbers can be used, but there isn't one specific known value for them 🫠.
Here's one good detection method 🔁: if a packet is dropped, the next packet will most likely be a retransmission of the previous packet. So if two packets are received with the same source and destination port numbers, and their sequence numbers are either the same or right after each other, then it's likely a TCP packet has been correctly detected ✅. This heuristic is most helpful when only one packet is outstanding. For example, if a TCP SYN packet is lost (or dropped because of policy), the next packet would always be a retransmission of the same TCP SYN packet 💖.
Existing deep-inspection engines usually do very good TCP flow checking already, including flow tracking, verification of sequence numbers, and reconstruction of the whole TCP flow 👑. Similar methods can be used here, but they're implementation dependent and not described here 🙈.
8.3.2. UDP Checks
The UDP header has even more problems than the TCP header, because UDP has even less known data 😭. The checksum has the same problem as the TCP checksum, due to NATs. The UDP length field might not match the overall packet length, because the sender is allowed to include TFC (traffic flow confidentiality; see Section 2.7 of "IP Encapsulating Security Payload" [RFC4303]) padding 🫶.
With UDP, similar multiple-packet methods can be used as with TCP. UDP protocols usually include several packets using the same port numbers going from one end node to another, so receiving multiple packets with a known pair of UDP port numbers is a good indication that the heuristics have passed ✨.
Some UDP protocols also use identical source and destination port numbers, so that's a good check too 🌸.
8.3.3. ICMP Checks
ICMP messages are usually sent as return packets for other packets, so they're not very common as the first packets for the SA. The ICMP ECHO_REQUEST message is a noteworthy exception 🧚. ICMP ECHO_REQUEST has a known type, code, identifier, and sequence number. The checksum, however, might be wrong again because of NATs 🙃.
For ICMP error messages, the ICMP message contains part of the original IP packet inside. Then, the same rules that are used to detect IPv4/IPv6 tunnel checks can be used 🔍.
8.3.4. SCTP Checks
SCTP [RFC4960] has a self-contained checksum, which is computed over the SCTP payload and is not affected by NATs unless the NAT is SCTP-aware 🦋. Even more than the TCP and UDP checksums, the SCTP checksum is expensive, and may be prohibitive even for deep packet inspections 💸.
SCTP chunks can be inspected to see if their lengths are consistent across the total length of the IP datagram, so long as TFC padding is not present 📏.
8.3.5. IPv4 and IPv6 Tunnel Checks
With tunneled traffic, the packet inside contains a full IPv4 or IPv6 packet, so there are lots of usable fields 🎀. For IPv4, those include version, header length, total length (again, TFC padding might confuse things there), protocol number, and 16-bit header checksum 🧁. In those cases, the intermediate device should give the decapsulated IP packet to the deep-inspection engine.
IPv6 has fewer usable fields, but the version number, packet length (modulo TFC confusion), and next header can all be used by deep packet inspection 🌈.
If all traffic going through the intermediate device is either from or to certain address blocks (for example, either to or from the company intranet prefix), this can also be checked by the heuristics 🏢.
9. Security Considerations ✨
Heads up, bestie: this section is serious business 🚨
Attackers can always bypass ESP-NULL deep packet inspection by using encrypted ESP (or some other encryption or tunneling method) instead, unless the intermediate node's policy requires dropping packets that it cannot inspect 😈. Ultimately, the responsibility for performing deep inspection, or allowing intermediate nodes to perform deep inspection, must rest on the end nodes 🫡. That is, if a server allows encrypted connections also, then an attacker who wants to attack the server and wants to bypass a deep-inspection device in the middle will use encrypted traffic. This means that the protection of the whole network is only as good as the policy enforcement and protection of the end node 🥀. One way to enforce deep inspection for all traffic is to forbid encrypted ESP completely, in which case ESP-NULL detection is easier, as all packets must be ESP-NULL based on the policy (heuristics may still be needed to find out the IV and ICV lengths, unless further policy restrictions eliminate the ambiguities) 🔐.
Section 3 discusses failure modes of the heuristics. An attacker can poison flows, tricking inspectors into ignoring legitimate ESP-NULL flows, but that is no worse than injecting fuzz 🐛.
Forcing the use of ESP-NULL everywhere inside the enterprise, so that accounting, logging, network monitoring, and intrusion detection all work, increases the risk of sending confidential information where eavesdroppers can see it 👀.
10. References ✨
10.1. Normative References
- [RFC2410] Glenn, R. and S. Kent, "The NULL Encryption Algorithm and Its Use With IPsec", RFC 2410, November 1998.
- [RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.
- [RFC4302] Kent, S., "IP Authentication Header", RFC 4302, December 2005.
- [RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", RFC 4303, December 2005.
10.2. Informative References
- [AUTH-ONLY-ESP] Hoffman, P. and D. McGrew, "An Authentication-only Profile for ESP with an IP Protocol Identifier", Work in Progress, August 2007.
- [ESP-NULL] Bhatia, M., "Identifying ESP-NULL Packets", Work in Progress, December 2008.
- [RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M. Stenberg, "UDP Encapsulation of IPsec ESP Packets", RFC 3948, January 2005.
- [RFC4359] Weis, B., "The Use of RSA/SHA-1 Signatures within Encapsulating Security Payload (ESP) and Authentication Header (AH)", RFC 4359, January 2006.
- [RFC4555] Eronen, P., "IKEv2 Mobility and Multihoming Protocol (MOBIKE)", RFC 4555, June 2006.
- [RFC4835] Manral, V., "Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH)", RFC 4835, April 2007.
- [RFC4960] Stewart, R., "Stream Control Transmission Protocol", RFC 4960, September 2007.
- [RFC5840] Grewal, K., Montenegro, G., and M. Bhatia, "Wrapped Encapsulating Security Payload (ESP) for Traffic Visibility", RFC 5840, April 2010.
Appendix A. Example Pseudocode ✨
This appendix is meant for the implementors 🛠️. It does not include all the required checks, and this is just example pseudocode, so a final implementation can be very different 🙃. It mostly lists things that need to be done, but implementations can optimize steps depending on their other parts 🧩. For example, an implementation might combine heuristics and deep inspection tightly together 🔥.
A.1. Fastpath 🚀
okay so this bit is the fastpath example pseudocode, which is the part of the packet processing engine that's usually built into hardware ⚡ it's giving speed, bestie. first up, the variables the code uses 💅
////////////////////////////////////////////////////////////
// This pseudocode uses following variables:
//
// SPI_offset: Number of bytes between start of protocol
// data and SPI. This is 0 for ESP and
// 8 for UDP-encapsulated ESP (i.e, skipping
// UDP header).
//
// IV_len: Length of the IV of the ESP-NULL packet.
//
// ICV_len: Length of the ICV of the ESP-NULL packet.
//
// State: State of the packet, i.e., ESP-NULL, ESP, or
// unsure.
//
// Also following data is taken from the packet:
//
// IP_total_len: Total IP packet length.
// IP_hdr_len: Header length of IP packet in bytes.
// IP_Src_IP: Source address of IP packet.
// IP_Dst_IP: Destination address of IP packet.
//
// UDP_len: Length of the UDP packet taken from UDP header.
// UDP_src_port: Source port of UDP packet.
// UDP_dst_port: Destination port of UDP packet.
//
// SPI: SPI number from ESP packet.
//
// Protocol: Actual protocol number of the protocol inside
// ESP-NULL packet.
// Protocol_off: Calculated offset to the protocol payload data
// inside ESP-NULL packet.Now the main processing code 🎀 It checks whether the packet needs ESP processing at all, and sends it down the right path from there.
////////////////////////////////////////////////////////////
// This is the main processing code for the packet
// This will check if the packet requires ESP processing,
//
Process packet:
* If IP protocol is ESP
* Set SPI_offset to 0 bytes
* Goto Process ESP
* If IP protocol is UDP
* Goto Process UDP
* If IP protocol is WESP
// For information about WESP processing, see WESP
// specification.
* Continue WESP processing
* Continue Non-ESP processingUDP packets get their own check 🦋 This code figures out whether a UDP packet is actually UDP-encapsulated UDP, UDP-encapsulated IKE, or a keepalive. Reassembly isn't strictly required here, and the author explains why: they could reassemble only after spotting UDP-encapsulated ESP, but that would make the pseudocode way messier, since it'd have to check whether the UDP header is in the packet. So they just reassemble first to keep things simple 🧚
////////////////////////////////////////////////////////////
// This code is run for UDP packets, and it checks if the
// packet is UDP encapsulated UDP packet, or UDP
// encapsulated IKE packet, or keepalive packet.
//
Process UDP:
// Reassembly is not mandatory here, we could
// do reassembly also only after detecting the
// packet being UDP encapsulated ESP packet, but
// that would complicate the pseudocode here
// a lot, as then we would need to add code
// for checking whether or not the UDP header is in this
// packet.
// Reassembly is to simplify things
* If packet is fragment
* Do full reassembly before processing
* If UDP_src_port != 4500 and UDP_dst_port != 4500
* Continue Non-ESP processing
* Set SPI_offset to 8 bytes
* If UDP_len > 4 and first 4 bytes of UDP packet are 0x000000
* Continue Non-ESP processing (pass IKE-packet)
* If UDP_len > 4 and first 4 bytes of UDP packet are 0x000002
* Continue WESP processing
* If UDP_len == 1 and first byte is 0xff
* Continue Non-ESP processing (pass NAT-Keepalive Packet)
* Goto Process ESPNext up, the ESP check 💖 This one looks at whether the IPsec flow is already known. If it isn't, it calls the heuristics. If it is, it keeps going based on policy.
////////////////////////////////////////////////////////////
// This code is run for ESP packets (or UDP-encapsulated ESP
// packets). This checks if IPsec flow is known, and
// if not calls heuristics. If the IPsec flow is known
// then it continues processing based on the policy.
//
Process ESP:
* If packet is fragment
* Do full reassembly before processing
* If IP_total_len < IP_hdr_len + SPI_offset + 4
// If this packet was UDP encapsulated ESP packet then
// this might be valid UDP packet that might
// be passed or dropped depending on policy.
* Continue normal packet processing
* Load SPI from IP_hdr_len + SPI_offset
* Initialize State to ESP
// In case this was UDP encapsulated ESP, use UDP_src_port and
// UDP_dst_port also when finding data from SPI cache.
* Find IP_Src_IP + IP_Dst_IP + SPI from SPI cache
* If SPI found
* Load State, IV_len, ICV_len from cache
* If SPI not found or State is unsure
* Call Autodetect ESP parameters (drop to slowpath)
* If State is ESP
* Continue Non-ESP-NULL processing
* Goto Check ESP-NULL packetESP-NULL packets 🌈 This section works out the protocol number and the offset to the data, which the deep-inspection engine needs, and then hands the packet over to that engine 👑
////////////////////////////////////////////////////////////
// This code is run for ESP-NULL packets, and this
// finds out the data required for deep-inspection
// engine (protocol number, and offset to data)
// and calls the deep-inspection engine.
//
Check ESP-NULL packet:
* If IP_total_len < IP_hdr_len + SPI_offset + IV_len + ICV_len
+ 4 (spi) + 4 (seq no) + 4 (protocol + padding)
// This packet was detected earlier as being part of
// ESP-NULL flow, so this means that either ESP-NULL
// was replaced with other flow or this is an invalid packet.
// Either drop or pass the packet, or restart
// heuristics based on the policy
* Continue packet processing
* Load Protocol from IP_total_len - ICV_len - 1
* Set Protocol_off to
IP_hdr_len + SPI_offset + IV_len + 4 (spi) + 4 (seq no)
* Do normal deep inspection on packet.Figure 3
A.2. Slowpath 🐢
okay now for the slowpath 💅 this is the heuristics part of the engine, and it's usually written in software. it's where the real detective work happens, so it's giving main character energy 🔥 first, the variables it uses (plenty of them come straight from the fastpath 🦋)
////////////////////////////////////////////////////////////
// This pseudocode uses following variables:
//
// SPI_offset, IV_len, ICV_len, State, SPI,
// IP_total_len, IP_hdr_len, IP_Src_IP, IP_Dst_IP
// as defined in fastpath pseudocode.
//
// Stored_Check_Bits:Number of bits we have successfully
// checked to contain acceptable values
// in the actual payload data. This value
// is stored/retrieved from SPI cache.
//
// Check_Bits: Number of bits we have successfully
// checked to contain acceptable values
// in the actual payload data. This value
// is updated during the packet
// verification.
//
// Last_Packet_Data: Contains selected pieces from the
// last packet. This is used to compare
// certain fields of this packet to
// same fields in previous packet.
//
// Packet_Data: Selected pieces of this packet, same
// fields as Last_Packet_Data, and this
// is stored as new Last_Packet_Data to
// SPI cache after this packet is processed.
//
// Test_ICV_len: Temporary ICV length used during tests.
// This is stored to ICV_len when
// padding checks for the packet succeed
// and the packet didn't yet have unsure
// status.
//
// Test_IV_len: Temporary IV length used during tests.
//
// Pad_len: Padding length from the ESP packet.
//
// Protocol: Protocol number of the packet inside ESP
// packet.
//
// TCP.*: Fields from TCP header (from inside ESP)
// UDP.*: Fields from UDP header (from inside ESP)Now the actual heuristics kick off 🔍 The fastpath has already loaded State, ICV_len, and IV_len if the flow was found in the SPI cache, which is the case when the flow was marked unsure. The autodetect code checks the next packet against the IV/ICV_len pair it already picked. If that doesn't work out, it tries the other combinations, going from shortest ICV to longest.
////////////////////////////////////////////////////////////
// This code starts the actual heuristics.
// During this the fastpath has already loaded
// State, ICV_len, and IV_len in case they were
// found from the SPI cache (i.e., in case the flow
// had unsure status).
//
Autodetect ESP parameters:
// First, we check if this is unsure flow, and
// if so, we check next packet against the
// already set IV/ICV_len combination.
* If State is unsure
* Call Verify next packet
* If State is ESP-NULL
* Goto Store ESP-NULL SPI cache info
* If State is unsure
* Goto Verify unsure
// If we failed the test, i.e., State
// was changed to ESP, we check other
// ICV/IV_len values, i.e., fall through
// ICV lengths are tested in order of ICV lengths,
// from shortest to longest.
* Call Try standard algorithms
* If State is ESP-NULL
* Goto Store ESP-NULL SPI cache info
* Call Try 128bit algorithms
* If State is ESP-NULL
* Goto Store ESP-NULL SPI cache info
* Call Try 192bit algorithms
* If State is ESP-NULL
* Goto Store ESP-NULL SPI cache info
* Call Try 256bit algorithms
* If State is ESP-NULL
* Goto Store ESP-NULL SPI cache info
// AUTH_DES_MAC and AUTH_KPDK_MD5 are left out from
// this document.
// If any of those test above set state to unsure
// we mark IPsec flow as unsure.
* If State is unsure
* Goto Store unsure SPI cache info
// All of the test failed, meaning the packet cannot
// be ESP-NULL packet, thus we mark IPsec flow as ESP
* Goto Store ESP SPI cache info
////////////////////////////////////////////////////////////
// Store ESP-NULL status to the IPsec flow cache.
//
Store ESP-NULL SPI cache info:
* Store State, IV_len, ICV_len to SPI cache
using IP_Src_IP + IP_Dst_IP + SPI as key
* Continue Check ESP-NULL packet
////////////////////////////////////////////////////////////
// Store encrypted ESP status to the IPsec flow cache.
//
Store ESP SPI cache info:
* Store State, IV_len, ICV_len to SPI cache
using IP_Src_IP + IP_Dst_IP + SPI as key
* Continue Check non-ESP-NULL packet
////////////////////////////////////////////////////////////
// Store unsure flow status to IPsec flow cache.
// Here we also store the Check_Bits.
//
Store unsure SPI cache info:
* Store State, IV_len, ICV_len,
Stored_Check_Bits to SPI cache
using IP_Src_IP + IP_Dst_IP + SPI as key
* Continue Check unknown packetAnd last but not least, the verify step 🫶 This checks the packet against the IV_len and ICV_len that were picked earlier. If the check fails, the flow goes back to ESP and the state gets reset, so the engine can start over. If it passes, the check bits go up and the stored data gets updated, so the next packet has something to compare against 💜
////////////////////////////////////////////////////////////
// Verify this packet against the previously selected
// ICV_len and IV_len values. This will either
// fail (and set state to ESP to mark we do not yet
// know what type of flow this is) or will
// increment Check_Bits.
//
Verify next packet:
// We already have IV_len, ICV_len, and State loaded
* Load Stored_Check_Bits, Last_Packet_Data from SPI Cache
* Set Test_ICV_len to ICV_len, Test_IV_len to IV_len
* Initialize Check_Bits to 0
* Call Verify padding
* If verify padding returned Failure
// Initial guess was wrong, restart
* Set State to ESP
* Clear IV_len, ICV_len, State,
Stored_Check_Bits, Last_Packet_Data
from SPI Cache
* Return
// Ok, padding check succeeded again
* Call Verify packet
* If verify packet returned Failure
// Guess was wrong, restart
* Set State to ESP
* Clear IV_len, ICV_len, State,
Stored_Check_Bits, Last_Packet_Data
from SPI Cache
* Return
// It succeeded and updated Check_Bits and Last_Packet_Data store
// them to SPI cache.
* Increment Stored_Check_Bits by Check_Bits
* Store Stored_Check_Bits to SPI Cache
* Store Packet_Data as Last_Packet_Data to SPI cache
* Return ////////////////////////////////////////////////////////////
// This will check if we have already seen enough bits
// acceptable from the payload data, so we can decide
// that this IPsec flow is ESP-NULL flow.
//
Verify unsure:
// Check if we have enough check bits.
* If Stored_Check_Bits > configured limit
// We have checked enough bits, return ESP-NULL
* Set State ESP-NULL
* Goto Store ESP-NULL SPI cache info
// Not yet enough bits, continue
* Continue Check unknown packet
////////////////////////////////////////////////////////////
// Check for standard 96-bit algorithms.
//
Try standard algorithms:
// AUTH_HMAC_MD5_96, AUTH_HMAC_SHA1_96, AUTH_AES_XCBC_96,
// AUTH_AES_CMAC_96
* Set Test_ICV_len to 12, Test_IV_len to 0
* Goto Check packet
////////////////////////////////////////////////////////////
// Check for 128-bit algorithms, this is only one that
// can have IV, so we need to check different IV_len values
// here too.
//
Try 128bit algorithms:
// AUTH_HMAC_SHA2_256_128, ENCR_NULL_AUTH_AES_GMAC
* Set Test_ICV_len to 16, Test_IV_len to 0
* If IP_total_len < IP_hdr_len + SPI_offset
+ Test_IV_len + Test_ICV_len
+ 4 (spi) + 4 (seq no) + 4 (protocol + padding)
* Return
* Call Verify padding
* If verify padding returned Failure
* Return
* Initialize Check_Bits to 0
* Call Verify packet
* If verify packet returned Failure
* Goto Try GMAC
// Ok, packet seemed ok, but go now and check if we have enough
// data bits so we can assume it is ESP-NULL
* Goto Check if done for unsure
////////////////////////////////////////////////////////////
// Check for GMAC MACs, i.e., MACs that have an 8-byte IV.
//
Try GMAC:
// ENCR_NULL_AUTH_AES_GMAC
* Set Test_IV_len to 8
* If IP_total_len < IP_hdr_len + SPI_offset
+ Test_IV_len + Test_ICV_len
+ 4 (spi) + 4 (seq no) + 4 (protocol + padding)
* Return
* Initialize Check_Bits to 0
* Call Verify packet
* If verify packet returned Failure
// Guess was wrong, continue
* Return
// Ok, packet seemed ok, but go now and check if we have enough
// data bits so we can assume it is ESP-NULL
* Goto Check if done for unsure
////////////////////////////////////////////////////////////
// Check for 192-bit algorithms.
//
Try 192bit algorithms:
// AUTH_HMAC_SHA2_384_192
* Set Test_ICV_len to 24, Test_IV_len to 0
* Goto Check packet
////////////////////////////////////////////////////////////
// Check for 256-bit algorithms.
//
Try 256bit algorithms:
// AUTH_HMAC_SHA2_512_256
* Set Test_ICV_len to 32, Test_IV_len to 0
* Goto Check packet
////////////////////////////////////////////////////////////
// This actually does the checking for the packet, by
// first verifying the length, and then self describing
// padding, and if that succeeds, then checks the actual
// payload content.
//
Check packet:
* If IP_total_len < IP_hdr_len + SPI_offset
+ Test_IV_len + Test_ICV_len
+ 4 (spi) + 4 (seq no) + 4 (protocol + padding)
* Return
* Call Verify padding
* If verify padding returned Failure
* Return
* Initialize Check_Bits to 0
* Call Verify packet
* If verify packet returned Failure
// Guess was wrong, continue
* Return
// Ok, packet seemed ok, but go now and check if we have enough
// data bits so we can assume it is ESP-NULL
* Goto Check if done for unsure
////////////////////////////////////////////////////////////
// This code checks if we have seen enough acceptable
// values in the payload data, so we can decide that this
// IPsec flow is ESP-NULL flow.
//
Check if done for unsure:
* If Stored_Check_Bits > configured limit
// We have checked enough bits, return ESP-NULL
* Set State ESP-NULL
* Set IV_len to Test_IV_len, ICV_len to Test_ICV_len
* Clear Stored_Check_Bits, Last_Packet_Data from SPI Cache
* Return
// Not yet enough bits, check if this is first unsure, if so
// store information. In case there are multiple
// tests succeeding, we always assume the first one
// (the one using shortest MAC) is the one we want to
// check in the future.
* If State is not unsure
* Set State unsure
// These values will be stored to SPI cache if
// the final state will be unsure
* Set IV_len to Test_IV_len, ICV_len to Test_ICV_len
* Set Stored_Check_Bits as Check_Bits
* Return
////////////////////////////////////////////////////////////
// Verify self describing padding
//
Verify padding:
* Load Pad_len from IP_total_len - Test_ICV_len - 2
* Verify padding bytes at
IP_total_len - Test_ICV_len - 1 - Pad_len ..
IP_total_len - Test_ICV_len - 2 are
1, 2, ..., Pad_len
* If Verify of padding bytes succeeded
* Return Success
* Return Failure
////////////////////////////////////////////////////////////
// This will verify the actual protocol content inside ESP
// packet.
//
Verify packet:
// We need to first check things that cannot be set, i.e., if any of
// those are incorrect, then we return Failure. For any
/ fields that might be correct, we increment the Check_Bits
// for a suitable amount of bits. If all checks pass, then
// we just return Success, and the upper layer will then
// later check if we have enough bits checked already.
* Load Protocol From IP_total_len - Test_ICV_len - 1
* If Protocol TCP
* Goto Verify TCP
* If Protocol UDP
* Goto Verify UDP
// Other protocols can be added here as needed, most likely same
// protocols as deep inspection does.
// Tunnel mode checks (protocol 4 for IPv4 and protocol 41 for
// IPv6) is also left out from here to make the document shorter.
* Return Failure
////////////////////////////////////////////////////////////
// Verify TCP protocol headers
//
Verify TCP:
// First we check things that must be set correctly.
* If TCP.Data_Offset field < 5
// TCP head length too small
* Return Failure
// After that, we start to check things that do not
// have one definitive value, but can have multiple possible
// valid values.
* If TCP.ACK bit is not set, then check
that TCP.Acknowledgment_number field contains 0
// If the ACK bit is not set, then the acknowledgment
// field usually contains 0, but I do not think
// RFCs mandate it being zero, so we cannot make
// this a failure if it is not so.
* Increment Check_Bits by 32
* If TCP.URG bit is not set, then check
that TCP.Urgent_Pointer field contains 0
// If the URG bit is not set, then urgent pointer
// field usually contains 0, but I do not think
// RFCs mandate it being zero, so we cannot make
// this failure if it is not so.
* Increment Check_Bits by 16
* If TCP.Data_Offset field == 5
* Increment Check_Bits by 4
* If TCP.Data_Offset field > 5
* If TCP options format is valid and it is padded correctly
* Increment Check_Bits accordingly
* If TCP options format was garbage
* Return Failure
* If TCP.checksum is correct
// This might be wrong because packet passed NAT, so
// we cannot make this failure case.
* Increment Check_Bits by 16
// We can also do normal deeper TCP inspection here, i.e.,
// check that the SYN/ACK/FIN/RST bits are correct and state
// matches the state of existing flow if this is packet
// to existing flow, etc.
// If there is anything clearly wrong in the packet (i.e.,
// some data is set to something that it cannot be), then
// this can return Failure; otherwise, it should just
// increment Check_Bits matching the number of bits checked.
//
// We can also check things here compared to the last packet
* If Last_Packet_Data.TCP.source port =
Packet_Data.TCP.source_port and
Last_Packet_Data.TCP.destination port =
Packet_Data.TCP.destination port
* Increment Check_Bits by 32
* If Last_Packet_Data.TCP.Acknowledgement_number =
Packet_Data.TCP.Acknowledgement_number
* Increment Check_Bits by 32
* If Last_Packet_Data.TCP.sequence_number =
Packet_Data.TCP.sequence_number
* Increment Check_Bits by 32
// We can do other similar checks here
* Return Success
////////////////////////////////////////////////////////////
// Verify UDP protocol headers
//
Verify UDP:
// First we check things that must be set correctly.
* If UDP.UDP_length > IP_total_len - IP_hdr_len - SPI_offset
- Test_IV_len - Test_ICV_len - 4 (spi)
- 4 (seq no) - 1 (protocol)
- Pad_len - 1 (Pad_len)
* Return Failure
* If UDP.UDP_length < 8
* Return Failure
// After that, we start to check things that do not
// have one definitive value, but can have multiple possible
// valid values.
* If UDP.UDP_checksum is correct
// This might be wrong because packet passed NAT, so
// we cannot make this failure case.
* Increment Check_Bits by 16
* If UDP.UDP_length = IP_total_len - IP_hdr_len - SPI_offset
- Test_IV_len - Test_ICV_len - 4 (spi)
- 4 (seq no) - 1 (protocol)
- Pad_len - 1 (Pad_len)
// If there is no TFC padding then UDP_length
// will be matching the full packet length
* Increment Check_Bits by 16
// We can also do normal deeper UDP inspection here.
// If there is anything clearly wrong in the packet (i.e.,
// some data is set to something that it cannot be), then
// this can return Failure; otherwise, it should just
// increment Check_Bits matching the number of bits checked.
//
// We can also check things here compared to the last packet
* If Last_Packet_Data.UDP.source_port =
Packet_Data.UDP.source_port and
Last_Packet_Data.destination_port =
Packet_Data.UDP.destination_port
* Increment Check_Bits by 32
* Return SuccessFigure 4
Authors' Addresses ✨
Meet the two legends who wrote this whole thing 🫶
Tero Kivinen
AuthenTec, Inc.
Fredrikinkatu 47
Helsinki FIN-00100
FI
EMail: kivinen@iki.fi
Daniel L. McDonald
Oracle Corporation
35 Network Drive
MS UBUR02-212
Burlington, MA 01803
USA
EMail: danmcd@opensolaris.org📦 where this page comes from
- ietf: rfc5879 2010-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms