RFC 4963

IPv4 Reassembly Errors at High Data Rates

📅 published July 2007, from ietf

RFC 4963: IPv4 Reassembly Errors at High Data Rates 🎀

okay bestie, this one is by J. Heffner, M. Mathis, and B. Chandler, all from PSC, dated July 2007 🌸 It's marked Category: Informational, so it's sharing info, not setting an Internet standard. The header doesn't list anything it obsoletes or updates ✨

Status of This Memo ✨

This memo gives information to the Internet community 💅 It does not specify an Internet standard of any kind. Distribution of this memo is unlimited 🦋

Copyright (C) The IETF Trust (2007).

Abstract 💜

IPv4 fragmentation is not robust enough for some conditions on today's Internet, honestly 😭 At high data rates, the 16-bit IP identification field is too small to stop frequent incorrectly assembled IP fragments. Then the TCP and UDP checksums aren't strong enough to stop the corrupted datagrams from reaching higher protocol layers 💀 This note walks through some easy-to-reproduce experiments showing the problem, and talks about what it means operationally 🔥

1. Introduction ✨

The IPv4 header was designed back when data rates were several orders of magnitude lower than what we get now 🧚 This doc describes a scale-related failure in the IP identification (ID) field. Fragments can get assembled wrong at a rate high enough to throw off our assumptions about how often data integrity fails 😭

That IP fragmentation wastes network capacity has been well documented [Kent87] 🍓 This note is about something different. It can cause big performance drops, and also frequent data corruption 💀 That's especially relevant now that UDP bulk transport tools are everywhere, and some of them fragment basically every datagram 🫶

There's also some network equipment that ignores the Don't Fragment (DF) bit in the IP header as a workaround for MTU discovery problems [RFC2923]. Ugh, yikes 🙈 That means properly implemented protocols and applications can end up exposed to corrupt data, even though they did everything right 💔

2. Wrapping the IP ID Field 🦋

The Internet Protocol standard [RFC0791] says:

"The choice of the Identifier for a datagram is based on the need to provide a way to uniquely identify the fragments of a particular datagram. The protocol module assembling fragments judges fragments to belong to the same datagram if they have the same source, destination, protocol, and Identifier. Thus, the sender must choose the Identifier to be unique for this source, destination pair and protocol for the time the datagram (or any fragment of it) could be alive in the Internet."

Okay so, if you follow that strictly, you can only send up to 65536 packets per protocol (like TCP, UDP, or ICMP) in one direction between any address pair, per maximum packet lifetime 🎀

Honestly, not every host follows this, because it would mean an unreasonably low maximum data rate 💅 For example, a host sending 1500-byte packets with a 30-second maximum packet lifetime could only send at about 26 Mbps before going past 65535 packets per packet lifetime. Or, to fill a 1 Gbps interface with 1500-byte packets, you'd need to send 65536 packets in under 1 second. That's an unreasonably short maximum packet lifetime, shorter than the round-trip time on some paths 😭 This requirement is widely ignored, and it's giving "nobody actually does this" 💀

Also, reusing values in the IP ID field once every 65536 datagrams is the best case 🙈 Some implementations randomize the IP ID to avoid leaking information out of the kernel [Bellovin02]. That makes reuse of the IP ID field happen probabilistically at all sending rates 🧚

IP receivers keep fragments in a reassembly buffer until all fragments in a datagram arrive, or until the reassembly timeout expires (15 seconds is suggested in [RFC0791]) ⏰ Fragments are linked to each other by their protocol number, the value in their ID field, and their source/destination address pair 🫶 If a sender wraps the ID field in less time than the reassembly timeout, fragments from different datagrams can get incorrectly spliced together ("mis-associated") and delivered to the upper layer protocol. Yikes 💀

A case that's especially concerning is when mis-association is self-propagating 😭 For example, this can happen when packets are reliably ordered and the first fragment of a datagram gets lost in the network. The rest of the fragments sit in the fragment reassembly buffer. When the sender wraps the ID field, the first fragment of the new datagram gets mis-associated with the rest of the old datagram. The new datagram is now incomplete (since it's missing its first fragment), so its remaining fragments get saved in the reassembly buffer too. That forms a cycle that repeats every 65536 datagrams 🔥 It's possible to have several simultaneous cycles, limited only by the size of the fragment reassembly buffer 💀

IPv6 is a lot less vulnerable to this, since its fragment header has a 32-bit identification field [RFC2460] ✨ Mis-association would only be a problem at packet rates 65536 times higher than for IPv4. Iconic 👑

3. Effects of Mis-Associated Fragments 💔

When mis-associated fragments get delivered, transport-layer checksumming should catch these datagrams as incorrect and discard them 🫶 But discarding them can create a performance problem for loss-feedback congestion control algorithms, especially when a large congestion window is needed, because it adds a certain amount of non-congestive loss 😭

Transport checksums, though, may not be built to handle error rates this high 💀 The TCP/UDP checksum is only 16 bits long. If these checksums follow a uniform random distribution, we'd expect mis-associated datagrams to get past the checksum at a rate of one per 65536. With just one mis-association cycle, we'd expect corrupt data to reach the application layer once per 2^32 datagrams 🙈

This number can get a lot higher with multiple concurrent cycles 🔥

With non-random data, the TCP/UDP checksum can be even weaker 😭 It's possible to build datasets where mis-associated fragments always have the same checksum. That case might seem unlikely, but it's worth thinking about 🧚 "Real" data is more likely than random data to cause checksum hot spots and raise the probability of a false checksum match [Stone98] 💅 Also, some applications or higher-level protocols turn off checksumming to go faster. That practice has been found to be dangerous for other reasons when data reliability matters [Stone00] 🙈

4. Experimental Observations 🧪

To test the real-world impact of fragmentation on UDP, we ran a series of experiments with a UDP bulk data transport protocol. It was designed as an alternative to TCP for moving large data sets over specialized networks 🚀 We picked Reliable Blast UDP (RBUDP), part of the QUANTA networking toolkit [QUANTA], because it had a clean interface that made automated experiments easier ✨ Choosing RBUDP had little to do with the details of the transport protocol itself. Any UDP transport protocol that has no other way to detect corruption, and that could be configured to use IP fragmentation, would get the same results 💜

To diagnose corruption in files sent with the UDP bulk transfer tool, we used a file format that included embedded sequence numbers and MD5 checksums in each fragment of each datagram 🔍 That way, it was possible to tell random corruption apart from corruption caused by mis-associated fragments. We used two types of files. One was built so that all the UDP checksums were constant (we'll call this the "constant" dataset). The other was built so that the UDP checksums were uniformly random (the "random" dataset). All tests used 400 MB files, sent in 1524-byte datagrams so they'd be fragmented on standard Fast Ethernet with a 1500-byte MTU 🎀

We used the UDP bulk file transport tool to send the datasets between a pair of hosts at slightly less than the available data rate (100 Mbps). Near the start of each flow, a brief secondary flow was started to induce packet loss in the primary flow 🌸 For the whole life of the primary flow, we typically saw mis-association rates on the order of a few hundredths of a percent 😭

Tests with the "constant" dataset caused corruption on all mis-associated fragments, meaning corruption on the order of a few hundredths of a percent 💀 Across roughly 10 TB of "random" datasets, we saw 8847668 UDP checksum errors and 121 corruptions of the data due to mis-associated fragments 🔥 Literally wild.

5. Preventing Mis-Association 🛡️

The most straightforward fix is to avoid fragmentation altogether, by implementing Path MTU Discovery [RFC1191] [RFC4821] 👑 But that's not always feasible for every application. Also, as a workaround for MTU discovery problems [RFC2923], some TCP implementations and communications gear give you ways to turn off path MTU discovery by clearing or ignoring the DF bit. Doing that exposes all protocols using IPv4 to mis-association errors, even the ones that take part in MTU discovery 💔

If IP fragmentation is in use, it might be possible to shorten the timeout enough that mis-association won't happen 🧚 But there are some difficulties with that approach. The sender controls the rate of packets sent and the selection of IP ID, while the receiver controls the reassembly timeout. So each party would need some mutual assurance about taking part in the scheme 🫶 It also isn't generally possible to set the timeout low enough that a fast sender's fragments avoid mis-association, but high enough that a slow sender's fragments aren't thrown away before they can be reassembled 😭 That means the timeout and IP ID selection would need to be handled on a per-peer basis. Also, NAT will likely break any per-peer tables keyed by IP address 🙈 This document isn't trying to recommend solutions to these problems. Still, we believe a per-peer adaptive timeout is likely to prevent mis-association in the circumstances where it would most commonly happen ✨

A case worth calling out is tunnels that encapsulate payload in IPv4 🚇 To deal with MTU Discovery difficulties [RFC4459], tunnels may rely on fragmentation between the two endpoints, even when the payload is marked with a DF bit [RFC4301]. In that mode, the two tunnel endpoints act like IP end hosts, and all tunneled traffic has the same protocol type 💅 So the aggregate rate of tunneled packets may not exceed 65536 per maximum packet lifetime, or the tunneled data becomes exposed to possible mis-association. Even protocols doing MTU discovery, like TCP, get affected 😭 Operators of tunnels should make sure the receiving end's reassembly timeout is short enough that mis-association can't happen at the tunnel's maximum rate 🧚

6. Mitigating Mis-Association ✨

okay bestie, so this one is about what to do when mis-association is basically unavoidable 😭🫶 it is genuinely hard to list every situation where fragments might get mixed up. even if an end host follows the spec perfectly and makes sure its IP IDs are unique, NATs or tunnels can still expose apps to IP ID space conflicts 💀 and devices the end hosts can't see or control, like tunnels, can cause mis-association too. even a fragmenting app that sends at a low rate might get caught when it runs at the same time as a non-fragmenting app that sends at a high rate 🙈

the receiver might do stuff to reduce or get rid of the chance of conflict (we talked about that above 💅), but there's no mechanism for a sender to find out what the receiver is doing about it. so there's no general way for an app using IPv4 fragmentation to know whether it's deterministically or statistically protected from mis-associated fragments. not gonna lie, that's a lot of uncertainty 🥺

⚠️ When it's impossible or impractical to prevent mis-association, you can mitigate the effects by using stronger integrity checking at any layer above IP. this is a natural side effect of using cryptographic authentication 🔐

for example, IPsec AH [RFC4302] will discard any corrupted datagrams so they never reach upper layers 🛡️ a stronger transport layer checksum like SCTP's, which is 32 bits long [RFC2960], may help a lot 💖 at the application layer, SSH message authentication codes [RFC4251] will stop corrupted data from being delivered, but since the TCP connection underneath isn't protected, it's considered invalid and the session gets terminated right away 😤 still, stronger integrity checking can prevent data corruption without preventing the performance hit we described earlier from non-congestive loss on congestion control at high congestion windows 📉

also, mis-association isn't the only way data can get corrupted above the network layer [Stone00] 👀 so most apps where data integrity is super important should use strong integrity checking no matter how exposed they are to mis-association 🎀

in general, apps that rely on IPv4 fragmentation should be written with these issues in mind, plus the ones documented in [Kent87] 📚 apps that rely on IPv4 fragmentation while sending at high speeds (on the order of 100 Mbps or higher), and devices that deliberately add fragmentation to traffic that wasn't fragmented (e.g., tunnels), should be extra careful and bring in strong mechanisms to keep data intact 🧚💜

7. Security Considerations 🔒

okay this section is giving thriller 😱 if a malicious entity knows that a pair of hosts are talking over a fragmented stream, it might get a chance to corrupt the flow. by sending "high" fragments (the ones with an offset greater than zero) with a forged source address, the attacker can deliberately cause corruption like we described above 💀 exploiting this only takes knowing the source and destination addresses of the flow, its protocol number, and the fragment boundaries. no port or sequence numbers needed 🙅‍♀️

if the attacker can see packets on the path, the attack looks a lot like injecting full segments 👁️ using this attack makes blind disruptions easier and might possibly be used to degrade service ⚠️ we believe only streams using IPv4 fragmentation are likely to be vulnerable 🌸 and because of the nature of the problems in this document, using IPv4 fragmentation for critical apps may not be advisable, no matter what the security concerns are 🫠

8. Informative References 📚

[Kent87] Kent, C. and J. Mogul, "Fragmentation considered harmful", Proc. SIGCOMM '87 vol. 17, No. 5, October 1987.

[RFC2923] Lahey, K., "TCP Problems with Path MTU Discovery", RFC 2923, September 2000.

[RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 1981.

[RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, November 1990.

[Stone98] Stone, J., Greenwald, M., Partridge, C., and J. Hughes, "Performance of Checksums and CRC's over Real Data", IEEE/ ACM Transactions on Networking vol. 6, No. 5, October 1998.

[Stone00] Stone, J. and C. Partridge, "When The CRC and TCP Checksum Disagree", Proc. SIGCOMM 2000 vol. 30, No. 4, October 2000.

[QUANTA] He, E., Alimohideen, J., Eliason, J., Krishnaprasad, N., Leigh, J., Yu, O., and T. DeFanti, "Quanta: a toolkit for high performance data delivery over photonic networks", Future Generation Computer Systems Vol. 19, No. 6, August 2003.

[Bellovin02] Bellovin, S., "A Technique for Counting NATted Hosts", Internet Measurement Conference, Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurement, November 2002.

[RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998.

[RFC2960] Stewart, R., Xie, Q., Morneault, K., Sharp, C., Schwarzbauer, H., Taylor, T., Rytina, I., Kalla, M., Zhang, L., and V. Paxson, "Stream Control Transmission Protocol", RFC 2960, October 2000.

[RFC4251] Ylonen, T. and C. Lonvick, "The Secure Shell (SSH) Protocol Architecture", RFC 4251, January 2006.

[RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.

[RFC4302] Kent, S., "IP Authentication Header", RFC 4302, December 2005.

[RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- Network Tunneling", RFC 4459, April 2006.

[RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU Discovery", RFC 4821, March 2007.

Appendix A. Acknowledgements 🎀

this work was supported by the National Science Foundation under Grant No. 0083285 🙏✨

Authors' Addresses 🏡

John W. Heffner Pittsburgh Supercomputing Center 4400 Fifth Avenue Pittsburgh, PA 15213 US

Phone: 412-268-2329 EMail: jheffner@psc.edu

Matt Mathis Pittsburgh Supercomputing Center 4400 Fifth Avenue Pittsburgh, PA 15213 US

Phone: 412-268-3319 EMail: mathis@psc.edu

Ben Chandler Pittsburgh Supercomputing Center 4400 Fifth Avenue Pittsburgh, PA 15213 US

Phone: 412-268-9783 EMail: bchandle@gmail.com

Copyright (C) The IETF Trust (2007).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 💜

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.

Acknowledgement 🫶

Funding for the RFC Editor function is currently provided by the Internet Society.

📦 where this page comes from