RFC 4459

MTU and Fragmentation Issues with In-the-Network Tunneling

📅 published April 2006, from ietf

MTU and Fragmentation Issues with In-the-Network Tunneling 🎀

Okay bestie, so this is RFC 4459 🦋 written by P. Savola from CSC/FUNET. It's an Informational doc (dated April 2006), and it doesn't obsolete or update any other RFCs, so it's giving standalone energy 💅.

Status of This Memo ✨

This memo gives information to the Internet community. It is not an Internet standard of any kind 🙈. Distribution of this memo is unlimited.

Copyright (C) The Internet Society (2006).

Abstract 🌸

Okay so tunneling stuff like IP-in-IP, when it's set up in the middle of the network (usually between routers), gets messy with big packets. The questions are: would those packets get fragmented and reassembled (and how?), would Path MTU Discovery be used, or can this whole situation be operationally avoided? This memo explains why it's a common and genuinely non-trivial problem, then goes deep on the different solutions and what each one is like, at some length 🧚.

Table of Contents 📜

1. Introduction ....................................................2
2. Problem Statement ...............................................3
3. Description of Solutions ........................................4
   3.1. Fragmentation and Reassembly by the Tunnel Endpoints .......4
   3.2. Signalling the Lower MTU to the Sources ....................5
   3.3. Encapsulate Only When There is Free MTU ....................6
   3.4. Fragmentation of the Inner Packet ..........................8
4. Conclusions .....................................................9
5. Security Considerations ........................................10
6. Acknowledgements ...............................................11
7. References .....................................................11
   7.1. Normative References ......................................11
   7.2. Informative References ....................................12

1. Introduction ✨

So there are a bunch of ways to wrap datagrams inside other packets. These are called tunneling mechanisms, and they've been specified over the years. Some examples are IP-in-IP (e.g., [1] [2], [3]), Generic Routing Encapsulation (GRE) [4], Layer 2 Tunneling Protocol (L2TP) [5], or IP Security (IPsec) [6] in tunnel mode. Any of these might run on top of IPv4, IPv6, or some other protocol, and they might carry the same protocol or a different one 💖.

All of these can be set up so the endpoints of the inner protocol live in the same place as the endpoints of the outer protocol. In a typical scenario, that's "host-to-host" tunneling. It's also possible for just one set of endpoints to be co-located, which is host-to-router or router-to-host tunneling. And lastly, lots of these mechanisms get used between routers for all or part of the traffic passing between them. That's router-to-router tunneling 🦋.

Here's the core issue for all of these protocols and scenarios. How does the source pick the maximum packet size so the packets still fit, even after encapsulation, inside the smallest Maximum Transmission Unit (MTU) on the path through the network? And if you can't change the packet sizes, what do you do so you can still encapsulate them? There are four main solutions, which get elaborated in Section 3 🔥:

  1. Fragmenting all too big encapsulated packets to fit in the paths, and reassembling them at the tunnel endpoints.
  2. Signalling all the sources whose traffic must be encapsulated and is larger than what fits, telling them to send smaller packets. For example, this can use Path MTU Discovery (PMTUD)[7][8].
  3. Making sure that in the specific environment, the encapsulated packets will fit in all the paths in the network. For example, by using an MTU bigger than 1500 in the backbone used for encapsulation.
  4. Fragmenting the original too big packets so their fragments will fit, even encapsulated, in the paths, and reassembling them at the destination nodes. Note that this approach is only available for IPv4 under certain assumptions (see Section 3.4) 👑.

It's also common to run multiple layers of encapsulation, like GRE or L2TP over IPsec 🌈. With nested tunnels in the network, the tunnel endpoints can be the same or different, and both the inner and outer tunnels may use different MTU handling strategies. In particular, signalling may be a scalable option for the outer tunnel or tunnels if the number of innermost tunnel endpoints is limited.

The tunneling packet size issues are pretty straightforward in host-to-host tunneling or host-to-router tunneling, where Path MTU Discovery only needs to signal one source node. They get significantly more difficult in router-to-router and certain router-to-host scenarios, and those are the focus of this memo 😭.

It's worth noting that most of this discussion applies to a more generic case, where there's a link with a lower MTU somewhere in the path. A concrete and widely deployed example is PPP over Ethernet (PPPoE) [11] on the customers' access link. These lower-MTU links, and PPPoE links especially, usually aren't deployed in topologies where fragmentation and reassembly might be unfeasible (e.g., a backbone). So this may be a slightly easier problem. However, this more generic case is considered out of scope for this memo 🙈.

There are also known challenges with specifying and implementing a mechanism that the tunnel endpoint would use to figure out the best packet size for encapsulation. If a static value is chosen, a lot of fragmentation might end up happening. On the other hand, if PMTUD is used, the implementation would need to update the discovered interface MTU based on the ICMP Packet Too Big messages, and also originate ICMP Packet Too Big message(s) back to the source(s) of the encapsulated packets. This also assumes that sufficient data has been piggybacked on the ICMP messages (beyond the required 64 bits after the IPv4 header). We'll talk about using PMTUD to signal the sources briefly in Section 3.2, but the in-depth specification and analysis are described elsewhere (e.g., in [4] and [2]) and are out of scope for this memo 🧚.

Section 2 includes a problem statement, Section 3 describes the different solutions with their drawbacks and advantages, and Section 4 presents conclusions ✨.

2. Problem Statement 💅

It's worth considering why exactly this is considered a problem 🤔.

It's possible to fix all the packet size issues using solution 1, which is fragmenting the resulting encapsulated packet and reassembling it at the tunnel endpoint. However, this is considered problematic for at least three reasons, as described in Section 3.1 💀.

So, it's desirable to avoid fragmentation and reassembly if possible. On the other hand, the other solutions may not be practical either. Especially in router-to-router or router-to-host tunneling, Path MTU Discovery might be very disadvantageous. Like, consider the case where a backbone router would send ICMP Packet Too Big messages to every source that tries to send packets through it. Fragmenting before encapsulation is also not available in IPv6, and it's not available when the Don't Fragment (DF) bit has been set (see Section 3.4 for more). Ensuring a high enough MTU so encapsulation is always possible is of course a valid approach, but it requires careful operational planning, and it may not be a feasible assumption for implementors 🫶.

This yields that there's no trivial solution to this problem. It needs to be explored further to consider the trade-offs, which is exactly what this memo does 🌸.

3. Description of Solutions 🔥

This section describes the potential solutions in a bit more detail 🦋.

3.1. Fragmentation and Reassembly by the Tunnel Endpoints 🎀

The seemingly simplest solution to tunneling packet size issues is fragmentation of the outer packet by the encapsulator, and reassembly by the decapsulator. However, this is highly problematic for at least three reasons:

  • Fragmentation causes overhead. Every fragment requires the IP header (20 or 40 bytes), and with IPv6, there's an additional 8 bytes for the Fragment Header 💅.
  • Fragmentation and reassembly require computation. Splitting datagrams into fragments is a non-trivial procedure, and so is their reassembly. For example, software router forwarding implementations may not be able to perform these operations at line rate 😭.
  • At the time of reassembly, all the information (i.e., all the fragments) is normally not available. When the first fragment arrives to be reassembled, a buffer of the maximum possible size may have to be allocated, because the total length of the reassembled datagram isn't known at that time. Furthermore, since fragments might get lost, reordered, or delayed, the reassembly engine has to wait with the partial packet for some time (e.g., 60 seconds [9]). When this would have to be done at line rate, with, for example, 10 Gbit/s speed, the length of the buffers that reassembly might require would be prohibitive 💀.

When you look at router-to-router tunneling, the third problem is likely the worst. Certainly, a hardware computation and implementation requirement would also be significant, but it's not all that difficult in the end. And the link capacity wasted in the backbones by additional overhead might not be a huge problem either 🌈.

However, the IPv4 identification header length is only 16 bits (compared to 32 bits in IPv6). So if a larger number of packets are being tunneled between two IP addresses, the ID is very likely to wrap and cause data misassociation 😱. This is a big deal because reassembly wrongly combining data from two unrelated packets causes data integrity problems and potentially a confidentiality violation. This problem is further described in [12] 🙈.

IPv6, and IPv4 with the DF bit set in the encapsulating header, allows the tunnel endpoints to optimize the tunnel MTU and minimize network-based reassembly. This also prevents fragmentation of the encapsulated packets on the tunnel path. If the IPv4 encapsulating header does not have the DF bit set, the tunnel endpoints will have to perform a significant amount of fragmentation and reassembly, while the use of PMTUD is minimized ✨.

As Appendix A describes, the MTU of the tunnel is also a factor in which packets require fragmentation and reassembly. The worst case happens if the tunnel MTU is "infinite" or equal to the physical interface MTUs. 🧚

So, if reassembly could be made to work sufficiently reliably, this would be one acceptable fallback solution, but only for IPv6 👑.

3.2. Signalling the Lower MTU to the Sources 📣

Okay so another approach 💅 is to use techniques like Path MTU Discovery (or potentially a future derivative [13]) to tell the sources whose packets will get encapsulated in the network to send smaller packets, so that they can be encapsulated. Especially when it's done on routers, this includes two separable functions:

  • a. Forwarding behaviour: when forwarding packets, if the IPv4-only DF bit is set, the router sends an ICMP Packet Too Big message to the source if the MTU of the egress link is too small. 🚫
  • b. Router's "host" behaviour: when the router receives an ICMP Packet Too Big message related to a tunnel, it (1) adjusts the tunnel MTU, and (2) originates an ICMP Packet Too Big message to the source address of the encapsulated packet. (2) can be done either immediately or by waiting for the next packet to trigger an ICMP; the former minimizes the packet loss due to MTU changes. 🧚

Note that this only works if the MTU of the tunnel is of reasonable size, and not, for example, 64 kilobytes: see Appendix A for more. 🙈

This approach would presuppose that PMTUD works. While it is currently working for IPv6, and critical for its operation, there is ample evidence that in IPv4, PMTUD is far from reliable 😭. For example, firewalls and other boxes are configured to inappropriately drop all the ICMP packets [14], or software bugs render PMTUD inoperational.

Furthermore, there are two scenarios where signalling from the network would be highly undesirable. 🫠 The first is when the encapsulation would be done in such a prominent place in the network that a very large number of sources would need to be signalled with this information (possibly even multiple times, depending on how long they keep their PMTUD state). The second is when the encapsulation is done for passive monitoring purposes (network management, lawful interception, etc.). In that case, it's critical that the sources whose traffic is being encapsulated are not aware of this happening. 🕵️‍♀️

When desiring to avoid fragmentation, IPv4 requires one of two alternatives [1]: copy the DF bit from the inner packets to the encapsulating header, or always set the DF bit of the outer header. The latter is better, especially in controlled environments, because it forces PMTUD to converge immediately. 👑

A related technique, which works with TCP under specific scenarios only, is so-called "MSS clamping" 🎀. With that technique (or rather, a "hack"), the TCP packets' Maximum Segment Size (MSS) is reduced by tunnel endpoints, so that the TCP connection automatically restricts itself to the maximum available packet size. Obviously, this does not work for UDP or other protocols that have no MSS. This approach is most applicable and used with PPPoE, but could be applied otherwise as well. It also assumes that all the traffic goes through tunnel endpoints that do MSS clamping. This is trivial for the single-homed access links, but could be a challenge otherwise.

A new approach to PMTUD is in the works [13], but it is uncertain whether that would fix the problems, at least not the passive monitoring requirements. 🤞

3.3. Encapsulate Only When There is Free MTU 🦋

The third approach is an operational one, depending on the environment where encapsulation and decapsulation are being performed. Say an ISP deploys tunneling in its backbone, which consists only of links supporting high MTUs (e.g., Gigabit Ethernet or SDH/SONET), but all its customers and peers have a lower MTU (e.g., 1500, or the backbone MTU minus the encapsulation overhead). This would imply that no packets (with the encapsulation overhead added) would have a larger MTU than the "backbone MTU", and all the encapsulated packets would always fit MTU-wise in the backbone links. Main character energy 👑

This approach is highly assumptive of the deployment scenario. 🙃 It may be desirable to build a tunnel to or from another ISP, for example, where this might no longer hold. Or there might be links in the network that cannot support the higher MTUs to satisfy the tunneling requirements. Or the tunnel might be set up directly between the customer and the ISP, in which case fragmentation would occur, with tunneled fragments terminating on the ISP and thus requiring reassembly capability from the ISP's equipment.

To restate, this approach can only be considered when tunneling is done inside a part of a specific kind of ISP's own network, not, for example, transiting an ISP. 🚧

Another, related approach might be having the sources use only a low enough MTU that would fit in all the physical MTUs. For example, IPv6 specifies the minimum MTU of 1280 bytes. If all the sources whose traffic would be encapsulated used this as the maximum packet size, there would probably always be enough free MTU for encapsulation in the network. However, this is not the case today, and it would be completely unrealistic to assume that this kind of approach could be made to work in general. 💀

It is worth remembering that while the IPv6 minimum MTU is 1280 bytes [10], there are scenarios where the tunnel implementation must implement fragmentation and reassembly [3]. For example, this happens when having an IPv6-in-IPv6 tunnel on top of a physical interface with an MTU of 1280 bytes, or when having two layers of IPv6 tunneling. This can only be avoided by ensuring that links on top of which IPv6 is being tunneled have a somewhat larger MTU (e.g., 40 bytes) than 1280 bytes. This conclusion can be generalized: because IP can be tunneled on top of IP, no single minimum or maximum MTU can be found such that fragmentation or signalling to the sources would never be needed. 🌈

All in all, while in certain operational environments it might be possible to avoid any problems by deployment choices, or by limiting the MTU that the sources use, this is probably not a sufficiently good general solution for the equipment vendors. Other solutions must also be provided. 🫶

Tea: deployment choices alone are not a general fix. Vendors need more solutions in their toolkit. 🍓

3.4. Fragmentation of the Inner Packet ✂️

A final possibility is fragmenting the inner packet, before encapsulation, in such a manner that the encapsulated packet fits in the tunnel's path MTU (discovered using PMTUD). However, one should note that only IPv4 supports this "in-flight" fragmentation. Furthermore, it isn't allowed for packets where the Don't Fragment bit has been set. Even if one could ignore IPv6 completely, so many IPv4 host stacks send packets with the DF bit set that this would seem unfeasible. 🥲

However, there are existing implementations that violate the standard in the following ways:

  • discard too big packets with the DF bit not set instead of fragmenting them (this is rare);
  • ignore the DF bit completely, for all or specified interfaces; or
  • clear the DF bit before encapsulation, in the egress of configured interfaces. This is typically done for all the traffic, not just too big packets (allowing configuring this is common).

This is non-compliant behaviour, but there are certainly uses for it, especially in certain tightly controlled passive monitoring scenarios. It also has potential for more generic applicability, to work around PMTUD issues. 🔧

Clearing the DF bit effectively disables the sender's PMTUD for the path beyond the tunnel. This may result in fragmentation later in the network. But as the packets have already been fragmented prior to encapsulation, this fragmentation later on does not make matters significantly worse. 🤷‍♀️

As this is an implemented and desired (by some) behaviour, the full impacts (e.g., for the functioning of PMTUD) should be analyzed, and the use of fragmentation-related IPv4 bits should be re-evaluated. 🔍

In summary, this approach provides a relatively easy fix for IPv4 problems, with potential for causing problems for PMTUD. As this would not work with IPv6, it could not be considered a generic solution. 💅

4. Conclusions 🏁

Fragmentation and reassembly by the tunnel endpoints are a clear and simple solution to the problem. But the hardware reassembly, when the packets get lost, may face significant implementation challenges that may be insurmountable. 😵 This approach does not seem feasible, especially for IPv4 with high data rates, due to problems with wrapping the fragment identification field [12]. Constant wrapping may occur when the data rate is in the order of MB/s for IPv4 and in the order of dozens of GB/s for IPv6. However, this reassembly approach is probably not a problem for passive monitoring applications. 👀

PMTUD techniques, at least at the moment and especially for IPv4, appear to be too unreliable or unscalable to be used in the backbones. It is an open question whether a future solution might work better in this aspect. 🔮

It is clear that in some environments, the operational approach to the problem is sufficient. That approach ensures that fragmentation is never necessary by keeping higher MTUs in the networks where encapsulated packets traverse. But this is unlikely to be enough in general, and for vendors that may not be able to make assumptions about the operators' deployments. 🧐

Fragmentation of the inner packet is only possible with IPv4, and is sufficient only if standards-incompliant behaviour, with potential for bad side-effects (e.g., for PMTUD), is adopted. It should not be used if there are alternatives. Fragmentation of the outer packet seems a better option for passive monitoring. 💖

However, if reassembly in the network must be avoided, there are basically two possibilities:

  1. For IPv6, use ICMP signalling or operational methods.
  2. For IPv4, packets for which the DF bit is not set can be fragmented before encapsulation (and the encapsulating header would have the DF bit set). Packets whose DF bit is set would need to get the DF bit cleared (though this is non-compliant). This also minimizes the need for (unreliable) Internet-wide PMTUD. 🌸

An interesting thing to explicitly note is that when tunneling is done in a high-speed backbone, one may typically be able to make assumptions about the environment. However, when reassembly is not performed in such a network, it might be done in software or with lower requirements. There exists either a reassembly implementation using PMTUD, or a separate approach for passive monitoring, so this might not be a real problem. ✨

In consequence, the critical questions at this point appear to be: 1) whether a higher MTU can be assumed in the high-speed networks that deploy tunneling, and 2) whether "slower-speed" networks could cope with a software-based reassembly, a less capable hardware-based reassembly, or the other workarounds. An important future task would be analyzing the observed incompliant behaviour about the DF bit, to note whether it has any unanticipated drawbacks. 🧠

5. Security Considerations 🔐

Okay bestie, this section is lowkey the most important one 😭. This document is about packet sizes and tunneling inside the network, and on its own it doesn't have security considerations. 💖

BUT different solutions can have traits that make them easier to attack 🙈. For example, if a router is the one doing fragment reassembly, an attacker can send a bunch of fragments and never send the rest. The reassembly then sits there stalled until it times out, and the router's reassembly buffer memory can run out 💀. Fragment attacks like this one (see [15]) have already been used against firewalls and host stacks, so implementers need to take them into account 🛡️.

Here's another spicy one 🌶️. Fragmenting the inner packet or the outer packet has a cryptographic cost, and that cost is usually bigger than the reassembly cost if it's done in software. If an outer fragment goes missing, no cryptographic operations have been performed yet. If an inner fragment goes missing, the cryptographic operations have already happened. So which approach is better also depends on whether cryptography or reassembly is already handled in hardware 🤔. For high-speed routers, at the very least, you should be able to assume that if a router is doing pretty heavy cryptography, it already has hardware support 🔥.

The solutions that use PMTUD (and therefore ICMP) also have to deal with attacks that use ICMP 🙃. An attacker could send ICMP Packet Too Big messages that claim a really low MTU. That can cut down throughput, and it can also be used as a fragmentation/reassembly denial-of-service attack 😤. This attack was described for TCP in [16] 💜.

6. Acknowledgements 🫶

This topic is far from new, but recent chats with W. Mark Townsley about L2TP fragmentation issues got the author to finally sit down and write everything up 📝. Michael Richardson and Mika Joutsenvirta gave useful feedback on the first version ✨. When the author asked for comments on the NANOG list, Carsten Bormann, Kevin Miller, Warren Kumari, Iljitsch van Beijnum, Alok Dube, and Stephen J. Wilcox gave useful feedback too 🌸. Later, Carlos Pignataro gave excellent input that helped improve the document 👑. Joe Touch also gave input on the memo 🧚.

7. References 📚

7.1. Normative References

[1] Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996.

[2] Nordmark, E. and R. Gilligan, "Basic Transition Mechanisms for IPv6 Hosts and Routers", RFC 4213, October 2005.

[3] Conta, A. and S. Deering, "Generic Packet Tunneling in IPv6 Specification", RFC 2473, December 1998.

[4] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, March 2000.

[5] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005.

[6] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005.

[7] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, November 1990.

[8] McCann, J., Deering, S., and J. Mogul, "Path MTU Discovery for IP version 6", RFC 1981, August 1996.

[9] Braden, R., "Requirements for Internet Hosts - Communication Layers", STD 3, RFC 1122, October 1989.

[10] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998.

7.2. Informative References

[11] Mamakos, L., Lidl, K., Evarts, J., Carrel, D., Simone, D., and R. Wheeler, "A Method for Transmitting PPP Over Ethernet (PPPoE)", RFC 2516, February 1999.

[12] Mathis, M., "Fragmentation Considered Very Harmful", Work in Progress, July 2004.

[13] Mathis, M. and J. Heffner, "Path MTU Discovery", Work in Progress, March 2006.

[14] Medina, A., Allman, M., and S. Floyd, "Measuring the Evolution of Transport Protocols in the Internet", Computer Communications Review, Apr 2005, <http://www.icir.org/tbit/>.

[15] Miller, I., "Protection Against a Variant of the Tiny Fragment Attack (RFC 1858)", RFC 3128, June 2001.

[16] Gont, F., "ICMP attacks against TCP", Work in Progress, February 2006.

Appendix A. MTU of the Tunnel 🎀

Different tunneling mechanisms can treat tunnel links as having different kinds of MTU values 🦋. Some just use the same default MTU as everything else. Some use the default MTU minus the expected IP overhead (for example, 20, 28, or 40 bytes). And some even treat the tunnel as having an "infinite MTU", like 64 kilobytes 😳.

As [2] describes, having an infinite MTU is a very bad idea 💀. That means always fragmenting the outer packet (and never the inner packet) and never doing PMTUD for the tunnel path. It's especially bad in host-to-router scenarios 🙅‍♀️. (It could be argued that if the nodes are sure this is a host-to-host tunnel, a larger MTU might make sense if fragmentation and reassembly are more efficient than just sending properly sized packets. But honestly? That seems like a stretch 🙄.)

Author's Address 🖋️

Pekka Savola CSC/FUNET Espoo Finland

EMail: psavola@funet.fi

Copyright (C) The Internet Society (2006).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property 💼

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document, or the extent to which any license under such rights might or might not be available. It also doesn't represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79 📜.

Copies of IPR disclosures made to the IETF Secretariat, any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr 🔗.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address that information to the IETF at ietf-ipr@ietf.org 💌.

Acknowledgement 🌈

Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA) 💖✨

📦 where this page comes from