RFC 7690

Close Encounters of the ICMP Type 2 Kind (Near Misses with ICMPv6 Packet Too Big (PTB))

📅 published January 2016, from ietf

RFC 7690: Close Encounters of the ICMP Type 2 Kind (Near Misses with ICMPv6 Packet Too Big (PTB)) 💖

okay bestie, this one is a whole mystery novel about ICMP packets getting lost on their way home 🕵️‍♀️✨ it was written by M. Byerly (Fastly), M. Hite (Evernote), and J. Jaeggli (Fastly), and it's an Informational doc, not a standards-track spec, so it's more of a "here's what we saw and what we tried" kind of read 🎀. It's dated January 2016, the ISSN is 2070-1721, and the title block doesn't list anything this RFC obsoletes or updates 🌸. It's published by the Internet Engineering Task Force (IETF) 🦋

Abstract

okay so the whole point is this: we're calling out the problem of getting ICMPv6 type 2 "Packet Too Big" (PTB) messages to the right destination (usually the server) in ECMP load-balanced or anycast setups 👀 the doc also talks about operational mitigations you can use to deal with this whole class of failures 🛠️💅

Status of This Memo

this is not an Internet Standards Track specification, it's just published for informational purposes 🧚

this doc is a product of the IETF, so it represents the consensus of the IETF community 🫶 it got public review and was approved for publication by the Internet Engineering Steering Group (IESG) ✅ not every IESG-approved doc is a candidate for any level of Internet Standard though, so check Section 2 of RFC 5741 if you're curious about that 🤓

for info about the current status, any errata, and how to give feedback on this doc, go to http://www.rfc-editor.org/info/rfc7690 💜

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. 📜

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were in effect on the date of publication 🍓 please actually read those, bestie, because they explain your rights and restrictions for this doc. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 🌈

Table of Contents

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
2.  Problem . . . . . . . . . . . . . . . . . . . . . . . . . . .   3
3.  Mitigation  . . . . . . . . . . . . . . . . . . . . . . . . .   4
  3.1.  Alternative Mitigations . . . . . . . . . . . . . . . . .   5
  3.2.  Implementation  . . . . . . . . . . . . . . . . . . . . .   5
    3.2.1.  Alternative Implementation  . . . . . . . . . . . . .   6
4.  Improvements  . . . . . . . . . . . . . . . . . . . . . . . .   7
5.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
6.  Informative References  . . . . . . . . . . . . . . . . . . .   8
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .   9
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   9

1. Introduction ✨

operators of popular Internet services are dealing with some seriously complex scaling problems 😩 one way they handle it is with equal-cost multipath (ECMP) routing, which does stateless distribution of incoming TCP or UDP sessions across multiple servers or middle boxes like load balancers 🎀

here's the tea: this kind of distribution causes problems when you're dealing with ICMP signaling 💀 specifically, an ICMP error is not guaranteed to hash via ECMP to the same destination as its corresponding TCP or UDP session. so the error can end up somewhere totally different, which is so frustrating 😭 operationally, this is especially painful for path MTU discovery (PMTUD) [RFC1981] 🙈

2. Problem 🔍

a super common use of stateless load balancing for TCP or UDP flows is doing an initial split of flows in front of a stateful load-balancer tier or multiple servers 🧚 that way the workload gets divided into manageable chunks of the total number of flows 💅 the split uses ECMP forwarding plus a stateless but sticky algorithm that hashes across the available paths (see [RFC2991] for background on ECMP routing) 🦋

for this kind of flow distribution, the next-hop selection is basically a constrained form of anycast topology 👑 all the anycast destinations are equidistant from the upstream router that makes the last next-hop forwarding decision before the flow reaches the destination device 🌸 the hash is calculated across some set of available protocol headers. typically that includes all or a subset of (IPv6) Flow-Label, IP-source, IP-destination, protocol, source-port, destination-port, and maybe others like ingress interface ✨

now the main issue: hashing messes with path MTU discovery 😤 an ICMPv6 type 2 PTB message made on an intermediate device for a packet sent from a server in an ECMP load-balanced service to a client gets the load-balanced anycast address as its destination. so it's statelessly load balanced to one of the servers, and it could land on the wrong one 💀 the PTB message includes as much of the packet that couldn't be forwarded as it can, but the payload headers aren't considered in the forwarding decision and get ignored 🙃 because the PTB message can't be identified as part of the original flow by the IP or upper-layer packet headers, its ICMPv6 ECMP hash is unlikely to land on the same next hop as packets matching the TCP or UDP ECMP hash of the flow 😭 that's the near miss

here's an example packet flow and topology. the packet that generated the PTB message was meant for the client 🧸

ptb -> router ecmp -> next hop L4/L7 load balancer -> destination

  router --> load balancer 1 --->
       \\--> load balancer 2 ---> load-balanced service
        \--> load balancer N --->

                              Figure 1

the router's ECMP decision gets used because it's part of the forwarding architecture, it can run at line rate, and it doesn't depend on shared state or coordination across a distributed forwarding system that might have multiple linecards or routers 🏎️ the ECMP routing decision is deterministic for packets with the same computed hash 💜

a typical case where ICMPv6 PTB messages show up at the load balancer is when the path MTU from the client to the load balancer is limited by a tunnel that the client itself doesn't know about 🫣

ngl this is the sneaky part. the client has no idea the tunnel exists, so it can't account for it, and the PTB message is what tells everyone something's off 🚨

direct experience says PTB messages are rare compared to total flows 🌈 one possible conclusion is that tunneled IPv6 deployments that can't carry 1500 MTU packets are relatively rare too 🦋 techniques clients use (for example, Happy Eyeballs [RFC6555]) might actually help the IPv6 client experience a bit, by preferring IPv4 in cases that might be identified as failures 🎀 still, operators expect PMTUD to work and expect unnecessary breakage of client traffic to be avoided 🙏

one last note about server tuning: it isn't always possible, even when it's potentially desirable, to independently set the TCP MSS (Maximum Segment Size) for different address families on some end systems 😩 on Linux platforms, advmss (advertised mss) can be set on a per-route basis for selected destinations, but only in cases where discrimination by route is possible 🐧

the problem as described also affects IPv4 🫠 however, a few things make it rare enough that some existing deployments have decided to ignore it: implementation of RFC 4821 [RFC4821] TCP MTU probing, the ability to fragment on the wire at tunnel ingress points, and the relative rarity of sub-1500-byte MTUs that aren't coupled to changes in client behavior. for example, endpoint VPN clients set the tunnel interface MTU accordingly to avoid fragmentation for performance reasons 🧚‍♀️

3. Mitigation 🛡️

mitigation for the chance that PTB messages get misdelivered comes down to making sure an ICMPv6 error message goes to the same anycast server that's responsible for the flow the error was generated for 💖 with the right hardware support, flows could be identified the same way hosts do it, by inspecting the payload of the ICMPv6 message 👀 the ECMP hash calculation can then use values pulled from the inner TCP flow parameters of the ICMPv6 message ✨

because the encapsulated IP header sits at a fixed offset in the ICMP message, it's not outside the realm of possibility that routers with enough header processing capability could parse that far into the payload 🤯 another option is a mediation device that handles parsing and distribution of PTB messages after policy routing, or on each load balancer / server 🍓

another approach is to send the PTB message to all anycast servers, assuming the one it was meant for will be able to match it to the flow, update the route cache with the new MTU, and that devices that can't match the flow will discard it 🌸 but this has potentially big implications for resource consumption and for self-inflicted denial of service (DOS) if it isn't carefully managed 😬 the good news is that we've observed the number of flows where this problem happens is pretty small in real-world deployments (for example, 10 or fewer pps on 1 Gbit/s or more worth of HTTPS) 🥹 so sensible ingress rate limiters that discard excessive message volume can protect even very large anycast server tiers, and the fallout is limited to circumstances of deliberate duress 🔥

3.1. Alternative Mitigations 🌷

another option is lowering the TCP MSS to 1220 so it can accommodate a 1280-byte MTU 🎀 we don't love this one 😕 hosts might not be able to set TCP MSS independently by address family, which would impact IPv4, or middle-boxes would need to be used to clamp the MSS independently from the end systems 🫠 extension headers could also change the lower bound the MSS would have to be set to, which makes clamping even more undesirable 💀

to be continued bestie 💜 part 2 picks up with Section 4 (Improvements), then Security Considerations, the References, and the Acknowledgements and Authors' Addresses 🦋

3.2. Implementation ✨

Okay so, here's how the actual build goes 🛠️💖

  1. Filter-based forwarding matches next-header ICMPv6 type 2 and matches a next hop on a particular subnet directly attached to one or more routers. The filter is policed to reasonable limits (we chose 1000 pps; more conservative rates might be required in other implementations) 🚦.
  2. The filter is applied on the input side of all external (Internet- or customer-facing) interfaces 🌐.
  3. A proxy located at the next hop forwards ICMPv6 type 2 packets it receives to an Ethernet broadcast address (example ff:ff:ff:ff:ff:ff) on all specified subnets. This was necessitated by router inability (in IPv6) to forward the same packet to multiple unicast next hops 📡. Routers just can't do that in IPv6, so the proxy has to do it 🥲.
  4. Anycasted servers receive the PTB error and process the packet as needed 🧚.

A simple Python scapy [SCAPY] script that can perform the ICMPv6 proxy reflection is included. Bestie, this is the vibe 👩‍💻✨

#!/usr/bin/python

from scapy.all import *

IFACE_OUT = ["p2p1", "p2p2"]

def icmp6_callback(pkt):
    if pkt.haslayer(IPv6) and (ICMPv6PacketTooBig in pkt) \
    and pkt[Ether].dst != 'ff:ff:ff:ff:ff:ff':
        del(pkt[Ether].src)
        pkt[Ether].dst = 'ff:ff:ff:ff:ff:ff'
        pkt.show()
        for iface in IFACE_OUT:
            sendp(pkt, iface=iface)

def main():
    sniff(prn=icmp6_callback, filter="icmp6 \
    and (ip6[40+0] == 2)", store=0)

if __name__ == '__main__':
    main()

This example script listens on all interfaces for IPv6 PTB errors being forwarded using filter-based forwarding 👀. It removes the existing Ethernet source and rewrites a new Ethernet destination of the Ethernet broadcast address 🪄. Then it sends the resulting frame out the p2p1 and p2p2 interfaces that are attached to VLANs where our anycast servers reside 💜.

3.2.1. Alternative Implementation

Alternatively, network designs in which a common layer 2 network exists on the ECMP hop could distribute the proxy onto the end systems, eliminating the need for policy routing 🧠. They could then rewrite the destination (for example, using iptables before forwarding the packet back to the network containing all of the server or load-balancer interfaces) 🔁. This implementation can be done entirely within the Linux iptables firewall 🐧. Because of the distributed nature of the filter, more conservative rate limits are required than when a global rate limit can be employed 🐢.

An example ip6tables/nftables rule to match icmp6 traffic, not match broadcast traffic, impose a rate limit of 10 pps, and pass to a target destination would resemble:

ip6tables -I INPUT -i lo -p icmpv6 -m icmpv6 --icmpv6-type 2/0 \
-m pkttype ! --pkt-type broadcast -m limit --limit 10/second \
-j TEE 2001:DB8::1

As with the scapy example, once the destination has been rewritten from a hardcoded ND entry to an Ethernet broadcast address (in this case to an IPv6 documentation address 2001:DB8::1), the traffic will be reflected to all the hosts on the subnet 🦋. Main character energy for the broadcast address 📣.

4. Improvements ✨

Okay, so there are several ways this could get even better at handling ECMP load balancing of ICMPv6 PTB messages 🎀. Little change to the Internet protocol specification is required 🙌. Instead, we foresee practical implementation change, which, insofar as we are aware, does not exist in current router, switch, or layer 3/4 load balancers 🫠. Alternatively, improved behavior on the part of client/server detection of path MTU in band could render the behavior of devices in the path irrelevant 💅. Honestly, that would be iconic.

  1. Routers with sufficient capacity within the lookup process could parse all the way through the L3 or L4 header in the ICMPv6 payload beginning at bit offset 32 of the ICMP header 🔍. By reordering the elements of the hash to match the inward direction of the flow, the PTB error could be directed to the same next hop as the incoming packets in the flow 🎯.
  2. The FIB (Forwarding Information Base) on the router could be programmed with a multicast distribution tree that includes all of the necessary next hops, and unicast ICMPv6 packets could be policy routed to these destinations 🌳.
  3. Ubiquitous implementation of RFC 4821 [RFC4821] Packetization Layer Path MTU Discovery would probably go a long way towards reducing dependence on ICMPv6 PTB by end systems 🥰. That's the glow-up we need.

5. Security Considerations

⚠️ Bestie, this part is serious, so read it carefully 💀.

The employed mitigation has the potential to greatly amplify the impact of a deliberately malicious sending of ICMPv6 PTB messages 😭. Sensible ingress rate limiting can reduce the potential for impact; legitimate PMTUD messages may be lost once the rate limit is reached 🚨. The scenario where drops of legitimate traffic occur is analogous to other cases where DOS traffic can crowd out legitimate traffic, however only a limited subset of overall traffic is impacted 🙏.

The proxy replication results in all devices on the subnet receiving ICMPv6 PTB errors, even those not associated with the flow 👀. This could arguably result in information disclosure due to the wide replication of the ICMPv6 PTB error on the subnet and the large fragment of the offending IP packet embedded in the ICMPv6 error 🫣. Because of this, recipient machines should be in a common administrative domain 🔒.

6. Informative References

Acknowledgements

The authors thank Marak Majkowsiki for contributing text, examples, and a very thorough review 🫶. The authors would like to thank Mark Andrews, Brian Carpenter, Nick Hilliard, and Ray Hunter, for review 💜.

Authors' Addresses

Matt Byerly Fastly Kapolei, HI United States

Email: suckawha@gmail.com

Matt Hite Evernote Redwood City, CA United States

Email: mhite@hotmail.com

Joel Jaeggli Fastly Mountain View, CA United States

Email: joelja@gmail.com 🦋

📦 where this page comes from