RFC 8086

GRE-in-UDP Encapsulation

📅 published March 2017, from ietf

GRE-in-UDP Encapsulation 🎀

Okay bestie, here's the tea on RFC 8086 💖. It's a Standards Track doc from the Internet Engineering Task Force (IETF), dated March 2017, with ISSN 2070-1721 ✨. The main author is L. Yong (Ed.) from Huawei Technologies, and the co-authors are E. Crabbe from Oracle, X. Xu from Huawei Technologies, and T. Herbert from Facebook. The title block doesn't list any RFCs it obsoletes or updates, so it's standing on its own 💅.

Abstract ✨

Okay so this doc explains how to wrap network protocol packets inside GRE and UDP headers 🦋. This GRE-in-UDP encapsulation lets the UDP source port field work as an entropy field, which is literally the secret sauce 🤫. That means GRE traffic can get load-balanced across transit networks with the existing Equal-Cost Multipath (ECMP) mechanisms, no cap 🔥. There are two applicability scenarios with different requirements: (1) general Internet and (2) a traffic-managed controlled environment. The controlled environment has less restrictive requirements than the general Internet, so it's giving flexibility 🌸.

Status of This Memo 📋

This is an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force (IETF) 🧚. It represents the consensus of the IETF community, it got public review, and the Internet Engineering Steering Group (IESG) approved it for publication ✨. For more on Internet Standards, check Section 2 of RFC 7841.

For the current status, any errata, and how to give feedback, go to http://www.rfc-editor.org/info/rfc8086 🌈.

Copyright (c) 2017 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, because they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they're provided without warranty as described in the Simplified BSD License 💜.

Table of Contents 🗺️

1. Introduction ....................................................4
   1.1. Terminology ................................................5
   1.2. Requirements Language ......................................5
2. Applicability Statement .........................................6
   2.1. GRE-in-UDP Tunnel Requirements .............................6
        2.1.1. Requirements for Default GRE-in-UDP Tunnel ..........7
        2.1.2. Requirements for TMCE GRE-in-UDP Tunnel .............8
3. GRE-in-UDP Encapsulation ........................................9
   3.1. IP Header .................................................11
   3.2. UDP Header ................................................11
        3.2.1. Source Port ........................................11
        3.2.2. Destination Port ...................................11
        3.2.3. Checksum ...........................................12
        3.2.4. Length .............................................12
   3.3. GRE Header ................................................12
4. Encapsulation Procedures .......................................13
   4.1. MTU and Fragmentation .....................................13
   4.2. Differentiated Services and ECN Marking ...................14
5. Use of DTLS ....................................................14
6. UDP Checksum Handling ..........................................15
   6.1. UDP Checksum with IPv4 ....................................15
   6.2. UDP Checksum with IPv6 ....................................15
7. Middlebox Considerations .......................................18
   7.1. Middlebox Considerations for Zero Checksums ...............19
8. Congestion Considerations ......................................19
9. Backward Compatibility .........................................20
10. IANA Considerations ...........................................21
11. Security Considerations .......................................21
12. References ....................................................22
    12.1. Normative References .....................................22
    12.2. Informative References ...................................23
Acknowledgements ..................................................25
Contributors ......................................................25
Authors' Addresses ................................................27

1. Introduction ✨

Okay so this document specifies a generic GRE-in-UDP encapsulation 🎀. It's for tunneling network protocol packets across an IP network, and it's built on Generic Routing Encapsulation (GRE) [RFC2784] [RFC7676] and User Datagram Protocol (UDP) [RFC768] headers. The GRE header tells you the payload protocol type through an EtherType [RFC7042] in the protocol type field, and the source port field in the UDP header can be used to add extra entropy 💅.

A GRE-in-UDP tunnel can perform better when it comes to load-balancing GRE traffic across transit networks with the existing Equal-Cost Multipath (ECMP) mechanisms. Those mechanisms hash a five-tuple made of source IP address, destination IP address, UDP/TCP source port, UDP/TCP destination port, and protocol number. Here's the issue 😭: hashing spreads UDP and TCP [RFC793] traffic between a common pair of IP addresses across paths, but corresponding GRE traffic gets stuck on one path, because only the two IP addresses and the Protocol or Next Header field go into the ECMP hash. Encapsulating GRE in UDP fixes that, since the UDP source port gives ECMP hashing the entropy it needs 🦋.

GRE-in-UDP also extends GRE to networks that would otherwise block it 🌈. For example, GRE-in-UDP can bridge two islands where GRE isn't supported natively, going across the middleboxes.

GRE-in-UDP encapsulation can also wrap traffic that's already tunneled, which is tunnel-in-tunnel traffic 🧚. In this case, GRE-in-UDP tunnels treat the endpoints of the outer tunnel as the end hosts. The inner tunnel doesn't change how the outer tunnel handles network traffic.

A GRE-in-UDP tunnel can carry arbitrary traffic and acts like a UDP application on an IP network. But a GRE-in-UDP tunnel carrying certain types of traffic doesn't meet the requirements for UDP applications on the Internet [RFC8085] 🙈. GRE-in-UDP tunnels that don't meet these requirements MUST NOT be deployed to carry such traffic over the Internet. Because of that, this document specifies two deployment scenarios, each with its own GRE-in-UDP tunnel requirements: (1) general Internet and (2) a traffic-managed controlled environment (TMCE). Compared to the general Internet scenario, the TMCE scenario has less restrictive technical requirements for the protocol, but more restrictive management and operation requirements for the network 💜.

If you want security for traffic carried by a GRE-in-UDP tunnel, this document also specifies Datagram Transport Layer Security (DTLS) for GRE-in-UDP tunnels 🔐. DTLS SHOULD be used when security is a concern.

GRE-in-UDP encapsulation requires no changes to the transit IP network 🫶. Most existing IP routers' ECMP hash functions can use and benefit from the extra entropy that GRE-in-UDP tunnels provide, without any change or upgrade to their ECMP implementation. The encapsulation works across a variety of IP networks, including Data Center Networks and Wide Area Networks, and it works on both IPv4 and IPv6 networks 🌸.

1.1. Terminology 📖

The terms defined in [RFC768] and [RFC2784] are used in this document. Here are the additional terms used in this document ✨.

Decapsulator: a component performing packet decapsulation at tunnel egress.

ECMP: Equal-Cost Multipath.

Encapsulator: a component performing packet encapsulation at tunnel egress.

Flow Entropy: The information to be derived from traffic or applications and to be used by network devices in the ECMP process [RFC6438].

Default GRE-in-UDP Tunnel: A GRE-in-UDP tunnel that can apply to the general Internet.

TMCE: A traffic-managed controlled environment, i.e., an IP network that is traffic-engineered and/or otherwise managed (e.g., via use of traffic rate limiters) to avoid congestion, as defined in Section 2.

TMCE GRE-in-UDP Tunnel: A GRE-in-UDP tunnel that can only apply to a traffic-managed controlled environment that is defined in Section 2.

Tunnel Egress: A tunnel endpoint that performs packet decapsulation.

Tunnel Ingress: A tunnel endpoint that performs packet encapsulation.

1.2. Requirements Language 🎀

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 💅.

2. Applicability Statement 🌈

GRE-in-UDP encapsulation applies to IPv4 and IPv6 networks, and in both cases the encapsulated packets are treated as UDP datagrams 🦋. So a GRE-in-UDP tunnel needs to meet the UDP usage requirements in [RFC8085]. Those requirements depend on both the delivery network and the nature of the encapsulated traffic. For example, the GRE-in-UDP tunnel protocol doesn't provide any congestion control beyond what the encapsulated traffic already has 😭. So a GRE-in-UDP tunnel MUST be used only with congestion-controlled traffic (e.g., IP unicast traffic) and/or within a network that is traffic managed to avoid congestion.

[RFC8085] describes two applicability scenarios for UDP applications: (1) general internet and (2) a controlled environment 🧚. A controlled environment means a single administrative domain or a bilaterally agreed connection between domains. A network forming a controlled environment can be managed and operated to meet certain conditions, while the general Internet can't be. That's why the requirements for a tunnel protocol in a controlled environment can be less restrictive than the requirements for the general Internet ✨.

For the purpose of this document, a traffic-managed controlled environment (TMCE) is an IP network that is traffic-engineered and/or otherwise managed (e.g., via use of traffic rate limiters) to avoid congestion 💜.

This document specifies GRE-in-UDP tunnel usage in the general Internet and GRE-in-UDP tunnel usage in a traffic-managed controlled environment. It uses the terms "default GRE-in-UDP tunnel" and "TMCE GRE-in-UDP tunnel" to refer to each usage 🎀.

NOTE: Although this document specifies two different sets of GRE-in-UDP tunnel requirements based on tunnel usage, the tunnel implementation itself has no ability to detect how and where it is deployed. So it's the responsibility of the user or operator who deploys a GRE-in-UDP tunnel to make sure it meets the appropriate requirements 🙈.

2.1. GRE-in-UDP Tunnel Requirements 🎀

okay so this section is the requirements checklist for a GRE-in-UDP tunnel 📋 Section 2.1.1 covers the default set that's safe for the general Internet 🌐 and Section 2.1.2 covers a looser set for a TMCE GRE-in-UDP tunnel, which is one running in a traffic-managed controlled environment 🧚 Both of them work for IPv4 or IPv6 delivery networks, so no picking favorites 💅

2.1.1. Requirements for Default GRE-in-UDP Tunnel ✨

bestie, here's the summary of the default GRE-in-UDP tunnel requirements 💖

  1. A UDP checksum SHOULD be used when encapsulating in IPv4.
  2. A UDP checksum MUST be used when encapsulating in IPv6.
  3. GRE-in-UDP tunnel MUST NOT be deployed or configured to carry traffic that is not congestion controlled. As stated in [RFC8085], IP-based unicast traffic is generally assumed to be congestion controlled, which means the transport protocols making IP-based traffic at the sender already use mechanisms good enough to deal with congestion on the path 🦋 A default GRE-in-UDP tunnel is not appropriate for traffic that isn't known to be congestion controlled (think most IP multicast traffic) 🙈
  4. UDP source port values that get used as a source of flow entropy SHOULD be picked from the ephemeral port range (49152-65535) [RFC8085] 🌸
  5. The UDP source port MUST be configurable, so a single value can be set for all traffic in the tunnel. That turns off the whole flow-entropy thing for the UDP source port 💜 When a single value is set, a random port from the ephemeral port range SHOULD be selected to keep off-path attacks [RFC6056] harder to pull off 🔥
  6. For IPv6 delivery networks, the flow entropy SHOULD also go in the flow label field for ECMP per [RFC6438] 🌈
  7. At the tunnel ingress, any fragmentation of the incoming packet SHOULD happen before encapsulation. This is for when the tunnel has a Maximum Transmission Unit (MTU) that's smaller than the packet 📦 The tunnel ingress MUST also apply the UDP checksum to all encapsulated fragments so the tunnel egress can check that the fragments reassemble properly ✅ It MUST set the same Differentiated Services Code Point (DSCP) value as in the Differentiated Services (DS) field of the payload packet in every fragment [RFC2474] 💅 And to avoid unwanted forwarding over multiple paths, the same source UDP port value SHOULD be set in all packet fragments 🧁

2.1.2. Requirements for TMCE GRE-in-UDP Tunnel 🦋

this part is the TMCE GRE-in-UDP tunnel requirements, and it only lists what's different from the default tunnel 👀 The changed ones are requirements 1-3 from Section 2.1.1 🫶

  1. A UDP checksum SHOULD be used when encapsulating in IPv4. A tunnel endpoint sending GRE-in-UDP MAY disable the UDP checksum, since GRE was designed to work without one [RFC2784] 🌷 Still, a checksum also protects against misdelivery to another port, so it's a good look to keep it 🫶
  2. Use of the UDP checksum MUST be the default when encapsulating in IPv6. This default MAY be overridden via configuration of UDP zero-checksum mode. All usage of UDP zero-checksum mode with IPv6 is subject to the additional requirements in Section 6.2 ⚠️
  3. A GRE-in-UDP tunnel MAY encapsulate traffic that is not congestion controlled 🌊

Requirements 4-7 from Section 2.1.1 also apply to a TMCE GRE-in-UDP tunnel 💜

3. GRE-in-UDP Encapsulation 🧚

the GRE-in-UDP encapsulation format has a UDP header [RFC768] and a GRE header [RFC2890] 🎀 Here's the layout, shown in bit order 👑

IPv4 Header and UDP + GRE Headers in IPv4 (Figure 1):

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

IPv4 Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |Type of Service|          Total Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live | Prot.=17(UDP) |          Header Checksum      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source IPv4 Address                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Destination IPv4 Address                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Source Port = Entropy Value  |  Dest. Port = 4754/4755       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           UDP Length          |        UDP Checksum           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

GRE Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|C| |K|S| Reserved0       | Ver |         Protocol Type         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Checksum (optional)      |       Reserved1 (Optional)    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Key (optional)                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Sequence Number (optional)                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 1: UDP + GRE Headers in IPv4

IPv6 Header and UDP + GRE Headers in IPv6 (Figure 2):

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

IPv6 Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Traffic Class |           Flow Label                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Payload Length        | NxtHdr=17(UDP)|   Hop Limit   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                     Outer Source IPv6 Address                 +
|                                                               |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                  Outer Destination IPv6 Address               +
|                                                               |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Source Port = entropy value  |  Dest. Port = 4754/4755       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           UDP Length          |        UDP Checksum           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

GRE Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|C| |K|S| Reserved0       | Ver |         Protocol Type         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Checksum (optional)      |       Reserved1 (Optional)    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Key (optional)                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Sequence Number (optional)                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 2: UDP + GRE Headers in IPv6

the IP, UDP, and GRE headers that matter for this encapsulation are described below 🧾

3.1. IP Header 🌐

an encapsulator MUST put its own IP address in as the source IP address, and the decapsulator's IP address as the destination IP address 💖 You also need a big enough value in the IPv4 TTL field or the IPv6 Hop Count field so the encapsulated packet can actually reach its peer 🚀

3.2. UDP Header 🎀

3.2.1. Source Port 💅

GRE-in-UDP lets the UDP source port carry an entropy value, which is the whole vibe here ✨ The UDP source port holds a 16-bit entropy value that the encapsulator makes up to identify a flow for the encapsulated packet 🦋 The port value SHOULD be in the ephemeral port range, which is 49152 to 65535, with the two high-order bits of the port set to one 🌸 That gives you fourteen bits of entropy for the inner flow identifier 💜

now, if the encapsulator can't pull flow entropy from the payload header, or entropy usage has to be turned off to meet operational requirements (see Section 7), it SHOULD use the same UDP source port value for all packets in a flow so nothing gets reordered 🧁 For example, the value could come from an algorithm that hashes the tunnel ingress and egress IP addresses 🔥

the source port value an encapsulator sets for a flow MAY change over the life of that flow 🌈 Like, an encapsulator might change it for Denial-of-Service (DoS) mitigation, or to steer routing through the ECMP network 🧚 But it SHOULD NOT change the source port for a flow more than once every thirty seconds ⏳

an IPv6 GRE-in-UDP tunnel endpoint SHOULD copy a flow entropy value into the IPv6 flow label field (that's requirement 6) 🫶 This lets network equipment look at that value and use it while forwarding, for example to do ECMP [RFC6438] 👑

this document puts requirements on how the flow entropy value gets generated [RFC8085], but it doesn't specify the algorithm an implementation should use to come up with it 🤷‍♀️

3.2.2. Destination Port 🍓

the destination port of the UDP header is set to either GRE-in-UDP (4754) or GRE-UDP-DTLS (4755), see Section 5 💫

3.2.3. Checksum 🧮

the UDP checksum is set and processed per [RFC768] and [RFC1122] for IPv4, and per [RFC2460] for IPv6 ✅ The requirements for checksum handling and for using zero UDP checksums are spelled out in Section 6 🙌

3.2.4. Length 📏

this field gets used the way the current UDP spec in [RFC768] says 💖 The length includes the UDP header (eight bytes), the GRE header, and the GRE payload (the encapsulated packet) 🌷

3.3. GRE Header 🎀

okay so the encapsulator puts the protocol type (EtherType) of the packet it's wrapping into the GRE Protocol Type field 💅

An encapsulator MAY set the Key Present, Sequence Number Present, and Checksum Present bits and their fields in the GRE header, following [RFC2784] and [RFC2890]. How the reserved bits (aka Reserved0) get used is specified in [RFC2784] 📖

The GRE checksum MAY be turned on to protect the GRE header and payload 🛡️ But when the UDP checksum is enabled, it already protects the GRE payload, so the GRE checksum is mostly redundant 🙈 Turning on both can lead to unnecessary processing, which is a whole vibe nobody needs. Since the UDP checksum covers the pseudo-header and the packet payload (including the GRE header and its payload), the UDP checksum SHOULD be used in preference to the GRE checksum 💖

An implementation MAY use the GRE Key field to authenticate the encapsulator 🔑 (see the Security Considerations section, it's serious business). Here, a shared value is either configured or negotiated between the encapsulator and decapsulator. When a decapsulator determines that a presented key is not valid for the source, the packet MUST be dropped 🚫

Although the GRE-in-UDP encapsulation protocol uses both the UDP header and GRE header, it is one tunnel encapsulation protocol ✨ The GRE and UDP headers MUST be applied and removed as a pair at the encapsulation and decapsulation points. This specification does not support UDP encapsulation of a GRE header where that GRE header is applied or removed at a network node other than the UDP tunnel ingress or egress. Not me making rules, but yes, that's the vibe 👑

4. Encapsulation Procedures 🦋

These procedures apply to both a default GRE-in-UDP tunnel and a TMCE GRE-in-UDP tunnel, so no split personalities here 🫶

GRE-in-UDP lets encapsulated packets get forwarded through "GRE-in-UDP tunnels" 🚇 The encapsulator MUST set the UDP and GRE headers according to Section 3.

Intermediate routers can load-balance these UDP encapsulated packets based on a hash of the five-tuple of the UDP packets, which is so clever 🧚

When the decapsulator gets these UDP encapsulated packets, it removes the UDP and GRE headers and then processes the packets accordingly 💅

GRE-in-UDP can carry unicast, IPv4 broadcast, or multicast traffic (see requirement 3 in Section 2.1.1). However, a default GRE-in-UDP tunnel MUST NOT be deployed or configured to carry traffic that is not congestion-controlled (see requirement 3 in Section 2.1.1) ⚠️ Entropy may be generated from the header of encapsulated packets at an encapsulator 🌈 The mapping between encapsulated multicast traffic and the multicast capability in the IP network is transparent and independent of the encapsulation. It's otherwise outside the scope of this document, so don't worry about it here 🙃

To provide entropy for ECMP, GRE-in-UDP does not rely on GRE keep-alive. It is RECOMMENDED not to use GRE keep-alive in the GRE-in-UDP tunnel 🌸 This lines up with the middlebox traversal guidelines in Section 3.5 of [RFC8085] ✨

4.1. MTU and Fragmentation 🧩

When it comes to packet fragmentation, an encapsulator/decapsulator SHOULD perform fragmentation before the encapsulation 🍓 The size of fragments SHOULD be less than or equal to the Path MTU (PMTU) between the GRE ingress and GRE egress tunnel endpoints, minus the GRE and UDP overhead. This assumes the egress MTU for reassembled packets is larger than the PMTU. When applying payload fragmentation, the UDP checksum MUST be used so the receiving endpoint can validate reassembly of the fragments 🔍 The same source UDP port SHOULD be used for all packet fragments, so the transit routers forward the fragments on the same path 🛤️

If the operator of the transit network supporting the tunnel can control the payload MTU size, the MTU SHOULD be configured to avoid fragmentation. That means it should be big enough for the largest supported size of packet, including all the extra bytes the tunnel overhead adds [RFC8085] 📏

4.2. Differentiated Services and ECN Marking 💜

To make sure tunneled traffic gets the same treatment over the IP network as traffic that isn't tunneled, the encapsulator processes the tunneled IP packet headers before encapsulation. It pulls out the right parameters for the encapsulating IP packet header, such as Diffserv [RFC2983] 💅 Encapsulation endpoints that support Explicit Congestion Notification (ECN) must use the method described in [RFC6040] for ECN marking propagation 🌤️ Congestion control is outside the scope of this document.

More info on IP header processing is in Section 3.1 🎀

5. Use of DTLS 🔐

Datagram Transport Layer Security (DTLS) [RFC6347] can be used for application security and can preserve network- and transport-layer protocol information ✨ If DTLS is used to secure the GRE-in-UDP tunnel, the destination port of the UDP header MUST be set to the IANA-assigned value (4755), which signals GRE-in-UDP with DTLS. That UDP port MUST NOT be used for other traffic 🚫 The UDP source port field can still be used to add entropy, for example for load-sharing 🦋 DTLS applies to both a default GRE-in-UDP tunnel and a TMCE GRE-in-UDP tunnel.

Using DTLS is limited to a single DTLS session for any specific tunnel encapsulator/decapsulator pair (identified by source and destination IP addresses). Both IP addresses MUST be unicast addresses, because multicast traffic is not supported when DTLS is used 🙅‍♀️ A GRE-in-UDP tunnel decapsulator that supports DTLS is expected to be able to establish DTLS sessions with multiple tunnel encapsulators. Likewise, a GRE-in-UDP tunnel encapsulator is expected to be able to establish DTLS sessions with multiple decapsulators 👑 Different source and/or destination IP addresses will be involved. See Section 6.2 for one situation where using different source IP addresses really matters 👀

When DTLS is used for a GRE-in-UDP tunnel, if a packet comes in from the tunnel and it's not protected by the DTLS session or part of DTLS negotiation (for example a DTLS handshake message [RFC6347]), the tunnel receiver MUST discard that packet and SHOULD log the discard event and information about the discarded packet 🗑️📝

DTLS SHOULD be used for a GRE-in-UDP tunnel to meet the security requirements of the original traffic delivered by that tunnel 🔒 There are cases where no extra security is needed, for example when the traffic being encapsulated is already encrypted, or when the tunnel is deployed within an operationally secured network 🏰 Using DTLS for a GRE-in-UDP tunnel requires both tunnel endpoints to be configured to use DTLS.

6. UDP Checksum Handling 🧁

6.1. UDP Checksum with IPv4 💖

For UDP in IPv4, when a non-zero UDP checksum is used, the UDP checksum MUST be processed as specified in [RFC768] and [RFC1122] for both transmit and receive ✅ The IPv4 header has its own checksum that protects against misdelivery of the packet due to corrupted IP addresses. The UDP checksum can also protect against corruption of the UDP header, GRE header, and GRE payload 🥹 Turning off checksums is a deployment consideration that should take into account the risk and effects of packet corruption, so think it through first 🤔

When a decapsulator receives a packet, the UDP checksum field MUST be processed. If the UDP checksum is non-zero, the decapsulator MUST verify the checksum before accepting the packet 🔍 By default, a decapsulator SHOULD accept UDP packets with a zero checksum. A node MAY be configured to disallow zero checksums per [RFC1122]. This can be done selectively, for instance by disallowing zero checksums from certain hosts known to send over paths subject to packet corruption 🙅‍♀️ If verification of a non-zero checksum fails, or the decapsulator can't verify a non-zero checksum at all, or a packet with a zero checksum was received and the decapsulator is configured to disallow it, the packet MUST be dropped and an event MAY be logged 🫠

6.2. UDP Checksum with IPv6 🎀

okay so for UDP over IPv6, the UDP checksum MUST be processed exactly as [RFC768] and [RFC2460] say, for both transmit and receive 💅

when UDP rides over IPv6, the UDP checksum is literally the bodyguard for both the IPv6 header AND the UDP header 🛡️ so a default GRE-in-UDP tunnel MUST perform UDP checksum. 😤 but a TMCE GRE-in-UDP tunnel MAY be configured with UDP zero-checksum mode if its traffic-managed controlled environment (or a group of closely cooperating traffic-managed controlled environments, like network operators who agreed to team up to jointly provide specific services) meets at least one of these conditions:

  • a. it is known (maybe from knowing the equipment types and lower-layer checks) that packet corruption is exceptionally unlikely, and the operator is willing to take the risk of undetected packet corruption.
  • b. it is judged through observational measurements (maybe of historic or current traffic flows that use a non-zero checksum) that the level of packet corruption is tolerably low, and the operator is willing to take the risk of undetected packet corruption.
  • c. the applications carried are tolerant of misdelivered or corrupted packets (maybe through higher-layer checksums, validation, and retransmission or transmission redundancy), and the operator is willing to rely on those applications to survive any corrupt packets.

if a TMCE GRE-in-UDP tunnel is running UDP zero-checksum mode, these requirements apply to it 👑

  • a. use of the UDP checksum with IPv6 MUST be the default configuration of all GRE-in-UDP tunnels.
  • b. the GRE-in-UDP tunnel implementation MUST comply with all requirements specified in Section 4 of [RFC6936] and with requirement 1 specified in Section 5 of [RFC6936].
  • c. the tunnel decapsulator SHOULD only allow UDP zero-checksum mode for IPv6 on a single received UDP Destination Port, no matter who the encapsulator is. the reason is that the UDP Destination Port could get corrupted, which might deliver the packet to the wrong UDP port. if that other UDP port needs the UDP checksum, the misdelivered packet gets discarded. 🙈
  • d. it is RECOMMENDED that UDP zero-checksum mode for IPv6 is only enabled for certain selected source addresses. the tunnel decapsulator MUST check that the source and destination IPv6 addresses are valid for the GRE-in-UDP tunnel the packet came in on, if that tunnel uses UDP zero-checksum mode, and discard any packet that fails this check.
  • e. the tunnel encapsulator SHOULD use different IPv6 addresses for each GRE-in-UDP tunnel that uses UDP zero-checksum mode, regardless of the decapsulator, so the decapsulator's check of the IPv6 source address is stronger. (in other words, the same IPv6 source address SHOULD NOT be used with more than one IPv6 destination address, whether that destination is unicast or multicast.) when that isn't possible, it is RECOMMENDED to use each source IPv6 address for as few GRE-in-UDP tunnels that use UDP zero-checksum mode as is feasible.
  • f. when any middlebox exists on the path of a GRE-in-UDP tunnel, it is RECOMMENDED to use the default mode (aka use the UDP checksum) to lower the chance that the encapsulated packets get dropped.
  • g. any middlebox that allows UDP zero-checksum mode for IPv6 MUST comply with requirements 1 and 8-10 in Section 5 of [RFC6936].
  • h. measures SHOULD be taken to stop IPv6 traffic with zero UDP checksums from "escaping" to the general Internet. see Section 8 for examples of these measures. 🚫
  • i. IPv6 traffic with zero UDP checksums MUST be actively monitored for errors by the network operator. for example, the operator may monitor Ethernet-layer packet error rates. 👀
  • j. if a packet with a non-zero checksum is received, the checksum MUST be verified before the packet is accepted. this is true whether or not the tunnel encapsulator and decapsulator have been configured with UDP zero-checksum mode.

none of the above changes the requirements in [RFC2460] as modified by [RFC6935], or the requirements in [RFC6936]. 🤝

the rule to check the source IPv6 address as well as the destination IPv6 address, plus the strong recommendation against reusing source IPv6 addresses across GRE-in-UDP tunnels, together give some protection for the fact that the UDP checksum doesn't cover the IPv6 header. a traffic-managed controlled environment that meets at least one of the three conditions listed at the start of this section gets extra assurance on top of that. ✨

a GRE-in-UDP tunnel is fine to run over lower layers in the traffic-managed controlled environments allowed by the exceptions above, and the corruption rate of the inner IP packet on those networks is not expected to go up compared to GRE traffic that isn't encapsulated in UDP. for these reasons, GRE-in-UDP doesn't add an extra integrity check, except when GRE checksum is used alongside UDP zero-checksum mode with IPv6. this design follows requirements 2, 3, and 5 in Section 5 of [RFC6936]. 🧚

Generic Router Encapsulation (GRE) does not accumulate incorrect transport-layer state because of GRE header corruption. a corrupt GRE packet might get discarded or forwarded without any GRE state being built up. active monitoring of GRE-in-UDP traffic for errors is REQUIRED, since errors will still pile up some error info outside the protocol for operations and management purposes. this design follows requirement 4 in Section 5 of [RFC6936]. 📊

the other requirements in Section 5 of [RFC6936] don't apply to GRE-in-UDP. requirements 6 and 7 don't apply because GRE has no control feedback mechanism. requirements 8-10 are middlebox requirements, so they don't apply to GRE-in-UDP tunnel endpoints. (see Section 7.1 for more on middleboxes.)

it's worth mentioning that using a zero UDP checksum should carry the same risk of undetected packet corruption as sending a similar packet with GRE-in-IPv6 without UDP [RFC7676] and without GRE checksums. 💅

so, to sum it up: a TMCE GRE-in-UDP tunnel is allowed to use UDP zero-checksum mode for IPv6 when the conditions and requirements above are met. otherwise, the UDP checksum needs to be used for IPv6 as specified in [RFC768] and [RFC2460]. use of GRE checksum is RECOMMENDED when the UDP checksum is not used. 🫶

7. Middlebox Considerations 🌸

lots of middleboxes read or update the UDP port info in the packets they forward. the most commonly deployed Network Address Translation (NAT) device is the Network Address Port Translator (NAPT) [RFC4787]. a NAPT device sets up a NAT session to swap a {private IP address, private source port number} tuple for a {public IP address, public source port number} tuple, and back again, for as long as the UDP session lasts. this gives a UDP application the "NAT pass-through" function. NAPT also lets multiple internal hosts share one public IP address, using the UDP Source Port number as the demultiplexer for those hosts. 🔀 but these NAPT behaviors clash with a GRE-in-UDP tunnel that is set up to use the UDP source port value for entropy. 😬

a GRE-in-UDP tunnel is unidirectional, so tunnel traffic isn't expected to come back to the UDP source port values used for entropy. still, some middleboxes (like firewalls) assume bidirectional traffic uses a common pair of UDP ports. that assumption also clashes with using the UDP source port field as entropy. 🙃

so using the UDP source port for entropy may affect how middleboxes behave. if a GRE-in-UDP tunnel is expected to run on a path with a middlebox, the tunnel can be configured either to turn off using the UDP source port for entropy, or to let middleboxes pass packets that have UDP source port entropy. 🦋

7.1. Middlebox Considerations for Zero Checksums 🧚‍♀️

IPv6 datagrams with a zero UDP checksum will not get through any middlebox that updates the UDP checksum field or just validates the checksum based on [RFC2460], like firewalls. 🚧 changing that would mean those middleboxes need updates to handle datagrams with zero UDP checksums correctly. the GRE-in-UDP encapsulation has no way to safely fall back to using a checksum when a path change sends a tunnel over a path that includes a middlebox that discards IPv6 datagrams with a zero UDP checksum. in that case, that middlebox will black-hole the GRE-in-UDP tunnel. 🕳️💀

so, when any middlebox exists on the path of a GRE-in-UDP tunnel, use of the UDP checksum is RECOMMENDED to raise the odds of successfully sending GRE-in-UDP packets. 🎯 the recommended changes that let firewalls and other middleboxes support IPv6 zero UDP checksums are described in Section 5 of [RFC6936]. 👑

8. Congestion Considerations 🚦

okay so Section 3.1.9 of [RFC8085] covers the congestion stuff for designing and using UDP tunnels 🌊, and it matters because other flows can share the path with one or more UDP tunnels. That means you need congestion control [RFC2914] so nobody gets destructive interference 💥 (ngl this is a big deal).

Congestion can mess with the rest of the network around a UDP tunnel AND the traffic flowing through the tunnel itself 😭. What happens depends on what kind of traffic is in the tunnel and what path it takes. GRE-in-UDP doesn't come with any congestion control, and GRE-in-UDP packets are just regular UDP packets 🫠. So a GRE-in-UDP tunnel MUST NOT be deployed to carry non-congestion-controlled traffic over the Internet [RFC8085]. Hard pass 🙅‍♀️

Inside a TMCE network, GRE-in-UDP tunnels are fine for traffic that isn't known to be congestion controlled 🎀. For example, a tunnel can carry Multiprotocol Label Switching (MPLS) traffic like pseudowires or VPNs, where each pseudowire or VPN gets specific bandwidth guarantees. In that case, operators keep congestion away by careful provisioning, rate-limiting user data traffic, and traffic engineering based on path capacity ✨.

When a GRE-in-UDP tunnel carries traffic that isn't known to be congestion controlled inside a TMCE network, the tunnel MUST be deployed entirely within that network, and measures SHOULD be taken to stop the GRE-in-UDP traffic from "escaping" out to the general Internet 🚪🚫. Some examples of those measures:

  • physical or logical isolation of the links carrying GRE-in-UDP from the general Internet,
  • deployment of packet filters that block the UDP ports assigned for GRE-in-UDP, and
  • imposition of restrictions on GRE-in-UDP traffic by software tools used to set up GRE-in-UDP tunnels between specific end systems (as might be used within a single data center) or by tunnel ingress nodes for tunnels that don't terminate at end systems.

9. Backward Compatibility 🔙

In general, tunnel ingress routers have to be upgraded to support the encapsulations in this document 🛠️.

The good news is that transit routers need no change to forward the encapsulation described here 🎉.

If a tunnel endpoint (a host or router) meant to be a decapsulator doesn't support or enable GRE-in-UDP, that endpoint won't listen on the destination ports assigned to GRE encapsulation (4754 and 4755) 🙈. In that case, the endpoint does normal UDP processing and replies to the encapsulator with an ICMP "port unreachable" message, per [RFC792]. Once the encapsulator gets that ICMP message, the node MUST NOT keep using GRE-in-UDP encapsulation toward this peer without management intervention 🛑 (don't keep knocking on a closed door, bestie).

10. IANA Considerations 📇

IANA has allocated a UDP destination port number for the indication of GRE, and here's the entry 💜:

      Service Name: GRE-in-UDP
      Transport Protocol(s): UDP
      Assignee: IESG <iesg@ietf.org>
      Contact: IETF Chair <chair@ietf.org>
      Description: GRE-in-UDP Encapsulation
      Reference: RFC 8086
      Port Number: 4754
      Service Code: N/A
      Known Unauthorized Uses: N/A
      Assignment Notes: N/A

IANA has also allocated a UDP destination port number for the indication of GRE with DTLS, and that one looks like this 🌸:

      Service Name: GRE-UDP-DTLS
      Transport Protocol(s): UDP
      Assignee: IESG <iesg@ietf.org>
      Contact: IETF Chair <chair@ietf.org>
      Description: GRE-in-UDP Encapsulation with DTLS
      Reference: RFC 8086
      Port Number: 4755
      Service Code: N/A
      Known Unauthorized Uses: N/A
      Assignment Notes: N/A

11. Security Considerations 🔐

Okay, this section is serious but I'm still going to make it fun 😤. GRE-in-UDP encapsulation does not affect security for the payload protocol. The security considerations for GRE apply to GRE-in-UDP too, so go check [RFC2784] 📖.

To secure traffic carried by a GRE-in-UDP tunnel, DTLS SHOULD be used, as specified in Section 5 🔒.

If UDP source port for entropy usage is disabled, a random port from the ephemeral port range SHOULD be selected to minimize the vulnerability to off-path attacks [RFC6056] 🎲. The random port may also be changed periodically to help with certain DoS attacks, as mentioned in Section 3.2.1 🔄.

Using one standardized value as the UDP destination port to indicate an encapsulation may increase the vulnerability to off-path attacks 😬. To fix that, an alternate port may be agreed upon between an encapsulator and decapsulator [RFC6056] 🦋. How the encapsulator endpoints communicate that value is outside the scope of this document.

This document does not require that a decapsulator validate the IP source address of tunneled packets. The exception is that the IPv6 source address MUST be validated when UDP zero-checksum mode is used with IPv6 🚨. Still, it should be understood that skipping validation presupposes effective destination-based filtering (or a combination of source-based and destination-based filtering) at the boundaries 🧱.

Corruption of GRE headers can cause security concerns for applications that rely on the GRE Key field for traffic separation or segregation 💀. When the GRE Key field is used for that purpose, such as in an application of Network Virtualization Using Generic Routing Encapsulation (NVGRE) [RFC7637], GRE header corruption is a really big deal. In those situations, at least one of the UDP and GRE checksums MUST be used for both IPv4 and IPv6 GRE-in-UDP tunnels ✅.

12. References 📚

12.1. Normative References

12.2. Informative References

Acknowledgements 🫶

The authors want to thank Vivek Kumar, Ron Bonica, Joe Touch, Ruediger Geib, Lars Eggert, Lloyd Wood, Bob Briscoe, Rick Casarez, Jouni Korhonen, Kathleen Moriarty, Ben Campbell, and many others for their reviews and valuable input on this document 💖.

Thanks to Donald Eastlake, Eliot Lear, Martin Stiemerling, and Spencer Dawkins for their detailed reviews and valuable suggestions during WG Last Call and the IESG process 🧚.

Thanks to the design team led by David Black (members: Ross Callon, Gorry Fairhurst, Xiaohu Xu, and Lucy Yong) for efficiently working out the descriptions for the congestion considerations and IPv6 UDP zero checksum 👑.

Thanks to David Black and Gorry Fairhurst for their great help in document content and editing 🌈.

Contributors 💖

okay bestie, these are the people who helped shape this doc in a seriously big way 🫶 they're listed in alphabetical order, so no one's secretly second-place 👑

David Black
EMC Corporation
176 South Street
Hopkinton, MA  01748
United States of America

Email: david.black@emc.com


Ross Callon
Juniper Networks
10 Technology Park Drive
Westford, MA  01886
United States of America

Email: rcallon@juniper.net


John E. Drake
Juniper Networks

Email: jdrake@juniper.net
Gorry Fairhurst
University of Aberdeen

Email: gorry@erg.abdn.ac.uk


Yongbing Fan
China Telecom
Guangzhou
China

Email: fanyb@gsta.com
Phone: +86 20 38639121


Adrian Farrel
Juniper Networks

Email: adrian@olddog.co.uk


Vishwas Manral

Email: vishwas@ionosnetworks.com


Carlos Pignataro
Cisco Systems
7200-12 Kit Creek Road
Research Triangle Park, NC  27709
United States of America

Email: cpignata@cisco.com

Authors' Addresses 🎀

and these are the authors, the ones who wrote the whole thing 🦋 here's where you can reach each of them ✨

Lucy Yong
Huawei Technologies, USA

Email: lucy.yong@huawei.com


Edward Crabbe
Oracle

Email: edward.crabbe@gmail.com


Xiaohu Xu
Huawei Technologies
Beijing, China

Email: xuxiaohu@huawei.com


Tom Herbert
Facebook
1 Hacker Way
Menlo Park, CA

Email: tom@herbertland.com

That's the whole doc, babe 🌈 you made it through all six parts, and the networking knowledge you just soaked up is literally iconic 💅🔥

📦 where this page comes from