RFC 8926

Geneve: Generic Network Virtualization Encapsulation

📅 published November 2020, from ietf

🎀 RFC 8926: Geneve, Generic Network Virtualization Encapsulation 🦋

okay bestie, settle in because this one is a whole vibe ✨. This is the Geneve spec, published November 2020 as a Standards Track document, which means it's the real deal and not just someone's opinion 💅. It's from the IETF, the Internet Engineering Task Force, and it was written by J. Gross (Ed.) at Intel... wait, that's the wrong column 🙈. Let me sort the title block properly:

  • Authors and editors: J. Gross, Ed., I. Ganga, Ed., and T. Sridhar, Ed. (the "Ed." means they're the editors who wrangled the whole thing together 🫶)
  • Affiliations: J. Gross is listed with no affiliation in the header, I. Ganga is at Intel, and T. Sridhar is at VMware 🌸
  • Category: Standards Track
  • ISSN: 2070-1721
  • Date: November 2020
  • Obsoletes and updates: the title block for this document doesn't list anything it obsoletes or updates, so it's a fresh start, no drama 🧚

Now the actual content. Geneve is an encapsulation protocol designed so tunnels can keep up with how fast network virtualization changes 🚀. The big idea is that the tunnel endpoints, transit fabrics, and controllers all have different strengths and evolve at different speeds, and that's treated as a feature, not a bug 💜.

1. Introduction ✨

1.1. Requirements Language 🎀

Okay, the BCP 14 key words. These are the words that carry real weight in a spec, so they only mean their official thing when they show up in ALL CAPS. The ones this document uses are "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" 🔥. They follow [RFC2119] and [RFC8174] when, and only when, they're capitalised like that.

1.2. Terminology 💅

Here's the glossary, and yes, it's giving definitions, not vibes 😭.

The Network Virtualization over Layer 3 (NVO3) Framework [RFC7365] defines a lot of the concepts used in network virtualization, so go read that for the background 🌈. On top of that, these terms have specific meanings in this document:

Checksum offload: an optimisation built into many NICs (Network Interface Controllers) that computes and verifies upper-layer protocol checksums in hardware, on transmit and receive respectively 🍓. This usually covers the IP and TCP/UDP checksums that the protocol stack would otherwise compute in software.

Clos network: a way of building network fabrics that are bigger than a single switch while keeping non-blocking bandwidth across the connection points 🦋. ECMP spreads traffic across all the links and switches in the fabric. People also call these "leaf and spine" or "fat tree" topologies.

ECMP: Equal Cost Multipath 🧚. It's a routing mechanism that picks from several best next-hop paths by hashing packet headers, which uses bandwidth well and avoids reordering packets within a flow.

Geneve: Generic Network Virtualization Encapsulation 💖. It's the tunneling protocol this document describes, and it's giving main character energy.

LRO: Large Receive Offload 🌸. It's the receiver-side version of LSO, where several protocol segments (mostly TCP) get stitched into bigger data units.

LSO: Large Segmentation Offload ✨. Plenty of commercial NICs offer this so data units bigger than the MTU can be handed to the NIC, which then chops them into segments that fit the MTU, with correct protocol headers. When it's specifically TCP/IP, people call it TSO (TCP Segmentation Offload) 🫶.

Middlebox: in this document, "middlebox" means network service functions or service interposition appliances that usually act as tunnel endpoints, terminating and re-encapsulating Geneve traffic 🎀.

NIC: Network Interface Controller, also called "Network Interface Card" or "Network Adapter" 🌈. A NIC can be part of a tunnel endpoint or a transit device, and it can process Geneve packets or help with processing them.

Transit device: a forwarding element, like a router or switch, along the tunnel's path that's part of the underlay network 🍓. A transit device might understand the Geneve packet format, but it doesn't originate or terminate Geneve packets.

Tunnel endpoint: a component that encapsulates and decapsulates packets, such as Ethernet frames or IP datagrams, in Geneve headers 👑. Because it's the final consumer of tunnel metadata, a tunnel endpoint has the strictest requirements for parsing and interpreting tunnel headers. These can be software, hardware, or a mix of both 💜. Tunnel endpoints are often part of a Network Virtualization Edge (NVE), but they can also live in middleboxes or other pieces of an NVO3 network.

VM: Virtual Machine 🧚.

💀 This is part 1 of 8, so more is coming, bestie. Stay tuned for the design requirements and the packet format next 🌸.

2. Design Requirements ✨

okay so bestie, Geneve was literally built for network virtualization in data centers 💅 Picture tunnels acting as a backplane between virtual switches living in hypervisors, physical switches, middleboxes, and other appliances. The underlay can be any IP network, but Clos networks built with ECMP links are the go-to, because they give you consistent bisectional bandwidth across every connection point 🦋 A lot of the vibes behind network virtualization overlays over IP networks are laid out in the NVO3 Framework [RFC7365]. Figure 1 shows a hypervisor, a top-of-rack switch talking to physical servers, and a WAN uplink, all linked with Geneve tunnels over a simplified Clos network. Those tunnels carry frames from the attached stuff, like VMs or physical links 🌸

     +---------------------+           +-------+  +------+
     | +--+  +-------+---+ |           |Transit|--|Top of|==Physical
     | |VM|--|       |   | | +------+ /|Router |  | Rack |==Servers
     | +--+  |Virtual|NIC|---|Top of|/ +-------+\/+------+
     | +--+  |Switch |   | | | Rack |\ +-------+/\+------+
     | |VM|--|       |   | | +------+ \|Transit|  |Uplink|   WAN
     | +--+  +-------+---+ |           |Router |--|      |=========>
     +---------------------+           +-------+  +------+
            Hypervisor

                 ()===================================()
                         Switch-Switch Geneve Tunnels

                     Figure 1: Sample Geneve Deployment

So the tunneling protocol has to be able to take advantage of the different (and constantly evolving) abilities of every device in the underlay and overlay networks 🧚 That leads to three requirements for the data plane tunneling protocol:

  • The data plane is generic and extensible enough to support current and future control planes 💜
  • Tunnel components are efficiently implementable in both hardware and software without restricting capabilities to the lowest common denominator ✨
  • High performance over existing IP fabrics is maintained 🔥

These requirements get the full breakdown in the subsections below 🎀

2.1. Control Plane Independence

Some protocols for network virtualization have included a control plane as part of the tunnel format spec. VXLAN [RFC7348] is the big example, since it prescribed a multicast-learning-based control plane 🙈 But these specs have mostly been treated as describing only the data format. The VXLAN packet format has actually ended up with a wild variety of control planes built on top of it 💅

Settling on a data format is a clear win, ngl. Most of these protocols are only superficially different, so there's little reason to duplicate effort. Control planes are a whole different story though, because they're diverse in very fundamental ways 🌈 Standardizing them is also less clear-cut, given how many different requirements, goals, and deployment scenarios are out there.

So Geneve is a pure tunnel format spec 🦋 It can meet the needs of many control planes by explicitly not picking any one of them. This keeps the data format shared and lowers the chance it becomes obsolete when future control plane upgrades show up. Galaxy brain move, periodt 👑

2.2. Data Plane Extensibility

To support current and future control planes, you need an options infrastructure, so new metadata types can be defined, deployed, and then either finalized or retired 💖 Options also let vendors differentiate their products by doing independent development in their own core specialty, which speeds up advancement overall ✨ The most common way to implement options is the Type-Length-Value (TLV) format 🎀

Here's the thing though: options can be used for non-wirespeed control packets, but they matter just as much in data packets for segregating and directing forwarding 👀 The examples from earlier, input-port-based security policies and terminating/re-encapsulating service interposition, both need tags placed on data packets. So even though limiting extensibility to only control packets would simplify the datapath, it would not satisfy the design requirements 🙅‍♀️

2.2.1. Efficient Implementation

Software flexibility and hardware performance clash a lot, and it's honestly hard to resolve 😭 For any set of functionality, maximizing performance is obviously the dream. But that doesn't mean new features that can't run at full speed today should be banned 🍓 So for a protocol to count as efficiently implementable, it's expected to have a set of common capabilities that can be reasonably handled across platforms, plus a graceful way to handle more advanced features when the situation calls for it 💅

A variable-length header and options can make people wonder whether the protocol is truly efficiently implementable in hardware. To answer that for Geneve, first split "hardware" into two categories: tunnel endpoints and transit devices 🧚

Tunnel endpoints have to parse the variable-length header, options included, and then take action. They're actively participating in the protocol, so they're the most affected by Geneve 💜 But since tunnel endpoints are the ultimate consumers of the data, transmitters can tailor their output to whatever the recipient can handle ✨

Transit devices might be able to interpret the options, but they're non-terminating devices, so they don't originate or terminate the Geneve packet 🦋 That's why they MUST NOT modify Geneve headers and MUST NOT insert or delete options, since that job belongs to tunnel endpoints. Options, if present in the packet, MUST only be generated and terminated by tunnel endpoints. Transit devices interpreting options is OPTIONAL 🌸

Also, either tunnel endpoints or transit devices MAY use NIC offload capabilities, like checksum offload, to speed up Geneve packet processing 🔥 A Geneve variable-length header should not stop tunnel endpoints and transit devices from using those offloads. Slay 💅

2.3. Use of Standard IP Fabrics

IP is the dominant transport mechanism, full stop 👑 Over time, a bunch of techniques have evolved to make it robust, efficient, and cheap. So it's totally natural to use IP fabrics as the transit network for Geneve 🎀 Luckily, IP encapsulation and addressing are enough to hit the main goal, which is delivering packets to the correct point in the network through standard switching and routing 🌈

Also, nearly all underlay fabrics are built to exploit parallelism in traffic, spreading load across multiple links without reordering packets inside individual flows 🦋 ECMP techniques usually parse and hash the addresses and port numbers in the packet to pick an outgoing link. The problem is that tunnels often give you poor ECMP performance 😭 Without extra knowledge of the protocol, the encapsulated traffic is hidden from the fabric by design, so only the tunnel endpoint addresses are left to hash.

Since Geneve should perform well on these existing fabrics, entropy from the encapsulated packets has to be exposed in the tunnel header ✨ The most common technique for this is using the UDP source port, which gets more detail in Section 3.3. Main character energy 💖

3. Geneve Encapsulation Details 🧚

The Geneve packet format is a compact tunnel header encapsulated in UDP over either IPv4 or IPv6 🌸 A small fixed tunnel header gives you control info plus a base level of functionality and interoperability, with a focus on simplicity. After that comes a set of variable-length options so the protocol can keep innovating 🦋 Last up, the payload is a protocol data unit of the indicated type, like an Ethernet frame 💜 Sections 3.1 and 3.2 show the Geneve packet format carried over (for example) Ethernet, along with an Ethernet payload, so you can see it in action 🫶

3.1. Geneve Packet Format over IPv4 ✨

okay bestie, this is where we get to see the whole Geneve packet stacked up over IPv4 🎀 the outer headers go on first, then the Geneve header, then the inner frame 🧚

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   Outer Ethernet Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Outer Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Outer Destination MAC Address |   Outer Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Outer Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Outer VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |    Ethertype = 0x0800 IPv4    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer IPv4 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version|  IHL  |Type of Service|          Total Length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Identification        |Flags|      Fragment Offset    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Time to Live |Protocol=17 UDP|         Header Checksum       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Outer Source IPv4 Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Outer Destination IPv4 Address              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer UDP Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Source Port = xxxx      |    Dest Port = 6081 Geneve    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Geneve Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Virtual Network Identifier (VNI)       |    Reserved   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      ~                    Variable-Length Options                    ~
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Inner Ethernet Header (example payload):
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Inner Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Inner Destination MAC Address |   Inner Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Inner Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Inner VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Payload:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Ethertype of Original Payload |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
      |                                  Original Ethernet Payload    |
      |                                                               |
      ~ (Note that the original Ethernet frame's preamble, start      ~
      | frame delimiter (SFD), and frame check sequence (FCS) are not |
      | included, and the Ethernet payload need not be 4-byte aligned)|
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Frame Check Sequence:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   New Frame Check Sequence (FCS) for Outer Ethernet Frame     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 2: Geneve Packet Format over IPv4 💜 that's the full IPv4 stack, bestie

3.2. Geneve Packet Format over IPv6 🦋

same vibe, but now the outer IP layer is IPv6 🌈 the rest of the layout lines up with the IPv4 version, so you can spot the differences right away 👑

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   Outer Ethernet Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Outer Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Outer Destination MAC Address |   Outer Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Outer Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Outer VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |    Ethertype = 0x86DD IPv6    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer IPv6 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version| Traffic Class |           Flow Label                  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Payload Length        | NxtHdr=17 UDP |   Hop Limit   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                     Outer Source IPv6 Address                 +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                  Outer Destination IPv6 Address               +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer UDP Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Source Port = xxxx      |    Dest Port = 6081 Geneve    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Geneve Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Virtual Network Identifier (VNI)       |    Reserved   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      ~                    Variable-Length Options                    ~
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Inner Ethernet Header (example payload):
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Inner Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Inner Destination MAC Address |   Inner Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Inner Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Inner VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Payload:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Ethertype of Original Payload |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
      |                                  Original Ethernet Payload    |
      |                                                               |
      ~ (Note that the original Ethernet frame's preamble, start      ~
      | frame delimiter (SFD), and frame check sequence (FCS) are not |
      | included, and the Ethernet payload need not be 4-byte aligned)|
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Frame Check Sequence:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   New Frame Check Sequence (FCS) for Outer Ethernet Frame     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 3: Geneve Packet Format over IPv6 🌈✨💖

3.3. UDP Header 🎀

okay bestie, the encapsulating UDP [RFC0768] header is what gives Geneve that connectionless vibe like Ethernet and IP, and it also gives routers doing ECMP some entropy to work with ✨ so here's how the header fields work:

  • Source Port: the source port gets picked by the originating tunnel endpoint. It SHOULD be the same for all packets in a single encapsulated flow so nothing gets reordered from taking different paths 🛤️ To spread flows evenly across multiple links, the source port SHOULD be calculated from a hash of the encapsulated packet headers, for example a traditional 5-tuple. The port is really a flow identifier and not a true UDP connection, so the entire 16-bit range MAY be used to max out the entropy 💅 Also, for IPv6, the flow label MAY be used for entropy too. For an example of using the IPv6 flow label for tunnel use cases, see [RFC6438].

If Geneve traffic is shared with other UDP listeners on the same IP address, tunnel endpoints SHOULD implement a mechanism so ICMP return traffic from network errors gets sent to the correct listener. The definition of that mechanism is beyond the scope of this document 🙈

  • Dest Port: IANA assigned port 6081 as the fixed well-known destination port for Geneve. The well-known value should be used by default, but it is RECOMMENDED that implementations make this configurable 🍓 The chosen port is used to identify Geneve packets and MUST NOT be reversed for the different ends of a connection the way TCP does it. The control plane is responsible for managing any reconfiguration of the assigned port and how respective devices interpret it. The definition of the control plane is beyond the scope of this document.
  • UDP Length: the length of the UDP packet, including the UDP header.
  • UDP Checksum: to protect the Geneve header, options, and payload from potential data corruption, the UDP checksum SHOULD be generated as specified in [RFC0768] and [RFC1122] when Geneve is encapsulated in IPv4. To protect the IP header, Geneve header, options, and payload from potential data corruption, the UDP checksum MUST be generated by default as specified in [RFC0768] and [RFC8200] when Geneve is encapsulated in IPv6, except under certain conditions, which are outlined in the next paragraph. When receiving such packets with a non-zero UDP checksum, the receiving tunnel endpoints MUST validate the checksum. If the checksum is not correct, the packet MUST be dropped. Otherwise, the packet MUST be accepted for decapsulation 🔥

Under certain conditions, the UDP checksum MAY be set to zero on transmit for packets encapsulated in both IPv4 and IPv6 [RFC8200]. See Section 4.3 for additional requirements that apply when using a zero UDP checksum with IPv4 and IPv6. Turning off UDP checksums is an operational consideration that should take into account the risks and effects of packet corruption ⭐

3.4. Tunnel Header Fields ✨

  • Ver (2 bits): the current version number is 0. Packets received by a tunnel endpoint with an unknown version MUST be dropped. Transit devices interpreting Geneve packets with an unknown version number MUST treat them as UDP packets with an unknown payload 🦋
  • Opt Len (6 bits): the length of the option fields, expressed in 4-byte multiples, not including the 8-byte fixed tunnel header. This results in a minimum total Geneve header size of 8 bytes and a maximum of 260 bytes. The start of the payload headers can be found using this offset from the end of the base Geneve header.

Transit devices MUST maintain consistent forwarding behavior irrespective of the value of Opt Len, including ECMP link selection 💜

  • O (1 bit): control packet. This packet contains a control message. Control messages are sent between tunnel endpoints. Tunnel endpoints MUST NOT forward the payload, and transit devices MUST NOT attempt to interpret it. Since control messages are less frequent, it is RECOMMENDED that tunnel endpoints direct these packets to a high-priority control queue (for example, to direct the packet to a general purpose CPU from a forwarding Application-Specific Integrated Circuit (ASIC) or to separate out control traffic on a NIC) 🧚 Transit devices MUST NOT alter forwarding behavior on the basis of this bit, such as ECMP link selection.
  • C (1 bit): critical options present. One or more options has the critical bit set (see Section 3.5). If this bit is set, then tunnel endpoints MUST parse the options list to interpret any critical options. On tunnel endpoints where option parsing is not supported, the packet MUST be dropped on the basis of the C bit in the base header. If the bit is not set, tunnel endpoints MAY strip all options using Opt Len and forward the decapsulated packet. Transit devices MUST NOT drop packets on the basis of this bit 👑
  • Rsvd. (6 bits): reserved field, which MUST be zero on transmission and MUST be ignored on receipt.
  • Protocol Type (16 bits): the type of protocol data unit appearing after the Geneve header. This follows the Ethertype [ETYPES] convention, with Ethernet itself being represented by the value 0x6558 🌈
  • Virtual Network Identifier (VNI) (24 bits): an identifier for a unique element of a virtual network. In many situations, this may represent an L2 segment; however, the control plane defines the forwarding semantics of decapsulated packets. The VNI MAY be used as part of ECMP forwarding decisions, or MAY be used as a mechanism to distinguish between overlapping address spaces contained in the encapsulated packet when load balancing across CPUs 💖
  • Reserved (8 bits): reserved field, which MUST be zero on transmission and ignored on receipt.

3.5. Tunnel Options 💅

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Option Class         |      Type     |R|R|R| Length  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                  Variable-Length Option Data                  ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                       Figure 4: Geneve Option

The base Geneve header is followed by zero or more options in Type-Length-Value format. Each option consists of a 4-byte option header and a variable amount of option data interpreted according to the type.

  • Option Class (16 bits): namespace for the Type field. IANA has created a "Geneve Option Class" registry to allocate identifiers for organizations, technologies, and vendors that have an interest in creating types for options. Each organization may allocate types independently to allow experimentation and rapid innovation. It is expected that, over time, certain options will become well known, and a given implementation may use option types from a variety of sources. In addition, IANA has reserved specific ranges for allocation by IETF Review and for Experimental Use (see Section 7) ✨
  • Type (8 bits): type indicating the format of the data contained in this option. Options are primarily designed to encourage future extensibility and innovation, and standardized forms of these options will be defined in separate documents.

The high-order bit of the option type indicates that this is a critical option. If the receiving tunnel endpoint does not recognize the option and this bit is set, then the packet MUST be dropped. If this bit is set in any option, then the C bit in the Geneve base header MUST also be set. Transit devices MUST NOT drop packets on the basis of this bit. The following figure shows the location of the C bit in the Type field:

   0 1 2 3 4 5 6 7 8
   +-+-+-+-+-+-+-+-+
   |C|    Type     |
   +-+-+-+-+-+-+-+-+
                Figure 5: 'C' Bit in the 'Type' Field

The requirement to drop a packet with an unknown option with the C bit set applies to the entire tunnel endpoint system and not a particular component of the implementation. For example, in a system comprised of a forwarding ASIC and a general purpose CPU, this does not mean that the packet must be dropped in the ASIC. An implementation may send the packet to the CPU using a rate-limited control channel for slow-path exception handling 🦋

  • R (3 bits): option control flags reserved for future use. These bits MUST be zero on transmission and MUST be ignored on receipt.
  • Length (5 bits): length of the option, expressed in 4-byte multiples, excluding the option header. The total length of each option may be between 4 and 128 bytes. A value of 0 in the Length field implies an option with only an option header and no option data. Packets in which the total length of all options is not equal to the Opt Len in the base header are invalid and MUST be silently dropped if received by a tunnel endpoint that processes the options 💀
  • Variable-Length Option Data: option data interpreted according to Type. 🫶

3.5.1. Options Processing 🧚

Okay so Geneve options are meant to be made and read by the tunnel endpoints 💖. But transit devices along the tunnel path are allowed to peek at them too, since options MAY be interpreted by transit devices 👀. Any transit device that isn't reading the Geneve headers (which may or may not have options in them) MUST handle the Geneve packet like any other UDP packet and keep its forwarding behavior consistent 🫶.

Inside the tunnel endpoints, the control plane decides how options get generated and interpreted. That's beyond the scope of this doc, so don't come for me about it 🙈. Still, so that totally different devices can talk to each other, there are some rules for options and for the devices that process them:

  • Receiving tunnel endpoints MUST drop packets containing unknown options with the 'C' bit set in the option type. Conversely, transit devices MUST NOT drop packets as a result of encountering unknown options, including those with the 'C' bit set. Main character energy for the transit devices 👑.
  • The contents of the options and their ordering MUST NOT be modified by transit devices. Hands off, bestie 🙅‍♀️.
  • If a tunnel endpoint receives a Geneve packet with an 'Opt Len' (the total length of all options) that exceeds the options-processing capability of the tunnel endpoint, then the tunnel endpoint MUST drop such packets. An implementation may raise an exception to the control plane in such an event. It is the responsibility of the control plane to ensure the communicating peer tunnel endpoints have the processing capability to handle the total length of options. The definition of the control plane is beyond the scope of this document. Ngl this one is so important 😭.

When you're designing a Geneve option, you have to think about how it'll evolve later 🔮. Once an option is defined, implementations may come to depend on a specific behavior, so the scope of any future changes must be carefully described upfront. Planning ahead is everything 💅.

Architecturally, options are meant to be self descriptive and independent ✨. That lets options get processed in parallel and keeps implementations simpler. The control plane may still impose certain ordering restrictions, as described in Section 4.5.1, so don't get too comfy 🌸.

Changing the length of an option that was defined to be a certain size can cause unexpectedly big interoperability issues, so be careful 🚨. Every option is either fixed-length (constant) or variable-length (it can change over time or for different use cases), and that property is part of the option's definition and is conveyed by the 'Type'. For fixed-length options, some implementations may choose to ignore the 'Length' field in the option header and parse based on the well-known length tied to the type instead. In that case, redefining the length affects not only that option's parsing but also any options that come after it. So options defined to be a fixed length MUST NOT be redefined to a different length. Instead, a new 'Type' should be allocated. The actual definition of the option type is beyond the scope of this document. The option type and its interpretation should be defined by the entity that owns the option class 🎀.

Options may be processed by NIC hardware using offloads (e.g., LSO and LRO), as described in Section 4.6. Careful consideration should be given to how the offload capabilities outlined in Section 4.6 affect an option's design 🍓.

4. Implementation and Deployment Considerations 🌈

4.1. Applicability Statement

Geneve is a UDP-based network virtualization overlay encapsulation protocol. Its job is to set up tunnels between NVEs over an existing IP network 🦋. It's meant for public or private data centers, where you deploy multi-tenant overlay networks on top of an existing IP underlay network.

As a UDP-based protocol, Geneve follows the UDP usage guidelines in [RFC8085]. How much those guidelines apply depends on the underlay IP network and on what kind of payload Geneve is carrying (for example, TCP/IP or IP/Ethernet) 💜.

Geneve is intended to run in a data center network operated by a single operator, or by a group of closely cooperating operators, that fits the definition of controlled environments in [RFC8085]. In a controlled environment, the network can be managed to behave a certain way, which you just can't do across the general Internet 🤷‍♀️. So the requirements for a tunneling protocol in a controlled environment can be less strict than the ones for the general Internet.

For the purpose of this document, a traffic-managed controlled environment (TMCE) is an IP network that is traffic engineered and/or otherwise managed (e.g., with traffic rate limiters) to avoid congestion. The TMCE concept is outlined in [RFC8086]. Big chunks of the text in Section 4.1 through Section 4.3 are based on [RFC8086], as they apply to Geneve 🧠.

It is the responsibility of the operator to make sure the guidelines and requirements in this section are followed, as they apply to their Geneve deployment(s) 🫡.

4.2. Congestion-Control Functionality

Geneve doesn't natively provide congestion-control functionality. It relies on the payload protocol's traffic for congestion control 😬. Because of that, Geneve MUST be used with congestion-controlled traffic or within a TMCE to avoid congestion. An operator of a TMCE can avoid congestion by provisioning their networks carefully, rate-limiting user data traffic, and managing traffic engineering according to path capacity 🚦.

4.3. UDP Checksum

The outer UDP checksum SHOULD be used with Geneve when it's transported over IPv4 💖. This gives integrity for the Geneve headers, options, and payload in case of data corruption, for example so the payload doesn't get misdelivered to different tenant systems 📦. The UDP checksum gives a statistical guarantee that a payload wasn't corrupted in transit. These integrity checks aren't strong from a coding or cryptographic perspective, and they aren't designed to detect physical-layer errors or malicious modification of the datagram (see Section 3.4 of [RFC8085]). Where that risk exists, an operator SHOULD use additional data integrity mechanisms, such as those offered by IPsec (see Section 6.2) 🔐.

An operator MAY choose to disable UDP checksums and use a zero UDP checksum if Geneve packet integrity is provided by other data integrity mechanisms (such as IPsec or additional checksums), or if one of the conditions (a, b, or c) in Section 4.3.1 is met 🙃.

By default, UDP checksums MUST be used when Geneve is transported over IPv6 🌸. A tunnel endpoint MAY be configured for use with a zero UDP checksum if the additional requirements in Section 4.3.1 are met.

4.3.1. Zero UDP Checksum Handling with IPv6

When Geneve is used over IPv6, the UDP checksum is what protects the IPv6 headers, UDP headers, and Geneve headers, options, and payload from potential data corruption ✨. So by default, Geneve MUST use UDP checksums when it's transported over IPv6. An operator MAY choose to configure a zero UDP checksum if they're operating in a TMCE as stated in Section 4.1, provided one of the following conditions is met:

  • a. It is known that packet corruption is exceptionally unlikely (perhaps based on knowledge of the equipment types in their underlay network), and the operator is willing to risk undetected packet corruption.
  • b. It is judged through observational measurements (perhaps through historic or current traffic flows that use a non-zero checksum) that the level of packet corruption is tolerably low, and the operator is willing to risk undetected corruption.
  • c. The Geneve payload is carrying applications that are tolerant of misdelivered or corrupted packets (perhaps through higher-layer checksum validation and/or reliability through retransmission).

Also, Geneve tunnel implementations that use a zero UDP checksum MUST meet the following requirements 🎀:

  1. Use of UDP checksum over IPv6 MUST be the default configuration for all Geneve tunnels.
  2. If Geneve is used with a zero UDP checksum over IPv6, then such a tunnel endpoint implementation MUST meet all the requirements specified in Section 4 of [RFC6936] and requirement 1 as specified in Section 5 of [RFC6936], since it is relevant to Geneve.
  3. The Geneve tunnel endpoint that decapsulates the tunnel SHOULD check that the source and destination IPv6 addresses are valid for the Geneve tunnel that is configured to receive a zero UDP checksum, and discard other packets for which such a check fails 🕵️‍♀️.
  4. The Geneve tunnel endpoint that encapsulates the tunnel MAY use different IPv6 source addresses for each Geneve tunnel that uses zero UDP checksum mode, to strengthen the decapsulator's check of the IPv6 source address. In other words, the same IPv6 source address is not to be used with more than one IPv6 destination address, whether that destination address is unicast or multicast. When that isn't possible, it is RECOMMENDED to use each source address for as few Geneve tunnels that use zero UDP checksum as is feasible.

Note that for requirements 3 and 4, the receiving tunnel endpoint can apply these checks only if it has out-of-band knowledge that the encapsulating tunnel endpoint is applying the indicated behavior. One way to get that out-of-band knowledge is through signaling by the control plane. The definition of the control plane is beyond the scope of this document 🧐.

  1. Measures SHOULD be taken to stop Geneve traffic over IPv6 with a zero UDP checksum from escaping into the general Internet 🚫. Examples of such measures include packet filters at the gateways or edge of the Geneve network, and/or keeping a logical or physical separation between the Geneve network and networks carrying general Internet traffic.

The above requirements don't change the requirements specified in either [RFC8200] or [RFC6936] 📜.

Using the source IPv6 address as well as the destination IPv6 address, plus the recommendation against reusing source IPv6 addresses among Geneve tunnels, together give some mitigation for the lack of UDP checksum coverage of the IPv6 header 🛡️. A traffic-managed controlled environment that satisfies at least one of the three conditions listed at the beginning of this section gives additional assurance 💜.

4.4. Encapsulation of Geneve in IP

As an IP-based tunneling protocol, Geneve shares a lot of properties and techniques with existing protocols 🤝. Some of these are described in further detail here, but in general, most concepts that apply to the IP layer or to IP tunnels generally also work in the context of Geneve ✨.

4.4.1. IP Fragmentation ✨

Okay bestie, first up is fragmentation, and the big vibe here is: please use Path MTU Discovery 💅 It is RECOMMENDED that you use it (see [RFC1191] and [RFC8201]) to prevent or minimize fragmentation. Running Path MTU Discovery on the transit network gives the encapsulating tunnel endpoint soft-state info about the link, and it can use that info to prevent or minimize fragmentation depending on what job it's doing in the virtualized network 🦋

The NVE can keep track of this state (the MTU size of the tunnel link(s) tied to the tunnel endpoint). So if a tenant system sends big packets that get too large once they're encapsulated, the tunnel endpoint can discard them and send exception messages back to the tenant system(s) 🙈 If the tunnel endpoint is part of a routing or forwarding function and/or can send ICMP messages, the encapsulating tunnel endpoint MAY send ICMP fragmentation needed [RFC0792] or Packet Too Big [RFC4443] messages to the tenant system(s) 🧚

When you're figuring out the MTU size of a tunnel link, the maximum length of options MUST be assumed, because options can change from packet to packet 🍓 Want guidance on fragmentation in similar overlay services? Pseudowire Emulation Edge-to-Edge (PWE3) has recommendations, and they're in Section 5.3 of [RFC3985] ✨

Quick heads-up: some implementations might not be able to handle fragmentation or other less common IP header features, like options and extension headers 😭 The issues around MTU size, fragmentation, and ICMP messages in IP tunneling are laid out in Section 4.2 of [INTAREA-TUNNELS] 💜

4.4.2. DSCP, ECN, and TTL 🌈

So when you're encapsulating IP packets (including ones riding over Ethernet) in Geneve, there are a few things to think about for carrying Differentiated Services Code Point (DSCP) and Explicit Congestion Notification (ECN) bits from the inner header into the tunnel on the way out, and back again on the way in 🎀

[RFC2983] gives guidance on mapping DSCP between the inner and outer IP headers 🦋 Network virtualization usually lines up more with the Pipe model, where the DSCP value on the tunnel header gets set by a policy. That policy could be a fixed value, one based on the inner traffic class, or some other way of grouping traffic 💅 Parts of the Uniform model (where the inner and outer DSCP values act like one field, copied on ingress and egress) might also apply, like being able to re-mark the inner header on tunnel egress based on transit marking ✨ Still, the Uniform model doesn't really vibe with network virtualization, since virtualization is all about strong isolation between encapsulated traffic and the physical network 🙈

[RFC6040] describes how to expose ECN capabilities on IP tunnels and pass congestion markers along to the inner packets 🌸 This behavior MUST be followed for IP packets encapsulated in Geneve. No skipping this one, bestie 👑

For TTL (or Hop Limit, for IPv6), either the Uniform or Pipe model could work when you're tunneling IP packets. But the Pipe model is more consistent with network virtualization 🧚 [RFC2003] covers handling TTL between the inner IP header and outer IP tunnels. That model is similar to the Pipe model, and it is RECOMMENDED for Geneve in network virtualization applications 💖

4.4.3. Broadcast and Multicast 📡

Geneve tunnels can be point-to-point unicast between two tunnel endpoints, or they can use broadcast or multicast addressing 🦋 The inner and outer addressing don't have to match on this front. For example, in physical networks that don't support multicast, encapsulated multicast traffic can be replicated into several unicast tunnels, or forwarded by policy to a unicast location (where it might get replicated again) 🌸

When the physical network does support multicast, it can be really handy to use that to get hardware replication for encapsulated packets ⭐ In that case, multicast addresses can be allocated in the physical network for tenants, for encapsulated multicast groups, or based on some other factor. How those groups get allocated is part of the control plane, so it's outside the scope of this document 🧚

Heads-up: when physical multicast is in use, you might have devices with different capabilities in the same group 🙈 Some options can only be interpreted by a subset of the devices. Other devices can safely ignore those options, unless the 'C' bit is set to mark the unknown option as critical. The requirements in Section 3.4 apply for critical options 💅

And [RFC8293] has examples of different mechanisms for multicast handling in network virtualization overlay networks 💜

4.4.4. Unidirectional Tunnels 🌊

Generally speaking, a Geneve tunnel is a unidirectional concept 💖 IP isn't connection-oriented, so two tunnel endpoints might talk to each other over different paths, or one side might not send anything at all 🙈 Since Geneve is an IP-based protocol, the tunnel layer inherits all of these same characteristics ✨

That said, a tunnel can encapsulate a connection-oriented protocol, like TCP, which keeps session state at its own layer 🎀 Also, implementations MAY model Geneve tunnels as connected, bidirectional links, for example to give you the abstraction of a virtual port 🦋 In both cases, the bidirectionality is handled at a higher layer and doesn't change how Geneve itself works 👑

4.5. Constraints on Protocol Features 🎀

Geneve is meant to be flexible for a wide range of current and future applications, bestie 🌈 Because of that, certain constraints may be placed on how you use metadata or other parts of the protocol, so you can optimize for a particular use case 💅 For example, some applications may limit which types of options they support, or enforce a maximum number or length of options. Other applications might only handle certain encapsulated payload types, like Ethernet or IP 🧚 These optimizations can happen globally (throughout the whole system) or locally (for example, just for certain classes of devices or network paths) ✨

These constraints can be communicated to tunnel endpoints explicitly through a control plane, or implicitly by the nature of the application 🌸 Geneve is defined as a data plane protocol that doesn't care which control plane you use, so defining those mechanisms is outside the scope of this document 💜

4.5.1. Constraints on Options 🍓

Geneve options are super flexible, but a control plane may restrict the number of option TLVs, plus the order and size of the TLVs between tunnel endpoints. That makes it simpler for a data plane implementation in software or hardware to handle them (see [NVO3-ENCAP]) 🦋 For example, there may be critical information, like a secure hash, that must be processed in a certain order to get the lowest latency. Or there may be other scenarios where options must be processed in a given order because of protocol semantics 🫶

A control plane may negotiate a subset of option TLVs and certain TLV ordering. It can also limit the total number of option TLVs in a packet, for example, to fit hardware that can only process fewer options 💅 So a control plane needs to be able to describe the supported TLV subset and its ordering to the tunnel endpoints 👑 Without a control plane, other configuration mechanisms may be used for this. Those mechanisms are outside the scope of this document 🙈

4.6. NIC Offloads 💻

Modern NICs already offer a bunch of offloads for processing packets efficiently 🔥 A lot of these offloads only need the encapsulated packet to be easy to parse (checksum offload is a good example). But optimizations like LSO and LRO do involve processing the options themselves, since they have to be replicated or merged across multiple packets 😭 In these situations, it's really nice not to need changes to the offload logic every time a new option shows up 🧚 So some constraints are placed on how options are defined, so the processing rules stay simple:

  • When performing LSO, a NIC MUST replicate the entire Geneve header and all options, including those unknown to the device, onto each resulting segment, unless an option allows an exception. On the flip side, when performing LRO, a NIC may assume that a binary comparison of the options (including unknown options) is sufficient to ensure equality, and MAY merge packets with equal Geneve headers 💖
  • Options MUST NOT be reordered during offload processing, including when merging packets for LRO 🙅
  • NICs performing offloads MUST NOT drop packets with unknown options, including those marked as critical, unless they've been explicitly configured to do so 🚫

No one is forcing a Geneve implementation to use the offloads listed as examples above 🌸 But since these offloads are already widely deployed in commercially available NICs, the rules here are meant to make it efficient to handle current and future options across all kinds of devices 🌈

4.7. Inner VLAN Handling 🎀

Geneve can encapsulate a wide range of protocols, so any given implementation will probably only support a small subset of them 🦋 But Ethernet is expected to be widely deployed, so it's useful to describe how VLANs behave inside encapsulated Ethernet frames ✨

As with any protocol, support for inner VLAN headers is OPTIONAL 💅 In lots of cases, encapsulated VLANs might be disallowed for security or implementation reasons 🙈 But in other cases, trunking VLAN frames across a Geneve tunnel can be super useful 🧚 So how inner VLAN tags get processed on ingress or egress at a tunnel endpoint depends on the tunnel endpoint's configuration and/or control plane. It's not explicitly defined as part of the data format 👑

5. Transition Considerations ✨

Okay so bestie, look at this 🦋. From the data plane's point of view, Geneve is totally compatible with existing IP networks, because to most devices it just looks like UDP packets. Iconic, right? 💅 But there are already a bunch of tunneling protocols out there in network virtualization environments, so the real question is how everyone transitions and coexists 🫶.

Here's the lowkey genius part 💖. Geneve builds on the base data plane functionality of the most common protocols used for network virtualization (VXLAN and NVGRE), so it should be straightforward to port an existing control plane to run on top of it with minimal effort. 🧚 Both the old and new packet formats support the same set of capabilities, so there's no need for a hard cutover. Tunnel endpoints that talk directly to each other can use any common protocol, and that protocol can even be different within a single overall system 🌈. Transit devices mostly forward packets based on the IP header, so all these protocols look pretty similar to them, and they don't add any extra interoperability concerns. No cap, it's a whole vibe ✨.

To make the transition smooth, it is strongly suggested that implementations support simultaneous operation of both Geneve and existing tunneling protocols 🎀. A single node is expected to talk to a mix of other nodes, so this is going to be super common 🙈. Eventually the older protocols may be phased out once nobody uses them anymore 🍓.

6. Security Considerations 🔥

Okay, this section is serious, bestie. Please read every word 🚨💀.

Since Geneve is encapsulated inside a UDP/IP packet, it does not have any inherent security mechanisms 😭. That means an attacker with access to the underlay network carrying the IP packets can snoop on packets, alter them, or inject them. Compromised tunnel endpoints or transit devices may also spoof identifiers in the tunnel header to get into networks owned by other tenants 💀.

Within a particular security domain, like a data center run by a single service provider, the most common and highest-performing security mechanism is isolation of trusted components 👑. Tunnel traffic can be carried over a separate VLAN and filtered at any untrusted boundaries 🛡️.

When crossing an untrusted link, like the general Internet, VPN technologies such as IPsec [RFC4301] should be used to provide authentication and/or encryption of the IP packets formed as part of Geneve encapsulation (see Section 6.1.1) 🔐.

Geneve does not otherwise affect the security of the encapsulated packets. As per the guidelines of BCP 72 [RFC3552], the following sections describe potential security risks that may apply to Geneve deployments, plus ways to mitigate them 🧁. Not all of these risks apply to every Geneve deployment, though. Only a subset may be relevant to certain deployments 🦋. An operator has to assess their own network environment, figure out which risks apply to their specific setup, and use suitable mitigation approaches where they apply ✨.

6.1. Data Confidentiality 🤫

Geneve is a network virtualization overlay encapsulation protocol designed to set up tunnels between NVEs over an existing IP network 🌸. It can be used to deploy multi-tenant overlay networks on top of an existing IP underlay network in a public or private data center. The overlay service is usually provided by a service provider, like a cloud service provider or a private data center operator. That may or may not be the same provider as the underlay service provider 🤷‍♀️.

Because of multi-tenancy, a tenant system may expect data confidentiality, meaning its packet data is not tampered with (an active attack) in transit and is not a target of unauthorized monitoring (a passive attack). The threat could come from other tenant systems or from the underlay service provider, for example 👀. A compromised network node or transit device within a data center may passively monitor Geneve packet data between NVEs, or route traffic somewhere for further inspection 🕵️‍♀️. A tenant may expect the overlay service provider to supply data confidentiality as part of the service. Or a tenant may bring its own confidentiality mechanisms, like IPsec or TLS, to protect the data end to end between its tenant systems 🔒. The overlay provider is expected to provide cryptographic protection when the underlay provider is not the same as the overlay provider, so the payload is not exposed to the underlay 💜.

If an operator decides data confidentiality is necessary in their environment based on their risk analysis (for example, in multi-tenant environments), then an encryption mechanism SHOULD be used to encrypt the tenant data end to end between the NVEs. The NVEs may use existing, well-established encryption mechanisms such as IPsec, DTLS, and so on 🫶.

6.1.1. Inter-Data Center Traffic 🌍

A tenant system in a customer premises (a private data center) may want to connect to tenant systems on its tenant overlay network in a public cloud data center. Or a tenant may want its tenant systems spread across multiple geographically separated data centers for high availability 🏢. Geneve data traffic between tenant systems across these separated networks should be protected from threats when it travels over public networks. Any Geneve overlay data leaving the data center network beyond the operator's security domain SHOULD be secured by encryption mechanisms, such as IPsec or other VPN technologies, to protect communications between the NVEs when they are geographically separated over untrusted network links 🔐✨. Specifying the data protection mechanisms used between data centers is out of scope for this document 🙅‍♀️.

The principles described in Section 4 about controlled environments still apply to the geographically separated data center usage outlined in this section 🧚.

6.2. Data Integrity 🧷

Geneve encapsulation is used between NVEs to set up overlay tunnels over an existing IP underlay network 🦋. In a multi-tenant data center, a rogue or compromised tenant system may try a passive attack, like monitoring other tenants' traffic, or an active attack, like injecting unauthorized Geneve encapsulated traffic (spoofing, replay, and so on) into the network 😱. To prevent this, an NVE MUST NOT propagate Geneve packets beyond the NVE to tenant systems, and SHOULD use packet-filtering mechanisms so it doesn't forward unauthorized traffic between tenant systems in different tenant networks 🚫. An NVE MUST NOT interpret Geneve packets from tenant systems other than as frames to be encapsulated 💅.

A compromised network node or transit device within a data center may launch an active attack to tamper with Geneve packet data between NVEs 🥺. Malicious tampering with Geneve header fields may cause a packet from one tenant to be forwarded to a different tenant network 😭. If an operator determines there is a possibility of such a threat in their environment, the operator may choose to use data integrity mechanisms between NVEs. To prevent these risks, a data integrity mechanism SHOULD be used in such environments to protect the integrity of Geneve packets, including packet headers, options, and payload, on communications between NVE pairs 🔒. A cryptographic data protection mechanism, such as IPsec, may be used to provide data integrity protection. A data center operator may choose to deploy any other data integrity mechanisms that are applicable and supported in their underlay networks, although non-cryptographic mechanisms may not protect the Geneve portion of the packet from tampering 🤍.

6.3. Authentication of NVE Peers 🪪

A rogue network device or a compromised NVE in a data center environment might be able to spoof Geneve packets as if they came from a legitimate NVE 👻. To reduce this risk, an operator SHOULD use an authentication mechanism, such as IPsec, to make sure the Geneve packet came from the intended NVE peer, in environments where the operator determines that spoofing or rogue devices are real threats 🛡️. Simpler source checks, like ingress filtering for VLAN/MAC/IP addresses, reverse path forwarding checks, and so on, may be used in certain trusted environments to make sure Geneve packets came from the intended NVE peer 🌷.

6.4. Options Interpretation by Transit Devices 🧭

Options, if present in the packet, are generated and terminated by tunnel endpoints. As indicated in Section 2.2.1, transit devices may interpret the options 👀. However, if the packet is protected by encryption from tunnel endpoint to tunnel endpoint (for example, through IPsec), transit devices will not be able to see the Geneve header or options in the packet 🙈. In that case, transit devices MUST handle Geneve packets as any other IP packet and maintain consistent forwarding behavior 💖. Where options are interpreted by transit devices, the operator MUST ensure that those transit devices are trusted and not compromised 🔥. The definition of a mechanism to ensure this trust is out of scope for this document 🙅‍♀️.

6.5. Multicast/Broadcast 📡

In typical data center networks where IP multicasting is not supported in the underlay network, multicasting may be supported using multiple unicast tunnels 🌈. The same security requirements described in the sections above can be used to protect Geneve communications between NVE peers 💜. If IP multicasting is supported in the underlay network and the operator chooses to use it for multicast traffic among tunnel endpoints, then the operator may use data protection mechanisms, such as IPsec with multicast extensions [RFC5374], to protect multicast traffic among Geneve NVE groups 🎀.

6.6. Control Plane Communications 🎛️

A Network Virtualization Authority (NVA), as outlined in [RFC8014], may be used as a control plane for configuring and managing the Geneve NVEs 🧚‍♀️. The data center operator is expected to use security mechanisms to protect communications between the NVA and NVEs, and to use authentication mechanisms to detect any rogue or compromised NVEs within their administrative domain 🔐. Data protection mechanisms for control plane communication, and authentication mechanisms between the NVA and NVEs, are out of scope for this document 🫶.

7. IANA Considerations 📋✨

okay so IANA is the one that handed out port 6081 to Geneve in the "Service Name and Transport Protocol Port Number Registry" [IANA-SN], so now Geneve has its own well-known spot on UDP 🎀 here's the exact entry:

Service Name:  geneve
Transport Protocol(s):  UDP
Assignee:  IESG <iesg@ietf.org>
Contact:  IETF Chair <chair@ietf.org>
Description:  Generic Network Virtualization Encapsulation (Geneve)
Reference:  [RFC8926]
Port Number:  6081

and ALSO, IANA made a brand new subregistry called "Geneve Option Class" 💅 for option classes. it lives under a new "Network Virtualization Overlay (NVO3)" heading in the IANA protocol registries [IANA-PR], so it's giving very organized. each entry is a 16-bit hexadecimal value with a descriptive string, assignee/contact info, and references 🦋

the registration rules for this new registry (as defined by [RFC8126]) are split into three ranges, and yes, the ranges have different vibes 👑

                +===============+=========================+
                | Range         | Registration Procedures |
                +===============+=========================+
                | 0x0000-0x00FF | IETF Review             |
                +---------------+-------------------------+
                | 0x0100-0xFEFF | First Come First Served |
                +---------------+-------------------------+
                | 0xFF00-0xFFFF | Experimental Use        |
                +---------------+-------------------------+

                   Table 1: Geneve Option Class Registry
                                   Ranges

TL;DR: 0x0000-0x00FF goes through IETF Review, 0x0100-0xFEFF is First Come First Served, and 0xFF00-0xFFFF is for Experimental Use 🧚

8. References 📚

8.1. Normative References 💜

8.2. Informative References 🌸

Acknowledgements 🫶

the authors wanna give a huge shoutout to Puneet Agarwal, David Black, Sami Boutros, Scott Bradner, Martín Casado, Alissa Cooper, Roman Danyliw, Bruce Davie, Anoop Ghanwani, Benjamin Kaduk, Suresh Krishnan, Mirja Kühlewind, Barry Leiba, Daniel Migault, Greg Mirksy, Tal Mizrahi, Kathleen Moriarty, Magnus Nyström, Adam Roach, Sabrina Tanamal, Dave Thaler, Éric Vyncke, Magnus Westerlund, and everyone else in the NVO3 Working Group 💖 for their reviews, comments, and suggestions. literally the whole group was reviewing 😭

the authors also thanked Sam Aldrin, Alia Atlas, Matthew Bocci, Benson Schliesser, and Martin Vigoureux 🌈 for their guidance the whole way through.

Contributors 🎀

the following people wrote an earlier version of this document and made huge contributions 👑

Pankaj Garg
Microsoft Corporation
1 Microsoft Way
Redmond, WA 98052
United States of America

Email: pankajg@microsoft.com
Chris Wright
Red Hat Inc.
1801 Varsity Drive
Raleigh, NC 27606
United States of America

Email: chrisw@redhat.com
Kenneth Duda
Arista Networks
5453 Great America Parkway
Santa Clara, CA 95054
United States of America

Email: kduda@arista.com
Dinesh G. Dutt
Independent

Email: didutt@gmail.com
Jon Hudson
Independent

Email: jon.hudson@gmail.com
Ariel Hendel
Facebook, Inc.
1 Hacker Way
Menlo Park, CA 94025
United States of America

Email: ahendel@fb.com

Authors' Addresses 💅

Jesse Gross (editor)

Email: jesse@kernel.org
Ilango Ganga (editor)
Intel Corporation
2200 Mission College Blvd.
Santa Clara, CA 95054
United States of America

Email: ilango.s.ganga@intel.com
T. Sridhar (editor)
VMware, Inc.
3401 Hillview Ave.
Palo Alto, CA 94304
United States of America

Email: tsridhar@utexas.edu

and that's the whole Geneve spec, bestie 🦋✨ we did it 💖

📦 where this page comes from