RFC 9638
Network Virtualization over Layer 3 (NVO3) Encapsulation Considerations
Network Virtualization over Layer 3 (NVO3) Encapsulation Considerations 🦋
okay bestie, RFC 9638 is here and it's giving IETF design-team drama 💅 Here's the tea: it's an Informational RFC (so it's a record, not a standards-track rule 📝), published September 2024 with the ISSN 2070-1721. The authors are S. Boutros, Ed. from Ciena Corporation and D. Eastlake 3rd, Ed. (independent). The header has no Obsoletes or Updates lines, so it doesn't replace or change any other RFC ✨
Abstract 💖
okay so the IETF Network Virtualization Overlays (NVO3) Working Group came up with considerations for one common encapsulation that tackles a bunch of network virtualization overlay problems 🌸 This doc is the official record of those considerations, starting from what the NVO3 encapsulation Design Team produced, and it's meant to help future working groups pick encapsulation formats 🧚
The vibe is that different encapsulations living together in real environments, both software and hardware, and across multiple data centers, is messy 😭 For example, Operations, Administration, and Maintenance (OAM) stuff like path MTU discovery gets really hard when you've got multiple encapsulations along the same data path 💀
So after all that thinking, the NVO3 WG decided that Generic Network Virtualization Encapsulation (Geneve), with a few changes, is the common one 🎀 Section 7 has the most detail, so go there if you're impatient 👑
Status of This Memo 🌈
this one's not an Internet Standards Track spec, it's just published for information 📖
It's still a real IETF product. It reflects the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so see Section 2 of RFC 7841 for that ✨
Errata, current status, and feedback info live at https://www.rfc-editor.org/info/rfc9638 🔗
Copyright Notice 💜
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they stand on the publication date. Please read these carefully, because they spell out your rights and restrictions for this doc 🧾 Code Components pulled out of this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as the Revised BSD License says 🫶
Table of Contents 🍓
1. Introduction
2. Design Team and Working Group Process
3. Terminology
4. Abbreviations, Acronyms, and Definitions
5. Encapsulation Issues and Background
5.1. Geneve
5.2. Generic UDP Encapsulation (GUE)
5.3. Generic Protocol Extension (GPE) for VXLAN
6. Common Encapsulation Considerations
6.1. Current Encapsulations
6.2. Useful Extensions Use Cases
6.2.1. Telemetry Extensions
6.2.2. Security/Integrity Extensions
6.2.3. Group-Based Policy
6.3. Hardware Considerations
6.4. Extension Size
6.5. Ordering of Extension Headers
6.6. TLV versus Bit Fields
6.7. Control Plane Considerations
6.8. Split NVE
6.9. Larger VNI Considerations
7. Recommendations
8. Security Considerations
9. IANA Considerations
10. References
10.1. Normative References
10.2. Informative References
Appendix A. Encapsulation Comparison
A.1. Overview
A.2. Extensibility
A.2.1. Innate Extensibility Support
A.2.2. Extension Parsing
A.2.3. Critical Extensions
A.2.4. Maximal Header Length
A.3. Encapsulation Header
A.3.1. Virtual Network Identifier (VNI)
A.3.2. Next Protocol
A.3.3. Other Header Fields
A.4. Comparison Summary
Acknowledgements
Contributors
Authors' Addresses1. Introduction ✨
the NVO3 WG is chartered to gather requirements and build solutions for network virtualization data planes, based on encapsulating virtual network traffic over an IP-based underlay data plane 🦋 The requirements include proper consideration for OAM and security 🔐 With those in hand, the WG was to select, extend, and/or develop one or more data plane encapsulation formats.
that led to WG Internet-Drafts and an RFC describing three encapsulations:
- "Geneve: Generic Network Virtualization Encapsulation" [RFC8926]
- "Generic UDP Encapsulation" [GUE]
- "Generic Protocol Extension for VXLAN (VXLAN-GPE)" [VXLAN-GPE]
discussion on the list and in face-to-face meetings flagged a bunch of technical problems with each one 😬 And there was clear consensus at the 96th IETF meeting in Berlin that the WG should only push forward one data plane encapsulation, to maximize interoperability 💅 To break the deadlock on the choice, the WG agreed to form a Design Team [RFC2418] to sort it out and provide initial considerations 🧚
2. Design Team and Working Group Process 💜
the Design Team was supposed to pick one of the proposed encapsulations and level it up to fix the technical concerns 🆙 The goals were simple evolution of deployed networks, plus applicability to every location in the NVO3 architecture 🌍 They were specifically looking for a design that allows future extensibility but isn't burdensome on hardware implementations 🔧 The chosen design also needed to play nicely with the Internet Control Message Protocol (ICMP) and in Equal-Cost Multipath (ECMP) environments 🌊 And if more extensibility is needed, it has to happen in a way that doesn't need permission from any entity outside the IETF 🕊️
the Design Team's output then went through the working group, which resulted in the working group consensus for this document 🫶
3. Terminology 🎀
the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are the BCP 14 [RFC2119] [RFC8174] words in this doc 🔑 They only mean what BCP 14 says when they show up in ALL CAPITALS, exactly like that. Otherwise they're just regular words 🙈
4. Abbreviations, Acronyms, and Definitions 🌸
the following abbreviations and acronyms are used in this document 📚
- ACL: Access Control List
- ECMP: Equal-Cost Multipath
- EVPN: Ethernet VPN [RFC8365]
- Geneve: Generic Network Virtualization Encapsulation [RFC8926]
- GPE: Generic Protocol Extension
- GUE: Generic UDP Encapsulation [GUE]
- HMAC: Hash-Based Message Authentication Code [RFC2104]
- IEEE: Institute for Electrical and Electronic Engineers (
<https://www.ieee.org/>) - NIC: Network Interface Card (refers to network interface hardware that is not necessarily a discrete "card") 🖧
- NSH: Network Service Header [RFC8300]
- NVA: Network Virtualization Authority
- NVE: Network Virtual Edge (refers to an NVE device)
- NVO3: Network Virtualization over Layer 3
- OAM: Operations, Administration, and Maintenance [RFC6291]
- PWE3: Pseudowire Emulation Edge-to-Edge
- TCAM: Ternary Content-Addressable Memory
- TLV: Type-Length-Value
- Transit device: Refers to underlay network devices between NVEs. 🚦
- UUID: Universally Unique Identifier
- VNI: Virtual Network Identifier
- VXLAN: Virtual eXtensible Local Area Network [RFC7348]
5. Encapsulation Issues and Background 🌈
the following subsections describe the issues with current encapsulations that the NVO3 WG talked about 💬 Lots of extensions and options have been designed for GUE and Geneve that might help with some of these problems, but the WG hasn't validated them yet 🧪
there are also diagrams and info on the candidate encapsulations. Most of them are copied from other docs 📎 Since each protocol is assumed to be sent over UDP, there's an initial UDP header shown, which would sit behind an IPv4 or IPv6 header 🧩
5.1. Geneve 🦋
the Geneve packet format, taken from [RFC8926], is shown in Figure 1 below 👀
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
Outer UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Dest Port = 6081 Geneve |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Geneve Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Ver| Opt Len |O|C| Rsvd. | Protocol Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Virtual Network Identifier (VNI) | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Variable-Length Options ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1: Geneve Headerthe type of payload being carried is shown by an Ethertype [RFC9542] in the Protocol Type field of the Geneve header 📦 Ethernet itself is Ethertype 0x6558 🧃 See [RFC8926] for details on the UDP header fields. The O bit marks an OAM packet 🔍 The Geneve C bit is the "Critical" bit, meaning the options must be processed or the packet gets discarded 🗑️
Geneve [RFC8926] has these issues:
- Geneve can't be implemented cost-effectively in all use cases, because the variable-length header and the order of the TLVs make it costly (in number of gates) to build in hardware 😤
- The header doesn't fit in the largest commonly available parse buffer (256 bytes in a NIC). So doubling the buffer size can't be justified unless it's mandatory for hardware to process extra option fields 📏
Geneve may have been picked despite these issues because of the Geneve design effort, assuming the Geneve header would typically get delivered to a server and parsed in software 💻✨
5.2. Generic UDP Encapsulation (GUE) 🎀
okay so GUE is basically "UDP, but make it extensible" 💅 Here's the layout:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Dest Port = 6080 GUE |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
GUE Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 0 |C| Hlen | Proto/ctype | Flags |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Extensions Fields (optional) ~
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: GUE HeaderSo the type of payload being carried is an IANA protocol number, and it lives in the Proto/ctype field 🦋 The GUE C bit (Control bit) is the one that tells you it's a control packet 💜
Issues with GUE [GUE] are as follows:
- There were a significant number of objections to GUE related to the complexity of its implementation in hardware, similar to those noted for Geneve above, such as the variable length and possible high maximum length of the header. 😭
5.3. Generic Protocol Extension (GPE) for VXLAN 🌸
Now GPE, which is VXLAN's glow-up attempt 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
Outer UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Dest Port = 4790 GPE |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
VXLAN-GPE Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|Ver|I|P|B|O| Reserved | Next Protocol |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Virtual Network Identifier (VNI) | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: GPE HeaderThe type of payload is indicated by the Next Protocol field, which uses a registry specific to VXLAN-GPE. The I bit says the VNI is valid, and the P bit says the Next Protocol field is valid. The B bit marks the packet as an ingress replicated Broadcast, Unknown Unicast, or Multicast packet, and the O bit marks an OAM packet ✨
Issues with VXLAN-GPE [VXLAN-GPE] are as follows:
- GPE is not day one backwards compatible with VXLAN [RFC7348]. Although the frame format is similar, it uses a different UDP port, so it would require changes to existing implementations even if the rest of the GPE frame were the same. 💀
- GPE is insufficiently extensible. It adds a Next Protocol field and some flag bits to the VXLAN header but is not otherwise extensible. 🙈
- As discussed in Section 6.2.2, security (e.g., of the VNI) has not been addressed by GPE. Although a shim header could be added for security and to support other extensions, this has not been defined yet. More study would be needed to understand the implication of such a shim on offloading in NICs. 🔥
6. Common Encapsulation Considerations 👑
6.1. Current Encapsulations
Appendix A includes a detailed comparison between the three proposed encapsulations 🍓 The comparison shows several common properties, but also three major differences:
- Extensibility: Geneve and GUE were defined with built-in extensibility, while VXLAN-GPE is not inherently extensible. Note that any of the three encapsulations can be extended using the Network Service Header (NSH) [RFC8300].
- Extension method: Geneve is extensible using Type-Length-Value (TLV) fields, while GUE uses a small set of possible extensions and a set of flags that indicate which extensions are present.
- Length field: Geneve and GUE include a Length field, indicating the length of the encapsulation header, while VXLAN-GPE does not include such a field. Thus, it may be harder to skip the encapsulation header with VXLAN-GPE
6.2. Useful Extensions Use Cases
Extensions that are not vendor-specific, such as TLVs, MUST follow the standardization process 💜 The following use cases show there is a strong requirement to support variable-length extensions with possible different subtypes, and honestly it's giving very real needs 😤
6.2.1. Telemetry Extensions
In several scenarios, it's super beneficial to make information available to the operator about the path a packet took through the network or through a network device, plus information about associated telemetry 🔍
This covers debugging, troubleshooting, and network planning and optimization, but also policy or service level agreement compliance checks ✅
Packet scheduling algorithms, especially for balancing traffic across equal-cost paths or links, often lean on information inside the packet, like protocol number, IP address, or Message Authentication Code (MAC) address. So probe packets would need to be either sent between the exact same endpoints with the exact same parameters, or artificially constructed as "fake" packets and inserted along the path. Both approaches are often not feasible from an operational perspective, because access to the end system is not feasible or the diversity of parameters and associated probe packets to be created is simply too large 😩 An extension providing an in-band telemetry mechanism [RFC9197] is an alternative in those cases ✨
6.2.2. Security/Integrity Extensions
Okay, this one is serious bestie 🚨 Since the currently proposed NVO3 encapsulations do not protect their headers, a single bit corruption in the VNI field could deliver a packet to the wrong tenant. Extension headers are needed to use any sophisticated security.
The possibility of VNI spoofing with an NVO3 protocol is made worse by using UDP. Systems typically have no restrictions on applications being able to send to any UDP port, so an unprivileged application can trivially spoof VXLAN [RFC7348] packets, using arbitrary VNIs, for instance 😱
One can envision support of an HMAC-like Message Authentication Code (MAC) [RFC2104] in an NVO3 extension to authenticate the header and the outer IP addresses, thereby preventing attackers from injecting packets with spoofed VNIs 🛡️
Another aspect of security is payload security. Essentially, this makes packets that look like the following:
IP|UDP|NVO3 Encap|DTLS/IPsec-ESP Extension|payload.This is desirable because:
- we still have the UDP header for ECMP,
- the NVO3 header is in plain text so it can be read by network elements, and
- different security or other payload transforms can be supported on a single UDP port (we don't need a separate UDP port for DTLS/IPsec; see [RFC9147] and [RFC6071], respectively). 🦋
6.2.3. Group-Based Policy
Another use case would be to carry the Group-Based Policy (GBP) source group information within a NVO3 header extension, in a similar manner to what has been implemented for VXLAN [VXLAN-GROUP] 💅 This lets you apply various forms of policy, such as access control and QoS, between abstract groups rather than tying them to specific endpoint addresses. Iconic 👑
6.3. Hardware Considerations
Hardware restrictions should be taken into consideration, along with future hardware enhancements that may provide more flexible metadata (MD) processing 🔧 However, the set of options that need to be, and will be, implemented in hardware is a subset of what's implemented in software. That's because software NVEs are likely to grow features, and hence option support, at a faster rate 🚀
It's hard to predict which options will land in which piece of hardware, and when. That depends on whether the hardware will be in the form of:
- a NIC providing increasing offload capabilities to software NVEs, or
- a switch chip being used as an NVE gateway towards non-NVO3 parts of the network, or even
- a transit device that participates in the NVO3 data plane, e.g., for OAM purposes.
So it doesn't look useful to prescribe some order to the options so that the ones likely to be implemented in hardware come first. We can't decide such an order when we define the options; however, a control plane can enforce such an order for some hardware implementations 🧠
We do know that hardware initially needs to be able to efficiently skip over the NVO3 header to find the inner payload. That's needed both for NICs implementing various TCP offload mechanisms, and for transit devices and NVEs applying policy or ACLs to the inner payload 🫶
6.4. Extension Size
Extension header length has a significant impact on hardware and software implementations. A maximum total header length that is too small will unnecessarily constrain software flexibility. A maximum total header length that is too large will place a nontrivial cost on hardware implementations. Thus, the DT recommends that there be a minimum and maximum total available extension header length specified. The maximum total header length is determined by the size of the bit field allocated for the total extension header length field. The risk with this approach is that it may be difficult to extend the total header size in the future. The minimum total header length is determined by a requirement in the specifications that all implementations must meet. The risk with this approach is that all implementations will only implement support for the minimum total header length, which would then become the de facto maximum total header length 🙃
The recommended minimum total available header length is 64 bytes 🍰
The size of an extension header should always be 4-byte aligned 📏
The maximum length of a single option should be large enough to meet the different extension use case requirements, e.g., for in-band telemetry and future use 🌈
6.5. Ordering of Extension Headers
To support hardware nodes at the target NVE, or at a transit device, that can process one or a few extension headers in TCAM, a control plane in such a deployment could signal a capability to ensure that a specific extension header always appears in a specific order. For example, that specific extension header could appear first in the packet 🎯
The order of the extension headers should be hardware friendly for both the sender and the receiver, and possibly some transit devices as well. This may require that the extension headers and their order be determined dynamically, based on the hardware of those devices 🧩
Transit devices don't participate in control plane communication between the endpoints and are not required to process the extension headers. However, if they do, they may need to process only a small subset of the extension headers that will be consumed by target NVEs 🦋
6.6. TLV versus Bit Fields 💅✨
okay so bestie, this is the TLV vs bit fields showdown and it's giving main character energy 🎀. if there's a well-known starter set of options that's likely to get built into both software AND hardware, a bit fields approach is super efficient to signal which extensions are present, just like GUE does 🦋. BUT (and this is the tea 🍵) as Section 6.3 explained, if options keep getting added over time and different subsets end up in different chips, the IETF would really struggle to decide which options deserve the early bit fields 😭. TLVs are way more flexible because you don't have to figure out the relative importance of each option up front 💖. the catch? general TLVs of any order, size, and repetition are a nightmare to implement in hardware 💀.
so the middle ground is TLVs with restrictions on their size and alignment 🌸. individual TLVs can have a fixed length, and the control plane can make sure an NVE only receives the options it actually needs and implements 🫶. the control plane can also be used to control the order of the TLVs sent to a particular NVE ✨. just note that transit devices usually don't take part in the control plane, so if they need to take part in option processing, some other method has to be used 🙈. transit devices would also have issues with future GUE bit fields being defined for future options 😬.
a benefit of TLVs from a hardware perspective is that they're self describing, i.e., all the information is in the TLV 🧚. with a bit field approach, the hardware has to look up the bit in some separate table to figure out the length of the data tied to that bit, and that adds hardware complexity 🤓.
there are also use cases where multiple software modules are running on one NVE 🍓. think a diagnostic module from one vendor that does packet sampling, plus a firewall module from a different vendor 🔥. with a TLV format it's way easier for different software modules to process different TLVs without stepping on each other's toes 👑. those TLVs can be standard extensions or vendor-specific extensions 🌈. this can help with hardware modularity too, and some implementations already have options that let different software modules, like MAC learning and security, process different options 💜.
6.7. Control Plane Considerations 🦋
okay so we want to allow a LOT of flexibility and extensibility (think software NVEs), but we also want to support important extensions in less flexible places like hardware NVEs 💅. that's why it's useful to think about the control plane 🧠. when we say "control plane" here, we mean protocols like EVPN [RFC8365] and others, plus deployment-specific configurations ✨.
if each NVE can say in the control plane that it only supports certain extensions (could be just one, or a few 🎀), and the source NVEs only put supported extensions into NVO3 packets, then the target NVE can use a simpler parser 🌸. (a TCAM might be usable to look for a single NVO3 extension, which is so clever 😭). the depth of the inner payload in the NVO3 packet also gets minimized 💖. and if the target NVE cares about a few extensions and can signal the order it wants them in via the control plane, the deployment gets useful functionality with simpler hardware requirements 🫶.
transit devices that don't know about NVO3 extensions still kinda benefit from this 🦋, since the inner payload is less deep in the packet when no extra extension headers are included 🧚. transit devices usually aren't part of the NVO3 control plane though 🙈. configuration mechanisms can still account for what the transit devices in a given deployment can handle 💅.
heads up: different NVEs might want different extensions or sets of extensions. that means the source NVE has to be able to put different sets of extensions into different NVO3 packets, and maybe in a different order 👑. it also assumes that underlay multicast or replication servers are NOT used together with NVO3 extension headers 💀.
there's a need to think about mandatory extensions versus optional extensions 🌈. mandatory extensions require the receiver to drop the packet if it doesn't know the extension 😱. a control plane mechanism can prevent that drop from happening, since unsupported extensions just wouldn't be included for target NVEs that don't support them 🫶.
the control planes defined today need to add the ability to describe the different encapsulations 💜. so, EVPN [RFC8365] and any other control plane protocol the IETF defines should have a way to indicate the supported NVO3 extensions and their order for each encapsulation they support 🔥.
a separate document giving guidance on option processing and control plane participation should be considered ✨. it should include examples and guidance on the range of usage models and deployment scenarios for specific options, plus examples of option ordering that's relevant for each deployment 🍓. this covers endpoints and middleboxes that use the options too 👑. having the control plane negotiate the constraints is the most appropriate and flexible way to handle all of this 💖.
6.8. Split NVE 🌸
if hosts need to send and receive options in a split NVE case [RFC8394] 🧚, it's possible with any of the existing extensible encapsulations (GPE with NSH, GUE, or Geneve) by defining a way to carry those over other transports 🦋. NSH can already be used over different transports, so it's already set ✨.
if this is needed with other encapsulations, it can be done by defining an Ethertype so the encapsulation can be carried over Ethernet and IEEE Std 802.1Q [IEEE802.1Q] 💅.
if other encapsulations need to be carried over MPLS, it would require an EVPN control plane to signal that other encapsulation headers and options will be present in front of the Layer 2 (L2) packet 🫶. the VNI can be ignored in the header, and the MPLS label is the one used to identify the EVPN L2 instance 🔥.
6.9. Larger VNI Considerations 🎀
so whether the VNI should be 32 bits or larger was one of the topics the group considered 👑. the benefit of a 24-bit VNI is avoiding unnecessary changes to existing proposals and implementations, which are almost all (if not all) using a 24-bit VNI 💜. if we do end up needing a larger VNI, maybe for a telemetry case, an extension can be used to support that ✨💖.
7. Recommendations 💅✨
Okay so the Design Team reported that Geneve was the most suitable starting point for a proposed standard for network virtualization, and the reasons are iconic 👑 The NVO3 Working Group backed this conclusion too.
- Okay, the big debate: should the VNI live in the base header or an extension header, and should it be a 24-bit or 32-bit field? (see Section 6.9) The group agreed that the VNI is critical info for network virtualization and it MUST be present in all packets 🔑 They also agreed that a 24-bit VNI, which is what Geneve uses, matches the widely used encapsulation formats, VXLAN [RFC7348] and Network Virtualization Using Generic Routing Encapsulation (NVGRE) [RFC7637]. So it's the more suitable choice going forward 🦋
- Geneve has a total options length in its header, which lets NIC offload operations and transit devices skip over the options and still look at flow information in the inner payload 🧚
- Next up: using an NSH [RFC8300] with VXLAN-GPE got considered 🤔 but an NSH is built for service chaining and carries service chaining info, so it's a worse fit for the network virtualization use case. VXLAN-GPE also has no header length field, which makes skipping over headers to reach the inner payload way harder. Geneve has a total options length, but VXLAN-GPE can't skip any options in the middle. A split between a base header and an options header sounded interesting, whether that options header is an NSH or a brand new header not tied to a service path. The group also looked at using an NSH for this or defining a new NVO3 options header. The catch is that it's slightly harder to find the inner payload, because the Length field isn't in the NVO3 header itself, so one more field would have to be extracted to compute where the inner payload starts. Plus, if the IPv6 extension headers experience is any guide, key hardware might not implement the options header, and then we'd be stuck calling for its deprecation later 😬 Making the options part of the base NVO3 header has fewer of those issues. You can't predict which option will be implemented ahead of time, but the option mechanism and the ability to skip options are likely to be broadly implemented 💜
- The group compared the TLV style and the bit field style of extension mechanisms 🍓 Parsing either TLVs or bit fields was deemed expensive. Bit fields may be simpler to parse, but they're more restrictive and you have to guess which extensions will be widely implemented in order to get early bit assignments. Half the bits are already assigned in GUE, so a widely deployed extension might show up as a flag extension, which means extra processing to dig the flag out of the flag extension and then look for the extension itself. Bit fields also aren't flexible enough for the OAM, telemetry, and security extension requirements, which need variable-length options and different subtypes of the same option. TLVs are more flexible, and a control plane can restrict the number of option TLVs, plus their order and size, to limit that flexibility and make them simpler for a data plane to handle ✨
- The multi-vendor NVE case got a quick discussion, along with the need to let vendors put their own extensions in the NVE header. TLVs make this possible 🌈
- The group agreed that the C bit (Critical bit) in Geneve is helpful 🔥 It indicates that the header includes options that must be parsed, or else the packet must be discarded. This lets a receiver NVE easily decide whether to process options (like a UUID-based packet trace) and figure out how an optional extension can be ignored. So a Critical bit makes it easy for the NVE to skip over options not marked with it. That's why the C bit should stay as defined in Geneve 🫶
- Some extensions of varying sizes are already being discussed (see Section 6.2) 🎀 Using Geneve options, you can get in-band parameters like switch id, ingress port, egress port, internal delay, and queue size, using TLV extensions for telemetry from switches. You can also add security extension TLVs like HMAC [RFC2104] and DTLS/IPsec (see [RFC9147] and [RFC6071], respectively) to authenticate the Geneve packet header and secure the Geneve packet payload, done by software or hardware tunnel endpoints 🔐 A Group-Based Policy extension TLV can be carried too.
- Geneve options are already implemented and deployed in production networks 🚀 There's new hardware that supports Geneve TLV parsing too. The P4.org In-band Telemetry (INT) specification [INT] is being developed and shows INT metadata carried over Geneve. Open Virtual Network (OVN) and Open vSwitch (OVS) [OVN] have also defined one or more option TLVs for Geneve.
- The usage requirements (see Section 6) have been addressed, while also considering requirements and implementations in general, including those for software and hardware 💖
There seems to be interest in standardizing some well-known secure option TLVs to secure the header and payload, guaranteeing encapsulation header integrity and tenant data privacy 🕵️♀️ The working group should consider standardizing such option(s).
The following enhancements to Geneve are recommended to make it more suitable for hardware while still giving software flexibility 🌸
- The following sort of text is recommended in Geneve documents: while TLVs are more flexible, a control plane can restrict the number of option TLVs as well as the order and size of the TLVs to make it simpler for a data plane implementation in software or hardware to handle. For example, there may be some critical information, such as a secure hash, that must be processed in a certain order at lowest latency.
- A control plane can negotiate a subset of option TLVs and a certain TLV ordering, and limit the total number of option TLVs present in the packet. For example, this lets hardware that can process fewer options keep up. So the control plane needs the ability to describe the supported TLV subset and their order 🧩
- The Geneve documents should specify that the subset and order of option TLVs SHOULD be configurable for each remote NVE in the absence of a protocol control plane.
- Geneve should follow the fragmentation recommendations in overlay services like PWE3 and the L2/L3 VPN recommendations, to guarantee larger MTUs for the tunnel overhead ([RFC3985], Section 5.3).
- The Geneve documents should provide a recommendation for C bit (Critical bit) processing. This text could specify how critical bits can be used with control planes and specify the critical options 🎯
- Since there's a telemetry option use case for a length of 256 bytes, it is recommended that Geneve increase the single TLV option length to 256 📏
- Geneve address requirements for OAM considerations, for alternate marking and for performance measurements that need a 2-bit field in the header, should be considered. The need for the current OAM bit in the Geneve header should be clarified 🦋
- The WG should work on security options for Geneve 🔐
8. Security Considerations 🚨
This document does not introduce any additional security constraints. However, Section 6.2.2 discusses security/integrity extensions, and this document suggests, in Section 7, that the NVO3 WG work on security options for Geneve 💀
9. IANA Considerations 🎀
This document has no IANA actions. 🌷
10. References 📚
10.1. Normative References 🦋
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997,
https://www.rfc-editor.org/info/rfc2119. - [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017,
https://www.rfc-editor.org/info/rfc8174.
10.2. Informative References 📚
Okay bestie, here's the reading list for the extra-credit stuff 💅
[GUE] Herbert, T., Yong, L., and O. Zia, "Generic UDP
Encapsulation", Work in Progress, Internet-Draft, draft-
ietf-intarea-gue-09, 26 October 2019,
<https://datatracker.ietf.org/doc/html/draft-ietf-intarea-
gue-09>.
[GUE-ENCAPSULATION]
Yong, L., Herbert, T., and O. Zia, "Generic UDP
Encapsulation (GUE) for Network Virtualization Overlay",
Work in Progress, Internet-Draft, draft-hy-nvo3-gue-4-nvo-
04, 28 October 2016,
<https://datatracker.ietf.org/doc/html/draft-hy-nvo3-gue-
4-nvo-04>.
[GUE-EXTENSIONS]
Herbert, T., Yong, L., and F. Templin, "Extensions for
Generic UDP Encapsulation", Work in Progress, Internet-
Draft, draft-ietf-intarea-gue-extensions-06, 8 March 2019,
<https://datatracker.ietf.org/doc/html/draft-ietf-intarea-
gue-extensions-06>.
[IEEE802.1Q]
IEEE, "IEEE Standard for Local and Metropolitan Area
Networks--Bridges and Bridged Networks", IEEE Std 802.1Q-
2022, DOI 10.1109/IEEESTD.2022.10004498, December 2022,
<https://doi.org/10.1109/IEEESTD.2022.10004498>.
[INT] P4.org Applications Working Group, "In-band Network
Telemetry (INT) Dataplane Specification", November 2020,
<https://p4.org/p4-spec/docs/INT_v2_1.pdf>.
[OVN] Linux Foundation, "Open vSwitch",
<https://www.openvswitch.org/>.
[RFC2104] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-
Hashing for Message Authentication", RFC 2104,
DOI 10.17487/RFC2104, February 1997,
<https://www.rfc-editor.org/info/rfc2104>.
[RFC2418] Bradner, S., "IETF Working Group Guidelines and
Procedures", BCP 25, RFC 2418, DOI 10.17487/RFC2418,
September 1998, <https://www.rfc-editor.org/info/rfc2418>.
[RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
Edge-to-Edge (PWE3) Architecture", RFC 3985,
DOI 10.17487/RFC3985, March 2005,
<https://www.rfc-editor.org/info/rfc3985>.
[RFC6071] Frankel, S. and S. Krishnan, "IP Security (IPsec) and
Internet Key Exchange (IKE) Document Roadmap", RFC 6071,
DOI 10.17487/RFC6071, February 2011,
<https://www.rfc-editor.org/info/rfc6071>.
[RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu,
D., and S. Mansfield, "Guidelines for the Use of the "OAM"
Acronym in the IETF", BCP 161, RFC 6291,
DOI 10.17487/RFC6291, June 2011,
<https://www.rfc-editor.org/info/rfc6291>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.
[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
Virtualization Using Generic Routing Encapsulation",
RFC 7637, DOI 10.17487/RFC7637, September 2015,
<https://www.rfc-editor.org/info/rfc7637>.
[RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed.,
"Network Service Header (NSH)", RFC 8300,
DOI 10.17487/RFC8300, January 2018,
<https://www.rfc-editor.org/info/rfc8300>.
[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.
[RFC8394] Li, Y., Eastlake 3rd, D., Kreeger, L., Narten, T., and D.
Black, "Split Network Virtualization Edge (Split-NVE)
Control-Plane Requirements", RFC 8394,
DOI 10.17487/RFC8394, May 2018,
<https://www.rfc-editor.org/info/rfc8394>.
[RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
"Geneve: Generic Network Virtualization Encapsulation",
RFC 8926, DOI 10.17487/RFC8926, November 2020,
<https://www.rfc-editor.org/info/rfc8926>.
[RFC9147] Rescorla, E., Tschofenig, H., and N. Modadugu, "The
Datagram Transport Layer Security (DTLS) Protocol Version
1.3", RFC 9147, DOI 10.17487/RFC9147, April 2022,
<https://www.rfc-editor.org/info/rfc9147>.
[RFC9197] Brockners, F., Ed., Bhandari, S., Ed., and T. Mizrahi,
Ed., "Data Fields for In Situ Operations, Administration,
and Maintenance (IOAM)", RFC 9197, DOI 10.17487/RFC9197,
May 2022, <https://www.rfc-editor.org/info/rfc9197>.
[RFC9542] Eastlake 3rd, D., Abley, J., and Y. Li, "IANA
Considerations and IETF Protocol and Documentation Usage
for IEEE 802 Parameters", BCP 141, RFC 9542,
DOI 10.17487/RFC9542, April 2024,
<https://www.rfc-editor.org/info/rfc9542>.
[VXLAN-GPE]
Maino, F., Ed., Kreeger, L., Ed., and U. Elzur, Ed.,
"Generic Protocol Extension for VXLAN (VXLAN-GPE)", Work
in Progress, Internet-Draft, draft-ietf-nvo3-vxlan-gpe-13,
4 November 2023, <https://datatracker.ietf.org/doc/html/
draft-ietf-nvo3-vxlan-gpe-13>.
[VXLAN-GROUP]
Smith, M. and L. Kreeger, "VXLAN Group Policy Option",
Work in Progress, Internet-Draft, draft-smith-vxlan-group-
policy-05, 22 October 2018,
<https://datatracker.ietf.org/doc/html/draft-smith-vxlan-
group-policy-05>.Appendix A. Encapsulation Comparison 🦋
A.1. Overview 💖
Okay so this appendix is a side-by-side of the three NVO3 encapsulation proposals: Geneve [RFC8926], GUE [GUE], and VXLAN-GPE [VXLAN-GPE]. All three ride on an outer UDP/IP transport 🌸. Geneve and VXLAN-GPE use an 8-octet header, while GUE uses a 4-octet header ✨. On top of the base header, you can also add optional extensions to the encapsulation, and we'll get into that in Appendix A.2 below 💅.
A.2. Extensibility 🧚
A.2.1. Innate Extensibility Support 💜
Geneve and GUE both let you tack optional headers onto the end of the base encapsulation header 🎀.
VXLAN-GPE doesn't have innate support for header extensions 🙈. BUT, as [VXLAN-GPE] explains, you can get some extensibility if you put the Network Service Header (NSH) [RFC8300] right after the VXLAN-GPE header. The NSH gives you either a fixed-size extension (MD Type 1) or a variable-size TLV-based extension (MD Type 2) 🌈. Heads up: NSH-over-VXLAN-GPE adds the 8-octet NSH on top of the VXLAN-GPE header, so there's extra overhead to account for 💀.
A.2.2. Extension Parsing 🌈
Geneve's variable-length options are defined as Type-Length-Value (TLV) extensions. VXLAN-GPE can include NSH TLV-based extensions too, when it's using an NSH 🦋. GUE is the different one: it defines a small set of possible extension fields (proposed in [GUE-EXTENSIONS] and [GUE-ENCAPSULATION]), plus a set of flags in the GUE header that say, for each extension type, whether it's present or not 🫶.
TLV-based extensions like the ones in Geneve give you a lot of flexibility for a big pile of extension types. Similar behaviour can be supported in NSH-over-VXLAN-GPE when using MD Type 2. The flag-based approach in GUE aims to keep implementations simple by defining a small number of possible extensions, used in a fixed order ✨.
The Geneve and GUE headers both include a Length field that defines the total length of the encapsulation, including the optional extensions. This Length field makes life easier for transit devices, since they can skip the encapsulation header without parsing its extensions 🧠.
A.2.3. Critical Extensions 🔥
The Geneve encapsulation header includes the C field, which says whether the current Geneve header includes critical options. Those are options which must be parsed by the target NVE 👀. If the endpoint isn't able to process a critical option, the packet gets discarded 💀.
A.2.4. Maximal Header Length 👑
The maximal header length in Geneve, options included, is 260 octets. GUE defines its maximal header as 128 octets. VXLAN-GPE uses a fixed-length 8-octet header, unless NSH-over-VXLAN-GPE is used, which gives you an encapsulation header of up to 264 octets 📏.
A.3. Encapsulation Header 🎀
A.3.1. Virtual Network Identifier (VNI) 🌸
The Geneve and VXLAN-GPE headers both have a 24-bit VNI field. GUE, on the other hand, can use a 32-bit field called VNID. That field isn't in the GUE header itself, but it was defined as an optional extension in [GUE-ENCAPSULATION] 🧚.
The VXLAN-GPE header has the I bit, which says the VNI field is valid in the current header. A similar indicator is defined as a flag in the GUE header [GUE-EXTENSIONS] 💖.
A.3.2. Next Protocol 💜
All three encapsulation headers have a field that says what type the next protocol header is, the one sitting after the NVO3 encapsulation header 📦. Geneve has a 16-bit field that uses the IEEE Ethertype convention. GUE uses an 8-bit field that follows IANA protocol numbering. The VXLAN-GPE header has an 8-bit Next Protocol field, using a VXLAN-GPE-specific registry defined in [VXLAN-GPE] 🌈.
The VXLAN-GPE header also has the P bit, which explicitly says whether the Next Protocol field is present in the current header 🍓.
A.3.3. Other Header Fields 🦋
The OAM bit, which is defined in Geneve and VXLAN-GPE, says whether the current packet is an OAM packet 🔧. The GUE header has a similar field but different terminology: the GUE C bit (Control bit) says whether the current packet is a control packet. Note that the GUE C bit can potentially be used in a large set of protocols that aren't OAM protocols 👀. The control packet examples in [GUE] are related to OAM, though.
Each of the three NVO3 encapsulation headers has a 2-bit Version field, which is currently defined to be zero 🔢.
The Geneve and VXLAN-GPE headers include reserved fields: 14 bits in the Geneve header and 27 bits in the VXLAN-GPE header are reserved 🙃.
A.4. Comparison Summary
okay bestie, here's the whole showdown between the three NVO3 encapsulations in one table 💅 when you see a plus sign ("+") or a minus sign ("-"), it means the header is stronger or weaker in that area ✨
+===============+=================+=============+===================+
| | Geneve | GUE | VXLAN-GPE |
+===============+=================+=============+===================+
| Outer | UDP/IP 6081 | UDP/IP 6080 | UDP/IP 4790 |
| transport UDP | | | |
| Port Number | | | |
+---------------+-----------------+-------------+-------------------+
| Base header | 8 octets | 4 octets | 8 octets (16 |
| length | | | octets using |
| | | | an NSH) |
+---------------+-----------------+-------------+-------------------+
| Extensibility | Variable-length | Extension | No innate |
| | options | fields | extensibility. |
| | | | Might use an |
| | | | NSH. |
+---------------+-----------------+-------------+-------------------+
| Extension | TLV-based | Flag-based | TLV-based |
| parsing | | | (using an NSH |
| method | | | with MD Type |
| | | | 2) |
+---------------+-----------------+-------------+-------------------+
| Extension | Variable | Fixed | Variable |
| order | | | (using an NSH) |
+---------------+-----------------+-------------+-------------------+
| Length field | + | + | - |
+---------------+-----------------+-------------+-------------------+
| Max header | 260 octets | 128 octets | 8 octets (264 |
| length | | | using an NSH) |
+---------------+-----------------+-------------+-------------------+
| Critical | + | - | - |
| extension bit | | | |
+---------------+-----------------+-------------+-------------------+
| VNI field | 24 bits | 32 bits | 24 bits |
| size | | (extension) | |
+---------------+-----------------+-------------+-------------------+
| Next Protocol | 16 bits | 8 bits | 8 bits New |
| field | Ethertype | Internet | registry |
| | registry | protocol | |
| | | registry | |
+---------------+-----------------+-------------+-------------------+
| Next protocol | - | - | + |
| indicator | | | |
+---------------+-----------------+-------------+-------------------+
| OAM / Control | OAM bit | Control bit | OAM bit |
| field | | | |
+---------------+-----------------+-------------+-------------------+
| Version field | 2 bits | 2 bits | 2 bits |
+---------------+-----------------+-------------+-------------------+
| Reserved bits | 14 bits | none | 27 bits |
+---------------+-----------------+-------------+-------------------+
Table 1: Encapsulations ComparisonAcknowledgements ✨
the authors want to give a huge shoutout 🙌 to Tom Herbert for the motivation behind the security/integrity extension and for his valuable comments 💖 T. Sridhar gets thanks for valuable comments and feedback 🫶 Anoop Ghanwani gets thanks for his extensive comments 💜 and Ignas Bagdonas gets a big thank you too 🌸
Contributors 🎀
these coauthors helped build this document, and they were absolutely iconic about it 👑
Ilango Ganga
Intel
Email: ilango.s.ganga@intel.com
Pankaj Garg
Microsoft
Email: pankajg@microsoft.com
Rajeev Manur
Broadcom
Email: rajeev.manur@broadcom.com
Tal Mizrahi
Huawei
Email: tal.mizrahi.phd@gmail.com
David Mozes
Email: mosesster@gmail.com
Erik Nordmark
ZEDEDA
Email: nordmark@sonic.net
Michael Smith
Cisco
Email: michsmit@cisco.com
Sam Aldrin
Google
Email: aldrin.ietf@gmail.comAuthors' Addresses 🦋
here's where you can find the editors, so you can slide into their inbox if you have questions 💌
Sami Boutros (editor)
Ciena Corporation
United States of America
Email: sboutros@ciena.com
Donald E. Eastlake 3rd (editor)
Independent
2386 Panoramic Circle
Apopka, FL 32703
United States of America
Phone: +1-508-333-2270
Email: d3e3e3@gmail.com📦 where this page comes from
- ietf: rfc9638 2024-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms