RFC 7637
NVGRE: Network Virtualization Using Generic Routing Encapsulation
RFC 7637: NVGRE, Network Virtualization Using Generic Routing Encapsulation 🎀
okay bestie, this one is the NVGRE spec, and it was written by P. Garg, Ed. and Y. Wang, Ed. from Microsoft 💅 It was published in September 2015 as RFC 7637, and its category is Informational. That means it's a published document, not an Internet Standards Track spec, so it's not a standard you're required to follow 🧚 It's an independent submission to the RFC Series, and it has ISSN 2070-1721. Nothing in the header says it obsoletes or updates another RFC.
Abstract ✨
okay so the big idea here is using the Generic Routing Encapsulation (GRE) header for Network Virtualization (NVGRE) in multi-tenant data centers 🏢 Network virtualization splits virtual networks and their addresses away from the physical network underneath, so lots of virtual networks can share the same physical network while staying isolated and running side by side 🦋 The doc also sketches a network virtualization framework to show the use cases, but the real focus is the data-plane part of NVGRE. the main character here is the data plane 💖
Status of This Memo ✨
this doc is not an Internet Standards Track spec, it's just published for info 🌸 It's a contribution to the RFC Series, independent of any other RFC stream. The RFC Editor picked to publish it at their discretion and makes no claims about how useful it is to implement or deploy 🙈 Docs the RFC Editor approves aren't candidates for any level of Internet Standard. (See Section 2 of RFC 5741 if you want the details.)
for the current status, errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc7637 🔗
Copyright Notice ✨
© 2015 IETF Trust and the persons identified as the document authors. All rights reserved 💜
this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they were on the publication date. please read these carefully, bestie, because they describe your rights and restrictions for this document 📜
Table of Contents 🧚
1. Introduction ....................................................2
1.1. Terminology ................................................4
2. Conventions Used in This Document ...............................4
3. Network Virtualization Using GRE (NVGRE) ........................4
3.1. NVGRE Endpoint .............................................5
3.2. NVGRE Frame Format .........................................5
3.3. Inner Tag as Defined by IEEE 802.1Q ........................8
3.4. Reserved VSID ..............................................8
4. NVGRE Deployment Considerations .................................9
4.1. ECMP Support ...............................................9
4.2. Broadcast and Multicast Traffic ............................9
4.3. Unicast Traffic ............................................9
4.4. IP Fragmentation ..........................................10
4.5. Address/Policy Management and Routing .....................10
4.6. Cross-Subnet, Cross-Premise Communication .................10
4.7. Internet Connectivity .....................................12
4.8. Management and Control Planes .............................12
4.9. NVGRE-Aware Devices .......................................12
4.10. Network Scalability with NVGRE ...........................13
5. Security Considerations ........................................14
6. Normative References ...........................................14
Contributors ......................................................16
Authors' Addresses ................................................171. Introduction 💖
okay so conventional data center designs are built for mostly static workloads, and that causes fragmentation of network and server capacity [6] [7] 😭 A few things stop capacity from being allocated dynamically and consolidated. Layer 2 networks use the Rapid Spanning Tree Protocol (RSTP), which is designed to kill loops by blocking redundant paths. those blocked paths turn into wasted capacity and a highly oversubscribed network 💀 There are alternatives, like the Transparent Interconnection of Lots of Links (TRILL), that tackle this problem [13].
the utilization problems get worse because VLANs fragment the network for broadcast isolation 🌈 VLANs are used for traffic management and also as the way to give security and performance isolation to services that belong to different tenants. The Layer 2 network gets carved into smaller subnets (usually one subnet per VLAN), with VLAN tags configured on every Layer 2 switch connected to the server racks that host a given tenant's services. The current VLAN limits theoretically allow for 4,000 such subnets. The broadcast domain is usually kept small because broadcast traffic has overhead. And the 4,000-subnet VLAN limit is no longer enough in a shared infrastructure serving multiple tenants 🙈
data center operators need to reach high utilization of server and network capacity, periodt 🔥 To get there, it should be possible to assign workloads that operate in a single Layer 2 network to any server in any rack. It should also be possible to migrate workloads to any server anywhere in the network while keeping their addresses. You can do this today by stretching VLANs, but when workloads migrate, the network has to be reconfigured, and that's usually error prone 😬 If you decouple the workload's location on the LAN from its network address, the admin configures the network once, not every single time a service moves. That decoupling lets any server become part of any server resource pool ✨
the key design objectives for next-gen data centers are:
- a) location-independent addressing
- b) the ability to scale the number of logical Layer 2 / Layer 3 networks, regardless of the underlying physical topology or the number of VLANs
- c) preserving Layer 2 semantics for services and letting them keep their addresses as they move within and across data centers
- d) providing broadcast isolation as workloads move around, without burdening the network control plane
this doc describes using the Generic Routing Encapsulation (GRE) header [3] [4] for network virtualization 🦋 Network virtualization decouples a virtual network from the physical network underneath by virtualizing network addresses. Combine that with a management and control plane for the virtual-to-physical mapping, and you get flexible VM placement and movement plus network isolation for a multi-tenant data center. it's giving clean separation 💅
network virtualization also lets customers bring their own address spaces into a multi-tenant data center. Meanwhile, data center admins can place customer VMs anywhere in the data center without reconfiguring their switches or routers, no matter what the customer address spaces are 👑
1.1. Terminology 🧚
please check RFCs 7364 [10] and 7365 [11] for more formal definitions of terminology. These are the terms this doc uses:
Customer Address (CA): This is the virtual IP address assigned and configured on the virtual Network Interface Controller (NIC) inside each VM. It's the only address VMs and the apps running inside them can see 👀
Network Virtualization Edge (NVE): This is an entity that does the network virtualization encapsulation and decapsulation 🫶
Provider Address (PA): This is the IP address used in the physical network. PAs are associated with VM CAs through the network virtualization mapping policy 🌸
Virtual Machine (VM): This is an instance of an OS running on top of the hypervisor over a physical machine or server. Multiple VMs can share the same physical server through the hypervisor, yet they're completely isolated from each other in terms of CPU usage, storage, and other OS resources 💜
Virtual Subnet Identifier (VSID): This is a 24-bit ID that uniquely identifies a virtual subnet or virtual Layer 2 broadcast domain ✨
2. Conventions Used in This Document 📜
the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [1].
in this document these words only carry that meaning when they're in ALL CAPS. Lowercase uses of these words don't carry the significance defined in RFC 2119 🙈
3. Network Virtualization Using GRE (NVGRE) ✨
this section describes Network Virtualization using GRE (NVGRE) 🦋 Network virtualization means creating virtual Layer 2 topologies on top of a physical Layer 3 network. Connectivity in the virtual topology comes from tunneling Ethernet frames in GRE over IP over the physical network 🌈
in NVGRE, every virtual Layer 2 network is tied to a 24-bit identifier called a Virtual Subnet Identifier (VSID) 🎀 The VSID is carried in an outer header, as defined in Section 3.2. That lets various devices in the network uniquely identify a tenant's virtual subnet. A 24-bit VSID supports up to 16 million virtual subnets in the same management domain, compared to only 4,000 with VLANs. Each VSID is a virtual Layer 2 broadcast domain, which can be used to identify a given tenant's virtual subnet. For multi-subnet virtual topologies, data center admins can configure routes so that virtual subnets of the same tenant can talk to each other 💖
GRE is a Proposed Standard from the IETF [3] [4], and it's a way to encapsulate an arbitrary protocol over IP 🧚 NVGRE uses the GRE header to carry VSID information in each packet. That VSID info can be used to build multi-tenant-aware tools for traffic analysis, traffic inspection, and monitoring 👑
the next sections cover the NVGRE packet format, describe what an NVGRE endpoint does, show typical traffic flow within and across data centers, and talk about address/policy management and deployment considerations 📜
3.1. NVGRE Endpoint 🫶
NVGRE endpoints are the ingress/egress points between the virtual and physical networks 🌸 They're the NVEs defined in the Network Virtualization over Layer 3 (NVO3) Framework document [11]. Any physical server or network device can be an NVGRE endpoint, and a common setup is to have the endpoint be part of a hypervisor 💅 The main job of the endpoint is to encapsulate and decapsulate Ethernet data frames to and from the GRE tunnel, keep Layer 2 semantics intact, and apply isolation policy scoped on VSID 🔥 The endpoint can optionally take part in routing and act as a gateway in the virtual topology ✨ To encapsulate an Ethernet frame, the endpoint needs to know the location information for the destination address in that frame. That info can be provisioned through a management plane, or obtained through a combo of control-plane distribution and data-plane learning approaches 🦋 This doc assumes the location information, including the VSID, is available to the NVGRE endpoint 💜
that's part 1 of 4 bestie, so more is coming 🎀 Next up is the NVGRE frame format and everything after it 🌈
3.2. NVGRE Frame Format ✨
okay bestie, NVGRE endpoints talk to each other using the GRE header from RFCs 2784 [3] and 2890 [4] 🧚 and NVGRE borrows the Key extension from RFC 2890 [4] to carry the VSID 💖. Here's the Layer 2 encapsulation in GRE, Figure 1, and it is giving detailed 😭🎀
Outer Ethernet Header:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Outer) Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|(Outer)Destination MAC Address | (Outer)Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Outer) Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype 0x0800 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer IPv4 Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| HL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol 0x2F | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Outer) Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Outer) Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
GRE Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| |1|0| Reserved0 | Ver | Protocol Type 0x6558 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Virtual Subnet ID (VSID) | FlowID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Inner Ethernet Header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Inner) Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|(Inner)Destination MAC Address | (Inner)Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| (Inner) Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype 0x0800 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Inner IPv4 Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| HL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Original IP Payload |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1: GRE Encapsulation Frame FormatNote: HL stands for Header Length. 🙈
The outer/delivery headers are the outer Ethernet header plus the outer IP header 🦋:
- The outer Ethernet header: the source Ethernet address in the outer frame is set to the MAC address associated with the NVGRE endpoint. The destination endpoint may or may not be on the same physical subnet. The destination Ethernet address is set to the MAC address of the next-hop IP address for the destination NVE. The outer VLAN tag information is optional and can be used for traffic management and broadcast scalability on the physical network. 🫶
- The outer IP header: both IPv4 and IPv6 can be used as the delivery protocol for GRE. The IPv4 header is shown for illustrative purposes. Henceforth, the IP address in the outer frame is referred to as the Provider Address (PA). There can be one or more PA associated with an NVGRE endpoint, with policy controlling the choice of which PA to use for a given Customer Address (CA) for a customer VM. 👑
Now the GRE header 🌸:
- The C (Checksum Present) and S (Sequence Number Present) bits in the GRE header MUST be zero. 🙅
- The K (Key Present) bit in the GRE header MUST be set to one. The 32-bit Key field in the GRE header is used to carry the Virtual Subnet ID (VSID) and the FlowID 🔥:
- Virtual Subnet ID (VSID): this is a 24-bit value that is used to identify the NVGRE-based Virtual Layer 2 Network. 💜
- FlowID: this is an 8-bit value that is used to provide per-flow entropy for flows in the same VSID. The FlowID MUST NOT be modified by transit devices. The encapsulating NVE SHOULD provide as much entropy as possible in the FlowID. If a FlowID is not generated, it MUST be set to all zeros. 🍓
- The Protocol Type field in the GRE header is set to
0x6558(Transparent Ethernet Bridging) [2]. ✨
And the inner headers, aka the headers of the GRE payload 🧁:
- The inner Ethernet frame comprises an inner Ethernet header, followed by an optional inner IP header, followed by the IP payload. The inner frame could be any Ethernet data frame, not just IP. Note that the inner Ethernet frame's Frame Check Sequence (FCS) is not encapsulated. 🌈
- For illustrative purposes, IPv4 headers are shown as the inner IP headers, but IPv6 headers may be used. Henceforth, the IP address contained in the inner frame is referred to as the Customer Address (CA). 💅
3.3. Inner Tag as Defined by IEEE 802.1Q 🎀
Okay, so the inner Ethernet header of NVGRE MUST NOT contain the tag as defined by IEEE 802.1Q [5]. 🙅♀️ The encapsulating NVE MUST remove any existing IEEE 802.1Q tag before encapsulation of the frame in NVGRE. A decapsulating NVE MUST drop the frame if the inner Ethernet frame contains an IEEE 802.1Q tag. 💀
No tags allowed inside the NVGRE bubble. Strip it going in, drop it coming out. 🫶
3.4. Reserved VSID 🦋
The VSID range from 0-0xFFF is reserved for future use. 🧚
The VSID 0xFFFFFF is reserved for vendor-specific NVE-to-NVE communication. The sender NVE SHOULD verify the receiver NVE's vendor before sending a packet using this VSID; however, such a verification mechanism is out of scope of this document. Implementations SHOULD choose a mechanism that meets their requirements. 👀💖
4. NVGRE Deployment Considerations 🌸
4.1. ECMP Support 💅
Equal-Cost Multipath (ECMP) may be used to provide load balancing. If ECMP is used, it is RECOMMENDED that the ECMP hash is calculated either using the outer IP frame fields and entire Key field (32 bits) or the inner IP and transport frame fields. ⭐
4.2. Broadcast and Multicast Traffic 📣
To support broadcast and multicast traffic inside a virtual subnet, one or more administratively scoped multicast addresses [8] [9] can be assigned for the VSID. All multicast or broadcast traffic originating from within a VSID is encapsulated and sent to the assigned multicast address. From an administrative standpoint, it is possible for network operators to configure a PA multicast address for each multicast address that is used inside a VSID; this facilitates optimal multicast handling. Depending on the hardware capabilities of the physical network devices and the physical network architecture, multiple virtual subnets may use the same physical IP multicast address. 🧠
Alternatively, based upon the configuration at the NVE, broadcast and multicast in the virtual subnet can be supported using N-way unicast. In N-way unicast, the sender NVE would send one encapsulated packet to every NVE in the virtual subnet. The sender NVE can encapsulate and send the packet as described in Section 4.3 ("Unicast Traffic"). This alleviates the need for multicast support in the physical network. 🍓
4.3. Unicast Traffic 🧚♀️
The NVGRE endpoint encapsulates a Layer 2 packet in GRE using the source PA associated with the endpoint, with the destination PA corresponding to the location of the destination endpoint. As outlined earlier, there can be one or more PAs associated with an endpoint, and policy will control which ones get used for communication. The encapsulated GRE packet is bridged and routed normally by the physical network to the destination PA. Bridging uses the outer Ethernet encapsulation for scope on the LAN. The only requirement is bidirectional IP connectivity from the underlying physical network. On the destination, the NVGRE endpoint decapsulates the GRE packet to recover the original Layer 2 frame. Traffic flows similarly on the reverse path. 🔁✨
4.4. IP Fragmentation 🧩
Section 5.1 of RFC 2003 [12] specifies mechanisms for handling fragmentation when encapsulating IP within IP. The subset of mechanisms NVGRE selects are intended to ensure that NVGRE-encapsulated frames are not fragmented after encapsulation en route to the destination NVGRE endpoint, and that traffic sources can leverage Path MTU discovery. 🛣️
A sender NVE MUST NOT fragment NVGRE packets. A receiver NVE MAY discard fragmented NVGRE packets. It is RECOMMENDED that the MTU of the physical network accommodates the larger frame size due to encapsulation. Path MTU or configuration via control plane can be used to meet this requirement. 📏💜
Don't fragment it, bestie. The sender NVE MUST NOT fragment, so the physical network needs room for the bigger frame. 🚫
4.5. Address/Policy Management and Routing 🗺️
Address acquisition is beyond the scope of this document and can be obtained statically, dynamically, or using stateless address autoconfiguration. CA and PA space can be either IPv4 or IPv6. In fact, the address families don't have to match; for example, a CA can be IPv4 while the PA is IPv6, and vice versa. 🌈✨
4.6. Cross-Subnet, Cross-Premise Communication 🌸
Okay so this framework is giving enterprises a seamless path to expand their VM hosting into public clouds 🚀. They can bring their entire IP subnet(s) and isolation policies over, which makes moving to or from the cloud way simpler 💖. It is possible to move portions of an IP subnet to the cloud, but that needs extra configuration on the enterprise network, and this document doesn't go into it 🙈. Enterprises can keep using existing models like site-to-site VPN to secure their traffic 🔐.
A VPN gateway sets up a secure site-to-site tunnel over the Internet, and all the enterprise services running in VMs in the cloud use that gateway to talk back to the enterprise ✨. To keep things simple, we use a VPN gateway configured as a VM (shown in Figure 2) to show cross-subnet, cross-premise communication 🎀.
+-----------------------+ +-----------------------+
| Server 1 | | Server 2 |
| +--------+ +--------+ | | +-------------------+ |
| | VM1 | | VM2 | | | | VPN Gateway | |
| | IP=CA1 | | IP=CA2 | | | | Internal External| |
| | | | | | | | IP=CAg IP=GAdc | |
| +--------+ +--------+ | | +-------------------+ |
| Hypervisor | | | Hypervisor| ^ |
+-----------------------+ +-------------------:---+
| IP=PA1 | IP=PA4 | :
| | | :
| +-------------------------+ | : VPN
+-----| Layer 3 Network |------+ : Tunnel
+-------------------------+ :
| :
+-----------------------------------------------:--+
| : |
| Internet : |
| : |
+-----------------------------------------------:--+
| v
| +-------------------+
| | VPN Gateway |
|---| |
IP=GAcorp| External IP=GAcorp|
+-------------------+
|
+-----------------------+
| Corp Layer 3 Network |
| (In CA Space) |
+-----------------------+
|
+---------------------------+
| Server X |
| +----------+ +----------+ |
| | Corp VMe1| | Corp VMe2| |
| | IP=CAe1 | | IP=CAe2 | |
| +----------+ +----------+ |
| Hypervisor |
+---------------------------+Figure 2: Cross-Subnet, Cross-Premise Communication
The packet flow is pretty similar to unicast traffic between VMs, but the big difference here is that the packet has to go to a VPN gateway before it gets forwarded to its destination 🦋. As part of the routing config in the CA space, a per-tenant VPN gateway gets provisioned for talking back to the enterprise 💅. The example shows an outbound connection from VM1 inside the data center to VMe1 inside the enterprise network 👑. When the outbound packet from CA1 to CAe1 hits the hypervisor on Server 1, the NVE on Server 1 can do the equivalent of a route lookup on it 🔍. The cross-premise packet matches the default gateway rule, since CAe1 isn't part of the tenant virtual network in the data center 🙃. The virtualization policy says to encapsulate the packet and send it to the PA of the tenant VPN gateway (PA4), which runs as a VM on Server 2 🧚. The packet gets decapsulated on Server 2 and delivered to the VM gateway 📦. The gateway then validates the packet and sends it down the site-to-site VPN tunnel back to the enterprise network 🔐. Since this communication is external to the data center, the PA address for the VPN tunnel is globally routable 🌍. The outer header of this packet is sourced from GAdc and destined to GAcorp 📮. It gets routed through the Internet to the enterprise VPN gateway, which is the other end of the site-to-site tunnel 🛣️. At that point, the VPN gateway decapsulates the packet and sends it inside the enterprise, where CAe1 is routable on the network 🏢. The reverse path works similarly once the packet reaches the enterprise VPN gateway 🔁.
4.7. Internet Connectivity 🌐
To get connected to the Internet, you need an Internet gateway that bridges the virtualized CA space to the public Internet address space 🌉. That gateway has to translate between the virtualized world and the Internet 🔄. For example, the NVGRE endpoint can be part of a load balancer or a NAT that replaces the VPN Gateway on Server 2 shown in Figure 2 ⚖️.
4.8. Management and Control Planes 🎛️
There are several protocols that can manage and distribute policy, but that's outside the scope of this document 🤷♀️. Implementations SHOULD choose a mechanism that meets their scale requirements. 💜
4.9. NVGRE-Aware Devices 🧠
One typical deployment looks like virtualized servers spread across multiple racks, connected by one or more layers of Layer 2 switches, which may in turn connect to a Layer 3 routing domain 🏗️. Routing in the physical infrastructure works without any modification with NVGRE, but devices that do specialized processing in the network need to parse GRE to get access to tenant-specific information 👀. Devices that understand and parse the VSID can provide rich multi-tenant-aware services inside the data center 💎. As we outlined earlier, it's imperative to exploit multiple paths inside the network using techniques like ECMP 🛤️. The Key field (a 32-bit field, including both the VSID and the optional FlowID) can give switches extra entropy to exploit path diversity inside the network ✨. A diverse ecosystem is expected to emerge as more and more devices become multi-tenant aware 🌈. In the meantime, without any hardware upgrades, you can exploit path diversity with GRE by associating multiple PAs with NVGRE endpoints, with policy controlling which PA gets used 🦄.
Communication is expected to span multiple data centers and also cross the virtual/physical boundary 🌍. Typical scenarios that need virtual-to-physical communication include access to storage and databases 🗄️. Scenarios demanding lossless Ethernet functionality may not be amenable to NVGRE, since the traffic is carried over an IP network 🚫. NVGRE endpoints mediate between the network-virtualized and non-network-virtualized environments 🤝. That functionality can be built into Top-of-Rack switches, storage appliances, load balancers, routers, etc., or made as a stand-alone appliance 📦.
It's imperative to consider the impact of any solution on host performance 🏎️. Today's server operating systems use sophisticated acceleration techniques like checksum offload, Large Send Offload (LSO), Receive Segment Coalescing (RSC), Receive Side Scaling (RSS), Virtual Machine Queue (VMQ), etc. 💅 These technologies should become NVGRE aware. IPsec Security Associations (SAs) can be offloaded to the NIC so that computationally expensive cryptographic operations run at line rate in the NIC hardware 🔥. These SAs are based on the IP addresses of the endpoints 📍. As each packet on the wire gets translated, the NVGRE endpoint SHOULD intercept the offload requests and do the appropriate address translation. 🙌 This keeps IPsec usable with network virtualization while still taking advantage of hardware offload for better performance 💖.
4.10. Network Scalability with NVGRE 📈
One of the key benefits of NVGRE is the IP address scalability, and in turn the MAC address table scalability, that you can get 😍. An NVGRE endpoint can use one PA to represent multiple CAs, which lowers the burden on MAC address table sizes at the Top-of-Rack switches 🧁. This is a big deal in server virtualization, which has ramped up the demands on the network infrastructure 📊. By embedding an NVGRE endpoint in a hypervisor, you can scale way up 🚀. This framework lets location information be preconfigured inside an NVGRE endpoint, so broadcast ARP traffic can be proxied locally 🪄. That approach scales to large virtual subnets 🐘. These virtual subnets can be spread across multiple Layer 3 physical subnets ✨. It lets workloads move around without putting a huge burden on the network control plane 🏃♀️. By eliminating most broadcast traffic and converting the rest to multicast, routers and switches can run more optimally by building efficient multicast trees 🌳. By using server and network capacity efficiently, you can drive down the cost of building and managing data centers 💸.
5. Security Considerations 🔐
This proposal extends the Layer 2 subnet across the data center and increases the scope for spoofing attacks. 😱 Mitigations for such attacks are possible with authentication/encryption using IPsec or any other IP-based mechanism 🛡️. The control plane for policy distribution is expected to be secured using any of the existing security protocols 🔒. Further, management traffic can be isolated in a separate subnet/VLAN 🙌.
The checksum in the GRE header is not supported. 💀 The mitigation is to deploy an NVGRE-based solution in a network that provides error detection along the NVGRE packet path, for example using Ethernet Cyclic Redundancy Check (CRC), IPsec, or any other error detection mechanism 🧯.
6. Normative References ✨
okay bestie, this is the reading list 📚💅 every doc it's citing has to be here, so no skipping, no cap. the titles and details are copied exactly, since you can't cite a vibe 🙈
[1] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.
[2] IANA, "IEEE 802 Numbers",
<http://www.iana.org/assignments/ieee-802-numbers>.
[3] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina,
"Generic Routing Encapsulation (GRE)", RFC 2784,
DOI 10.17487/RFC2784, March 2000,
<http://www.rfc-editor.org/info/rfc2784>.
[4] Dommety, G., "Key and Sequence Number Extensions to GRE",
RFC 2890, DOI 10.17487/RFC2890, September 2000,
<http://www.rfc-editor.org/info/rfc2890>.
[5] IEEE, "IEEE Standard for Local and metropolitan area
networks--Media Access Control (MAC) Bridges and Virtual Bridged
Local Area Networks", IEEE Std 802.1Q.
[6] Greenberg, A., et al., "VL2: A Scalable and Flexible Data Center
Network", Communications of the ACM,
DOI 10.1145/1897852.1897877, 2011.
[7] Greenberg, A., et al., "The Cost of a Cloud: Research Problems
in Data Center Networks", ACM SIGCOMM Computer Communication
Review, DOI 10.1145/1496091.1496103, 2009.
[8] Hinden, R. and S. Deering, "IP Version 6 Addressing
Architecture", RFC 4291, DOI 10.17487/RFC4291, February 2006,
<http://www.rfc-editor.org/info/rfc4291>.
[9] Meyer, D., "Administratively Scoped IP Multicast", BCP 23,
RFC 2365, DOI 10.17487/RFC2365, July 1998,
<http://www.rfc-editor.org/info/rfc2365>.
[10] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L., Kreeger,
L., and M. Napierala, "Problem Statement: Overlays for Network
Virtualization", RFC 7364, DOI 10.17487/RFC7364, October 2014,
<http://www.rfc-editor.org/info/rfc7364>.
[11] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. Rekhter,
"Framework for Data Center (DC) Network Virtualization",
RFC 7365, DOI 10.17487/RFC7365, October 2014,
<http://www.rfc-editor.org/info/rfc7365>.
[12] Perkins, C., "IP Encapsulation within IP", RFC 2003,
DOI 10.17487/RFC2003, October 1996,
<http://www.rfc-editor.org/info/rfc2003>.
[13] Touch, J. and R. Perlman, "Transparent Interconnection of Lots
of Links (TRILL): Problem and Applicability Statement",
RFC 5556, DOI 10.17487/RFC5556, May 2009,
<http://www.rfc-editor.org/info/rfc5556>.Contributors 🎀
these legends helped make the doc, and they're all listed here with their addresses 🫶
Murari Sridharan
Microsoft Corporation
1 Microsoft Way
Redmond, WA 98052
United States
Email: muraris@microsoft.com
Albert Greenberg
Microsoft Corporation
1 Microsoft Way
Redmond, WA 98052
United States
Email: albert@microsoft.com
Narasimhan Venkataramiah
Microsoft Corporation
1 Microsoft Way
Redmond, WA 98052
United States
Email: navenkat@microsoft.com
Kenneth Duda
Arista Networks, Inc.
5470 Great America Pkwy
Santa Clara, CA 95054
United States
Email: kduda@aristanetworks.com
Ilango Ganga
Intel Corporation
2200 Mission College Blvd.
M/S: SC12-325
Santa Clara, CA 95054
United States
Email: ilango.s.ganga@intel.com
Geng Lin
Google
1600 Amphitheatre Parkway
Mountain View, CA 94043
United States
Email: genglin@google.com
Mark Pearson
Hewlett-Packard Co.
8000 Foothills Blvd.
Roseville, CA 95747
United States
Email: mark.pearson@hp.com
Patricia Thaler
Broadcom Corporation
3151 Zanker Road
San Jose, CA 95134
United States
Email: pthaler@broadcom.com
Chait Tumuluri
Emulex Corporation
3333 Susan Street
Costa Mesa, CA 92626
United States
Email: chait@emulex.comAuthors' Addresses 💖
and last but definitely not least, the editors who held this whole thing together 👑✨
Pankaj Garg (editor)
Microsoft Corporation
1 Microsoft Way
Redmond, WA 98052
United States
Email: pankajg@microsoft.com
Yu-Shun Wang (editor)
Microsoft Corporation
1 Microsoft Way
Redmond, WA 98052
United States
Email: yushwang@microsoft.comthat's the whole RFC, bestie 🌈🦋 we made it through all the parts 🥹💜
📦 where this page comes from
- ietf: rfc7637 2015-09
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms