RFC 7348
Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks
RFC 7348: VXLAN, Part 1 of 5 🦋
Okay bestie, let's get into it. ✨ This is RFC 7348, and it's an Independent Submission, so it came through the independent RFC stream rather than the IETF standards track 💅. The title is "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks." It was written by M. Mahalingam (Storvisor), D. Dutt (Cumulus Networks), K. Duda (Arista), P. Agarwal (Broadcom), L. Kreeger (Cisco), T. Sridhar (VMware), M. Bursell (Intel), and C. Wright (Red Hat). The category is Informational, the ISSN is 2070-1721, and it's dated August 2014. The header doesn't list anything it obsoletes or updates. 🌸
Abstract 🫶
This doc describes VXLAN, which exists to meet the need for overlay networks inside virtualized data centers that hold multiple tenants 🏢. The scheme and its related protocols can be used in networks run by cloud service providers and in enterprise data centers. This memo documents the VXLAN protocol as it's actually deployed, so the wider Internet community can learn from it 🌈.
Status of This Memo 📜
This document is not an Internet Standards Track specification. It's published for informational purposes, so no need to panic 😌.
This is a contribution to the RFC Series, independent of any other RFC stream. The RFC Editor chose to publish it at their discretion, and they make no statement about its value for implementation or deployment 🙈. Documents the RFC Editor approves for publication are not candidates for any level of Internet Standard. See Section 2 of RFC 5741 for the details.
Information about this document's current status, any errata, and how to give feedback can be found at http://www.rfc-editor.org/info/rfc7348 🔗.
Copyright Notice ©️
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication. Please read these documents carefully, because they spell out your rights and restrictions with respect to this document 📖.
Table of Contents 🗂️
1. Introduction ....................................................3
1.1. Acronyms and Definitions ...................................4
2. Conventions Used in This Document ...............................4
3. VXLAN Problem Statement .........................................5
3.1. Limitations Imposed by Spanning Tree and VLAN Ranges .......5
3.2. Multi-tenant Environments ..................................5
3.3. Inadequate Table Sizes at ToR Switch .......................6
4. VXLAN ...........................................................6
4.1. Unicast VM-to-VM Communication .............................7
4.2. Broadcast Communication and Mapping to Multicast ...........8
4.3. Physical Infrastructure Requirements .......................9
5. VXLAN Frame Format .............................................10
6. VXLAN Deployment Scenarios .....................................14
6.1. Inner VLAN Tag Handling ...................................18
7. Security Considerations ........................................18
8. IANA Considerations ............................................19
9. References .....................................................19
9.1. Normative References ......................................19
9.2. Informative References ....................................20
10. Acknowledgments ...............................................211. Introduction 🌟
Server virtualization is putting a lot of pressure on the physical network. One physical server can now host several Virtual Machines (VMs), and each VM has its own Media Access Control (MAC) address 🖥️. That means the switched Ethernet network needs much bigger MAC address tables, since it might have to handle hundreds of thousands of VMs attaching to it and talking to each other 😵.
When VMs in a data center are grouped by their Virtual LAN (VLAN), you might need thousands of VLANs to split traffic by whichever group a VM may belong to. The current VLAN limit of 4094 just isn't enough for that 💀.
Data centers often have to host multiple tenants, and each one wants its own isolated network domain 🏠. Dedicated infrastructure for every tenant isn't economical, so network admins choose to isolate tenants over a shared network instead. The catch is that each tenant might assign MAC addresses and VLAN IDs on its own, which can lead to duplicates on the physical network 👯.
Virtualized environments on a Layer 2 physical infrastructure have an important requirement: the Layer 2 network has to scale across the whole data center, or even across several data centers, so compute, network, and storage resources can be allocated efficiently 🚀. With traditional approaches like the Spanning Tree Protocol (STP) keeping the topology loop-free, a lot of links can end up disabled 🔌.
The last scenario is when the network operator prefers to use IP to interconnect the physical infrastructure. For example, that gets multipath scalability through Equal-Cost Multipath (ECMP) and avoids disabled links. Even then, there's still a need to keep the Layer 2 model for inter-VM communication 🧠.
All of these scenarios add up to a need for an overlay network. The overlay carries MAC traffic from individual VMs in an encapsulated format over a logical "tunnel" 🚇.
This document details a framework called "Virtual eXtensible Local Area Network (VXLAN)." It provides an encapsulation scheme that addresses all of the requirements above. The memo documents the VXLAN protocol as deployed, for the benefit of the Internet community 💜.
1.1. Acronyms and Definitions 📚
ACL Access Control List
ECMP Equal-Cost Multipath
IGMP Internet Group Management Protocol
IHL Internet Header Length
MTU Maximum Transmission Unit
PIM Protocol Independent Multicast
SPB Shortest Path Bridging
STP Spanning Tree Protocol
ToR Top of Rack
TRILL Transparent Interconnection of Lots of Links
VLAN Virtual Local Area Network
VM Virtual Machine
VNI VXLAN Network Identifier (or VXLAN Segment ID)
VTEP VXLAN Tunnel End Point. An entity that originates and/or
terminates VXLAN tunnels
VXLAN Virtual eXtensible Local Area Network
VXLAN Segment
VXLAN Layer 2 overlay network over which VMs communicate
VXLAN Gateway
an entity that forwards traffic between VXLANs2. Conventions Used in This Document 🎀
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 🤓
3. VXLAN Problem Statement 🧩
This section gets into the details of the areas VXLAN is meant to fix. The focus is on the networking infrastructure inside the data center and the issues that come with it 🏗️.
3.1. Limitations Imposed by Spanning Tree and VLAN Ranges 🌳
Current Layer 2 networks use the IEEE 802.1D Spanning Tree Protocol (STP) [802.1D] to avoid loops caused by duplicate paths. STP blocks links so frames don't get replicated and looped 🔁. Some data center operators see this as a general problem with Layer 2 networks, because with STP they're effectively paying for more ports and links than they can really use 💸. Resiliency through multipathing isn't available in the STP model either 😬. Newer efforts, like TRILL [RFC6325] and SPB [802.1aq], have been proposed to help with multipathing and to get past some of the STP problems 🛣️. You can also dodge some STP limits by putting the servers in a rack on the same Layer 3 network, with switching at Layer 3 both inside the rack and between racks. The catch is that this clashes with a Layer 2 model for inter-VM communication 🙃.
A key trait of Layer 2 data center networks is using Virtual LANs (VLANs) for broadcast isolation. A 12-bit VLAN ID in the Ethernet data frames divides the bigger Layer 2 network into multiple broadcast domains 📡. That's worked fine for plenty of data centers that need fewer than 4094 VLANs. But virtualization keeps pushing on that upper limit 📈. On top of that, STP means several data centers cap how many VLANs they can use 🚧. Multi-tenant environments make the need for bigger VLAN limits even more urgent, as discussed in Section 3.3 ⭐.
3.2. Multi-tenant Environments 👑
Cloud computing is all about on-demand, elastic provisioning of resources for multi-tenant environments ☁️. The most common example is the public cloud, where a cloud service provider offers those elastic services to multiple customers or tenants on the same physical infrastructure 🛍️.
Tenant isolation of network traffic can be done with Layer 2 or Layer 3 networks. On Layer 2, VLANs are often used to segregate traffic, so a tenant might be identified by its own VLAN, for example 🧁. A cloud provider can serve a huge number of tenants, so the 4094 VLAN limit is often inadequate 😩. And there's often a need for multiple VLANs per tenant, which makes the problem worse 💥.
A related use case is cross-pod expansion. A pod usually consists of one or more racks of servers with their network and storage connectivity attached 🧱. A tenant may start out on one pod and then, as things grow, need servers or VMs on other pods. That's especially true when tenants on those other pods aren't using all of their resources 🌱. This use case calls for a "stretched" Layer 2 environment that connects the individual servers and VMs 🧵.
Layer 3 networks don't fully solve multi-tenancy either 🙅♀️. Two tenants might use the same set of Layer 3 addresses inside their networks, so the cloud provider has to provide isolation some other way. Also, requiring every tenant to use IP shuts out customers who rely on direct Layer 2 or non-IP Layer 3 protocols for inter-VM communication 🚫.
3.3. Inadequate Table Sizes at ToR Switch
Okay so virtualized setups are extra demanding on the MAC address tables of Top-of-Rack (ToR) switches, the ones that plug into the servers 🧚. Before, each server link only needed one MAC address. Now the ToR has to learn the MAC address of every single VM, and that can be hundreds per server 😭. This is necessary because all the traffic between the VMs and the rest of the physical network has to cross that server-to-switch link. A typical ToR switch can connect to 24 or 48 servers, depending on how many server-facing ports it has. A data center is usually several racks, so every ToR switch would need an address table covering the communicating VMs across all those physical servers. Literally way more table capacity than in non-virtualized environments 💀.
If the table overflows, the switch may stop learning new addresses until idle entries age out, and that leads to a lot of flooding of unknown destination frames 🌊.
4. VXLAN ✨
VXLAN (Virtual eXtensible Local Area Network) is the answer to the Layer 2 and Layer 3 data center network needs we just talked about, when you've got VMs in a multi-tenant environment 🎀. It runs on top of the networking infrastructure you already have, and it gives you a way to "stretch" a Layer 2 network. TL;DR: VXLAN is a Layer 2 overlay on a Layer 3 network 🦋. Each overlay is called a VXLAN segment, and only VMs in the same VXLAN segment can talk to each other. Every segment is identified by a 24-bit segment ID called the "VXLAN Network Identifier (VNI)", which means up to 16 M VXLAN segments can coexist within the same administrative domain 👑.
The VNI tells you the scope of the inner MAC frame that the VM originated. So you can have overlapping MAC addresses across segments and still never get traffic "crossing over", because the VNI keeps everything isolated 💜. The VNI lives in an outer header that wraps the inner MAC frame the VM sent. In the following sections, "VXLAN segment" and "VXLAN overlay network" mean the same thing.
Because of all this encapsulation, VXLAN could also be called a tunneling scheme for overlaying Layer 2 networks on top of Layer 3 networks 🌈. The tunnels are stateless, so every frame gets encapsulated according to a set of rules. The tunnel endpoint (VXLAN Tunnel End Point, or VTEP) that we'll be talking about lives inside the hypervisor on the server that hosts the VM. So the VNI and the VXLAN tunnel / outer header encapsulation are known only to the VTEP, and the VM never sees any of it (see Figure 1) 🙈. VTEPs can also live on a physical switch or a physical server, and they can be implemented in software or hardware. One use case where the VTEP is a physical switch is covered in Section 6 on VXLAN deployment scenarios.
The next sections walk through typical traffic flow scenarios in a VXLAN environment using one type of control scheme, which is data plane learning. Here, the association between a VM's MAC and its VTEP's IP address is discovered through source-address learning. Multicast is used to carry unknown destination, broadcast, and multicast frames 📡.
There are other ways to distribute the mapping from VTEP IP to VM MAC besides a learning-based control plane. You could use a central authority or directory that the individual VTEPs look things up in, or the central authority could push the mapping information out to the VTEPs, and so on. These get described as push and pull models, respectively. This document focuses on data plane learning as the control plane for VXLAN 🧠.
4.1. Unicast VM-to-VM Communication
Picture a VM inside a VXLAN overlay network. This VM has no idea VXLAN exists 🥹. To talk to a VM on a different host, it just sends a MAC frame addressed to the target, like normal. The VTEP on the physical host looks up the VNI that this VM is associated with. Then it checks whether the destination MAC is on the same segment and whether there's a mapping from the destination MAC address to the remote VTEP. If both are true, an outer header made of an outer MAC, an outer IP header, and a VXLAN header (see Figure 1 in Section 5 for the frame format) gets prepended to the original MAC frame. The encapsulated packet is then forwarded toward the remote VTEP 🚀.
When the packet arrives, the remote VTEP checks that the VNI is valid and that there's a VM on that VNI using a MAC address that matches the inner destination MAC address. If there is, the packet gets stripped of its encapsulating headers and passed along to the destination VM. The destination VM never knows about the VNI, and it never knows the frame traveled with VXLAN encapsulation. Main character energy, zero awareness 💅.
Here's the bonus part. Besides forwarding the packet to the destination VM, the remote VTEP learns the mapping from the inner source MAC to the outer source IP address, and it stores that mapping in a table. That way, when the destination VM sends a response, there's no need to flood it as an "unknown destination" frame 🎉.
Figuring out the destination VM's MAC address before the source VM transmits works the same as in non-VXLAN environments, except as described in Section 4.2. Broadcast frames are still used, but they get encapsulated inside a multicast packet, as detailed in Section 4.2 🫶.
4.2. Broadcast Communication and Mapping to Multicast
Let's say the VM on the source host wants to talk to the destination VM using IP. If they're both on the same subnet, the source VM sends out an Address Resolution Protocol (ARP) broadcast frame. In a non-VXLAN environment, that frame would go out as a MAC broadcast across every switch carrying that VLAN 📣.
With VXLAN, a header that includes the VXLAN VNI gets inserted at the front of the packet, along with the IP header and UDP header. The difference is that this broadcast packet gets sent to the IP multicast group that the VXLAN overlay network is realized on 🌐.
To make this work, we need a mapping between the VXLAN VNI and the IP multicast group it will use. That mapping is handled at the management layer and handed to each VTEP over a management channel 🔧. Using this mapping, the VTEP can send IGMP membership reports to the upstream switch or router to join and leave the VXLAN-related IP multicast groups as needed. This lets leaf nodes get pruned for specific multicast traffic addresses, based on whether a member is available on this host for that particular multicast address (see [RFC4541]). On top of that, multicast routing protocols like Protocol Independent Multicast - Sparse Mode (PIM-SM, see [RFC4601]) build efficient multicast trees within the Layer 3 network 🌳.
The VTEP will use (*,G) joins 💫. This is needed because the set of VXLAN tunnel sources is unknown and can change often, since VMs come up and go down across different hosts. Side note: since each VTEP can act as both the source and the destination for multicast packets, a protocol like bidirectional PIM (BIDIR-PIM, see [RFC5015]) would be more efficient 🤓.
The destination VM sends a standard ARP response using IP unicast. This frame gets encapsulated back to the VTEP connected to the originating VM, using IP unicast VXLAN encapsulation. That works because the mapping from the ARP response's destination MAC to the VXLAN tunnel endpoint IP was already learned during the ARP request 💖.
Multicast frames and "unknown MAC destination" frames also get sent using the multicast tree, same as broadcast frames 🍓.
4.3. Physical Infrastructure Requirements
When IP multicast is used inside the network infrastructure, a multicast routing protocol like PIM-SM can run on the individual Layer 3 IP routers and switches. It's used to build efficient multicast forwarding trees, so multicast frames only get sent to the hosts that asked to receive them ✨.
There's also no requirement that the network connecting the source VM and the destination VM be a Layer 3 network. VXLAN can work over Layer 2 networks too. Either way, efficient multicast replication within a Layer 2 network can be achieved using IGMP snooping 🦋.
VTEPs MUST NOT fragment VXLAN packets. Intermediate routers may fragment encapsulated VXLAN packets because the frame is bigger now. The destination VTEP MAY silently discard such VXLAN fragments. To make sure end-to-end delivery works without fragmentation, it is RECOMMENDED that the MTUs (Maximum Transmission Units) across the physical network infrastructure be set to a value that fits the larger frame size from the encapsulation. Other techniques, like Path MTU discovery (see [RFC1191] and [RFC1981]), MAY also be used to address this requirement 🛣️.
ngl this is the part to memorize 🫠
5. VXLAN Frame Format 🎀
The VXLAN frame format is shown below. If you parse it from the bottom of the frame up, above the outer Frame Check Sequence (FCS) you'll find an inner MAC frame with its own Ethernet header. That header has the source and destination MAC addresses plus the Ethernet type, and optionally a VLAN. See Section 6 for more details on how inner VLAN tags get handled 🧁.
The inner MAC frame gets encapsulated with the following four headers (starting from the innermost one):
- VXLAN Header: This is an 8-byte field that has:
- Flags (8 bits): The I flag MUST be set to 1 for a valid VXLAN Network ID (VNI). The other 7 bits (designated "R") are reserved fields and MUST be set to zero on transmission and ignored on receipt.
- VXLAN Segment ID / VXLAN Network Identifier (VNI): A 24-bit value that designates the individual VXLAN overlay network the communicating VMs are on. VMs in different VXLAN overlay networks cannot communicate with each other 🫧.
- Reserved fields (24 bits and 8 bits): These MUST be set to zero on transmission and ignored on receipt.
- Outer UDP Header: This is the outer UDP header, with a source port provided by the VTEP and a destination port that's a well-known UDP port.
- Destination Port: IANA has assigned the value 4789 for the VXLAN UDP port, and this value SHOULD be used by default as the destination UDP port. Some early VXLAN implementations used other values for the destination port. To enable interoperability with those implementations, the destination port SHOULD be configurable ⚙️.
- Source Port: It is recommended that the UDP source port number be calculated using a hash of fields from the inner packet. One example is a hash of the inner Ethernet frame's headers. This adds a level of entropy for ECMP / load balancing of the VM-to-VM traffic across the VXLAN overlay. When the UDP source port number is calculated this way, it is RECOMMENDED that the value be in the dynamic/private port range 49152-65535 [RFC6335] 🌸.
- UDP Checksum: bestie, this one is giving "keep it at zero" 🦋 It SHOULD be transmitted as zero. When a packet is received with a UDP checksum of zero, it MUST be accepted for decapsulation. Optionally, if the encapsulating end point includes a non-zero UDP checksum, it MUST be correctly calculated across the entire packet including the IP header, UDP header, VXLAN header, and encapsulated MAC frame. When a decapsulating end point receives a packet with a non-zero checksum, it MAY choose to verify the checksum value. If it chooses to perform such verification, and the verification fails, the packet MUST be dropped. If the decapsulating destination chooses not to perform the verification, or performs it successfully, the packet MUST be accepted for decapsulation. 💅
Outer IP Header: this is the outer IP header, and its source IP address is the IP address of the VTEP where the communicating VM is hanging out (that VM is identified by the inner source MAC address) 🌸 The destination IP address can be a unicast or multicast IP address (see Sections 4.1 and 4.2). When it's unicast, it's the IP address of the VTEP that connects the communicating VM, identified by the inner destination MAC address. For multicast destination IP addresses, check out the scenarios in Section 4.2 🧚
Outer Ethernet Header (example): Figure 1 shows an inner Ethernet frame wrapped up in an outer Ethernet + IP + UDP + VXLAN header, and it's giving layers 🎀 The outer destination MAC address in this frame may be the address of the target VTEP or of an intermediate Layer 3 router. The outer VLAN tag is optional. If it's present, it may be used for delineating VXLAN traffic on the LAN ✨
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
Outer Ethernet Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Destination MAC Address | Outer Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|OptnlEthtype = C-Tag 802.1Q | Outer.VLAN Tag Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype = 0x0800 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer IPv4 Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live |Protocl=17(UDP)| Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Source IPv4 Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Destination IPv4 Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Dest Port = VXLAN Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
VXLAN Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| VXLAN Network Identifier (VNI) | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Inner Ethernet Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Inner Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Inner Destination MAC Address | Inner Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Inner Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|OptnlEthtype = C-Tag 802.1Q | Inner.VLAN Tag Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Payload:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype of Original Payload | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| Original Ethernet Payload |
| |
|(Note that the original Ethernet Frame's FCS is not included) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Frame Check Sequence:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| New FCS (Frame Check Sequence) for Outer Ethernet Frame |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1: VXLAN Frame Format with IPv4 Outer HeaderThe frame format above shows tunneling of Ethernet frames using IPv4 for transport. Use of VXLAN with IPv6 transport is detailed below 🌈
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
Outer Ethernet Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Destination MAC Address | Outer Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Outer Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|OptnlEthtype = C-Tag 802.1Q | Outer.VLAN Tag Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype = 0x86DD |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer IPv6 Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Traffic Class | Flow Label |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Payload Length | NxtHdr=17(UDP)| Hop Limit |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ Outer Source IPv6 Address +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ Outer Destination IPv6 Address +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer UDP Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Dest Port = VXLAN Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| UDP Length | UDP Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
VXLAN Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| VXLAN Network Identifier (VNI) | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Inner Ethernet Header:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Inner Destination MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Inner Destination MAC Address | Inner Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Inner Source MAC Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|OptnlEthtype = C-Tag 802.1Q | Inner.VLAN Tag Information |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Payload:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Ethertype of Original Payload | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| Original Ethernet Payload |
| |
|(Note that the original Ethernet Frame's FCS is not included) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Frame Check Sequence:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| New FCS (Frame Check Sequence) for Outer Ethernet Frame |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: VXLAN Frame Format with IPv6 Outer Header6. VXLAN Deployment Scenarios ✨
okay so VXLAN is mostly used in data centers 🏢, running on virtualized hosts that can be spread across a bunch of racks 🗄️. Those racks might sit in different Layer 3 networks, or they could all live in one Layer 2 network. Either way, the VXLAN segments (a.k.a. overlay networks) get layered on top of whichever Layer 2 or Layer 3 network is there 💅.
Check out Figure 3 🦋. It shows two virtualized servers hanging off a Layer 3 infrastructure. The servers could be in the same rack, different racks, or even different data centers, as long as they're in the same administrative domain 🌈. There are four VXLAN overlay networks here, identified by VNIs 22, 34, 74, and 98 💜. Let's look at VM1-1 on Server 1 and VM2-4 on Server 2. They're both on the VXLAN overlay network with VNI 22, so they're besties in that network 🫶. The VMs have no idea the overlay or the transport method even exists, because the encapsulation and decapsulation happen invisibly at the VTEPs on Servers 1 and 2 🧚. The rest of the squad is just as iconic: VM1-2 on Server 1 and VM2-1 on Server 2 both sit on VNI 34 🌸. VM1-3 on Server 1 and VM2-2 on Server 2 share VNI 74 🍓. And VM1-4 on Server 1 and VM2-3 on Server 2 are on VNI 98 ⭐.
+------------+-------------+
| Server 1 |
| +----+----+ +----+----+ |
| |VM1-1 | |VM1-2 | |
| |VNI 22 | |VNI 34 | |
| | | | | |
| +---------+ +---------+ |
| |
| +----+----+ +----+----+ |
| |VM1-3 | |VM1-4 | |
| |VNI 74 | |VNI 98 | |
| | | | | |
| +---------+ +---------+ |
| Hypervisor VTEP (IP1) |
+--------------------------+
|
|
|
| +-------------+
| | Layer 3 |
|---| Network |
| |
+-------------+
|
|
+-----------+
|
|
+------------+-------------+
| Server 2 |
| +----+----+ +----+----+ |
| |VM2-1 | |VM2-2 | |
| |VNI 34 | |VNI 74 | |
| | | | | |
| +---------+ +---------+ |
| |
| +----+----+ +----+----+ |
| |VM2-3 | |VM2-4 | |
| |VNI 98 | |VNI 22 | |
| | | | | |
| +---------+ +---------+ |
| Hypervisor VTEP (IP2) |
+--------------------------+Figure 3: VXLAN Deployment - VTEPs across a Layer 3 Network
One deployment option is when the tunnel endpoint is a physical server that actually speaks VXLAN 🖥️. Another option is when nodes on a VXLAN overlay need to talk to nodes on legacy networks, which might be VLAN based 🕰️. Those nodes could be physical machines or VMs 💻. To make that chat happen, the network can include VXLAN gateways 🌉. Figure 4 shows a switch doing this job as a VXLAN gateway, forwarding traffic between the VXLAN world and the non-VXLAN world 🔥.
Let's go through Figure 4 🎀. When frames come in on the VXLAN-connected interface, the gateway strips off the VXLAN header and forwards the frame out a physical port, based on the destination MAC address of the inner Ethernet frame. Decapsulated frames carrying an inner VLAN ID SHOULD be discarded, unless you've explicitly configured them to be passed on to the non-VXLAN interface 🙈. Going the other way, frames arriving on the non-VXLAN interfaces get mapped to a specific VXLAN overlay network based on their VLAN ID 🌈. Unless you've explicitly configured the VLAN ID to be kept in the encapsulated VXLAN frame, it gets removed before the frame is encapsulated for VXLAN 💜.
These gateways that handle the VXLAN tunnel termination can be ToR/access switches, or they can sit higher up in the data center topology, like core switches or even WAN edge devices 🏰. The WAN edge case could involve a Provider Edge (PE) router that terminates VXLAN tunnels in a hybrid cloud environment ☁️. In every one of these cases, the gateway functionality can be implemented in software or in hardware 💅.
+---+-----+---+ +---+-----+---+
| Server 1 | | Non-VXLAN |
(VXLAN enabled)<-----+ +---->| server |
+-------------+ | | +-------------+
| |
+---+-----+---+ | | +---+-----+---+
|Server 2 | | | | Non-VXLAN |
(VXLAN enabled)<-----+ +---+-----+---+ +---->| server |
+-------------+ | |Switch acting| | +-------------+
|---| as VXLAN |-----|
+---+-----+---+ | | Gateway |
| Server 3 | | +-------------+
(VXLAN enabled)<-----+
+-------------+ |
|
+---+-----+---+ |
| Server 4 | |
(VXLAN enabled)<-----+
+-------------+Figure 4: VXLAN Deployment - VXLAN Gateway
6.1. Inner VLAN Tag Handling
Inner VLAN tag handling in a VTEP and a VXLAN gateway should conform to the following 💖:
Decapsulated VXLAN frames carrying an inner VLAN tag SHOULD be discarded unless configured otherwise 🙅♀️. On the encapsulation side, a VTEP SHOULD NOT put an inner VLAN tag on tunnel packets unless configured otherwise 🚫. And when a VLAN-tagged packet is a candidate for VXLAN tunneling, the encapsulating VTEP SHOULD strip the VLAN tag unless configured otherwise ✂️.
7. Security Considerations 🔐
Okay, this section is kind of a big deal 😭. Traditionally, Layer 2 networks can only be attacked from "within" by rogue end points 👀. Those rogues might get inappropriate access to a LAN and snoop on traffic, inject spoofed packets to "take over" another MAC address, or flood the network to cause denial of service 💀. A MAC-over-IP mechanism for delivering Layer 2 traffic significantly extends this attack surface 📈. Rogues can inject themselves into the network by subscribing to one or more multicast groups that carry broadcast traffic for VXLAN segments 📡. They can also source MAC-over-UDP frames into the transport network to inject spurious traffic, possibly to hijack MAC addresses 🏴☠️.
This document doesn't bake in specific defenses against these attacks. It relies on other traditional mechanisms layered on top of IP instead 🧱. This section sketches out some possible approaches to security in a VXLAN environment 🗺️.
Traditional Layer 2 attacks by rogue end points can be mitigated by limiting the management and administrative scope of who deploys and manages VMs and gateways in a VXLAN environment 🔒. On top of that, those administrative measures can be boosted with schemes like 802.1X [802.1X] for admission control of individual end points 🚪. Also, the UDP-based encapsulation of VXLAN lets you configure and use the 5-tuple-based ACL (Access Control List) functionality in physical switches 🧠.
Tunneled traffic over the IP network can be secured with traditional security mechanisms like IPsec, which authenticate and optionally encrypt VXLAN traffic 🔏. Of course, this has to be coupled with an authentication infrastructure so authorized end points can obtain and distribute credentials 🔑.
VXLAN overlay networks are designated and operated over the existing LAN infrastructure 🏠. To make sure VXLAN end points and their VTEPs are authorized on the LAN, it is recommended that a VLAN be designated for VXLAN traffic. The servers/VTEPs then send VXLAN traffic over that VLAN, which provides a measure of security 🛡️.
Finally, VXLAN requires proper mapping of VNIs and VM membership in these overlay networks 🗂️. It is expected that this mapping gets done and communicated to the management entity on the VTEP and the gateways using existing secure methods 🤝.
8. IANA Considerations 📋
A well-known UDP port (4789) has been assigned by IANA in the Service Name and Transport Protocol Port Number Registry for VXLAN 🎉. Check out Section 5 for the discussion of the port number 🧐.
9. References 📚
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.9.2. Informative References
[802.1aq] IEEE, "Standard for Local and metropolitan area networks --
Media Access Control (MAC) Bridges and Virtual Bridged
Local Area Networks -- Amendment 20: Shortest Path
Bridging", IEEE P802.1aq-2012, 2012.
[802.1D] IEEE, "Draft Standard for Local and Metropolitan Area
Networks/ Media Access Control (MAC) Bridges", IEEE
P802.1D-2004, 2004.
[802.1X] IEEE, "IEEE Standard for Local and metropolitan area
networks -- Port-Based Network Acces Control", IEEE Std
802.1X-2010, February 2010.
[RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
November 1990.
[RFC1981] McCann, J., Deering, S., and J. Mogul, "Path MTU Discovery
for IP version 6", RFC 1981, August 1996.
[RFC4541] Christensen, M., Kimball, K., and F. Solensky,
"Considerations for Internet Group Management Protocol
(IGMP) and Multicast Listener Discovery (MLD) Snooping
Switches", RFC 4541, May 2006.
[RFC4601] Fenner, B., Handley, M., Holbrook, H., and I. Kouvelas,
"Protocol Independent Multicast - Sparse Mode (PIM-SM):
Protocol Specification (Revised)", RFC 4601, August 2006.
[RFC5015] Handley, M., Kouvelas, I., Speakman, T., and L. Vicisano,
"Bidirectional Protocol Independent Multicast (BIDIR-PIM)",
RFC 5015, October 2007.
[RFC6325] Perlman, R., Eastlake 3rd, D., Dutt, D., Gai, S., and A.
Ghanwani, "Routing Bridges (RBridges): Base Protocol
Specification", RFC 6325, July 2011.
[RFC6335] Cotton, M., Eggert, L., Touch, J., Westerlund, M., and S.
Cheshire, "Internet Assigned Numbers Authority (IANA)
Procedures for the Management of the Service Name and
Transport Protocol Port Number Registry", BCP 165, RFC
6335, August 2011.10. Acknowledgments 🫶
The authors want to give a shoutout to Ajit Sanzgiri for his contributions to the Security Considerations section and his editorial inputs 🙌. Big thanks also to Joseph Cheng, Margaret Petrus, Milin Desai, Nial de Barra, Jeff Mandin, and Siva Kollipara for their editorial reviews, inputs, and comments 💕.
Authors' Addresses ✨
Here's where you can find the squad behind this spec, bestie 🫶 Every one of them is a networking legend, and they're all reachable at the spots below 💌
Mallik Mahalingam 🦋 Storvisor, Inc. 640 W. California Ave, Suite #110 Sunnyvale, CA 94086. USA
EMail: mallik_mahalingam@yahoo.com
Dinesh G. Dutt 💅 Cumulus Networks 140C S. Whisman Road Mountain View, CA 94041 USA
EMail: ddutt.ietf@hobbesdutt.com
Kenneth Duda 👑 Arista Networks 5453 Great America Parkway Santa Clara, CA 95054 USA
EMail: kduda@arista.com
Puneet Agarwal 🌸 Broadcom Corporation 3151 Zanker Road San Jose, CA 95134 USA
EMail: pagarwal@broadcom.com
Lawrence Kreeger 🎀 Cisco Systems, Inc. 170 W. Tasman Avenue San Jose, CA 95134 USA
EMail: kreeger@cisco.com
T. Sridhar 🧚 VMware, Inc. 3401 Hillview Palo Alto, CA 94304 USA
EMail: tsridhar@vmware.com
Mike Bursell 🍓 Intel Bowyer's, North Road Great Yeldham Halstead Essex. C09 4QD UK
EMail: mike.bursell@intel.com
Chris Wright 🌈 Red Hat, Inc. 100 East Davie Street Raleigh, NC 27601 USA
EMail: chrisw@redhat.com
📦 where this page comes from
- ietf: rfc7348 2014-08
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms