RFC 8014
An Architecture for Data-Center Network Virtualization over Layer 3 (NVO3)
RFC 8014: An Architecture for Data-Center Network Virtualization over Layer 3 (NVO3) 💖
Bestie, this one is the NVO3 architecture doc. It's RFC 8014, published December 2016 by the IETF as an Informational document (ISSN 2070-1721). The authors are D. Black (Dell EMC), J. Hudson, L. Kreeger, M. Lasserre (all Independent), and T. Narten (IBM). 🎀 It doesn't say anything about obsoleting or updating other RFCs, so I'm not making that up for you 🙈
Abstract ✨
okay so this doc lays out a high-level architecture for building data-center Network Virtualization over Layer 3 (NVO3) networks 🌸 It shows the big components of the whole system. The main goal is to split everything into smaller pieces that can each be built on their own, with clear interfaces between them. The idea is that you should be able to build and implement one piece in isolation and have it still work with other pieces that other people built 🫶 Implementers get room to optimize and innovate inside their own piece, without anyone else needing to change their code. Iconic, honestly 💅
Status of This Memo ✨
This document is not an Internet Standards Track specification; it is published for informational purposes.
ngl this part is the legal-ish bit 🙈 It's a product of the IETF and represents the consensus of the IETF community. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, though. See Section 2 of RFC 7841 for that 🧚
For the current status, any errata, and how to send feedback, check http://www.rfc-editor.org/info/rfc8014 🌈
Copyright Notice ✨
Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved 💜
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication. Please read these carefully, because they spell out your rights and restrictions for this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🫶
Table of Contents ✨
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1. VN Service (L2 and L3) . . . . . . . . . . . . . . . . . 7
3.1.1. VLAN Tags in L2 Service . . . . . . . . . . . . . . . 8
3.1.2. Packet Lifetime Considerations . . . . . . . . . . . 8
3.2. Network Virtualization Edge (NVE) Background . . . . . . 9
3.3. Network Virtualization Authority (NVA) Background . . . . 10
3.4. VM Orchestration Systems . . . . . . . . . . . . . . . . 11
4. Network Virtualization Edge (NVE) . . . . . . . . . . . . . . 12
4.1. NVE Co-located with Server Hypervisor . . . . . . . . . . 12
4.2. Split-NVE . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2.1. Tenant VLAN Handling in Split-NVE Case . . . . . . . 14
4.3. NVE State . . . . . . . . . . . . . . . . . . . . . . . . 14
4.4. Multihoming of NVEs . . . . . . . . . . . . . . . . . . . 15
4.5. Virtual Access Point (VAP) . . . . . . . . . . . . . . . 16
5. Tenant System Types . . . . . . . . . . . . . . . . . . . . . 16
5.1. Overlay-Aware Network Service Appliances . . . . . . . . 16
5.2. Bare Metal Servers . . . . . . . . . . . . . . . . . . . 17
5.3. Gateways . . . . . . . . . . . . . . . . . . . . . . . . 17
5.3.1. Gateway Taxonomy . . . . . . . . . . . . . . . . . . 18
5.3.1.1. L2 Gateways (Bridging) . . . . . . . . . . . . . 18
5.3.1.2. L3 Gateways (Only IP Packets) . . . . . . . . . . 18
5.4. Distributed Inter-VN Gateways . . . . . . . . . . . . . . 19
5.5. ARP and Neighbor Discovery . . . . . . . . . . . . . . . 20
6. NVE-NVE Interaction . . . . . . . . . . . . . . . . . . . . . 20
7. Network Virtualization Authority (NVA) . . . . . . . . . . . 21
7.1. How an NVA Obtains Information . . . . . . . . . . . . . 21
7.2. Internal NVA Architecture . . . . . . . . . . . . . . . . 22
7.3. NVA External Interface . . . . . . . . . . . . . . . . . 22
8. NVE-NVA Protocol . . . . . . . . . . . . . . . . . . . . . . 24
8.1. NVE-NVA Interaction Models . . . . . . . . . . . . . . . 24
8.2. Direct NVE-NVA Protocol . . . . . . . . . . . . . . . . . 25
8.3. Propagating Information Between NVEs and NVAs . . . . . . 25
9. Federated NVAs . . . . . . . . . . . . . . . . . . . . . . . 26
9.1. Inter-NVA Peering . . . . . . . . . . . . . . . . . . . . 29
10. Control Protocol Work Areas . . . . . . . . . . . . . . . . . 29
11. NVO3 Data-Plane Encapsulation . . . . . . . . . . . . . . . . 29
12. Operations, Administration, and Maintenance (OAM) . . . . . . 30
13. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
14. Security Considerations . . . . . . . . . . . . . . . . . . . 31
15. Informative References . . . . . . . . . . . . . . . . . . . 32
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 34
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 351. Introduction ✨
okay bestie, this doc gives a high-level architecture for building data-center Network Virtualization over Layer 3 (NVO3) networks 🦋 It shows the major components of the whole system at a high level. The big goal is to break the space into smaller individual components that can each be implemented on their own, with clear interfaces and interactions between them. You should be able to build one component in isolation and have it interoperate with other components that other people implemented independently. That gives implementers flexibility: they can optimize and innovate inside their own components without anyone else having to change theirs 💅
The motivation for overlay networks is covered in "Problem Statement: Overlays for Network Virtualization" [RFC7364] 🌸 "Framework for Data Center (DC) Network Virtualization" [RFC7365] gives a framework for talking about overlay networks in general and the different components that have to work together to build these systems 🧚 This doc is different from the framework doc because it doesn't try to cover every possible approach in the general design space. Instead, it describes one particular approach that the NVO3 WG has focused on ✨
2. Terminology ✨
This document uses the same terminology as [RFC7365] 🎀 On top of that, these terms are used too:
- NV Domain: A Network Virtualization Domain is an administrative construct. It defines a Network Virtualization Authority (NVA), the set of Network Virtualization Edges (NVEs) associated with that NVA, and the set of virtual networks the NVA manages and supports. NVEs are associated with a (logically centralized) NVA, and any NVE supports communication for any of the virtual networks in the domain 💜
- NV Region: A region where information about a set of virtual networks gets shared. The degenerate case, a single NV Domain, is an NV Region that matches that domain. The more interesting case is when two or more NV Domains share information about some or all of the virtual networks they manage. Two NVAs share information about particular virtual networks so that tenants in different NV Domains can connect to each other. NVAs can share information about a whole NV Domain, or just individual virtual networks 🌈
- Tenant System Interface (TSI): The interface to a Virtual Network (VN) as presented to a Tenant System (TS, see [RFC7365]). The TSI logically connects to the NVE through a Virtual Access Point (VAP). To the Tenant System, the TSI looks like a Network Interface Card (NIC). It presents itself to a Tenant System as a normal network interface 🫶
- VLAN: Unless stated otherwise, the terms "VLAN" and "VLAN Tag" are used in this document to mean a Customer VLAN (C-VLAN) [IEEE.802.1Q]. The two terms are used interchangeably to make things easier to read 🍓
3. Background ✨
okay bestie, let's get into the good stuff 🫶. Overlay networks are how we give network virtualization services to a bunch of Tenant Systems (TSs) [RFC7365] 🦋. With overlays, the data traffic between tenants gets tunneled across the underlying data center's IP network, and the tunnels are a whole mood because they decouple the network the tenants see from the physical network they're actually using 💅. More NVO3 use cases are over in [USECASES] if you want to go deeper 🌸.
Tenant Systems connect to Virtual Networks (VNs). Each VN has its own attributes that define its properties, like which members are allowed to connect to it 🎀. Tenant Systems on the same VN usually talk to each other freely, but talking to anything outside the VN (whether that's another VN or the Internet) is carefully controlled and governed by policy 🔒. The NVO3 architecture doesn't put any restrictions on applying policy controls, even inside a VN, so you can be as strict as you want 💜.
A Network Virtualization Edge (NVE) [RFC7365] is the entity that actually does the overlay work ✨. It sits at the boundary between a Tenant System and the overlay network, as shown in Figure 1 🧚. An NVE also creates and maintains local state about each VN it's serving on behalf of a Tenant System, so it always knows what's going on 👑.
+--------+ +--------+
| Tenant +--+ +----| Tenant |
| System | | (') | System |
+--------+ | ................ ( ) +--------+
| +-+--+ . . +--+-+ (_)
| | NVE|--. .--| NVE| |
+--| | . . | |---+
+-+--+ . . +--+-+
/ . .
/ . L3 Overlay . +--+-++--------+
+--------+ / . Network . | NVE|| Tenant |
| Tenant +--+ . .- -| || System |
| System | . . +--+-++--------+
+--------+ ................
|
+----+
| NVE|
| |
+----+
|
|
=====================
| |
+--------+ +--------+
| Tenant | | Tenant |
| System | | System |
+--------+ +--------+
Figure 1: NVO3 Generic Reference ModelThe next few subsections dig into the key parts of an overlay system 🔍. Section 3.1 covers the service model (Ethernet vs. IP) that gets handed to Tenant Systems 🎀. Section 3.2 goes deeper on NVEs 💅. Section 3.3 introduces the Network Virtualization Authority, which is where NVEs go to get info about virtual networks 🌈. And Section 3.4 gives background on Virtual Machine (VM) orchestration systems and how they use virtual networks 🍓.
3.1. VN Service (L2 and L3)
A VN gives connected tenants either Layer 2 (L2) or Layer 3 (L3) service 🦋. With L2 service, the VN transports Ethernet frames, and the Tenant System gets something that's basically the same as being plugged into a specific L2 C-VLAN 💖. L2 broadcast frames usually get delivered to all of the other Tenant Systems on the VN, and multicast frames go to a subset of them 📣. From the Tenant System's point of view, it looks like a regular L2 Ethernet link 🌸. Inside NVO3, tenant frames get tunneled to remote NVEs based on the Media Access Control (MAC) addresses in the frame headers, the ones originated by the Tenant System 🎀. On the underlay, NVO3 packets are forwarded between NVEs based on the outer addresses of the tunneled packets 🧚.
With L3 service, the VN is a routed network that transports IP datagrams, and the Tenant System gets a service that only supports IP traffic 🌈. Inside NVO3, tenant frames get tunneled to remote NVEs based on the IP addresses of the packet that the Tenant System originated, and any L2 destination addresses the Tenant System provides are effectively ignored by the NVEs and the overlay network 🙈. For L3 service, the Tenant System gets configured with an IP subnet that's effectively a point-to-point link, meaning it only has the Tenant System and a next-hop router address on it 🦋.
L2 service is meant for systems that need native L2 Ethernet service and the ability to run protocols directly over Ethernet (i.e., not based on IP) 💅. L3 service is meant for systems where all the traffic can safely be assumed to be IP 💜. It's important to note that whether an NVO3 network gives a Tenant System L2 or L3 service, the Tenant System doesn't generally need to know the difference 🫶. In both cases, the virtual network shows up to the Tenant System as an L2 Ethernet interface ✨. Ethernet is used in both cases simply because it's a widely supported interface type that basically every Tenant System already supports 👑. So, no special software is needed on Tenant Systems to use an L3 or an L2 overlay service, which is honestly so convenient 🎀.
NVO3 can also provide a combined L2 and L3 service to tenants 🌸. A combined service provides L2 service for intra-VN communication, and it also provides L3 service for L3 traffic entering or leaving the VN 🔥. Architecturally, the combined L2/L3 handling in NVO3 is meant to match what people commonly do today in non-overlay environments with devices that provide a combined bridge/router service 🧚. With combined service, the virtual network keeps the semantics of L2 service, and all traffic is processed according to those L2 semantics 🌈. On top of that, traffic that needs IP processing also gets processed at the IP level 💖.
The IP processing for a combined service can happen on a standalone device attached to the virtual network (for example, an IP router), or it can be implemented locally on the NVE (see Section 5.4 on Distributed Inter-VN Gateways) 🦋. For unicast traffic, an NVE doing combined service might deliver a packet to another Tenant System attached to the same NVE (on either the same or a different VN), tunnel it to a remote NVE, or even forward it outside the NV Domain 💅. For multicast or broadcast packets, the combination of NVE L2 and L3 processing may result in copies of the packet getting both L2 and L3 treatment so they can reach all of the destinations involved 📣. This distributed NVE implementation of IP routing has the same network delivery behavior as if the L2 processing of the packet included delivering it to an IP router attached to the L2 VN as a Tenant System, with that router having additional network attachments to other networks, whether virtual or not 🌸. Basically, it's the same result, just way more distributed 👑.
3.1.1. VLAN Tags in L2 Service
An NVO3 L2 virtual network service may include encapsulated L2 VLAN tags provided by a Tenant System 🎀. However, it does not use those encapsulated tags when deciding where and how to forward traffic 🙈. The tags can be passed through, so Tenant Systems that send or expect to receive them can be supported as appropriate 🧚.
How an NVE handles VLAN tags it receives from a TS is controlled by settings associated with the VAP 💅. Just like ports on Ethernet switches, a number of settings are possible 🌈. For example, Customer VLAN Tags (C-TAGs) can be passed through transparently, always stripped upon receipt from a Tenant System, compared against a list of explicitly configured tags, and so on 💖.
Heads up, there are additional considerations when VLAN tags are used to identify both the VN and a Tenant System VLAN within that VN ⚠️. Those are described in Section 4.2.1 🦋.
3.1.2. Packet Lifetime Considerations
For L3 service, Tenant Systems should expect the IPv4 Time to Live (TTL) or IPv6 Hop Limit in the packets they send to be decremented by at least 1 ⏳. For L2 service, neither the TTL nor the Hop Limit (when the packet is IP) gets modified 🫶. The underlay network manages TTLs and Hop Limits in the outer IP encapsulation, and the values in those fields could be independent from or related to the values in the same fields of the tenant IP packets 🌈. So the tenant's TTL and the underlay's TTL are separate things, and that's lowkey important to remember 💜.
3.2. Network Virtualization Edge (NVE) Background 💖
okay so let's talk about the NVE, bestie 🧚. Tenant Systems plug into NVEs through a Tenant System Interface (TSI) 🔌. The TSI hooks up to the NVE through a Virtual Access Point (VAP), and each VAP belongs to exactly one VN, see Figure 2 👀. From the Tenant System's point of view, the TSI is literally like a NIC 💅. It shows up as a normal network interface, no drama. On the NVE side, a VAP is a logical network port (virtual or physical) that leads into one specific virtual network 🦋. Also, ngl this is so cute: two different Tenant Systems (and their TSIs) hanging off the same NVE can share a VAP (think TS1 and TS2 in Figure 2) as long as they're connecting to the same VN 🫶.
| Data-Center Network (IP) |
| |
+-----------------------------------------+
| |
| Tunnel Overlay |
+------------+---------+ +---------+------------+
| +----------+-------+ | | +-------+----------+ |
| | Overlay Module | | | | Overlay Module | |
| +---------+--------+ | | +---------+--------+ |
| | | | | |
NVE1 | | | | | | NVE2
| +--------+-------+ | | +--------+-------+ |
| | VNI1 VNI2 | | | | VNI1 VNI2 | |
| +-+----------+---+ | | +-+-----------+--+ |
| | VAP1 | VAP2 | | | VAP1 | VAP2|
+----+----------+------+ +----+-----------+-----+
| | | |
|\ | | |
| \ | | /|
-------+--\-------+-------------------+---------/-+-------
| \ | Tenant | / |
TSI1 |TSI2\ | TSI3 TSI1 TSI2/ TSI3
+---+ +---+ +---+ +---+ +---+ +---+
|TS1| |TS2| |TS3| |TS4| |TS5| |TS6|
+---+ +---+ +---+ +---+ +---+ +---+
Figure 2: NVE Reference ModelThe Overlay Module is the part that does the actual encapsulation and decapsulation of tunneled packets 📦✨. The NVE keeps track of the virtual networks it belongs to, so it can hand the Overlay Module the good stuff: the destination address of the NVE to tunnel a packet to, and the Context ID that goes in the encapsulation header to say which virtual network the tunneled packet belongs to 🧠.
On the side facing the data-center network, the NVE sends and receives native IP traffic 🌐. When traffic comes in from a Tenant System, the NVE figures out which egress NVE the packet should go to, adds an overlay encapsulation header, and sends it out on the underlay network 🚀. When traffic arrives from a remote NVE, the NVE strips off the encapsulation header and delivers the (original) packet to the right Tenant System 🎀. And if the source and destination Tenant Systems are on the same NVE? No encapsulation needed, the NVE just forwards the traffic directly 💁♀️.
Conceptually, the NVE is one single entity doing the NVO3 functionality 👑. In practice though, there are a bunch of different implementation scenarios, all detailed in Section 4 🌸.
3.3. Network Virtualization Authority (NVA) Background 🧚♀️
Address dissemination is the process of learning, building, and distributing the mapping and forwarding information that NVEs need so they can tunnel traffic to each other for communicating Tenant Systems 💌. For example, to send traffic to a remote Tenant System, the sending NVE has to know the destination NVE for that Tenant System 🗺️.
One way to build and maintain mapping tables is learning, the same way 802.1 bridges do [IEEE.802.1Q] 🧠. When forwarding traffic to multicast or unknown unicast destinations, an NVE could just flood the traffic 🌊. Flooding works, sure, but it can create traffic hot spots and cause problems in bigger networks (for example, way too much flooded traffic) 😭.
So, to shrink the area where flooding has to happen, or to get rid of it entirely, NVEs can use a Network Virtualization Authority (NVA) 🦸♀️. An NVA is the entity that provides address mapping and other information to NVEs. NVEs talk to an NVA to get whatever address-mapping information they need in order to forward traffic properly on behalf of tenants 💅. The term "NVA" refers to the overall system, no matter its scope or how it's implemented 🫧. NVAs provide a service, and NVEs access that service through an NVE-NVA protocol, which we'll get into in Section 8 📖.
Even when an NVA is present, Ethernet bridge MAC address learning could still be used as a fallback, in case the NVA can't provide an answer or for other reasons 🔁. This document doesn't go deep on flooding approaches, because there are a lot of benefits to relying on an NVA 🙈.
For the rest of this document, it's assumed that an NVA exists and will be used 🫶. NVAs get the full treatment in Section 7 💜.
3.4. VM Orchestration Systems 🎀
VM orchestration systems manage server virtualization across a set of servers 🖥️. VM management is its own topic, separate from network virtualization, but honestly the two are super close 💕. Managing the creation, placement, and movement of VMs also means creating, attaching to, and detaching from virtual networks 🔗. Plenty of existing VM orchestration systems have already folded aspects of virtual network management into what they do ✨.
Also, even though this section keeps saying "VM" and "hypervisor", the same issues apply to other virtualization approaches too, like Linux Containers (LXC), BSD Jails, Network Service Appliances as discussed in Section 5.1, etc. 🧩. From an NVO3 perspective, wherever the document says "VM" and "hypervisor", assume the discussion also covers other systems, where, e.g., the host operating system plays the hypervisor role and a container plays the equivalent of a VM 🐳.
When a new VM image starts up, the VM orchestration system decides where the VM should be placed, works with the hypervisor on the target server to load and start the VM, and controls when a VM should be shut down or migrated somewhere else 🚚. These systems also know how a VM should connect to a network, possibly including the name of the virtual network the VM is supposed to join 🪄. The VM orchestration system can pass that info to the hypervisor when the VM is instantiated 📨. These systems have significant (and sometimes global) knowledge over the domain they manage 🌍. They typically know which servers a VM is running on, and metadata attached to VM images can be really useful from a network virtualization perspective 💡. For example, the metadata may include the addresses (MAC and IP) the VMs will use and the name(s) of the virtual network(s) they connect to 🏷️.
VM orchestration systems run a protocol with an agent on the hypervisor of each server they manage 🤝. That protocol can also carry info about which virtual network a VM is associated with 📡. When the orchestrator instantiates a VM on a hypervisor, the hypervisor talks to the NVE so it can attach the VM to the virtual networks it has access to 🔌. In general, the hypervisor needs to communicate significant VM state changes to the NVE 📣. In the other direction, the NVE may need to send network connectivity information back to the hypervisor 🔙. Examples of deployed VM orchestration systems include VMware's vCenter Server, Microsoft's System Center Virtual Machine Manager, and systems based on OpenStack and its associated plugins (e.g., Nova and Neutron) 🌈. Each one can pass info about which virtual networks a VM connects to down to the hypervisor 🛎️. The protocol used between the VM orchestration system and hypervisors is generally proprietary 🔒.
It's worth noting that VM orchestration systems may not have direct access to all the networking-related information a VM uses 👀. For example, a VM may use additional IP or MAC addresses that the VM management system doesn't know about 💀.
4. Network Virtualization Edge (NVE) 🏰
As introduced in Section 3.2, an NVE is the entity that implements the overlay functionality ✨. This section goes into more detail about NVEs. An NVE has two external interfaces:
- Facing the Tenant System: On the side facing the Tenant System, an NVE talks to the hypervisor (or equivalent entity) to provide the NVO3 service 💖. An NVE needs to be notified when a Tenant System "attaches" to a virtual network, so it can validate the request and set up any state needed to send and receive traffic on behalf of that Tenant System on that VN 🛂. Likewise, an NVE needs to be informed when the Tenant System "detaches" from the virtual network, so it can reclaim state and resources appropriately 🧹.
- Facing the Data-Center Network: On the side facing the data-center network, an NVE interfaces with the data-center underlay network, sending and receiving tunneled packets to and from the underlay 🛣️. The NVE may also run a control protocol with other entities on the network, such as the Network Virtualization Authority 🤙.
4.1. NVE Co-located with Server Hypervisor ✨
okay so when server virtualization is in use, the whole NVE thing usually lives inside the hypervisor and/or virtual switch on the server 🖥️💖. The Tenant System talks to the hypervisor, and the hypervisor talks to the NVE. Since all of that interaction happens in software on the server, there's no "on-the-wire" protocol between the Tenant Systems (or the hypervisor) and the NVE that needs to be standardized 🙌. There might be APIs between the NVE and the hypervisor to support that interaction, but the details of those APIs are not in scope for the NVO3 WG at the time of publication of this memo 📝.
Doing all of the NVE work on the server has a downside, though 😭. Server CPU resources have to be spent on the NVO3 functionality. Experiments with overlay approaches, plus past experience with TCP and checksum adapter offloads, suggest that offloading certain NVE operations (e.g., encapsulation and decapsulation) onto the physical network adapter can give performance wins 🚀. Like checksum and/or TCP server offload and other optimization tricks, there may be benefits to offloading common operations onto adapters where possible. Just as important, adding an overlay header can disable existing adapter offload features, because those features aren't usually ready to handle a new header or the other operations an NVE brings with it 💀.
How exactly to split NVE functionality between a server and its network adapters is an implementation matter and outside the scope of IETF standardization 🫶. Still, the NVO3 architecture should be cognizant of that separation and support it. Ideally, it may even be possible to bypass the hypervisor completely on critical data-path operations, so packets between a Tenant System and its VN can be sent and received without the hypervisor being involved in every single packet operation 🦋✨.
4.2. Split-NVE 🌸
Another scenario calls for a split-NVE implementation 💅. An NVE running on a server (e.g., within a hypervisor) could support NVO3 service toward the tenant without doing all of the NVE functions (e.g., encapsulation) directly on the server. Some of the actual NVO3 functionality could be implemented on (i.e., offloaded to) an adjacent switch that the server is attached to 🔀. There are lots of possible link types between a server and the NVE, but one simple deployment has a server and NVE separated by a plain L2 Ethernet link. A more complicated setup has them separated by a bridged access network, like when the NVE lives on a Top of Rack (ToR) switch with an embedded switch sitting between the servers and the ToR switch 🧚.
In the split-NVE case, protocols are needed so the hypervisor and NVE can negotiate and set up the state they need. That way, traffic sent across the access link between a server and the NVE can be associated with the correct virtual network instance 🎀. Specifically, on the access link, traffic belonging to a specific Tenant System gets tagged with a specific VLAN C-TAG that identifies which NVO3 virtual network instance it connects to. The hypervisor-NVE protocol negotiates which VLAN C-TAG to use for a particular virtual network instance 💜. More details on the protocol requirements for functionality between hypervisors and NVEs can be found in [NVE-NVA].
4.2.1. Tenant VLAN Handling in Split-NVE Case 🎀
Preserving tenant VLAN tags across an NVO3 VN, as described in Section 3.1.1, gets extra complicated in the split-NVE case 😵💫. The part of the NVE that does the encapsulation needs access to the specific VLAN tags the Tenant System is using, so it can include them in the encapsulated packet. When an NVE is implemented entirely within the hypervisor, the NVE has access to the complete original packet (VLAN tags included) that the tenant sent. In the split-NVE case, though, the VLAN tag used between the hypervisor and the offloaded portions of the NVE normally only identifies the specific VN that the traffic belongs to 🏷️. So if a tenant wants to keep VLAN information end to end between Tenant Systems in the split-NVE case, additional mechanisms are needed. One option is to carry an additional VLAN tag, by carrying both a C-TAG and a Service VLAN Tag (S-TAG) as specified in [IEEE.802.1Q]. The C-TAG identifies the tenant VLAN end to end, and the S-TAG identifies the VN locally, between each Tenant System and the corresponding NVE 🫧.
4.3. NVE State 🗂️
NVEs keep internal data structures and state so they can send and receive tenant traffic 💾. An NVE may need some or all of the following information:
- An NVE keeps track of which attached Tenant Systems are connected to which virtual networks 👀. When a Tenant System attaches to a virtual network, the NVE will need to create or update the local state for that virtual network. When the last Tenant System detaches from a given VN, the NVE can reclaim the state associated with that VN 🧹.
- For tenant unicast traffic, an NVE maintains a per-VN table of mappings from Tenant System (inner) addresses to remote NVE (outer) addresses 🗺️.
- For tenant multicast (or broadcast) traffic, an NVE maintains a per-VN table of mappings and other information on how to deliver tenant multicast (or broadcast) traffic 📡. If the underlying network supports IP multicast, the NVE could use IP multicast to deliver tenant traffic. In that case, the NVE would need to know which IP underlay multicast address to use for a given VN. Alternatively, if the underlying network doesn't support multicast, a source NVE could use unicast replication to deliver traffic. In that case, an NVE would need to know which remote NVEs are participating in the VN 🔁. An NVE could use both approaches, switching from one mode to the other depending on factors such as bandwidth efficiency and group membership sparseness. [FRAMEWORK-MCAST] discusses multicast handling in NVO3 in further detail 📚.
- An NVE maintains the necessary information to encapsulate outgoing traffic, including what type of encapsulation to use and what value to use for a Context ID to identify the VN within the encapsulation header 📦.
- To deliver incoming encapsulated packets to the correct Tenant Systems, an NVE maintains the necessary information to map incoming traffic to the appropriate VAP (i.e., TSI) 🎯.
- An NVE may find it convenient to maintain additional per-VN information, such as QoS settings, Path MTU information, Access Control Lists (ACLs), etc. 💅✨
4.4. Multihoming of NVEs 🌈
NVEs may be multihomed 🙌. That means an NVE may have more than one IP address associated with it on the underlay network. Multihoming happens in two different scenarios. First, an NVE may have multiple interfaces connecting it to the underlay. Each of those interfaces will typically have a different IP address, so a specific Tenant Address (on a specific VN) can be reachable through the same NVE but through more than one underlay IP address. Second, a specific Tenant System may be reachable through more than one NVE, with each NVE having one or more underlay addresses 🗺️. In both cases, NVE address-mapping functionality needs to support one-to-many mappings, and it needs to enable a sending NVE to (at a minimum) fail over from one IP address to another, e.g., should a specific NVE underlay address become unreachable 🔄.
Multihomed NVEs also add complications when source unicast replication is used to implement tenant multicast, as described in Section 4.3. Specifically, an NVE should only receive one copy of a replicated packet 🫣.
Multihoming is needed for some important use cases 💖. First, a bare metal server may have multiple uplink connections to either the same or different NVEs. Having only a single physical path to an upstream NVE, or having all traffic flow through a single NVE, would be considered unacceptable in highly resilient deployment scenarios that aim to avoid single points of failure 🚫. Moreover, in today's networks, multiple paths would need to be usable in an active-active fashion (e.g., for load balancing) ⚖️.
4.5. Virtual Access Point (VAP) 🚪
The VAP is the NVE side of the interface between the NVE and the TS. Traffic to and from the tenant flows through the VAP 🌊. If an NVE runs into trouble sending traffic it received on the VAP, it may need to signal those errors back to the VAP. But the VAP is an emulation of a physical port, so its ability to signal NVE errors is limited and lacks the granularity to reflect every possible error an NVE may hit (e.g., being unable to reach a particular destination) 🙃. Some errors, like an NVE losing all of its connections to the underlay, could be reflected back to the VAP by effectively disabling it. That state change would show up on the TS as an interface going down, letting the TS handle interface errors (e.g., failover) the same way it would when a physical interface gets disabled 🔌✨.
5. Tenant System Types 👑
This section describes a number of special Tenant System types and how they fit into an NVO3 system 🧁.
5.1. Overlay-Aware Network Service Appliances 🛡️
Some Network Service Appliances [NVE-NVA] (virtual or physical) provide tenant-aware services 🥹. That is, the specific service they provide depends on the identity of the tenant using it. For example, firewalls are now available that support multitenancy, where a single firewall provides virtual firewall service on a per-tenant basis, using per-tenant configuration rules and maintaining per-tenant state 🔥. Such appliances are aware of the VN an activity corresponds to while processing requests. Unlike server virtualization, which shields VMs from needing to know about multitenancy, a Network Service Appliance may explicitly support multitenancy. In those cases, the Network Service Appliance itself is aware of network virtualization and either embeds an NVE directly or implements a split-NVE as described in Section 4.2 💕. Unlike server virtualization, however, the Network Service Appliance may not be running a hypervisor, and the VM orchestration system may not interact with the Network Service Appliance 🤷♀️. The NVE on such appliances will need to support a control plane to obtain the information it needs to fully participate in an NV Domain 🧠💜.
5.2. Bare Metal Servers 🖥️
okay so not every server is gonna be a glam virtualized vibe 💅 a bunch of data centers will keep some bare metal servers running a traditional OS and workload. on those machines there's zero NVE functionality, and the server has no idea what NVO3 is (or even whether overlays are in use at all) 🙈 so where does the NVE stuff live? on the first-hop physical switch! 🔌 the network admin manually configures the switch to turn on the right NVO3 functionality on the port that plugs into the server, and links that port to a specific virtual network. ngl that config is basically static, since the server isn't virtualized and once it's set up it's not gonna change much 😌 so this scenario doesn't need any new protocol or standards work. easy peasy 🎀
5.3. Gateways 🌉
gateways are the bouncers of a VN 🚪 they relay traffic onto and off of a virtual network. when a Tenant System wants to reach something outside its local VN, it goes through a gateway. the gateway gets encapsulated traffic from one VN, strips off the encapsulation header, and sends the native packet out onto the data-center network for delivery. outside traffic entering a VN goes through the same process, just in reverse 🔄
gateways can be virtual (basically a VM) or physical (a standalone device) ✨ for performance reasons, standalone hardware gateways can be the move in some cases. they could be a simple switch forwarding traffic from a VN onto the local data-center network, or they could have router functionality built in 🛜 on these gateways, the network interfaces that connect to virtual networks will (at least conceptually) have NVE (or split-NVE) functionality embedded in them. just like with Network Service Appliances, gateways might not run a hypervisor, so they'll need an appropriate control-plane protocol to get the info they need to provide NVO3 service 🧚
gateways handle lots of different use cases, bestie 💖 one is connecting systems that support overlays with systems that don't (like bare metal servers). gateways can also connect legacy systems that use, e.g., L2 VLANs to specific virtual networks, which makes them effectively part of the same virtual network 🫶 they can forward traffic between a virtual network and other hosts on the data-center network, or relay traffic between different VNs. and finally, gateways can provide external connectivity like Internet or VPN access 🌈
5.3.1. Gateway Taxonomy 🗂️
so as you can see from everything above, there are several types of gateways that can exist in an NVO3 environment. this section breaks them down into the types that could be supported. also, each type below could be implemented in a centralized way, or distributed so it coexists with the NVEs 💅
5.3.1.1. L2 Gateways (Bridging) 🌊
L2 Gateways act as Layer 2 bridges, forwarding Ethernet frames based on the MAC addresses in them.
- L2 VN to Legacy L2: this one bridges traffic between L2 VNs and other legacy L2 networks, like VLANs or L2 VPNs 🔗
- L2 VN to L2 VN: the main reason for this one is to create separate groups of Tenant Systems using L2 VNs, so the gateway can enforce network policies between each L2 VN 🛡️
5.3.1.2. L3 Gateways (Only IP Packets) 🧭
L3 Gateways forward IP packets based on the IP addresses in the packets. no frames for these girlies, just IP 💜
- L3 VN to Legacy L2: this one forwards packets between L3 VNs and legacy L2 networks, like VLANs or L2 VPNs. the original sender's destination MAC address in any frames that the gateway forwards from a legacy L2 network would be the MAC address of the gateway 🍓
- L3 VN to Legacy L3: this one forwards packets between L3 VNs and legacy L3 networks. those legacy L3 networks could be local to the data center, out in the WAN, or an L3 VPN 🌍
- L3 VN to L2 VN: this one forwards packets between L3 VNs and L2 VNs. the original sender's destination MAC address in any frames that the gateway forwards from an L2 VN would be the MAC address of the gateway 🎀
- L2 VN to L2 VN: this one acts kinda like a traditional router that forwards between L2 interfaces. the original sender's destination MAC address in any frames that the gateway forwards from any of the L2 VNs would be the MAC address of the gateway 🦋
- L3 VN to L3 VN: the main motivation here is to create separate groups of Tenant Systems using L3 VNs, so the gateway can enforce network policies between each L3 VN 🔥
5.4. Distributed Inter-VN Gateways 🌐
relaying traffic from one VN to another deserves some special attention 👀 whether traffic is allowed to flow from one VN to another is a matter of policy, and by default it's not allowed unless someone explicitly turns it on. NVAs are the logical place to keep the policy info about which inter-VN communication is allowed 📋 policy enforcement for inter-VN communication can be handled in (at least) two different ways. one option is explicit gateways acting as the central point for enforcement, with all inter-VN traffic forwarded to them for processing. the other option is for the NVA to hand that info straight to NVEs, either by giving them a mapping for a target Tenant System (TS) on another VN, or by indicating that the communication is disallowed by policy 💅
when inter-VN gateways are centralized, traffic between TSs on different VNs can take suboptimal paths. this is triangular routing, which means paths always go through the gateway 🔺 in the worst case, traffic between two TSs connected to the same NVE can get hairpinned through an external gateway 😭 as an optimization, individual NVEs can be part of a distributed gateway that does this relaying, which reduces or completely eliminates triangular routing ✨ in a distributed gateway, each ingress NVE can do the relaying itself, as long as it has access to the policy info needed to decide whether cross-VN communication is allowed. being part of a distributed gateway lets NVEs tunnel traffic directly to the destination NVE without taking suboptimal paths 🦋
the NVO3 architecture supports distributed gateways for inter-VN communication. that support requires NVO3 control protocols to include mechanisms for maintaining and distributing policy info about what type of cross-VN communication is allowed, so NVEs acting as distributed gateways can tunnel traffic from one VN to another as appropriate 🫶
distributed gateways could also be used to spread other traditional router services out to individual NVEs. the NVO3 architecture doesn't block those implementations, but it doesn't define or require them either, since they're outside the scope of the NVO3 architecture 🙃
5.5. ARP and Neighbor Discovery 📣
strictly speaking, for an L2 service, special processing of the Address Resolution Protocol (ARP) [RFC826] and IPv6 Neighbor Discovery (ND) [RFC4861] isn't required 🙌 ARP requests are broadcast, and an NVO3 can deliver them to all members of a given L2 virtual network, the same way it handles any packet sent to an L2 broadcast address. ND requests go out via IP multicast, which NVO3 can support by delivering them via L2 multicast 📡 but here's the perf glow-up: an NVE can intercept ARP (or ND) requests from its attached TSs and respond directly using info in its mapping tables ⚡ since an NVE has mechanisms for figuring out the NVE address associated with a given TS, it can use those same mechanisms to suppress sending ARP and ND requests for that TS to other members of the VN. the NVO3 architecture supports this capability 💖
6. NVE-NVE Interaction ✨
individual NVEs need to talk to each other to tunnel and deliver traffic to remote TSs 🤝 at a minimum, a control protocol may be needed for tunnel setup and maintenance. for example, tunneled traffic might need to be encrypted or integrity protected, in which case appropriate security associations have to be set up between NVE peers 🔐 it can also be useful to run tunnel maintenance (like continuity checks) on a tunnel, so you can detect when a remote NVE becomes unreachable 👀 these generic tunnel setup and maintenance functions aren't really NVO3-specific, so the NVO3 architecture plans to lean on existing tunnel maintenance protocols instead of making new ones. no need to reinvent the wheel, bestie 🛞
some NVE-NVE interactions are NVO3-specific (especially ones tied to info kept in mapping tables) and don't care which tunnel type is being used 🧃 for example, when tunneling traffic for TS-X to a remote NVE, it's possible that TS-X isn't currently associated with that remote NVE. normally this shouldn't happen, but race conditions can occur where the info an NVE learned from the NVA is out of date compared to what's actually happening 😬 in that case, the remote NVE could return an error or warning indication, so the sending NVE can attempt a recovery or otherwise try to mitigate the situation 🚑
the NVE-NVE interaction could signal a range of indications, for example:
- "No such TS here", upon receipt of a tunneled packet for an unknown TS
- "TS-X not here, try the following NVE instead" (i.e., a redirect)
- "Delivered to correct NVE but could not deliver packet to TS-X"
when an NVE gets info from a remote NVE that conflicts with what's in its own mapping tables, it should consult the NVA to resolve the conflict 🧐 in particular, it should confirm its info is up to date, and it might flag the error to the NVA to nudge the NVA into following up (as appropriate) 📣 it might make sense for an NVE to temporarily update its mapping table in response to an error from a remote NVE, but any changes have to be handled carefully. doing that can raise security considerations if the received info can't be authenticated 🚨 that said, a sending NVE might still take steps to mitigate a problem, like applying rate limiting to data traffic headed toward a particular NVE or TS 🚦
7. Network Virtualization Authority (NVA) 👑
before sending traffic to and receiving traffic from a virtual network, an NVE has to get the info it needs to build its internal forwarding tables and state, as listed in Section 4.3. 📝 an NVE can get that info from a Network Virtualization Authority (NVA) 🙌
the NVA is the entity that's expected to provide address mapping and other info to NVEs. NVEs can talk to an NVA to get any info they need to properly forward traffic on behalf of tenants 💅 the term "NVA" refers to the overall system, no matter its scope or how it's implemented ✨
7.1. How an NVA Obtains Information ✨
Okay so there are basically two main ways an NVA can get the address dissemination info it manages 📬: from the VM orchestration system, and/or straight from the NVEs themselves 💅.
On virtualized systems, the NVA may be able to grab address-mapping info for VMs right from the VM orchestration system. If that system holds the master database for all the virtualization info, pulling from it directly is the natural move 🧚. Honestly, the NVA could even be co-located with the VM orchestration system itself 👑. In those setups, the orchestration system talks to the NVE indirectly, through the hypervisor 🫶.
BUT here's the plot twist 😭: not every NVE is associated with a hypervisor, as we covered in Section 4. For those, NVAs can't use VM orchestration protocols to interact with the NVE, so they'll have to peer directly with it 🦋. Peering directly lets the NVA learn about the TSs connected to that NVE, and tell the NVE which VNs those TSs are associated with 💖. For example, whenever a Tenant System attaches to an NVE, that NVE notifies the NVA that the TS is now associated with it ✨. And when a TS detaches, the NVE informs the NVA 🌸. By talking directly, both the NVA and the NVE stay up to date on all the active tenants and the NVEs they're attached to. Main character energy for the data 🎀.
7.2. Internal NVA Architecture 🧩
For reliability and fault tolerance, an NVA would be built in a distributed or replicated way with no single points of failure 🛡️. How the NVA is implemented inside, though, doesn't really matter to an NVE, as long as the NVA gives it a consistent and well-defined interface 💜. For example, an NVA could use database techniques, where a server stores address-mapping info in a traditional (possibly replicated) database 🗄️. Or an NVA could be distributed, using an existing (or modified) routing protocol to maintain and distribute mappings 🌈. As long as there's a clear interface between the NVE and NVA, how the NVA is architected and implemented isn't the NVE's problem 🙈.
A bunch of architectural approaches could be used to build NVAs themselves. NVAs manage address bindings and distribute them to wherever they need to go 🚀. One approach would be to use the Border Gateway Protocol (BGP) [RFC4364] (possibly with extensions) and route reflectors. Another approach could use a transaction-based database model with replicated servers. Because the implementation details are local to an NVA, there's no need to pick exactly one solution tech, so long as the external interfaces to the NVEs (and remote NVAs) are well defined enough to get interoperability ✨.
7.3. NVA External Interface 🦋
Conceptually, from an NVE's point of view, an NVA is a single entity 💖. An NVE interacts with the NVA, and it's the NVA's job to make sure those interactions lead to consistent behavior across the NVA and all the other NVEs using the same NVA 💅. Because an NVA is built from multiple internal components, the NVA has to make sure information flows to all of those internal components properly 🧠.
One architectural question is how the NVA presents itself to the NVE 🤔. For example, an NVA could be required to provide access via a single IP address. If NVEs only have one IP address to talk to, it's on the NVA to handle component failures, e.g., by using a "floating IP address" that migrates among NVA components so the NVA can always be reached via that one address 🪽. Having all NVA access go through a single IP address, though, adds constraints to doing robust failover, load balancing, and so on 😬.
In the NVO3 architecture, an NVA is accessed through one or more IP addresses (or an IP address/port combination) 🌸. If multiple IP addresses are used, each one provides equivalent functionality, which means an NVE can use any of the provided addresses to interact with the NVA 👑. If one address stops working, an NVE is expected to fail over to another one 🔄. Even though the addresses give equivalent functionality, one might respond more quickly than another, e.g., due to network conditions, load on the server, etc. 🐢
To get some control over load balancing, NVA addresses may have an associated priority 🎀. Addresses are used in order of priority, with no explicit preference among NVA addresses that have the same priority. To provide basic load balancing among NVAs of equal priority, NVEs could use some randomization input to pick among them 🎲. This kind of priority scheme makes failover and load balancing easier, for example by letting a network operator specify a set of primary and backup NVAs 🫶.
It may be desirable to have individual NVA addresses responsible for a subset of the information about an NV Domain 🗂️. In that case, NVEs would use different NVA addresses for getting or updating information about particular VNs or TS bindings. The key questions with this approach are how the information would be partitioned, and how an NVE could figure out which address to use to get the info it needs 🧩.
Another possibility is to treat the info about which NVA addresses to use as cached (soft-state) information at the NVEs 💾. That way any NVA address can be used to get any information, but NVEs are told their preferences for which addresses to use for particular VN or TS binding info. That preference info would be cached for later use to improve behavior 🚀. For example, if all requests for a specific subset of VNs get forwarded to one specific NVA component, the NVE can optimize future requests in that subset by sending them directly to that component via its address 🏎️.
8. NVE-NVA Protocol 💌
As outlined in Section 4.3, an NVE needs certain info to do its job 🧃. To get that info from an NVA, you need an NVE-NVA protocol 🔐. This protocol does two things. First, it lets an NVE find out about the location and status of other TSs it needs to talk to 📍. Second, it gives NVEs a way to send updates to the NVA about the TSs attached to them (e.g., when a TS attaches or detaches from the NVE), or about communication errors they ran into while sending traffic to remote NVEs 🚨. For example, an NVE could say that a destination it's trying to reach at a destination NVE is unreachable for some reason 🙅♀️.
A direct NVE-NVA protocol might seem straightforward, but the existence of VM orchestration systems complicates the choices an NVE has for interacting with the NVA 🌀.
8.1. NVE-NVA Interaction Models 🎭
An NVE interacts with an NVA in at least two (quite different) ways:
- NVEs embedded within the same server as the hypervisor can get all the info they need entirely through the hypervisor-facing side of the NVE 🖥️. This is a natural extension of existing VM orchestration systems that support server virtualization, because there's already a protocol between the hypervisor and the VM orchestration system that can be leveraged to get whatever's needed 💅. Specifically, VM orchestration systems used to create, terminate, and migrate VMs already use well-defined (though typically proprietary) protocols to handle the interactions between the hypervisor and the orchestration system 🔒. So for these systems, it's a natural extension to use the existing orchestration protocol as a sort of proxy protocol for handling the interactions between an NVE and the NVA 🧚. Honestly, existing implementations can already do this ✅.
- Alternatively, an NVE can get the info it needs by interacting directly with an NVA through a protocol running over the data-center underlay network 🌐. This approach is needed to support NVEs that aren't tied to systems doing server virtualization (e.g., a standalone gateway), or where the NVE needs to talk directly to the NVA for other reasons 🏠.
The NVO3 architecture will focus on supporting the second model above 💜. Existing virtualization environments are already using the first model, but it isn't enough to cover standalone gateways. Those gateways may not support virtualization and don't interface with existing VM orchestration systems 🥲.
8.2. Direct NVE-NVA Protocol 🧑💻
An NVE can interact directly with an NVA via an NVE-NVA protocol 📡. That protocol can be either independent of the NVA internal protocol, or an extension of it. Using a purpose-specific protocol gives you architectural separation and independence between the NVE and NVA 🧱. The NVE and NVA interact in a well-defined way, so changes in the NVA (or the NVE) don't have to impact each other 🫶. A dedicated protocol also means both NVE and NVA implementations can evolve independently, without depending on each other 🦋. That independence matters because the upgrade paths for NVEs and NVAs are really different 😩. Upgrading all the NVEs at a site will likely be harder in practice than upgrading NVAs, because there are so many of them, one on each end device 📦. So it's smart to assume that once an NVE has been implemented and deployed, it may be tough to get later NVE extensions and changes implemented and deployed 🐌. An NVA (and its associated internal protocols), on the other hand, is more likely to evolve over time as experience is gained from usage, and its upgrades will involve fewer nodes 🚀.
Requirements for a direct NVE-NVA protocol can be found in [NVE-NVA] 📚.
8.3. Propagating Information Between NVEs and NVAs ✨
Okay so info flows both ways here 🔁 The NVA keeps track of every VN in the NV Domain, so NVEs don't have to do it all themselves 🙌 NVEs ask the NVA where a given remote TS destination lives, and NVAs collect info from NVEs about the individual TSs plugged into those NVEs 💅
Pushing everything about every virtual network to every NVE sounds fun, but it scales badly and it's honestly unnecessary 😭 A given NVE really only needs to know about the VNs it's attached to, so an NVE should be able to subscribe to updates just for the virtual networks it cares about 💜 The NVO3 architecture supports a model where an NVE does not have to hold full mapping tables for every virtual network in an NV Domain 🦋
Before an NVE sends unicast traffic to a remote TS (or to TSs for broadcast or multicast traffic), it needs to know where those remote TS(s) are right now 👀 When a TS attaches to a virtual network, the NVE gets info about that VN from the NVA 🫶 The NVA can hand over that info right when the TS attaches, either because the NVE asks for it during the attach operation, or because the VM orchestration system kicked off the attach and gave the NVE the mapping info at the same time 🎀
Sometimes an NVE wants to ask the NVA about individual mappings inside a VN 🕵️♀️ For example, when it's sending traffic to a remote TS on a remote NVE, that TS might be gone (maybe it migrated somewhere else or got shut down, and then the remote NVE may send back an error indication) 😬 In situations like that, the NVE may need to query the NVA for updated mapping info about that specific TS, or just to double-check that what it has is still correct despite the error 🔍 Also, that kind of query could be used by the NVA as a hint that something might be inconsistent in the network, so it should go verify that its info about the current state and location of that specific TS is still correct 🚨
Big virtual networks can mean a LOT of state for an NVE to keep track of for a single VN 🥹 Plus, an NVE might only be talking to a tiny handful of the TSs on that VN. In that case, it can be super smart for the NVE to only keep state for the destinations it's actively chatting with 💖 So an NVE in these scenarios may not want full mapping info about every destination on a VN. But if it needs to reach a destination it has no mapping for, it needs to be able to ask the NVA on demand for that missing info, one destination at a time 🧚
The NVO3 architecture is going to need to support a bunch of operations between the NVE and NVA 📋 The requirements for those operations can be found in [NVE-NVA] 📚
9. Federated NVAs 🌐
An NVA serves the set of NVEs in its NV Domain 🏠 Each NVA manages network virtualization info for the virtual networks inside its own NV Domain, and an NV Domain is run by a single entity 👑
Sometimes a specific VN, or even a whole NV Domain, needs to stretch beyond a single NVA 🌈 Picture an admin running several data centers who wants all of them to act like one single NV Region 🤩 That works by having different NVAs peer with each other and swap mapping info about specific VNs 🦋 NVAs operate in a federated way, meaning a set of NVAs works like a loosely coupled federation of individual NVAs 🧸 If a virtual network spans multiple NVAs (say, they're at different data centers) and an NVE needs to deliver tenant traffic to an NVE that belongs to a different NV Domain, the NVE still only talks to its own NVA 💅 Even when it's getting mappings for NVEs that are part of a different NV Domain, it's still only interacting with its NVA, periodt ✨
Figure 3 shows two separate NV Domains (A and B) sharing info about a VN 🎀 VM1 and VM2 both hook up to the same VN, even though the two VMs live in separate NV Domains 🫶 There are two cases to think about. In the first case, NV Domain B doesn't let NVE-A tunnel traffic directly to NVE-B 🙅♀️ There could be a bunch of reasons for that. For example, NV Domains A and B might not share a common address space (meaning traversal through a NAT device is required), or a domain might have a policy that says all traffic between separate NV Domains has to be funneled through one particular device, like a firewall 🔥 In cases like this, NVA-2 will advertise to NVA-1 that VM1 on the VN is available, and it will direct that traffic between the two nodes to be forwarded via IP-G (an IP Gateway) 🚪 IP-G then decapsulates the traffic it gets from one NV Domain, translates it appropriately for the other domain, and re-encapsulates the packet for delivery 🦋
xxxxxx xxxx +-----+
+-----+ xxxxxx xxxxxx xxxxxx xxxxx | VM2 |
| VM1 | xx xx xxx xx |-----|
|-----| xx x xx x |NVE-B|
|NVE-A| x x +----+ x x +-----+
+--+--+ x NV Domain A x |IP-G|--x x |
+-------x xx--+ | x xx |
x x +----+ x NV Domain B x |
+---x xx xx x---+
| xxxx xx +->xx xx
| xxxxxxxx | xx xx
+---+-+ | xx xx
|NVA-1| +--+--+ xx xxx
+-----+ |NVA-2| xxxx xxxx
+-----+ xxxxxFigure 3: VM1 and VM2 in Different NV Domains
NVAs at one site share info and interact with NVAs at other sites, but only in a controlled way 🔐 Policy and access control are expected to be applied at the boundaries between sites (and NVAs), so as to minimize dependencies on external NVAs that could mess with operations inside a site 🛡️ It's an architectural principle that operations involving NVAs at one site are not immediately impacted by failures or errors at another site 💪
(Of course, communication between NVEs in different NV Domains may still be impacted by those failures or errors 🙃) It is a strong requirement that an NVA keeps operating properly for its local NVEs even if external communication is interrupted, for example if communication between a local and remote NVA fails 🫡
At a high level, a federation of interconnected NVAs has some analogies to BGP and Autonomous Systems 🧠 Just like an Autonomous System, NVAs at one site are managed by a single administrative entity and don't interact with external NVAs except as policy allows 🍓 The interface between NVAs at different sites is well defined, so the internal details of operations at one site are mostly hidden from other sites 🙈 Finally, an NVA only peers with other NVAs it has a trusted relationship with, i.e., where a VN is meant to span multiple NVAs 🤝
Here are the reasons for using a federated model 📝
- Provide isolation among NVAs operating at different sites at different geographic locations. 🌍
- Control the quantity and rate of information updates that flow (and have to be processed) between different NVAs in different data centers. 📊
- Control the set of external NVAs (and external sites) a site peers with. A site will only peer with other sites that are cooperating in providing an overlay service. 🤙
- Allow policy to be applied between sites. A site will want to carefully control what info it exports (and to whom), as well as what info it is willing to import (and from whom). 🎛️
- Allow different protocols and architectures to be used for intra-NVA vs. inter-NVA communication. For example, within a single data center, a replicated transaction server using database techniques might be an attractive implementation option for an NVA, and protocols optimized for intra-NVA communication would likely be different from protocols involving inter-NVA communication between different sites. 🗄️
- Allow for optimized protocols rather than using a one-size-fits-all approach. Within a data center, networks tend to have lower latency, higher speed, and higher redundancy compared with WAN links interconnecting data centers. The design constraints and trade-offs for a protocol operating within a data-center network are different from those operating over WAN links. While a single protocol could be used for both cases, there could be advantages to using different and more specialized protocols for the intra- and inter-NVA case. ⚡
9.1. Inter-NVA Peering 🤝
To get NVAs peering with each other, an inter-NVA protocol is needed 📡 That protocol defines what info gets exchanged between NVAs 💌 It's assumed the protocol will be used to share addressing info between data centers, and it has to scale well over WAN links 🌊
10. Control Protocol Work Areas 🧰
The NVO3 architecture has two major distinct entities: NVEs and NVAs 👯 To keep them isolated and independent from each other, the NVO3 architecture calls for well-defined protocols for interfacing between them 🔌 For an individual NVA, the architecture calls for a logically centralized entity that could be implemented in a distributed or replicated fashion 🧩 The IETF may choose to define one or more specific architectural approaches for building individual NVAs, but there's little need to pick exactly one approach and rule out all the others 🤷♀️ An NVA for a single domain will likely be deployed as a single vendor product, so there's not much benefit in standardizing the internal structure of an NVA 🙅♀️
Individual NVAs peer with each other in a federated manner, and the NVO3 architecture calls for a well-defined interface between NVAs 🫂
Finally, a hypervisor-NVE protocol is needed to cover the split-NVE scenario described in Section 4.2 💫
11. NVO3 Data-Plane Encapsulation 📦
When tunneling tenant traffic, NVEs add an encapsulation header to the original tenant packet 🎁 The exact encapsulation to use for NVO3 doesn't seem to be critical 🤔 The main requirement is that the encapsulation supports a Context ID of sufficient size 💜 A number of encapsulations already exist that provide a VN Context of sufficient size for NVO3 ✨ For example, Virtual eXtensible Local Area Network (VXLAN) [RFC7348] has a 24-bit VXLAN Network Identifier (VNI) 🌸 Network Virtualization using Generic Routing Encapsulation (NVGRE) [RFC7637] has a 24-bit Tenant Network ID (TNI) 🦋 MPLS-over-GRE provides a 20-bit label field 🏷️ There's widespread recognition that a 12-bit VN Context would be too small (only 4096 distinct values), but it's generally agreed that 20 bits (1 million distinct values) and 24 bits (16.8 million distinct values) are sufficient for a wide variety of deployment scenarios 🎀
12. Operations, Administration, and Maintenance (OAM) 💅
Okay bestie, the simplicity of operating and debugging overlay networks is going to be critical for successful deployment 🫶
Overlay networks are built on tunnels between NVEs, so the Operations, Administration, and Maintenance (OAM) [RFC6291] framework for overlay networks can borrow from earlier IETF OAM work on tunnel-based networks, specifically L2VPN OAM [RFC6136]. RFC 6136 focuses on Fault Management and Performance Management as the basics of L2VPN service delivery, and leaves the Configuration Management, Accounting Management, and Security Management parts of the Open Systems Interconnection (OSI) Fault, Configuration, Accounting, Performance, and Security (FCAPS) taxonomy [M.3400] for later study 🦋 This section does the same for NVO3 OAM, but those three areas still matter for complete OAM functionality for NVO3 ✨
The relationship between the overlay and underlay networks is something to consider for fault and performance management. A fault in the underlay may show up as fault and/or performance issues in the overlay 🙈 Diagnosing and fixing these issues gets complicated because NVO3 abstracts the underlay network away from the overlay network (e.g., intermediate nodes on the underlay network path between NVEs are hidden from overlay VNs) 💀
So, NVO3-specific OAM techniques, protocol constructs, and tools are needed to see past this abstraction, so we can diagnose and correct problems that show up in the overlay 🔥 Two examples are underlay-aware traceroute [TRACEROUTE-VXLAN] and ping protocol constructs for overlay networks [VXLAN-FAILURE] [NVO3-OVERLAY] 🌸
NVO3-specific tools and techniques are best viewed as complements to (i.e., not as replacements for) single-network tools that apply to the overlay and/or underlay networks 💜 Coordination among the individual network tools (for the overlay and underlay networks) and NVO3-aware, dual-network tools is required to achieve effective monitoring and fault diagnosis. For example, the defect detection intervals and performance measurement intervals ought to be coordinated among all tools involved, so the results stay consistent and comparable 🎀
For further discussion of NVO3 OAM requirements, see [NVO3-OAM] 📖
13. Summary 🌈
This document presents the overall architecture for NVO3, and it's giving clean and organized 💖 The architecture calls for three main areas of protocol work:
- A hypervisor-NVE protocol to support split-NVEs as discussed in Section 4.2
- An NVE-NVA protocol for disseminating VN information (e.g., inner to outer address mappings)
- An NVA-NVA protocol for exchange of information about specific virtual networks between federated NVAs
It should be noted that existing protocols or extensions of existing protocols are applicable 🧚
14. Security Considerations 🔐
Okay, this part is serious bestie 😭 The data plane and control plane described in this architecture will need to address potential security threats.
For the data plane, tunneled application traffic may need protection against being misdelivered, being modified, or having its content exposed to an inappropriate third party. In all cases, encryption between authenticated tunnel endpoints (e.g., via use of IPsec [RFC4301]) and enforcing policies that control which endpoints and VNs are permitted to exchange traffic can be used to mitigate risks 🛡️
For the control plane, a combination of authentication and encryption can be used between NVAs, between the NVA and NVE, as well as between different components of the split-NVE approach. All entities will need to properly authenticate with each other and enable encryption for their interactions as appropriate to protect sensitive information 🔒
Leakage of sensitive information about users or other entities associated with VMs whose traffic is virtualized can also be covered by using encryption for the control-plane protocols and enforcing policies that control which NVO3 components are permitted to exchange control-plane traffic 🙅♀️
Control-plane elements such as NVEs and NVAs need to collect performance and other data in order to carry out their functions. This data can sometimes be unexpectedly sensitive, for example, allowing non-obvious inferences of activity within a VM 👀 This provides a reason to minimize the data collected in some environments, in order to limit potential exposure of sensitive information. As noted briefly in RFC 6973 [RFC6973] and RFC 7258 [RFC7258], there is an inevitable tension between being privacy sensitive and taking into account network operations in NVO3 protocol development. It's a real tradeoff, so no cap, it's a lot 💀
See the NVO3 framework security considerations in RFC 7365 [RFC7365] for further discussion 📚
15. Informative References 📚
[FRAMEWORK-MCAST]
Ghanwani, A., Dunbar, L., McBride, M., Bannai, V., and R.
Krishnan, "A Framework for Multicast in Network
Virtualization Overlays", Work in Progress,
draft-ietf-nvo3-mcast-framework-05, May 2016.
[IEEE.802.1Q]
IEEE, "IEEE Standard for Local and metropolitan area
networks--Bridges and Bridged Networks", IEEE 802.1Q-2014,
DOI 10.1109/ieeestd.2014.6991462,
<http://ieeexplore.ieee.org/servlet/
opac?punumber=6991460>.
[M.3400] ITU-T, "TMN management functions", ITU-T
Recommendation M.3400, February 2000,
<https://www.itu.int/rec/T-REC-M.3400-200002-I/>.
[NVE-NVA] Kreeger, L., Dutt, D., Narten, T., and D. Black, "Network
Virtualization NVE to NVA Control Protocol Requirements",
Work in Progress, draft-ietf-nvo3-nve-nva-cp-req-05, March
2016.
[NVO3-OAM] Chen, H., Ed., Ashwood-Smith, P., Xia, L., Iyengar, R.,
Tsou, T., Sajassi, A., Boucadair, M., Jacquenet, C.,
Daikoku, M., Ghanwani, A., and R. Krishnan, "NVO3
Operations, Administration, and Maintenance Requirements",
Work in Progress, draft-ashwood-nvo3-oam-requirements-04,
October 2015.
[NVO3-OVERLAY]
Kumar, N., Pignataro, C., Rao, D., and S. Aldrin,
"Detecting NVO3 Overlay Data Plane failures", Work in
Progress, draft-kumar-nvo3-overlay-ping-01, January 2014.
[RFC826] Plummer, D., "Ethernet Address Resolution Protocol: Or
Converting Network Protocol Addresses to 48.bit Ethernet
Address for Transmission on Ethernet Hardware", STD 37,
RFC 826, DOI 10.17487/RFC0826, November 1982,
<http://www.rfc-editor.org/info/rfc826>.
[RFC4301] Kent, S. and K. Seo, "Security Architecture for the
Internet Protocol", RFC 4301, DOI 10.17487/RFC4301,
December 2005, <http://www.rfc-editor.org/info/rfc4301>.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, <http://www.rfc-editor.org/info/rfc4364>.
[RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
"Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
DOI 10.17487/RFC4861, September 2007,
<http://www.rfc-editor.org/info/rfc4861>.
[RFC6136] Sajassi, A., Ed. and D. Mohan, Ed., "Layer 2 Virtual
Private Network (L2VPN) Operations, Administration, and
Maintenance (OAM) Requirements and Framework", RFC 6136,
DOI 10.17487/RFC6136, March 2011,
<http://www.rfc-editor.org/info/rfc6136>.
[RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu,
D., and S. Mansfield, "Guidelines for the Use of the "OAM"
Acronym in the IETF", BCP 161, RFC 6291,
DOI 10.17487/RFC6291, June 2011,
<http://www.rfc-editor.org/info/rfc6291>.
[RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J.,
Morris, J., Hansen, M., and R. Smith, "Privacy
Considerations for Internet Protocols", RFC 6973,
DOI 10.17487/RFC6973, July 2013,
<http://www.rfc-editor.org/info/rfc6973>.
[RFC7258] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an
Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May
2014, <http://www.rfc-editor.org/info/rfc7258>.
[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<http://www.rfc-editor.org/info/rfc7348>.
[RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
Kreeger, L., and M. Napierala, "Problem Statement:
Overlays for Network Virtualization", RFC 7364,
DOI 10.17487/RFC7364, October 2014,
<http://www.rfc-editor.org/info/rfc7364>.
[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network
Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
2014, <http://www.rfc-editor.org/info/rfc7365>.
[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
Virtualization Using Generic Routing Encapsulation",
RFC 7637, DOI 10.17487/RFC7637, September 2015,
<http://www.rfc-editor.org/info/rfc7637>.
[TRACEROUTE-VXLAN]
Nordmark, E., Appanna, C., Lo, A., Boutros, S., and A.
Dubey, "Layer-Transcending Traceroute for Overlay Networks
like VXLAN", Work in Progress, draft-nordmark-nvo3-
transcending-traceroute-03, July 2016.
[USECASES]
Yong, L., Dunbar, L., Toy, M., Isaac, A., and V. Manral,
"Use Cases for Data Center Network Virtualization Overlay
Networks", Work in Progress, draft-ietf-nvo3-use-case-15,
December 2016.
[VXLAN-FAILURE]
Jain, P., Singh, K., Balus, F., Henderickx, W., and V.
Bannai, "Detecting VXLAN Segment Failure", Work in
Progress, draft-jain-nvo3-vxlan-ping-00, June 2013.Acknowledgements 🫶
Helpful comments and improvements to this document have come from Alia Atlas, Abdussalam Baryun, Spencer Dawkins, Linda Dunbar, Stephen Farrell, Anton Ivanov, Lizhong Jin, Suresh Krishnan, Mirja Kuehlwind, Greg Mirsky, Carlos Pignataro, Dennis (Xiaohong) Qin, Erik Smith, Takeshi Takahashi, Ziye Yang, and Lucy Yong. Thank you all, you're literally the best 🌟
Authors' Addresses 💌
David Black
Dell EMC
Email: david.black@dell.com
Jon Hudson
Independent
Email: jon.hudson@gmail.com
Lawrence Kreeger
Independent
Email: lkreeger@gmail.com
Marc Lasserre
Independent
Email: mmlasserre@gmail.com
Thomas Narten
IBM
Email: narten@us.ibm.com📦 where this page comes from
- ietf: rfc8014 2016-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms