RFC 7364

Problem Statement: Overlays for Network Virtualization

📅 published October 2014, from ietf

RFC 7364: Problem Statement: Overlays for Network Virtualization 💖

okay bestie, this is the RFC, and here's the title block tea 🫶

  • RFC number: 7364
  • Title: Problem Statement: Overlays for Network Virtualization
  • Category: Informational 🧚 (published for information, not a Standards Track spec)
  • Date: October 2014
  • ISSN: 2070-1721
  • Obsoletes / Updates: no Obsoletes or Updates entry appears in the title block
  • Authors:
    • T. Narten, Ed. (IBM) 💜
    • E. Gray, Ed. (Ericsson) 💜
    • D. Black (EMC) 💜
    • L. Fang (Microsoft) 💜
    • L. Kreeger (Cisco) 💜
    • M. Napierala (AT&T) 💜

Abstract ✨

okay so this one is about multi-tenancy in huge data centers 🏢 and how an overlay-based network virtualization approach can help. the big vibe is traffic isolation, so one tenant's traffic is not visible to any other tenant 🙈. the other big vibe is address space isolation, so different tenants can use the same address space inside different virtual networks 🎀. traffic and address space isolation is achieved by giving each tenant one or more virtual networks, and traffic can only cross into another virtual network in a controlled way (e.g., via a configured router and/or a security gateway) 🚪. on top of that, extra functionality is needed to provision virtual networks, to link a virtual machine's network interface(s) to the right virtual network, and to keep that link working while the VM is activated, migrated, and/or deactivated 🧚. and using an overlay-based approach is what lets all this scale across large network infrastructures 📈.

Status of This Memo

this doc is not an Internet Standards Track specification. it's published just for info 🌸

it's a product of the Internet Engineering Task Force (IETF) 💅. it reflects the consensus of the IETF community, got public review, and was approved for publication by the Internet Engineering Steering Group (IESG). not every doc the IESG approves is a candidate for any level of Internet Standard, so see Section 2 of RFC 5741 for that 🤓.

for the current status, any errata, and how to give feedback, check http://www.rfc-editor.org/info/rfc7364 🔗.

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. all rights reserved ©️.

this doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication. please read these carefully, because they spell out your rights and restrictions for this doc 📜. code components pulled from this doc must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, as described in the Simplified BSD License 💖.

Table of Contents

   1. Introduction ....................................................4
   2. Terminology .....................................................6
   3. Problem Areas ...................................................6
      3.1. Need for Dynamic Provisioning ..............................6
      3.2. Virtual Machine Mobility Limitations .......................7
      3.3. Inadequate Forwarding Table Sizes ..........................7
      3.4. Need to Decouple Logical and Physical Configuration ........7
      3.5. Need for Address Separation between Virtual Networks .......8
      3.6. Need for Address Separation between Virtual Networks and ...8
      3.7. Optimal Forwarding .........................................9
   4. Using Network Overlays to Provide Virtual Networks .............10
      4.1. Overview of Network Overlays ..............................10
      4.2. Communication between Virtual and Non-virtualized
           Networks ..................................................12
      4.3. Communication between Virtual Networks ....................12
      4.4. Overlay Design Characteristics ............................13
      4.5. Control-Plane Overlay Networking Work Areas ...............14
      4.6. Data-Plane Work Areas .....................................15
   5. Related IETF and IEEE Work .....................................15
      5.1. BGP/MPLS IP VPNs ..........................................16
      5.2. BGP/MPLS Ethernet VPNs ....................................16
      5.3. 802.1 VLANs ...............................................17
      5.4. IEEE 802.1aq -- Shortest Path Bridging ....................17
      5.5. VDP .......................................................17
      5.6. ARMD ......................................................18
      5.7. TRILL .....................................................18
      5.8. L2VPNs ....................................................18
      5.9. Proxy Mobile IP ...........................................19
      5.10. LISP .....................................................19
   6. Summary ........................................................19
   7. Security Considerations ........................................19
   8. References .....................................................20
      8.1. Normative Reference .......................................20
      8.2. Informative References ....................................20
   Acknowledgments ...................................................22
   Contributors ......................................................22
   Authors' Addresses ................................................23

1. Introduction ✨

data centers are getting consolidated and outsourced so apps can deploy faster and ops costs go down 💸. this lines up with apps wanting more and more compute, storage, and network resources 📈. to scale all of that, physical resources get abstracted away from their logical representation, which is what people call server, storage, and network virtualization 🧚. virtualization can show up at different layers of computer systems or networks, so it's not just one thing 🌈.

server virtualization is blowing up in data centers 💥. with it, each physical server runs multiple virtual machines (VMs), and each VM has its own operating system, middleware, and applications 💻. virtualization is a major enabler of workload agility, which means any server can host any app, and you get the flexibility to add, shrink, or move services around the physical infrastructure 🦋. server virtualization also brings higher utilization, increased security, reduced user downtime, reduced power usage, etc. ✨.

multi-tenant data centers are taking this and making a whole new kind of hosting: a virtual hosted data center 🏠. in these, each tenant could belong to a different company (for a public provider) or a different department (for an internal company data center) 🏢. every tenant expects a level of security and privacy that keeps their resources separate from everyone else's 🔒. so one tenant's traffic must never be exposed to another, except through carefully controlled interfaces, like a security gateway (e.g., a firewall) 🔥.

to a tenant, a virtual data center looks a lot like a physical one, with end stations attached to a network and services like load balancers and firewalls included 🎀. but unlike a physical data center, the Tenant Systems connect to a virtual network (VN) 🧚. to Tenant Systems, a virtual network looks like a normal network (for example, one providing an Ethernet or L3 service), except that the only end stations connected to it are the ones that belong to that tenant's specific virtual network 🫶.

a tenant is the administrative entity on whose behalf one or more specific virtual network instances, and their associated services (virtual or physical), are managed 👑. in a cloud environment, a tenant would be the customer using a particular virtual network 💅. a tenant might also find it useful to make several different virtual network instances, so there's a one-to-many mapping between tenants and virtual network instances 🌸. a single tenant can run multiple virtual network instances, each one tied to a different service 🍓.

how a virtual network gets implemented doesn't really matter to the tenant 🤷‍♀️. what matters is that the service (Layer 2 (L2) or Layer 3 (L3)) has the right semantics, performance, etc. ✨. it could be built on a pure routed network, a pure bridged network, or a mix of bridged and routed networks 🌈. the key requirement is that each virtual network instance is isolated from the others, with traffic crossing from one virtual network to another only when policy allows it 🚦.

for data center virtualization, there are two key issues that have to be addressed 🧠. first, address space separation between tenants must be supported 🙈. second, it must be possible to place (and migrate) VMs anywhere in the data center, without restricting VM addressing to match the subnet boundaries of the underlying data center network 🦋.

this document lays out the problems you run into when you try to scale the number of isolated virtual networks in a data center 📈. it also covers the issues in managing those virtual networks for operations like virtual network creation/deletion and end-node membership change 🛠️. and finally, it makes the case that an overlay-based approach has a bunch of advantages over traditional, non-overlay approaches 💜. the purpose here is to identify the set of issues that any solution has to address when building multi-tenant data centers 🎯. the goal is to allow the construction of standardized, interoperable implementations, which in turn allows the construction of multi-tenant data centers 🫶.

this document is the problem statement for the "Network Virtualization over Layer 3" (NVO3) Working Group 🧚. NVO3 is focused on building overlay networks that run over an IP (L3) underlay transport network 🌐. NVO3 expects to provide both L2 service and IP service to Tenant Systems, though maybe as two different solutions 🎀. some deployments need an L2 service, others need an L3 service, and some may need both 💖.

Section 2 gives terminology. Section 3 describes the problem space in detail. Section 4 goes deeper into overlay networks. Section 5 reviews related and further work, and Section 6 wraps everything up with a summary 🌸.

2. Terminology ✨

okay so this section is basically our glossary bestie 📖 it says we're using the same terms as [RFC7365], plus a few extras 💅

Overlay Network: a virtual network where the separation of tenants is hidden from the underlying physical infrastructure 🦋 the underlying transport network doesn't need to know about tenancy separation to forward traffic correctly. IEEE 802.1 Provider Backbone Bridging (PBB) [IEEE-802.1Q] is an example of an L2 overlay network ✨ PBB uses MAC-in-MAC encapsulation (where "MAC" means "Media Access Control"), and the underlying transport network only forwards traffic using the Backbone MAC (B-MAC) and Backbone VLAN Identifier (B-VID) in the outer header 🌸 the underlay has no idea about the tenancy separation provided by, for example, a 24-bit Backbone Service Instance Identifier (I-SID). it's giving "the underlay is clueless and it's fine" 💀

C-VLAN: this document refers to Customer VLANs (C-VLANs) as implemented by many routers 🎀 so basically an L2 virtual network identified by a Customer VLAN Identifier (C-VID). an end station (like a VM) in this context that's part of an L2 virtual network will effectively belong to a C-VLAN 🫶 within an IEEE 802.1Q-2011 network, other tags may be used too, but that usage is generally not visible to the end station 👀 and if you want the deep dive on VLANs defined by [IEEE-802.1Q], go check Section 5.3 🧚

this document uses the phrase "virtual network instance" with its ordinary meaning, to represent an instance of a virtual network 💜 its usage may differ from the "VNI" acronym defined in the framework document [RFC7365] ⭐ and the "VNI" acronym is not used in this document at all, so no mixing them up, bestie 🙈

3. Problem Areas 🌈

okay this is the "here's why this stuff is hard" section 😭 the subsections describe aspects of multi-tenant data center networking that cause problems for network infrastructure ✨ different problem aspects may show up depending on the network architecture and scale 🍓

3.1. Need for Dynamic Provisioning

some service providers offer services to multiple customers where the services are dynamic 🔥 the resources assigned to support them have to be able to change quickly as demand changes 💖 and right now it can be hard to provision resources for individual tenants (e.g., QoS) so that those provisioned properties automatically move along when services get shifted around inside the data center to optimize workloads 😩 it's a whole vibe check on the systems 💅

3.2. Virtual Machine Mobility Limitations

okay so a key benefit of server virtualization is virtual machine (VM) mobility 🦋 a VM can be migrated from one server to another live, meaning while it keeps running, without needing to shut down and restart at the new location ✨ a key requirement for live migration is that the VM retains its critical network state at the new location, including its IP and MAC address(es) 🫶 preserving MAC addresses may be necessary, for example, when software licenses are bound to MAC addresses 💀 more generally, any change in the VM's MAC addresses from a move would be visible to the VM and could cause unexpected disruptions 😭 and retaining IP addresses after a move is necessary to stop existing transport connections (e.g., TCP) from breaking and needing to restart 🙅‍♀️

in data center networks, servers usually get IP addresses based on their physical location 🌸 for example, based on the Top-of-Rack (ToR) switch for the server rack, or the C-VLAN configured on the server 🎀 servers can only move to other locations within the same IP subnet 🥺 this constraint isn't a big deal for physical servers, which move infrequently, but it restricts where VMs can be placed and moved within the data center 😤 so any solution for a scalable multi-tenant data center has to let a VM be placed (or moved) anywhere in the data center without being constrained by the subnet boundary concerns of the host servers 👑

3.3. Inadequate Forwarding Table Sizes

today's virtualized environments put extra demands on the forwarding tables of forwarding nodes in the physical infrastructure 💅 the core problem is that location independence means specific end-state information gets pushed into the forwarding system (e.g., /32 host routes in IPv4 networks, or MAC addresses in IEEE 802.3 Ethernet networks) 🧚 in L2 networks, for instance, instead of one address per server, the network infrastructure may have to learn the addresses of the individual VMs (which could range into the hundreds per server) 😱 that increases the demand on a forwarding node's table capacity compared to non-virtualized environments 🫠

3.4. Need to Decouple Logical and Physical Configuration

data center operators need to achieve high utilization of server and network capacity 🔥 for efficient and flexible allocation, operators should be able to spread a virtual network instance across servers in any rack in the data center ✨ it should also be possible to migrate compute workloads to any server anywhere in the network while keeping the workload's addresses 🦋

in lots of network types (e.g., IP subnets, MPLS VPNs, VLANs, etc.), moving servers elsewhere in the network may mean expanding the scope of a portion of the network (e.g., subnet, VPN, VLAN, etc.) beyond its original boundaries 😵‍💫 this can be done, but it needs potentially complex network configuration changes, and in some cases (like a VLAN or L2VPN) it may clash with the desire to keep broadcast domains bounded in size 💀 plus, when VMs migrate, the physical network (e.g., access lists) may need reconfiguring, which can be time consuming and error prone 😩

an important use case is cross-pod expansion 🌈 a pod typically consists of one or more racks of servers with associated network and storage connectivity 🍓 a tenant's virtual network may start off on one pod and, due to expansion, need servers/VMs on other pods 👀 this especially happens when other pods aren't fully utilizing all their resources 💖 this use case requires virtual networks to span multiple pods so every tenant's servers/VMs stay connected 🫶 such expansion can be hard to achieve when tenant addressing is tied to the addressing used by the underlay network, or when the expansion requires the scope of the underlying C-VLAN to grow beyond its original pod boundary 🙃

3.5. Need for Address Separation between Virtual Networks

individual tenants need control over the addresses they use within a virtual network 💜 but it can be problematic when different tenants want to use the same addresses, or even when the same tenant wants to reuse the same addresses in different virtual networks 😬 so virtual networks must let tenants use whatever addresses they want, without worrying about what addresses other tenants or other virtual networks are using 🎀

3.6. Need for Address Separation between Virtual Networks and Infrastructure

same energy as the last one 🙈 a tenant needs to be able to use whatever addresses it wants in a virtual network, independent of what addresses the underlying data center network is using ✨ tenants (and the underlay infrastructure provider) should be able to use whatever addresses make sense for them, without having to worry about collisions between tenant addresses and the addresses used by the underlay data center network 🦋

3.7. Optimal Forwarding

another problem area is the optimal forwarding of traffic between peers that aren't connected to the same virtual network 🧭 this forwarding happens when a host on a virtual network talks to a host that isn't on any virtual network (e.g., an Internet host), and also when a host on a virtual network talks to a host on a different virtual network 💅 a virtual network may have two (or more) gateways for forwarding traffic onto and off of it 🌸 the best choice of gateway may depend on the set of available paths between the communicating peers 🛣️ and the available gateways may not be equally "close" to a given destination 😤 the issue shows up both when a VM is first instantiated on a virtual network and when a VM migrates or moves to a different location 🔁 after a migration, for instance, a VM's best-choice gateway may change, meaning the VM could get better service by switching to the "closer" gateway, which may also improve utilization of network resources 📈

IP implementations in network endpoints typically don't distinguish between multiple routers on the same subnet 🙃 there may only be a single default gateway in use, and any use of multiple routers usually treats all of them as one hop away 😵 routing protocol functionality is constrained by the need to cope with these endpoint limitations 🥲 for example, the Virtual Router Redundancy Protocol (VRRP) has one router serve as the master to handle all outbound traffic 👑 this can get especially acute when the virtual network spans multiple data centers 🌍 because a VM is likely to get significantly better service when forwarding external traffic through a local router, compared to using a router at a remote data center 💖

the optimal forwarding problem applies to both outbound and inbound traffic 🔄 for outbound traffic, the choice of outbound router determines the path of outgoing traffic from the VM, which may be sub-optimal after a VM move 🙁 for inbound traffic, the location of the VM within its IP subnet isn't visible to the routers beyond the virtual network 👀 so the routing infrastructure has no info on which of the two externally visible gateways leading into the virtual network would be the better choice for reaching a particular VM 🫠

the issue gets even messier when middleboxes (e.g., load balancers, firewalls, etc.) have to be traversed 🧱 middleboxes may hold session state that must be preserved for ongoing communication, and traffic has to keep flowing through the middlebox no matter which router is "closest" 🔒

4. Using Network Overlays to Provide Virtual Networks 🦋

okay so virtual networks exist to isolate a tenant's traffic from everyone else's 🫶 (or even from traffic inside the same tenant network that needs isolating 👀) there are two main characteristics of virtual networks, and they're both iconic 👑

  1. Virtual networks isolate the address space used in one virtual network from the address space used by another virtual network 🌈 the same network addresses may be used in different virtual networks at the same time 😌 and the address space used by a virtual network is independent from the one used by the underlying physical network 🧚
  2. Virtual networks limit the scope of packets sent on the virtual network 💅 packets sent by Tenant Systems attached to a virtual network are delivered as expected to other Tenant Systems on that same virtual network, and they may exit a virtual network only through controlled exit points, such as a security gateway 🚪 likewise, packets sourced from outside the virtual network may enter it only through controlled entry points, such as a security gateway 🔐 main character energy for sure 💖

4.1. Overview of Network Overlays 💖

Okay so to fix all the problems from Section 3 (bestie we just went through those 🥲), the move is a network overlay 🦋

The idea is lowkey simple. Each virtual network instance gets built as an overlay. The original packet gets encapsulated by the first-hop network device, called a Network Virtualization Edge (NVE), and then tunneled to a remote NVE. The encapsulation tells you where the device doing the decapsulation is (aka the egress NVE for the tunneled packet) before the original packet gets delivered to the endpoint. The rest of the network just forwards based on the encapsulation header and can be totally oblivious to the payload inside 🙈 no cap.

Overlays run on what's commonly called a "map-and-encap" architecture 🧚 When it comes to processing and forwarding packets, there are three distinct, logically separable steps:

  1. The first-hop overlay device does a mapping operation that figures out where the encapsulated packet should go to reach its intended destination VM. Specifically, the mapping function takes the destination address (either L2 or L3) of a packet received from a VM and maps it to the matching destination address of the egress NVE device. That destination address is the underlay address of the NVE device doing the decapsulation, and it's an IP address.
  2. Once the mapping is sorted, the ingress overlay NVE device wraps the received packet in an overlay header.
  3. Last step: actually forward the (now encapsulated) packet to its destination. The underlay (aka the IP network) forwards it based entirely on its outer address. When it arrives, the egress overlay NVE device decapsulates the original packet and delivers it to the intended recipient VM ✨

Each of these steps is logically distinct, though an implementation might combine them for efficiency or other reasons. Also, in L3 BGP/VPN terminology, these steps are commonly known as "forwarding" or "virtual forwarding" 💅

The first-hop NVE device can be a traditional switch or router, or the virtual switch living inside a hypervisor. And the endpoint can be a VM or a physical server. Examples of architectures based on network overlays include BGP/MPLS IP VPNs [RFC4364], Transparent Interconnection of Lots of Links (TRILL) [RFC6325], the Locator/ID Separation Protocol (LISP) [RFC6830], and Shortest Path Bridging (SPB) [IEEE-802.1aq] 🌈

In the data plane, the overlay header gives you a place to carry either the virtual network identifier or an identifier that's locally significant to the edge device. Either way, the identifier in the overlay header says which specific virtual network the data packet belongs to. Since a virtual data center can support both routed and bridged semantics, the original packet inside the overlay header can be an Ethernet frame or just the IP packet 🎀

A key aspect of overlays is decoupling the "virtual" MAC and/or IP addresses VMs use from the physical network infrastructure and the infrastructure IP addresses the data center uses. If a VM changes location, the overlay edge devices just update their mapping tables to reflect the VM's new spot within the data center's infrastructure space. Because an overlay network is used, a VM can be located anywhere in the data center the overlay reaches, without caring about the traditional constraints imposed by the underlay network, like the C-VLAN scope or the IP subnet scope 🦋👑

Multi-tenancy is supported by isolating one virtual network instance's traffic from another's. Traffic from one virtual network instance cannot be delivered to another instance without (conceptually) exiting the instance and entering the other via an entity (e.g., a gateway) with connectivity to both virtual network instances. Without a gateway entity, tenant traffic stays isolated within each individual virtual network instance 🧸

Overlays are designed to let a set of VMs sit within a single virtual network instance, whether that instance provides a bridged network or a routed network 🌸

4.2. Communication between Virtual and Non-virtualized Networks 🙃

Not all communication is going to be between devices on virtualized networks. Devices using overlays will keep accessing devices and using services on non-virtualized networks, whether those are in the data center, on the public Internet, or at remote/branch campuses. Any virtual network solution must be capable of interoperating with existing routers, VPN services, load balancers, intrusion-detection services, firewalls, etc., on external networks.

Communication between devices on a virtual network and devices on non-virtualized networks is handled architecturally by specialized gateway devices. They receive packets from a virtualized network, decapsulate them, process them as regular (i.e., non-virtualized) traffic, and then forward them to their proper destination (and vice versa) 🚪

Lots of implementation approaches are possible. Overlay gateway functionality could be combined with other network functionality in a single network device that implements the overlay functionality and then forwards traffic between other internal components that implement things like full router service, load balancing, firewall support, VPN gateway, etc. 🧁

4.3. Communication between Virtual Networks 🤝

Communication between devices on different virtual networks is handled architecturally by adding specialized interconnect functionality among the otherwise isolated virtual networks. For a virtual network providing an L2 service, that interconnect could be IP forwarding configured as part of the "default gateway" for each virtual network. For a virtual network providing L3 service, the interconnect could be IP forwarding configured as part of routing between IP subnets, or it could be based on configured inter-virtual-network traffic policies.

In both cases, the implementation of the interconnect functionality could be distributed across the NVEs and could be combined with other network functionality (e.g., load balancing and firewall support) that is applied to traffic forwarded between virtual networks 🌟

4.4. Overlay Design Characteristics 📝

Below are some of the characteristics of environments that must be taken into account by the overlay technology.

  1. Highly distributed systems: The overlay should work in an environment where there could be many thousands of access switches (e.g., residing within the hypervisors) and many more Tenant Systems (e.g., VMs) connected to them. This leads to a distributed mapping system that puts a low overhead on the overlay tunnel endpoints. 🌐
  2. Many highly distributed virtual networks with sparse membership: Each virtual network could be highly dispersed inside the data center. Also, along with the expectation of many virtual networks, the number of Tenant Systems connected to any one virtual network is expected to be relatively low; therefore, the percentage of NVEs participating in any given virtual network would also be expected to be low. For this reason, efficient delivery of multi-destination traffic within a virtual network instance should be taken into consideration. 🌸
  3. Highly dynamic Tenant Systems: Tenant Systems connected to virtual networks can be very dynamic, both in terms of creation/deletion/power-on/power-off and in terms of mobility from one access device to another. 💫
  4. Be incrementally deployable, without necessarily requiring major upgrade of the entire network: The first-hop device (or end system) that adds and removes the overlay header may require new software and may require new hardware (e.g., for improved performance). The rest of the network should not need to change just to enable the use of overlays. 🧩
  5. Work with existing data center network deployments without requiring major changes in operational or other practices: For example, some data centers have not enabled multicast beyond link-local scope. Overlays should be capable of leveraging underlay multicast support where appropriate, but not require its enablement in order to use an overlay solution. 🫶
  6. Network infrastructure administered by a single administrative domain: This is consistent with operation within a data center, and not across the Internet. 🏢

4.5. Control-Plane Overlay Networking Work Areas ✨

Okay so, there are three separate work areas on the control-plane side that you'd need to make an overlay solution actually work 🔥. Each one is about a different "on-the-wire" protocol, where different entities talk to each other 🫶.

The first area is how an NVE learns the address information it needs to build and keep up its mapping tables, so it knows where to send encapsulated packets 📬. One option is to build those tables purely by learning, the way 802.1 networks do 🧠. Another option is a dedicated control-plane protocol 💅. Reusing an existing protocol to maintain the mapping tables has some perks, but there's a catch: a lot of NVEs are going to live inside hypervisors, so the CPU and memory they can spare for this stuff is limited 😭.

Architecturally, the address-mapping problem splits into two pieces that can be pulled apart 🦋. The first piece is a back-end Network Virtualization Authority (NVA), which is in charge of distributing and maintaining the mapping info for the whole overlay system 👑. In this document, "NVA" just means whatever entity hands out the answers, no matter how it figured them out 🤷‍♀️. The second piece is the on-the-wire protocols an NVE uses when it talks to the NVA 💌.

So the first two work areas are: describing the NVA function, and defining the NVA-NVE interactions 🌸.

The back-end NVA could be built in lots of different ways, with high performance, high resiliency, failover, and so on 🍓. One model is a traditional, centralized "directory-based" database that uses replicated instances for reliability and failover 🗂️. Another model uses an existing routing protocol, possibly extended (for example BGP, IS-IS, and so on) 🛣️. Since different architectural models are possible, it makes sense to have one standard protocol for NVE-NVA interaction, while letting the NVA itself use whatever protocols and approaches it likes 🎀. Keeping the two separate means NVEs can work with either kind of NVA described above without any drama ✨. Separate protocols could also let an NVE be simple, only asking the NVA about the mapping entries it actually needs 🧚. And the NVA (plus its protocols) can keep evolving over time without messing up the NVEs much at all 🌈.

The third work area is about attaching and detaching VMs (or Tenant Systems [RFC7365], more generally) from a specific virtual network instance 🖥️. When a VM attaches, the NVE links that VM to a specific overlay so it can tunnel traffic going to or coming from the VM 🚇. When a VM disconnects, the NVE should tell the NVA that the Tenant System to NVE address mapping isn't valid anymore 📢. And if that VM was the last one left in the virtual network, the NVE can also shut down any tunnels it used to deliver tenant multi-destination packets within the VN to that NVE 🧹. If the NVE and the hypervisor are on separate physical devices with an access network in between, a standardized protocol might be needed 🤔.

To recap, there are three potential areas of work 💖. The first is implementing the NVA function itself, plus any protocols it needs (for example, if it's built in a distributed way) 🏗️. The second is the interaction between the NVA and the NVEs 🤝. The third is the protocols for attaching and detaching a VM from a particular virtual network instance 🔌. All three matter for building solutions that scale and work together across vendors 👑💅.

4.6. Data-Plane Work Areas

The data plane carries the encapsulated packets for Tenant Systems 📦. The data-plane encapsulation header carries a VN Context identifier [RFC7365] that tells you which virtual network the data packet belongs to 🏷️. There are already lots of encapsulation or tunneling protocols out there that can be reused ♻️. Unless there's a strong and compelling reason, it wouldn't really seem necessary or helpful to invent yet another encapsulation format just for NVO3 🙅‍♀️.

Next up, the related IETF and IEEE work! The subsections below talk about it, but they aren't meant to cover every IETF or IEEE effort related to data centers, so don't treat them as the full picture 🙈. Each one tackles particular limitations of today's data center networks 🧩. Scaling is a recurring theme across all of them, and so are multi-tenancy and VM mobility 📈. Comparing and evaluating how far each listed work area has gotten is out of scope for this document 🚫. The point of this section is to give interested readers a reference to go find more 🔎. And note that NVO3 is scoped to running over an IP/L3 underlay network 🌊.

5.1. BGP/MPLS IP VPNs

BGP/MPLS IP VPNs [RFC4364] support multi-tenancy, VPN traffic isolation, address overlapping, and address separation between tenants and network infrastructure 🏰. The BGP/MPLS control plane distributes the VPN labels and the tenant IP addresses that identify the tenants (or, more precisely, the particular VPN/virtual network) and those tenant IP addresses 🏷️. Enterprise L3 VPNs have been shown to scale to thousands of VPNs and millions of VPN prefixes, which is honestly iconic 🤯. BGP/MPLS IP VPNs are already deployed in some large enterprise data centers 🏢. The catch when you try to deploy them in data center environments is whether it's practical to run BGP in the data center, especially reaching into servers or hypervisors 😬. There may be workforce skill set issues, equipment support issues, and potential new scaling challenges too 🫠. A mix of BGP and lighter-weight IP signaling protocols, such as the Extensible Messaging and Presence Protocol (XMPP), has been proposed to extend these solutions into the data center environment [END-SYSTEM] 💡. That approach still gets the built-in VPN features and their rich policy support, and it's especially useful for inter-tenant connectivity 🎀.

5.2. BGP/MPLS Ethernet VPNs

Ethernet Virtual Private Networks (E-VPNs) [EVPN] give you an emulated L2 service where each tenant gets its own Ethernet network over a shared IP or MPLS infrastructure 💜. A BGP/MPLS control plane distributes the tenant MAC addresses and the MPLS labels that identify the tenants and their MAC addresses 📇. Inside that control plane, a 32-bit Ethernet tag identifies the broadcast domains (VLANs) tied to a given L2 VLAN service instance, and those Ethernet tags get mapped to VLAN IDs that the tenant understands at the service edges 🗺️. So any VLAN-based limitation on the customer site is tied to an individual tenant service edge, which means a much higher level of scalability 📈✨. Interconnection between tenants is also allowed, just in a controlled way 🔐.

VM mobility [MOBILITY] brings in the idea of a combined L2/L3 VPN service, so individual virtual machines (VMs) can move between data centers connected over a shared IP or MPLS infrastructure 🚚🦋.

5.3. 802.1 VLANs

VLANs are a super well-known construct in networking 🫶. They provide an L2 service over a physical network where the tenant forwarding information is part of the physical network infrastructure itself 🧱. A VLAN is an L2 bridging construct that gives you the virtual network semantics mentioned earlier: a MAC address can be unique within a VLAN, but it isn't necessarily unique across VLANs 🔑. Traffic scoped to a VLAN (including broadcast and multicast) can stay inside the VLAN it came from 🏠. Traffic going from one VLAN to another typically involves router (L3) processing 🚦. The forwarding table lookup may be keyed on {VLAN, MAC address} tuples 🔍.

VLANs are purely an L2 bridging construct, and the VLAN identifiers ride along with the data frames so every forwarding point knows which VLAN a frame belongs to 🎟️. There are several types of VLANs available today, and they can even be used together for network virtualization 🌈. The C-VLAN, Service VLAN (S-VLAN), and Backbone VLAN (B-VLAN) IDs [IEEE-802.1Q] are all 12 bits 🙃. The 24-bit I-SID [IEEE-802.1aq] supports more than 16 million virtual networks, which is giving main character energy 👑🔥.

5.4. IEEE 802.1aq -- Shortest Path Bridging

Shortest Path Bridging (SPB) [IEEE-802.1aq] is an overlay based on IS-IS that runs over L2 Ethernets 🛤️. SPB supports multipathing and fixes a bunch of shortcomings of the original Ethernet Spanning Tree Protocol 🙌. Shortest Path Bridging Mac (SPBM) uses IEEE 802.1ah PBB (MAC-in-MAC) encapsulation and supports a 24-bit I-SID, which can identify virtual network instances 🎀. SPBM gives you multipathing and makes it easy to create or update virtual networks 💫.

SPBM extends IS-IS so that link-state routing happens among the core SPBM nodes, which means those core nodes don't need bridge learning to talk to each other 🧠. Learning is still used to build and maintain the mapping tables at the edge nodes, so they can encapsulate Tenant System traffic for transport across the SPBM core 📦.

SPB is compatible with all the other 802.1 standards, so you can borrow other features like VSI Discovery Protocol (VDP), Operations, Administration, and Maintenance (OAM), or scalability solutions 🧰💖.

5.5. VDP

VDP is the Virtual Station Interface (VSI) Discovery and Configuration Protocol specified by IEEE P802.1Qbg [IEEE-802.1Qbg] 🪪. It's a protocol that supports associating a VSI with a port 🔗.

VDP runs between the end station (for example, a server running a hypervisor) and its adjacent switch (the device on the edge of the network) 🖧. It's used, for example, to tell the switch that a virtual machine (virtual station) is moving, which means it's designed for VM migration 🚚💜.

5.6. ARMD

The Address Resolution for Massive numbers of hosts in the Data center (ARMD) WG looked at data center scaling issues, focusing on address resolution, and wrote up a problem statement document [RFC6820] 📝. An overlay-based approach may fix some of the "pain points" ARMD raised (for example, better support for multi-tenancy) 🩹. But analysis is still needed to understand the scaling trade-offs of an overlay-based approach compared with existing approaches 🧮. And on the other hand, existing IP-based approaches like proxy ARP may help with some concerns 🙋‍♀️.

5.7. TRILL

TRILL is a network protocol that provides an Ethernet L2 service to end systems, and it's designed to run over any L2 link type 🌍. TRILL sets up forwarding paths using IS-IS routing and encapsulates traffic within its own TRILL header 🎁. As originally defined, TRILL only supports the standard (and limited) 12-bit C-VID identifier 🥲. Work to extend TRILL to support more than 4094 VLANs has recently wrapped up and is defined in [RFC7172] 🎉✨.

5.8. L2VPNs 🌸

Okay so the IETF has specified a bunch of ways to connect L2 domains together, and those live in the L2VPN Working Group 💅. BUT, and this is a big but, that group has mostly focused on provider-provisioned L2 VPNs, where the service provider helps with management and provisioning of the VPN. A lot of their target setups are about carrying L2 traffic over WANs 🌍. Overlay approaches like the ones in this document are meant to be used inside data centers, where the data center operator manages the overlay network instead of some outside party 🏢. Overlays can run across the Internet too, sure, but they reach way deeper into the data center (think all the way up to and including hypervisors 🖥️) and cover a huge number of machines inside the data center itself. Main character energy for real ✨.

Other L2VPN approaches, like the Layer 2 Tunneling Protocol (L2TP) [RFC3931], need a lot of tunnel state at the encapsulating and decapsulating endpoints 😭. Overlays need less tunnel state than those other approaches, and that's the whole reason overlays can scale to hundreds of thousands of endpoints 🚀🦋. It's assumed that smaller switches (like virtual switches in hypervisors, or the adjacent devices that VMs connect to) will be part of the overlay network and be responsible for encapsulating and decapsulating packets 🧚.

5.9. Proxy Mobile IP 💜

Proxy Mobile IP [RFC5213] [RFC5844] uses the Generic Routing Encapsulation (GRE) Key Field [RFC5845] [RFC6245], but not in a way that supports multi-tenancy 🙈. Not it, sorry Proxy Mobile IP 💀.

5.10. LISP 🍓

LISP [RFC6830] basically gives you an IP-over-IP overlay 🎀. The internal addresses are end station identifiers, and the outer IP addresses show where the end station sits in the core IP network topology 🗺️. The LISP overlay header has a 24-bit Instance ID, which is what lets it support overlapping inner IP addresses. Iconic design, honestly 👑.

6. Summary 🦋

This document argued that network virtualization using overlays helps address a bunch of issues that data centers are running into as they scale up 📈. Careful study of today's data center problems is also needed so we can write proper requirements and standard solutions 🧐.

This document identifies three potential control protocol work areas 🌈. The first one is about a back-end NVA and how it learns and distributes the mapping information that NVEs use when they process tenant traffic. The second is about the protocol an NVE would use to talk to the back-end NVA to get that mapping information. The third is about the interactions that happen when a VM attaches to or detaches from a specific virtual network instance 💖.

There are a number of approaches that provide some, if not all, of the semantics virtual networks are supposed to have ✨. Each approach needs to be analyzed in detail to figure out how well it satisfies the requirements 🔍.

7. Security Considerations 🔒

Okay, real talk, this one is serious 😤. This document describes the problem space for virtualizing networks in complex, large-scale data center networks, so it doesn't introduce any security risks by itself. BUT it's very clear that security has to be a consideration for any solution proposed for this problem space 🛡️.

Solutions will need to address both data-plane and control-plane security concerns 💜.

In the data plane, keeping virtual network traffic isolated from other virtual networks is the big one 🚧. For NVO3, this isolation may be based on VN identifiers that aren't involved in underlay network packet forwarding between overlay edges (NVEs). Using a VN identifier in the overlay means the underlay network plays a smaller role in isolating virtual networks, compared to approaches where VN identifiers are part of packet forwarding (like the 802.1 VLANs in Section 5.3) 🎀.

On top of isolation, assurances against spoofing, snooping, transit modification, and denial of service are other important data-plane considerations 😱. Some limited environments may even need confidentiality 🔐.

In the control plane, the main security concern is making sure an unauthorized party can't compromise the control-plane protocol in ways that improperly affect the data plane 🚨. Some environments may also care about confidentiality of the control plane 🫣.

More generally, denial-of-service concerns may also come into play 🌊. For example, a tenant on one virtual network could eat up excessive network resources in a way that degrades services for tenants on other virtual networks 💀.

8. References 📚

8.1. Normative Reference

8.2. Informative References

Acknowledgments 🫶

Huge thanks to Lou Berger, John Drake, Ilango Ganga, Ariel Hendel, Vinit Jain, Petr Lapukhov, Thomas Morin, Benson Schliesser, Qin Wu, Xiaohu Xu, Lucy Yong, and many others on the NVO3 mailing list for helpful comments and improvements to this document 💕.

Special thanks to Janos Farkas for his persistence and his many detailed comments about the lack of precision in the text relating to IEEE 802.1 technologies 🙌.

Contributors 🌟

Dinesh Dutt and Murari Sridharin were the original co-authors of the Internet-Draft that led to the BoF that formed the NVO3 WG 🧁. That original draft eventually became the basis for this document 🌷.

Authors' Addresses 💌

Thomas Narten (editor)
IBM
Research Triangle Park, NC
United States
EMail: narten@us.ibm.com


Eric Gray (editor)
Ericsson
EMail: eric.gray@ericsson.com


David Black
EMC Corporation
176 South Street
Hopkinton, MA  01748
United States
EMail: david.black@emc.com


Luyuan Fang
Microsoft
5600 148th Ave NE
Redmond, WA  98052
United States
EMail: lufang@microsoft.com


Lawrence Kreeger
Cisco
170 W. Tasman Avenue
San Jose, CA  95134
United States
EMail: kreeger@cisco.com


Maria Napierala
AT&T
200 S. Laurel Avenue
Middletown, NJ  07748
United States
EMail: mnapierala@att.com
📦 where this page comes from