RFC 7365

Framework for Data Center (DC) Network Virtualization

📅 published October 2014, from ietf

Framework for Data Center (DC) Network Virtualization 🎀

okay bestie, this is RFC 7365, published October 2014 by the IETF 💅

  • Request for Comments: 7365
  • Category: Informational (it's published for information, not as a standards track spec 📚)
  • ISSN: 2070-1721
  • Authors: M. Lasserre and F. Balus (Alcatel-Lucent), T. Morin (Orange), N. Bitar (Verizon), Y. Rekhter (Juniper)
  • Date: October 2014
  • Obsoletes / Updates: this header doesn't have an Obsoletes or Updates line, so this RFC doesn't list any 🙈

Abstract ✨

this doc gives you a framework for Data Center (DC) Network Virtualization over Layer 3 (NVO3) 🌸 and it defines a reference model plus the logical components you need to design a solution. main character energy fr 👑

Status of This Memo 💜

this document is not an Internet Standards Track specification. it's published for informational purposes, so no pressure to treat it like a rulebook 🫶

it's a product of the Internet Engineering Task Force (IETF) and it represents the consensus of the IETF community. it got public review and was approved for publication by the Internet Engineering Steering Group (IESG). ngl, not every IESG-approved doc is up for any level of Internet Standard. see Section 2 of RFC 5741 for the deets 🧚

for the current status, any errata, and how to give feedback, check http://www.rfc-editor.org/info/rfc7365 🦋

Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. 🔥

this document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the publication date. please actually read those, bestie, they're your rights and restrictions 🙏

if you extract Code Components from this document, they must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions. they come without warranty, as described in the Simplified BSD License ⚠️

Table of Contents 🍓

   1. Introduction ....................................................4
      1.1. General Terminology ........................................4
      1.2. DC Network Architecture ....................................7
   2. Reference Models ................................................8
      2.1. Generic Reference Model ....................................8
      2.2. NVE Reference Model .......................................10
      2.3. NVE Service Types .........................................11
           2.3.1. L2 NVE Providing Ethernet LAN-Like Service .........11
           2.3.2. L3 NVE Providing IP/VRF-Like Service ...............11
      2.4. Operational Management Considerations .....................12
   3. Functional Components ..........................................12
      3.1. Service Virtualization Components .........................12
           3.1.1. Virtual Access Points (VAPs) .......................12
           3.1.2. Virtual Network Instance (VNI) .....................12
           3.1.3. Overlay Modules and VN Context .....................14
           3.1.4. Tunnel Overlays and Encapsulation Options ..........14
           3.1.5. Control-Plane Components ...........................14
                  3.1.5.1. Distributed vs. Centralized
                           Control Plane .............................14
                  3.1.5.2. Auto-provisioning and Service Discovery ...15
                  3.1.5.3. Address Advertisement and Tunnel Mapping ..15
                  3.1.5.4. Overlay Tunneling .........................16
      3.2. Multihoming ...............................................16
      3.3. VM Mobility ...............................................17
   4. Key Aspects of Overlay Networks ................................17
      4.1. Pros and Cons .............................................18
      4.2. Overlay Issues to Consider ................................19
           4.2.1. Data Plane vs. Control Plane Driven ................19
           4.2.2. Coordination between Data Plane and Control Plane ..19
           4.2.3. Handling Broadcast, Unknown Unicast, and
                  Multicast (BUM) Traffic ............................20
           4.2.4. Path MTU ...........................................20
           4.2.5. NVE Location Trade-Offs ............................21
           4.2.6. Interaction between Network Overlays and
                  Underlays ..........................................22
   5. Security Considerations ........................................22
   6. Informative References .........................................24
   Acknowledgments ...................................................26
   Authors' Addresses ................................................26

1. Introduction ✨

this doc gives you a framework for Data Center (DC) Network Virtualization over Layer 3 (NVO3) tunnels 🦋 the whole point is to help standardize protocols and mechanisms that support large-scale network virtualization in data centers. it's giving big infrastructure energy 🏗️💖

[RFC7364] explains the rationale for using overlay networks to build large multi-tenant data center networks. so basically, why overlays at all? that's the tea ☕ compute, storage, and network virtualization all show up together in these huge data centers, supporting a large number of communication domains and end systems. literally so much going on 😭

the rest of this document covers reference models and functional components for data center overlay networks, plus a discussion of the technical issues that still have to be sorted out 🧚💜

1.1. General Terminology

okay bestie, let's get the vocab down first 💅 this doc uses the following terms ✨

NVO3 Network: an overlay network that gives a Layer 2 (L2) or Layer 3 (L3) service to Tenant Systems over an L3 underlay network, built using the architecture and protocols defined by the NVO3 Working Group 🦋

Network Virtualization Edge (NVE): the network entity sitting at the edge of an underlay network that does the L2 and/or L3 network virtualization stuff 🌸 its underlay-facing side uses the underlying L3 network to tunnel tenant frames to and from other NVEs, and its tenant-facing side sends and receives Ethernet frames to and from individual Tenant Systems. an NVE could live inside a virtual switch in a hypervisor, be a physical switch or router, or be a Network Service Appliance. it could also be split across multiple devices 🧚

Virtual Network (VN): a logical abstraction of a physical network that provides L2 or L3 network services to a set of Tenant Systems. it's also known as a Closed User Group (CUG) 🎀

Virtual Network Instance (VNI): a specific instance of a VN from the perspective of an NVE 💜

Virtual Network Context (VN Context) Identifier: a field in an overlay encapsulation header that says which specific VN the packet belongs to. the egress NVE uses the VN Context identifier to deliver the packet to the correct Tenant System 🔥 this identifier can be locally significant or globally unique, either one is fine 😌

Underlay or Underlying Network: the network that provides connectivity among NVEs, and that NVO3 packets get tunneled over. an NVO3 packet carries an NVO3 overlay header followed by a tenant packet 📦 the underlay doesn't need to know it's carrying NVO3 packets, and addresses on the underlay show up as "outer addresses" in encapsulated NVO3 packets 🫶 the underlay can totally use a different protocol and address family from the overlay. with NVO3, the underlay is IP 🌈

Data Center (DC): a physical complex housing physical servers, network switches and routers, network service appliances, and networked storage ✨ its whole purpose is to provide application, compute, and/or storage services. one of those services is virtualized infrastructure data center services, also called "Infrastructure as a Service" ☁️

Virtual Data Center (Virtual DC): a container for virtualized compute, storage, and network services 👑 a virtual DC is tied to a single tenant and can hold multiple VNs and Tenant Systems connected to one or more of those VNs 💅

Virtual Machine (VM): a software version of a physical machine that runs programs as if they were on a real, non-virtualized machine 🍓 apps usually have no idea they're on a VM instead of a "bare metal" host or server. some systems do offer para-virtualization, though, where the OS or app knows about the virtualization so it can optimize things 🙈

Hypervisor: software running on a server that lets multiple VMs run on the same physical server 🖥️ it manages and shares computation, memory, storage, and network connectivity with the VMs it hosts. hypervisors often embed a virtual switch (see below) 🔌

Server: a physical end-host machine that runs user applications 💖 a standalone (or "bare metal") server runs a conventional operating system hosting a single-tenant application. a virtualized server runs a hypervisor supporting one or more VMs 🌸

Virtual Switch (vSwitch): a function inside a hypervisor (usually done in software) that gives forwarding services similar to a physical Ethernet switch 🦋 a vSwitch forwards Ethernet frames between VMs on the same server, or between a VM and a physical Network Interface Card (NIC) that connects the server to a physical Ethernet switch or router. it also enforces network isolation between VMs that aren't allowed to talk to each other by policy (e.g., by honoring VLANs) 🔒 a vSwitch may be bypassed when an NVE is enabled on the host server 🧚

Tenant: the customer using a virtual network and any associated resources (e.g., compute, storage, and network) 💼 a tenant could be an enterprise, or a department/organization within an enterprise 🎀

Tenant System: a physical or virtual system that can act as a host or as a forwarding element like a router, switch, firewall, etc. 🧩 it belongs to a single tenant and connects to one or more VNs of that tenant 💜

Tenant Separation: isolating traffic of different tenants so that traffic from one tenant isn't visible to or delivered to another tenant, unless policy allows it 🫣 it also covers address space separation, meaning different tenants can use the same address space without conflict 🙌

Virtual Access Points (VAPs): logical connection points on the NVE for connecting a Tenant System to a virtual network 🌈 Tenant Systems connect to VNIs at an NVE through VAPs. VAPs can be physical ports or virtual ports identified through logical interface identifiers (e.g., a VLAN ID or an internal vSwitch Interface ID connected to a VM) 🔗

End Device: a physical device that connects directly to the DC underlay network ⭐ this is different from a Tenant System, which connects to a tenant VN 🤍 an End Device is administered by the DC operator instead of a tenant, and it's part of the DC infrastructure 🏗️ an End Device may implement NVO3 technology to support NVO3 functions. examples include hosts (e.g., a server or server blade), storage systems (e.g., file servers and iSCSI storage systems), and network devices (e.g., a firewall, load-balancer, and IPsec gateway) 🛡️

Network Virtualization Authority (NVA): the entity that provides reachability and forwarding information to NVEs 🧠

1.2. DC Network Architecture

okay now the big picture 🖼️ a generic architecture for data centers looks like this:

                            ,---------.
                          ,'           `.
                         (  IP/MPLS WAN )
                          `.           ,'
                            `-+------+'
                             \      /
                      +--------+   +--------+
                      |   DC   |+-+|   DC   |
                      |gateway |+-+|gateway |
                      +--------+   +--------+
                            |       /
                            .--. .--.
                          (    '    '.--.
                        .-.' Intra-DC     '
                       (     network      )
                        (             .'-'
                         '--'._.'.    )\ \
                         / /     '--'  \ \
                        / /      | |    \ \
               +--------+   +--------+   +--------+
               | access |   | access |   | access |
               | switch |   | switch |   | switch |
               +--------+   +--------+   +--------+
                  /     \    /    \     /      \
               __/_      \  /      \   /_      _\__
         '--------'   '--------'   '--------'   '--------'
         :  End   :   :  End   :   :  End   :   :  End   :
         : Device :   : Device :   : Device :   : Device :
         '--------'   '--------'   '--------'   '--------'

         Figure 1: A Generic Architecture for Data Centers

an example of a multi-tier DC network architecture is shown in Figure 1 ✨ it gives you a view of the physical components inside a DC 🏢

a DC network is usually made up of intra-DC networks and network services, plus inter-DC network and network connectivity services 🌐

DC networking elements can act as strict L2 switches and/or provide IP routing capabilities, including network service virtualization 🔀

in some DC architectures, some tier layers could provide L2 and/or L3 services 💅 also, some tier layers may be collapsed, and Internet connectivity, inter-DC connectivity, and VPN support may be handled by a smaller number of nodes 🫠 still, you can assume the network functional blocks in a DC fit into the architecture in Figure 1 🥰

the following components can be present in a DC:

  • Access switch: a hardware-based Ethernet switch that aggregates all the Ethernet links from the End Devices in a rack. it's the entry point into the physical DC network for the hosts 🚪 it may also provide routing functionality, virtual IP network connectivity, or Layer 2 tunneling over IP, for instance. access switches are usually multihomed to aggregation switches in the Intra-DC network 🧵 a typical example is a Top-of-Rack (ToR) switch. other deployment scenarios may use an intermediate Blade Switch before the ToR, or an End-of-Row (EoR) switch, to provide similar functions to a ToR 🍭
  • Intra-DC Network: a network made up of high-capacity core nodes (Ethernet switches/routers) 💪 core nodes may provide virtual Ethernet bridging and/or IP routing services 🌀
  • DC Gateway (DC GW): the gateway to the outside world, providing DC interconnect and connectivity to Internet and VPN customers 🌍 in the current DC network model, this may be simply a router connected to the Internet and/or an IP VPN/L2VPN PE. some network implementations may dedicate DC GWs to different connectivity types (e.g., one DC GW for Internet and another for VPN) 🛣️

note that End Devices may be single-homed or multihomed to access switches 🫶

2. Reference Models 🌟

2.1. Generic Reference Model ✨

okay so Figure 2 is the big picture 🌸 It's a DC reference model for network virtualization overlays, where NVEs act like a logical interconnect between the Tenant Systems that belong to one specific VN 🫶

         +--------+                                    +--------+
         | Tenant +--+                            +----| Tenant |
         | System |  |                           (')   | System |
         +--------+  |    .................     (   )  +--------+
                     |  +---+           +---+    (_)
                     +--|NVE|---+   +---|NVE|-----+
                        +---+   |   |   +---+
                        / .    +-----+      .
                       /  . +--| NVA |--+   .
                      /   . |  +-----+   \  .
                     |    . |             \ .
                     |    . |   Overlay   +--+--++--------+
         +--------+  |    . |   Network   | NVE || Tenant |
         | Tenant +--+    . |             |     || System |
         | System |       .  \ +---+      +--+--++--------+
         +--------+       .....|NVE|.........
                               +---+
                                 |
                                 |
                       =====================
                         |               |
                     +--------+      +--------+
                     | Tenant |      | Tenant |
                     | System |      | System |
                     +--------+      +--------+

      Figure 2: Generic Reference Model for DC Network Virtualization
                                 Overlays

So to get reachability info, NVEs can just chat with each other directly through a control-plane protocol. In that setup, there's a control-plane module living in every single NVE 💅

Another option is that NVEs talk to an external Network Virtualization Authority (NVA) to get their reachability and forwarding info. Then a protocol is used between the NVEs and the NVA(s) to swap that information 🧚

Also, NVAs can be grouped into clusters for redundancy and scalability, and they can look like one logically centralized controller from the outside 👑 When that happens, inter-NVA communication is needed to sync state among the nodes in a cluster or to share info across clusters. The info that gets exchanged between NVAs in the same cluster could be different from what gets exchanged across clusters. Iconic multitasking, honestly 🔥

Now, a Tenant System can hook up to an NVE in a couple of ways:

  • locally, by being co-located in the same End Device
  • remotely, via a point-to-point connection or a switched network

When an NVE is co-located with a Tenant System, the Tenant System's state can be figured out without any protocol help. Like, the operational status of a VM can just be passed along through a local API 🙈 When an NVE is connected to a Tenant System remotely, the state of the Tenant System or the NVE has to be exchanged directly, or through a management entity, using a control-plane protocol or API, or directly via a data-plane protocol. Lots of options, bestie 🦋

Quick reality check: the functional pieces in Figure 2 don't always line up neatly with the physical pieces in Figure 1 🎀 For example, an End Device can be a server blade with VMs and a virtual switch. A VM can be a Tenant System, and the NVE functions might be handled by the host server. In that case, the Tenant System and the NVE function are co-located. Another example: the End Device is the Tenant System, and the NVE function is implemented by the connected ToR. In that case, they are not co-located. Main character energy, the mapping is flexible ✨

Underlay nodes use L3 technologies to interconnect the NVE nodes. These nodes forward based on the outer L3 header info, and they generally don't keep state for each tenant service. Albeit, some applications (e.g., multicast) may need control-plane or forwarding-plane info that's about a tenant, a group of tenants, a tenant service, or a set of services that belong to one or more tenants. So mechanisms to control how much state gets kept in the underlay may be needed 🌈

2.2. NVE Reference Model

Figure 3 shows the NVE reference model 💜 One or more VNIs can be instantiated on an NVE. A Tenant System talks to its matching VNI through a VAP. The overlay module handles the tunneling overlay functions (e.g., encapsulation and decapsulation of tenant traffic, tenant identification, and mapping, etc.).

                     +-------- L3 Network -------+
                     |                           |
                     |        Tunnel Overlay     |
         +------------+---------+       +---------+------------+
         | +----------+-------+ |       | +---------+--------+ |
         | |  Overlay Module  | |       | |  Overlay Module  | |
         | +---------+--------+ |       | +---------+--------+ |
         |           |VN Context|       | VN Context|          |
         |           |          |       |           |          |
         |  +--------+-------+  |       |  +--------+-------+  |
         |  | |VNI|   .  |VNI|  |       |  | |VNI|   .  |VNI|  |
    NVE1 |  +-+------------+-+  |       |  +-+-----------+--+  | NVE2
         |    |   VAPs     |    |       |    |    VAPs   |     |
         +----+------------+----+       +----+-----------+-----+
              |            |                 |           |
              |            |                 |           |
             Tenant Systems                 Tenant Systems

                  Figure 3: Generic NVE Reference Model

Note that some NVE functions (e.g., data-plane and control-plane functions) may live in one device, or they may be implemented separately in different devices 🫶

2.3. NVE Service Types

An NVE provides different types of virtualized network services to multiple tenants, i.e., an L2 service or an L3 service ✨ An NVE may be capable of providing both L2 and L3 services for a tenant, so it's a two-in-one situation. This section defines the service types and their associated attributes 🍓

2.3.1. L2 NVE Providing Ethernet LAN-Like Service

An L2 NVE implements Ethernet LAN emulation, which is an Ethernet-based multipoint service similar to an IETF Virtual Private LAN Service (VPLS) [RFC4761][RFC4762] or Ethernet VPN [EVPN] service. The Tenant Systems look like they're interconnected by a LAN environment over an L3 overlay, and that's the whole vibe 🦋 So an L2 NVE provides a per-tenant virtual switching instance (L2 VNI), plus L3 (IP/MPLS) tunneling encapsulation of tenant Media Access Control (MAC) frames across the underlay. Note that the control plane for an L2 NVE could be implemented locally on the NVE, or in a separate control entity 💖

2.3.2. L3 NVE Providing IP/VRF-Like Service

An L3 NVE provides a virtualized IP forwarding service, similar to IETF IP VPN (e.g., BGP/MPLS IP VPN [RFC4364]) from a service definition perspective 🌸 In other words, an L3 NVE provides a per-tenant forwarding and routing instance (L3 VNI), plus L3 (IP/MPLS) tunneling encapsulation of tenant IP packets across the underlay. Note that routing could be performed locally on the NVE, or in a separate control entity 🧚

2.4. Operational Management Considerations

NVO3 services are overlay services over an IP underlay 🌈

As far as the IP underlay is concerned, existing IP Operations, Administration, and Maintenance (OAM) facilities are used 💅

With regard to the NVO3 overlay, both L2 and L3 services can be offered. It's expected that existing fault and performance OAM facilities will be used. Sections 4.1 and 4.2.6 provide further discussion of additional fault and performance management issues to consider 👑

As far as configuration goes, the DC environment is driven by the need to bring new services up rapidly, and it's typically very dynamic, especially in the context of virtualized services. So it is critical to automate the configuration of NVO3 services 🔥

3. Functional Components ✨

This section breaks the network virtualization architecture down into the functional components described in Figure 3, to make it easier to discuss solution options for these components 💜

3.1. Service Virtualization Components

3.1.1. Virtual Access Points (VAPs)

Tenant Systems are connected to VNIs through Virtual Access Points (VAPs) 🎀

VAPs can be physical ports or virtual ports identified through logical interface identifiers (e.g., VLAN ID and internal vSwitch Interface ID connected to a VM) 🦋

3.1.2. Virtual Network Instance (VNI)

A VNI is a specific VN instance on an NVE. Each VNI defines a forwarding context that contains reachability information and policies 🌸

3.1.3. Overlay Modules and VN Context

Mechanisms for identifying each tenant service are required, so multiple tenant services can be overlaid at the same time over the same underlay L3 network topology 😭 In the data plane, each NVE, upon sending a tenant packet, must be able to encode the VN Context for the destination NVE, in addition to the L3 tunneling information (e.g., source IP address identifying the source NVE and destination IP address identifying the destination NVE, or MPLS label). This lets the destination NVE identify the tenant service instance, and therefore process and forward the tenant packet appropriately 🧚

The overlay module provides tunneling overlay functions: tunnel initiation/termination, as in the case of stateful tunnels (see Section 3.1.4), and/or encapsulation/decapsulation of frames from the VAPs/L3 underlay 💅

In a multi-tenant context, tunneling aggregates frames from/to different VNIs. Tenant identification and traffic demultiplexing are based on the VN Context identifier 🫶

The following approaches can be considered:

  • VN Context identifier per Tenant: This is a globally unique (on a per-DC administrative domain) VN identifier used to identify the corresponding VNI. Examples of such identifiers in existing technologies are IEEE VLAN IDs and Service Instance IDs (I-SIDs) that identify virtual L2 domains when using IEEE 802.1Q and IEEE 802.1ah, respectively. Note that multiple VN identifiers can belong to a tenant 👑
  • One VN Context identifier per VNI: Each VNI value is automatically generated by the egress NVE, or by a control plane associated with that NVE, and is usually distributed by a control-plane protocol to all the related NVEs. An example of this approach is the use of per-VRF MPLS labels in IP VPN [RFC4364]. The VNI value is therefore locally significant to the egress NVE 🌈
  • One VN Context identifier per VAP: A value locally significant to an NVE is assigned, and usually distributed by a control-plane protocol, to identify a VAP. An example of this approach is the use of per-CE MPLS labels in IP VPN [RFC4364] 🍓

Note that when using one VN Context per VNI or per VAP, an additional global identifier (e.g., a VN identifier or name) may be used by the control plane to identify the tenant context 💖

3.1.4. Tunnel Overlays and Encapsulation Options

okay so once the VN Context identifier is tucked into the frame, an L3 tunnel encapsulation takes over and carries it to the destination NVE 🚀✨

You've got options here, bestie! IP tunneling like Generic Routing Encapsulation (GRE), the Layer 2 Tunneling Protocol (L2TP), and IPsec are all on the table, plus MPLS tunneling. Tunneling can be stateless or stateful, and the difference is kinda everything 💅

Stateless tunneling is the chill one: you just wrap the tenant packet in whatever header it needs to get across the underlay (think IP tunneling over an IP underlay) and that's it 🦋

Stateful tunneling means the tunnel endpoints (the NVEs) are keeping tunnel state. An ingress NVE can send tenant packets through those tunnels to an egress NVE by slapping on the matching tunneling header. That state can be configured by hand or set up dynamically, no drama 🌸

Solutions should say which tunneling technology they use and whether it's stateful or stateless. In this doc, though, "tunneling" and "tunneling encapsulation" mean the same thing: wrapping a tenant packet with the tunneling header it needs to travel from an ingress NVE to an egress NVE across the underlay 🎀

ngl, stateful tunneling (especially when there's a lot of configuration involved) comes with real management overhead and scale constraints 😭 Keep that in your back pocket.

When confidentiality is required, the use of opportunistic security [OPPSEC] can be used as a stateless tunneling solution. Main character move for privacy, iykyk 🔐

3.1.5. Control-Plane Components

3.1.5.1. Distributed vs. Centralized Control Plane

Control-plane and management-plane entities can be centralized or distributed, and both approaches have been used a LOT in the past 🌈 The routing model of the Internet is the iconic example of a distributed approach. Transport networks, on the other hand, have usually gone with a centralized approach for managing transport paths 💜

You can also mix the two and go hybrid 🫶 A global view of network state has lots of perks, but it doesn't stop you from using distributed protocols inside the network. Centralized models give you a place to keep global state and push it out to the network. Pair them with distributed protocols and you can get better efficiency, improved reliability, and robustness. The balance between centralized and distributed depends on the domain and/or deployment constraints, so it's a vibe check for each network ✨

3.1.5.2. Auto-provisioning and Service Discovery

NVEs have to figure out the right VNI for each Tenant System 🧚 That info usually comes from external entities. For example, when a VM is the Tenant System, the VM orchestration systems hold this info, because they're the only ones who know which VM belongs to which tenant 👑

A mechanism for passing this information to the NVE is required. VAPs need to get created and mapped to the correct VNI. Depending on the implementation, this control interface could be an auto-discovery protocol running between Tenant Systems and their local NVE, or it could go through management entities. Either way, appropriate security and authentication mechanisms are required to make sure Tenant System information can't be spoofed or altered. This is a critical piece for integrity and tenant isolation in the system, so don't skip it 🔥

NVEs may learn reachability information for VNIs on other NVEs through a control protocol that swaps that info among NVEs, or through a management-control entity 🍓

3.1.5.3. Address Advertisement and Tunnel Mapping

When traffic hits an ingress NVE on a VAP, a lookup happens to figure out which NVE or local VAP the packet should go to. If it's headed to another NVE, the packet gets encapsulated with a tunnel header that carries the destination info (a destination IP address or an MPLS label) of the egress NVE. Intermediate nodes between the ingress and egress NVEs switch or route the traffic based on that tunnel destination info 🦋

The key step here is working out which destination NVE the packet should be tunneled to 🎀 NVEs are responsible for keeping forwarding or mapping tables that hold the bindings between destination VM and egress NVE addresses. There are a few ways to fill these tables: control plane driven, management plane driven, or data plane driven.

When a control-plane protocol is distributing address reachability and tunneling information, auto-provisioning and service discovery could be handled by that same protocol 💅 In that setup, they can be combined with (or even inferred from) the address advertisement and the associated tunnel mapping. Plus, a control-plane protocol that carries both MAC and IP addresses gets rid of the need for the Address Resolution Protocol (ARP), which helps address one of the issues with explosive ARP handling as discussed in [RFC6820] 🙈

3.1.5.4. Overlay Tunneling

For overlay tunneling, depending on the tunneling technology used to encapsulate Tenant System packets, it may be enough to have one or more local NVE addresses assigned and used in the source and destination fields of a tunneling encapsulation header 🌸 Other info that's part of the tunneling encapsulation header might also need to be configured. Sometimes local NVE configuration is enough, and other times some tunneling-related info has to be shared among NVEs. What needs sharing depends on the technology. Potential info includes tunnel identity, encapsulation type, and/or tunnel resources 🧚

In certain cases, like when IP multicast is used in the underlay, tunnels that interconnect NVEs may need to be set up. When tunneling info has to be exchanged or shared among NVEs, a control-plane protocol may be required. For example, it might be necessary to share active/standby status, up/down status, or pruning/grafting info for multicast tunnels, and more 🌈

Also, a control plane may be required to set up the tunnel path for some tunneling technologies. This applies to both unicast and multicast tunneling ✨

3.2. Multihoming

Multihoming techniques can make an NVO3 network way more reliable 💖 It's also super important to account for physical diversity in an NVO3 network so you don't end up with single points of failure 😭

Multihoming can be turned on at a bunch of different nodes: from Tenant Systems into ToRs, from ToRs into core switches/routers, and from core nodes into DC GWs 🌐

The NVO3 underlay nodes (that's everything from NVEs to DC GWs) rely on IP routing techniques or MPLS re-rerouting capabilities to reroute traffic when something fails 🔥

When a Tenant System is co-located with the NVE, the Tenant System is effectively single-homed to the NVE through a virtual port 🎀 When they're separated, the Tenant System connects to the NVE through a logical L2 construct like a VLAN, and it can be multihomed to various NVEs 🦋 An NVE may provide an L2 service to the end system or an l3 service 💅 An NVE may also be multihomed to the next layer in the DC, at L2 or L3. When an NVE provides an L2 service and isn't co-located with the end system, loop-avoidance techniques must be used. When the NVE provides L3 service, similar dual-homing techniques can be used too 🧚 If an NVE is giving L3 service to the end system, there might not be any dynamic routing protocol running between them. The end system can also be multihomed to multiple physically separated L3 NVEs over multiple interfaces, so when one of the links to an NVE fails, the other interfaces can still reach the end system 🫶

External connectivity from a DC can be handled by two or more DC gateways 🌸 Each gateway gives access to external networks like VPNs or the Internet. A gateway may also connect to two or more edge nodes in the external network for redundancy. When a connection to an upstream node is lost, the alternative connection gets used and the failed route is withdrawn ✨

3.3. VM Mobility

In DC environments that use VM technologies, a big feature is that VMs can move from one server to another, whether that's in the same L2 physical domain or a different one (within or across DCs), and do it seamlessly 🦋💖

A VM can be moved from one server to another in a stopped or suspended state (that's "cold" VM mobility) or in a running/active state (that's "hot" VM mobility) 🌈 With "hot" mobility, the VM's L2 and L3 addresses need to be preserved. With "cold" mobility, it may be desired to preserve at least the VM's L3 addresses 🎀

Solutions for keeping connectivity while a VM moves are necessary for "hot" mobility. That means connectivity among VMs has to be preserved. For L2 VNs, for instance, ARP caches get updated accordingly 💅

When a VM moves, NVE policies that define connectivity among VMs must be maintained 🔐

During VM mobility, the path to the VM's default gateway is expected to assure adequate QoS for VM applications, meaning QoS that matches the expected service-level agreement for those applications 👑

4. Key Aspects of Overlay Networks ✨

The intent of this section is to highlight specific issues that proposed overlay solutions need to address 🧚💜

4.1. Pros and Cons 🎀

Okay so, an overlay network is basically a layer of virtual network topology sitting on top of the physical network. 🦋

Overlay networks have some seriously iconic perks:

  • Unicast tunneling state management and association of Tenant Systems reachability are handled at the edge of the network (at the NVE). Intermediate transport nodes are unaware of such state. Note that when multicast is enabled in the underlay network to build multicast trees for tenant VNs, there would be more state related to tenants in the underlay core network. 💅
  • Tunneling is used to aggregate traffic and hide tenant addresses from the underlay network, so it offers the advantage of minimizing the amount of forwarding state required within the underlay network. 🧚
  • Decoupling of the overlay addresses (MAC and IP) used by VMs from the underlay network provides tenant separation and separation of the tenant address spaces from the underlay address space. 👑
  • Overlay networks support a large number of virtual network identifiers. 🌈

Overlay networks also create some challenges, and ngl, these are where it gets messy 💀

  • Overlay networks typically have no control of underlay networks and lack underlay network information (e.g., underlay utilization):
    • Overlay networks and/or their associated management entities typically probe the network to measure link or path properties, such as available bandwidth or packet loss rate. It is difficult to accurately evaluate network properties. It might be preferable for the underlay network to expose usage and performance information. 🙈
    • Miscommunication or lack of coordination between overlay and underlay networks can lead to an inefficient usage of network resources.
    • When multiple overlays co-exist on top of a common underlay network, the lack of coordination between overlays can lead to performance issues and/or resource usage inefficiencies.
  • Traffic carried over an overlay might fail to traverse firewalls and NAT devices. 😭
  • Multicast service scalability: Multicast support may be required in the underlay network to address tenant flood containment or efficient multicast handling. The underlay may also be required to maintain multicast state on a per-tenant basis or even on a per-individual multicast flow of a given tenant. Ingress replication at the NVE eliminates that additional multicast state in the underlay core, but depending on the multicast traffic volume, it may cause inefficient use of bandwidth. 🔥

4.2. Overlay Issues to Consider ✨

4.2.1. Data Plane vs. Control Plane Driven 🌸

In the case of an L2 NVE, it is possible to dynamically learn MAC addresses against VAPs. It is also possible that such addresses be known and controlled via management or a control protocol for both L2 NVEs and L3 NVEs. Dynamic data-plane learning implies that flooding of unknown destinations be supported, and hence implies that broadcast and/or multicast be supported, or that ingress replication be used as described in Section 4.2.3. Multicasting in the underlay network for dynamic learning may lead to significant scalability limitations. Specific forwarding rules must be enforced to prevent loops from happening. This can be achieved using a spanning tree, a shortest path tree, or a split-horizon mesh.

It should be noted that the amount of state to be distributed is dependent upon network topology and the number of virtual machines. Different forms of caching can also be utilized to minimize state distribution between the various elements. The control plane should not require an NVE to maintain the locations of all the Tenant Systems whose VNs are not present on the NVE. The use of a control plane does not imply that the data plane on NVEs has to maintain all the forwarding state in the control plane. 🧠

4.2.2. Coordination between Data Plane and Control Plane 🫶

For an L2 NVE, the NVE needs to be able to determine MAC addresses of the Tenant Systems connected via a VAP. This can be achieved via data-plane learning or a control plane. For an L3 NVE, the NVE needs to be able to determine the IP addresses of the Tenant Systems connected via a VAP.

In both cases, coordination with the NVE control protocol is needed, such that when the NVE determines that the set of addresses behind a VAP has changed, it triggers the NVE control plane to distribute this information to its peers. 🔁

4.2.3. Handling Broadcast, Unknown Unicast, and Multicast (BUM) Traffic 📣

There are several options to support the packet replication needed for broadcast, unknown unicast, and multicast. Typical methods include:

  • Ingress replication
  • Use of underlay multicast trees

There is a bandwidth vs. state trade-off between the two approaches. Depending upon the degree of replication required (i.e., the number of hosts per group) and the amount of multicast state to maintain, trading bandwidth for state should be considered. 🍓

When the number of hosts per group is large, the use of underlay multicast trees may be more appropriate. When the number of hosts is small (e.g., 2-3) and/or the amount of multicast traffic is small, ingress replication may not be an issue. ✨

Depending upon the size of the data center network, and hence the number of (S,G) entries, and also the duration of multicast flows, the use of underlay multicast trees can be a challenge. 🥲

When flows are well known, it is possible to pre-provision such multicast trees. However, it is often difficult to predict application flows ahead of time, so programming (S,G) entries for short-lived flows could be impractical. 🙃

A possible trade-off is to use shared multicast trees in the underlay, as opposed to dedicated multicast trees. 🌈

4.2.4. Path MTU 📏

When using overlay tunneling, an outer header is added to the original frame. This can cause the MTU of the path to the egress tunnel endpoint to be exceeded.

It is usually not desirable to rely on IP fragmentation for performance reasons. Ideally, the interface MTU as seen by a Tenant System is adjusted such that no fragmentation is needed. 🎯

It is possible for the MTU to be configured manually or to be discovered dynamically. Various Path MTU discovery techniques exist in order to determine the proper MTU size to use:

Tenant Systems rely on ICMP messages to discover the MTU of the end-to-end path to their destination. This method is not always possible, such as when traversing middleboxes (e.g., firewalls) that disable ICMP for security reasons. 🚧

  • Extended Path MTU Discovery techniques such as those defined in [RFC4821]

Tenant Systems send probe packets of different sizes and rely on confirmation of receipt, or lack thereof, from receivers to allow a sender to discover the MTU of the end-to-end paths. 🔍

While it could also be possible to rely on the NVE to perform segmentation and reassembly operations without relying on the Tenant Systems to know about the end-to-end MTU, this would lead to undesired performance and congestion issues, as well as significantly increase the complexity of hardware NVEs required for buffering and reassembly logic. 😵‍💫

Preferably, the underlay network should be designed in such a way that the MTU can accommodate the extra tunneling and possibly additional NVO3 header encapsulation overhead. 🛣️

4.2.5. NVE Location Trade-Offs 📍

In the case of DC traffic, traffic originated from a VM is native Ethernet traffic. This traffic can be switched by a local virtual switch or ToR switch, and then by a DC gateway. The NVE function can be embedded within any of these elements. 🏢

There are several criteria to consider when deciding where the NVE function should happen:

  • Processing and memory requirements
    • Datapath (e.g., lookups, filtering, and encapsulation/decapsulation)
    • Control-plane processing (e.g., routing, signaling, and OAM) and where specific control-plane functions should be enabled
  • FIB/RIB size
  • Multicast support
    • Routing/signaling protocols
    • Packet replication capability
    • Multicast FIB
  • Fragmentation support
  • QoS support (e.g., marking, policing, and queuing)
  • Resiliency

4.2.6. Interaction between Network Overlays and Underlays 🤝

When multiple overlays co-exist on top of a common underlay network, resources (e.g., bandwidth) should be provisioned to ensure that traffic from overlays can be accommodated and QoS objectives can be met. Overlays can have partially overlapping paths (nodes and links). 🌀

Each overlay is selfish by nature. It sends traffic so as to optimize its own performance without considering the impact on other overlays, unless the underlay paths are traffic engineered on a per-overlay basis to avoid congestion of underlay resources. 😤

Better visibility between overlays and underlays, or general coordination in placing overlay demands on an underlay network, may be achieved by providing mechanisms to exchange performance and liveliness information between the underlay and overlay(s), or by the use of such information by a coordination system. Such information may include:

  • Performance metrics (throughput, delay, loss, jitter) such as defined in [RFC3148], [RFC2679], [RFC2680], and [RFC3393]. 📊
  • Cost metrics 💸

5. Security Considerations 🔐

okay bestie, security is where things get serious, so buckle up 💅 There are three points of view when we think about security for NVO3. 🎀 First, the service a provider runs over NVO3 for a tenant has to meet the security requirements they agreed on together. Second, a network doing NVO3 has to be able to trust the virtual network identity attached to packets it gets from a tenant. Third, an NVO3 network has to think about the security that comes with running as an overlay on top of the underlay network. 🧚

To meet a tenant's security needs, the NVO3 service has to deliver packets from the tenant to the destination(s) they point at, both in the overlay network and in external networks. 🦋 The NVO3 service gives you data confidentiality through data separation. ✨ Using both VNIs and tunneling of tenant traffic by NVEs keeps NVO3 data in its own separate context, so it stays apart from every other tenant's traffic. Super iconic 👑 The infrastructure that supports an NVO3 service (think management systems, NVEs, NVAs, and the underlay networks in between) should be limited to authorized access, so data integrity can be expected. 🛡️ And if a tenant wants its data to stay confidential, the Tenant System can choose to encrypt the data before sending it into the NVO3 service. 💜

Next up: an NVO3 service has to be able to verify the VNI it gets on a packet from the tenant. 😭 To make that happen, it's not just tenant data that needs to be secured, but also NVO3 control data. That means control traffic between NVAs and NVEs, between NVAs, and between NVEs. 🫶 NVEs and NVAs are the stars of NVO3, so it's critical that secure access to them is ensured, so no unauthorized access is possible. 🔥 As we talked about in Section 3.1.5.2, identifying Tenant Systems relies on state that's often provided by management systems (for example, a VM orchestration system in a virtualized environment). 🌸 Secure access to those management systems also has to be ensured. When an NVE receives data from a Tenant System, the tenant's identity needs to be verified, so we know they're allowed to access the corresponding VN. 🍓 In some cases, you can do this by matching incoming packets against specific VAPs. In other situations, authentication might be necessary. 💀 Once that verification is done, the packet gets let into the NVO3 overlay, and no integrity protection is provided on the overlay packet encapsulation (for example, the VNI, the destination NVE, and so on). 🙈 > That last bit is a big deal, so keep it in mind when you're designing your setup 🎀

Since an NVO3 service can run across all kinds of underlay networks, data encryption is needed to make sure packets are delivered correctly when the underlay network isn't trusted to provide at least data integrity. 🌈 It's also a good idea to limit the types of information (like the topology information we covered in Section 4.2.6) that can be swapped between an NVO3 service and underlay networks, based on the security requirements they agreed on. 🔒 Periodt 💅

6. Informative References 📚

Here's the reading list, bestie 🧚 These are the references that the document points to, with all their details:

Acknowledgments 💖

Shoutout to the authors, and also to the people who helped out with this document: Dimitrios Stiliadis, Rotem Salomonovitch, Lucy Yong, Thomas Narten, Larry Kreeger, and David Black. 🫶 Literal legends ✨

Authors' Addresses 🦋

Marc Lasserre
Alcatel-Lucent
EMail: marc.lasserre@alcatel-lucent.com


Florin Balus
Alcatel-Lucent
777 E. Middlefield Road
Mountain View, CA 94043
United States
EMail: florin.balus@alcatel-lucent.com


Thomas Morin
Orange
EMail: thomas.morin@orange.com


Nabil Bitar
Verizon
50 Sylvan Road
Waltham, MA 02145
United States
EMail: nabil.n.bitar@verizon.com


Yakov Rekhter
Juniper
EMail: yakov@juniper.net

That's the whole RFC, bestie! 🌈👑 Ate, served, and left no crumbs 💅

📦 where this page comes from