RFC 8394
Split Network Virtualization Edge (Split-NVE) Control-Plane Requirements
RFC 8394: Split Network Virtualization Edge (Split-NVE) Control-Plane Requirements 🦋
Okay bestie, this one is from the Internet Engineering Task Force (IETF) 💅 and it was published in May 2018 🌸. It's numbered RFC 8394 and it's labelled Informational, so it's sharing ideas, not setting a rule you have to follow ✨. The ISSN is 2070-1721. The authors are Y. Li and D. Eastlake 3rd from Huawei Technologies, L. Kreeger from Arrcus, Inc., T. Narten from IBM, and D. Black from Dell EMC 👑. The title block doesn't list any RFCs it obsoletes or updates, so it's a fresh start, no cap 🧚.
Abstract ✨
Okay so in the Split Network Virtualization Edge (Split-NVE) setup, the NVE's jobs get split between a server and a piece of outside network gear called an "External NVE" 🎀. The control-plane stuff that lives on the server sits in control software, which can be part of hypervisor or container-management software. To keep things simple, this document just calls the hypervisor the "location" of that software 💜.
One or more control-plane protocols run between a hypervisor and its External NVE(s) 🫶. The hypervisor uses them to pass its virtual-machine networking state over to the External NVE(s) so they can handle it from there 🔥. This doc shows what that kind of control-plane signaling protocol needs to do, and it lays out the high-level requirements 💖. It also sums up virtual-machine states and how they change, so the protocol requirements make way more sense 🌈.
Status of This Memo 🌸
This document is not an Internet Standards Track specification; it is published for informational purposes.
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841.
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8394.
Copyright Notice 💅
Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
Table of Contents 📖
1. Introduction ....................................................3
1.1. Terminology ................................................4
1.2. Target Scenarios ...........................................6
2. VM Lifecycle ....................................................7
2.1. VM Creation Event ..........................................8
2.2. VM Live Migration Event ....................................8
2.3. VM Termination Event .......................................9
2.4. VM Pause, Suspension, and Resumption Events ...............10
3. Hypervisor-to-NVE Control-Plane Protocol Functionality .........10
3.1. VN_Connect and VN_Disconnect ..............................10
3.2. TSI Associate and Activate ................................12
3.3. TSI De-Associate and Deactivate ...........................15
4. Hypervisor-to-NVE Control-Plane Protocol Requirements ..........16
5. VDP Applicability and Enhancement Needs ........................17
6. Security Considerations ........................................19
7. IANA Considerations ............................................20
8. References .....................................................21
8.1. Normative References ......................................21
8.2. Informative References ....................................22
Appendix A. VDP Illustrations (per IEEE 802.1Q) (for Information
Only) .................................................23
Acknowledgements ..................................................25
Authors' Addresses ................................................261. Introduction ✨
Okay, so in the Split Network Virtualization Edge (Split-NVE) architecture shown in Figure 1, the NVE's functionality is split between an end device that supports virtualization and an external network device called an "External NVE" 🌸. The part of the NVE that lives on the end device is called the "tNVE" (terminal-side NVE), and the part on the External NVE is called the "nNVE" (network-side NVE) in this document 🦋. Overlay encapsulation and decapsulation are usually offloaded to the nNVE on the External NVE 💜.
+------------ Split-NVE ---------+
| |
| |
+-----------------|-----+ |
| +---------------|----+| |
| | +--+ \|/ || |
| | |V |TSI +-------+ || +------|-------------+
| | |M |-----+ | || | \|/ |
| | +--+ | | || |+--------+ |
| | +--+ | tNVE | ||-------------------|| | |
| | |V |TSI | | || || nNVE | |
| | |M |-----| | || || | |
| | +--+ +-------+ || |+--------+ |
| | || +--------------------+
| +-----Hypervisor-----+|
+-----------------------+
End Device External NVE
Figure 1: Split-NVE StructureThe tNVE is usually part of a hypervisor or container, and/or a virtual switch in a virtualized end device 🎀. This document says "hypervisor" all the time when it talks about the Split-NVE setup, where part of the NVE gets offloaded to a separate device from the "hypervisor" that holds a VM (Virtual Machine) connected to a VN (Virtual Network) 🧚. In this context, "hypervisor" covers any device type where part of the NVE gets offloaded like this, for example a Network Service Appliance or a Linux Container ✨.
The Network Virtualization over Layer 3 (NVO3) problem statement [RFC7364] talks about needing a control-plane protocol (or protocols) to fill each NVE with the state it needs to do its jobs 💅. In one scenario, an NVE gives overlay encapsulation and decapsulation packet-forwarding services to Tenant Systems that live on the same end device as the NVE (for example, when the NVE is built into a hypervisor or a Network Service Appliance) 🌈. In those cases, there's no need for a standardized protocol between the hypervisor and the NVE, since the two talk through software on a single device 🙈. But in the Split-NVE architecture scenarios shown in Figures 2 through 4 (see Section 1.2), one or more control-plane protocols between a hypervisor and its associated External NVE(s) are required, so the hypervisor can send the VM's networking states to the NVE(s) for further handling 🔥. This protocol is an NVE-internal protocol and runs between the tNVE and nNVE logical entities 🫶. It's the protocol mentioned in the "third work area" text in Section 4.5 of the NVO3 problem statement [RFC7364] 👑.
VM states and state transitions get summarized in this document, showing the events where the NVE needs to take specific actions 💖. Those events might match up with what the control-plane signaling protocol or protocols need to do between the tNVE and the nNVE in the Split-NVE scenario 🦋. The high-level requirements are listed in Section 4 ✨.
To describe the requirements, this document uses VMs as an example of Tenant Systems, even though a VM is just one kind of Tenant System that can connect to a VN 💜. For example, a service instance inside a Network Service Appliance is another kind of Tenant System, and so are systems running on OS-level virtualization technologies like containers 🌸. The fact that VMs have lifecycles (they can be created and destroyed, moved, and started or stopped) leads to a general set of protocol requirements 🎀. Most of those apply to other forms of Tenant Systems too, although not every requirement applies to every form 🧚.
Section 2 describes VM states and state transitions across the VM's lifecycle 💅. Section 3 introduces hypervisor-to-NVE control-plane protocol functionality that comes from VM operations and network events 🌈. Section 4 outlines the requirements the control-plane protocol needs to meet to deliver that functionality 🫶.
1.1. Terminology 📚
Okay bestie, the BCP 14 key words are the big ones here. "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are to be read as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they show up in all capitals, just like here. 🎀
This document uses the same terminology as [RFC7365], so you already know the vibe. It also defines some extra terms of its own. Let's go 💅
Split-NVE: A type of NVE (Network Virtualization Edge) where the functionalities are split across an end device that supports virtualization and an external network device. 🦋
tNVE: Terminal-side NVE. This is the part of the Split-NVE functionalities that lives on the end device supporting virtualization. The tNVE talks to a Tenant System through an internal interface inside that end device. 💖
nNVE: Network-side NVE. This is the part of the Split-NVE functionalities that lives on the network device which is directly or indirectly connected to the end device holding the matching tNVE. The nNVE normally does the encapsulation to and decapsulation from the overlay network. ✨
External NVE: The physical network device that contains the nNVE. 🌸
Hypervisor: The logical collection of software, firmware, and/or hardware that lets you create and run server or service appliance virtualization. The tNVE sits under a hypervisor. The word "hypervisor" is used loosely in this document to mean the end device that supports the virtualization. For simplicity, we also use "hypervisor" to cover both the hypervisor and the container. 🧚
Container: Please see "Hypervisor:" above. 🫶
VN Profile: This is metadata tied to a VN and applied to any attachment point to the VN. Think of it as the VAP (Virtual Access Point) properties that apply to every VAP attached to a given VN, which an NVE uses when it ingresses or egresses packets to and from that specific VN. The metadata could include things like Access Control Lists (ACLs) and QoS settings. The VN Profile holds parameters for the VN as a whole. Control protocols between the NVE and the NVA (Network Virtualization Authority) can use the VN ID or VN Name to grab the VN Profile. It's giving organisation, honestly 🗂️💜
VSI: Virtual Station Interface. See [IEEE802.1Q]. 🌈
VDP: VSI Discovery and Configuration Protocol. See [IEEE802.1Q]. 🍓
1.2. Target Scenarios 🌍
In the Split-NVE architecture, an External NVE can offload the encapsulation and decapsulation functions, handle network policy enforcement, and take over overhead from the VN overlay protocol. This offloading may improve performance and/or save resources on the end device (for example, a hypervisor) that uses the External NVE. Main character energy for the hypervisor, no cap 👑
Figures 2 through 4 give example scenarios for the Split-NVE architecture. Here they are:
Hypervisor Access Switch
+------------------+ +-----+-------+
| +--+ +-------+ | | | |
| |VM|---| | | VLAN | | |
| +--+ | tNVE |---------+ nNVE| +--- Underlying
| +--+ | | | Trunk | | | Network
| |VM|---| | | | | |
| +--+ +-------+ | | | |
+------------------+ +-----+-------+
Figure 2: Hypervisor with an External NVE
Hypervisor L2 Switch
+---------------+ +-----+ +----+---+
| +--+ +----+ | | | | | |
| |VM|---| | |VLAN | |VLAN | | |
| +--+ |tNVE|-------+ +-----+nNVE| +--- Underlying
| +--+ | | |Trunk| |Trunk| | | Network
| |VM|---| | | | | | | |
| +--+ +----+ | | | | | |
+---------------+ +-----+ +----+---+
Figure 3: Hypervisor with an External NVE
Connected through an Ethernet Access Switch
Network Service Appliance Access Switch
+-----------------------------+ +-----+-------+
| +---------------+ | \ | | | |
| |Network Service|----| \ | | | |
| |Instance | | \ | VLAN | | |
| +---------------+ |tNVE| |------+nNVE | +--- Underlying
| +---------------+ | | | Trunk| | | Network
| |Network Service|----| / | | | |
| |Instance | | / | | | |
| +---------------+ | / | | | |
+-----------------------------+ +-----+-------+
Figure 4: Physical Network Service Appliance with an External NVETenant Systems connect to External NVEs through a Tenant System Interface (TSI). The TSI logically connects to the External NVE via a VAP [RFC8014]. The External NVE may provide Layer 2 or Layer 3 forwarding. In the Split-NVE architecture, the External NVE may be able to reach multiple Media Access Control (MAC) addresses and IP addresses through a single TSI. An IP address can be in either IPv4 or IPv6 format. 🌐
For example, Tenant Systems that provide network services (such as a transparent firewall, load balancer, or VPN gateway) are likely to have a complex address hierarchy. That implies that if a given TSI de-associates from one VN, all of its MAC and/or IP addresses de-associate too. There's no need to signal the deletion of every MAC or IP address when the TSI is brought down or deleted. Efficient and iconic 🙌 In the majority of cases, though, a VM is just a simple host with a single TSI and a single MAC and IP address visible to the External NVE. Simple, but make it a whole vibe 🌸
Figures 2 through 4 show VLANs being used to separate traffic for multiple VNs between the tNVE and the nNVE. VLANs aren't strictly necessary if only one VN is involved, but multiple VNs are expected in most cases. So this document assumes VLANs are present. 🦋✨
2. VM Lifecycle 🔄
[RFC7666] has a figure (Figure 2 of [RFC7666]) that shows the states and transitions of a VM. Some of those VM states are of interest to the External NVE. This section walks through the relevant phases and events in the VM lifecycle. Heads up: the subsections below are NOT exhaustive descriptions of VM lifecycle states. They're illustrative examples, relevant to the Split-NVE architecture, and not prescriptive text. The goal is to set enough context for the signaling-protocol functionality and requirements described in the sections that follow. Context is everything, bestie 🫶
2.1. VM Creation Event 🎂
The VM creation event moves the VM state from the "preparing" state to the "shutdown" state and then to the "running" state [RFC7666]. The end device allocates and initializes local virtual resources, like storage, while the VM is in its preparing state. In the shutdown state, the VM has everything ready, except that CPU execution isn't scheduled by the hypervisor and the VM's memory isn't resident in the hypervisor. Moving from shutdown to running normally needs human action or a system-triggered event. The running state means the VM is in its normal execution state. As part of entering the running state, the hypervisor must also provision network connectivity for the VM's TSI(s) so that Ethernet frames can be sent and received correctly. Right at the start, while in the running state, no ongoing migration, suspension, or shutdown is in progress. Clean slate 🧼✨
During the VM creation phase, the VM's TSI has to be associated with the External NVE. "Association" here means that the hypervisor and the External NVE have signaled each other and reached some form of agreement. Relevant networking parameters or information have been provisioned properly. The External NVE should be informed of the VM's TSI MAC address and/or IP address. On top of external network connectivity, the hypervisor may provide local network connectivity between the VM's TSI and the TSIs of other VMs co-resident on the same hypervisor. When that intra- or inter-hypervisor connectivity is extended to the External NVE, a locally significant tag (e.g., VLAN ID) should be used between the hypervisor and the External NVE to tell each VN's traffic apart. Both the hypervisor and the External NVE sides must agree on that tag value for traffic identification, isolation, and forwarding. Teamwork makes the dream work 🤝💖
The External NVE may need to do some prep before it signals a successful association with the TSI. That prep may include locally saving the states and binding information of the TSI and its VN, or talking to the NVA for network provisioning. 📝
A TSI association should be performed before the VM enters the running state, preferably in the shutdown state. If the association with an External NVE fails, the VM should not go into the running state. No association, no running. Period 🙅♀️💅
2.2. VM Live Migration Event 🚚
Live migration is sometimes called "hot" migration, because from an outside viewpoint the VM seems to keep running while it moves to another server (TCP connections generally survive this class of migration). Compare that with "cold" migration, which means shutting down VM execution on one server and restarting it on another. Cold energy versus hot energy 🥶🔥 For simplicity, the abstract summary of live migration below assumes shared storage, so the VM's storage is accessible to both the source and destination servers. Say the VM "live migrates" from hypervisor 1 to hypervisor 2. That migration involves state transitions on both the source hypervisor 1 and the destination hypervisor 2. On the source, the VM state goes from the running state to the "migrating" state and then to the shutdown state [RFC7666]. On the destination, the VM state goes from the shutdown state to the migrating state and then to the running state. A whole glow-up arc 🦋
The External NVE connected to destination hypervisor 2 has to associate the migrating VM's TSI with itself. It does this by discovering the TSI's MAC and/or IP addresses, discovering its VN, discovering its locally significant VLAN ID (if any), and provisioning the other network-related parameters of the TSI. The External NVE may also be told about the VM's peer VMs, storage devices, and other network appliances that the VM needs to talk to, or is already talking to. Networking group chat, but make it organised 📇💜 The migrated VM on destination hypervisor 2 should not go to the running state until all the network provisioning and binding have been done.
The VM state on both the source hypervisor and the destination hypervisor will be the migrating state during the transfer of VM execution. The migrating VM should not be in the running state on the source hypervisor and the destination hypervisor at the same time during migration. Only one running copy, bestie, no duplicates 🙅♀️👯 The VM on the source hypervisor does not transition to the shutdown state until the VM successfully enters the running state on the destination hypervisor. It's possible that the VM on the source hypervisor stays in the migrating state for a while after the VM on the destination hypervisor enters the running state. Lingering a little is still a vibe 🌙🌸
2.3. VM Termination Event 💀
Okay so a VM termination event is also called "powering off" a VM 🔌. It moves the VM into the shutdown state. Per [RFC7666], there are two possible causes of VM termination:
- A running VM has undergone a normal "power-off".
- The VM has been migrated to another hypervisor, and the VM image on the source hypervisor has to stop executing and be shut down.
When a VM gets terminated, the External NVE connected to that VM needs to deprovision it, i.e., delete the network parameters tied to that VM. In other words, the External NVE has to de-associate the VM's TSI. The whole cleanup is giving spring cleaning 🧹✨
2.4. VM Pause, Suspension, and Resumption Events ⏸️
A VM pause event moves the VM from the running state to the "paused" state. Paused means the VM is still chilling in memory, but the hypervisor isn't scheduling CPU execution for it [RFC7666]. The VM can be easily reactivated from paused back to running, so it's not gone, just on pause 🦋
A VM suspension event moves the VM from running to the "suspended" state, and a VM resumption event moves it from suspended back to running. While suspended, the memory and CPU execution state of the VM are saved to persistent storage. During this state, the hypervisor doesn't schedule CPU execution for the VM [RFC7666]. Literally a save state, bestie 💾
In the Split-NVE architecture, the External NVE should not de-associate the paused or suspended VM, because the VM can return to the running state at any time. Don't ghost a VM that's about to come back 🫶💜
3. Hypervisor-to-NVE Control-Plane Protocol Functionality 🧚
The following subsections show illustrative examples of the state transitions of an External NVE that matter for hypervisor-to-NVE signaling-protocol functionality. Note: This is not prescriptive text for the full state machine. So it's a sneak peek, not the whole blueprint 👑
3.1. VN_Connect and VN_Disconnect 🎀
In the Split-NVE scenario, the end device (e.g., hypervisor) and the External NVE it's using need a protocol between them. That way, the External NVE knows about the changing VN membership requirements of the Tenant Systems inside the end device. It's the group chat that keeps everyone updated 💬
A key reason to use a protocol instead of static configuration of the External NVE is that VN connectivity requirements can change super often. VMs get brought up, moved, and brought down on various hypervisors all over the data center or external cloud. Static config would be so not it 😭
Figure 5 shows the state transition for a VAP on the External NVE. An NVE that supports the hypervisor-to-NVE control-plane protocol should support one instance of the state machine for each active VN. The state transition on the External NVE is normally triggered by events and behaviors on the hypervisor-facing side. Some of the interleaved interactions between the NVE and the NVA will be illustrated to better explain the whole procedure, while other interactions will not be shown. Main character energy with the state machine 🌈
+----------------+ Receive VN_Connect; +----------------------+
|VN_Disconnected | return Local_Tag value |VN_Connected |
+----------------+ for VN if successful; +----------------------+
|VN_ID; |-------------------------->|VN_ID; |
|VN_State= | |VN_State=VN_Connected;|
|VN_Disconnected;| |Num_TSI_Associated; |
| |<--Receive VN_Disconnect---|Local_Tag; |
+----------------+ |VN_Context; |
+----------------------+
Figure 5: State Transition Example of a VAP Instance
on an External NVEThe External NVE must be notified when an end device requires a connection to a particular VN and when it no longer requires a connection. Connection cleanup for the failed devices should be employed. Note that this topic is out of scope for the protocol specified in this document. That's a different plotline 🙈
In addition, the External NVE should provide a local tag value for each connected VN to the end device. The end device uses it to exchange packets with the External NVE (e.g., a locally significant tag value per [IEEE802.1Q]). How "local" that tag is depends on whether:
- the hypervisor has a direct physical connection to the External NVE (in which case the significance is local to the physical link) or
- there is an Ethernet switch (e.g., a blade switch) connecting the hypervisor to the NVE (in which case the significance is local to the intervening switch and all the links connected to it).
These VLAN tags are how different VNs get differentiated as packets cross the shared-access network to the External NVE. When the External NVE receives packets, it uses the VLAN tag to identify their VN coming from a given TSI, strips the tag, adds the appropriate overlay encapsulation for that VN, and sends the packet toward the corresponding remote NVE across the underlying IP network. Tag, you're it 🏷️🦋
The Identification of the VN in this protocol could be through either a VN Name or a VN ID. A globally unique VN Name makes it way easier to port a tenant's virtual data center around ✈️🌸 Once an External NVE receives a VN_Connect message, the NVE needs a way to get a VN_Context allocated (or to receive the already-allocated VN_Context) for a given VN Name or VN ID, plus any other info needed to transmit encapsulated packets. How this happens is the subject of the NVE-to-NVA protocol; see the "first two areas of work" text in Section 4.5 of [RFC7364]. The External NVE needs to synchronize the mapping information of the local tag and VN Name or VN ID with the NVA. Sync or swim 🔄
The VN_Connect message can be explicit or implicit. "Explicit" means the hypervisor sends a request message explicitly for the connection to a VN. "Implicit" means the External NVE receives other messages, e.g., the very first TSI Associate message (see the next subsection) for a given VN, that implicitly signal its interest in connecting to a VN. Some messages just speak in subtext 🙈💅
A VN_Disconnect message indicates that the NVE can release all the resources for that disconnected VN and transition to the VN_Disconnected state. The local tag assigned for that VN can possibly be reclaimed for use by another VN. Bye bye, resources, the tag is free to be someone else's main character 👋💖
3.2. TSI Associate and Activate ✨
okay so bestie, typically a TSI gets ONE MAC address, and every frame sent and received on that TSI uses that same MAC 🎀. but as we said earlier, a Tenant System can also swap frames using multiple MAC addresses, or send packets with multiple IP addresses. it's giving flexible 💅
now, especially when a Tenant System is forwarding frames or packets for OTHER Tenant Systems, the External NVE needs to tell the NVA about the mapping between the NVE's IP address on the underlying network and ALL the addresses that Tenant System is forwarding on behalf of the corresponding VN. ALL of them, no skipping 😭
there are two ways the NVE can figure out which tenant addresses should be forwarded to a given end device (and ultimately to the Tenant System inside that end device) 🦋
- it can glean the addresses by looking at the source addresses in packets it gets from the end device.
- the hypervisor can explicitly signal the address associations of a TSI to the External NVE. an address association includes all the MAC and/or IP addresses that might be used as source addresses in a packet sent from the hypervisor to the External NVE. the External NVE can then use this info to filter future traffic from the hypervisor 🧚
to use that second approach, the control-plane protocol running between the hypervisor and the NVE MUST support end devices communicating new tenant-address associations for a given TSI within a given VN. wait, not MUST, it's the other one, let me fix that: the control-plane protocol running between the hypervisor and the NVE must support end devices communicating new tenant-address associations for a given TSI within a given VN. 🫶
Figure 6 shows an example of a state transition for a TSI connecting to a VAP on the External NVE. an NVE that supports the hypervisor-to-NVE control-plane protocol MAY support one instance of the state machine for each TSI connecting to a given VN 🌸
De-Associate +--------+ De-Associate
+--------------->| Init |<--------------------+
| +--------+ |
| | | |
| | | |
| +--------+ |
| | | |
| Associate | | Activate |
| +-----------+ +-----------+ |
| | | |
| | | |
| \|/ \|/ |
+--------------------+ +---------------------+
| Associated | | Activated |
+--------------------+ +---------------------+
|TSI_ID; | |TSI_ID; |
|Port; |-----Activate---->|Port; |
|VN_ID; | |VN_ID; |
|State=Associated; | |State=Activated; |-+
+-|Num_Of_Addr; |<---Deactivate ---|Num_Of_Addr; | |
| |List_Of_Addr; | |List_Of_Addr; | |
| +--------------------+ +---------------------+ |
| /|\ /|\ |
| | | |
+---------------------+ +-------------------+
add/remove/updt addr; add/remove/updt addr;
or update port; or update port;
Figure 6: State Transition Example of a TSI Instance
on an External NVEokay the Associated state of a TSI on an External NVE means all the addresses for that TSI are already associated with the VAP on a given port (say port p, for a given VN) 🍓 but NO real traffic to or from the TSI is expected or allowed to pass through yet. the NVE has reserved all the resources it needs for that TSI. an External NVE MAY report the mapping of its underlay IP address and the associated TSI addresses to the NVA, and relevant network nodes MAY save that info in their mapping tables but NOT in their forwarding tables. an NVE MAY create ACLs or filter rules based on the associated TSI addresses on port p, but it doesn't enable them yet 🙈 and the local tag for the VN that corresponds to the TSI instance SHOULD be provisioned on port p so it can receive packets.
the VM migration event (we covered it in Section 2) MAY make the hypervisor send an Associate message to the NVE connected to the destination hypervisor of the migration. a VM creation event MAY also trigger that same scenario 💜
the Activated state is the main character 👑 it means all the addresses for that TSI are working correctly on a given port (like port p) and traffic can be received from and sent to that TSI through the NVE. the mappings of the NVE's underlay IP address and the associated TSI addresses SHOULD be added to the forwarding table instead of the mapping table on the relevant network nodes. ACLs or filter rules based on the associated TSI addresses on the attached port p are enabled. and the local tag for the VN that corresponds to the TSI instance MUST be provisioned on port p to receive packets. no cap, that MUST is serious 🔥
the Activate message moves the state from Init or Associated to Activated. VM creation, VM migration, and VM resumption events (Section 2 again 🦋) MAY trigger the hypervisor to send the Activate message to the External NVE.
TSI info can get updated in either the Associated or the Activated state 🌈 the following count as updates: adding or removing associated addresses, updating the current associated addresses (like updating the IP address for a given MAC address), and updating the NVE port info based on where the NVE receives messages. these updates do NOT change the state of the TSI. when any address associated with a TSI changes, the NVE SHOULD tell the NVA to update the mapping info for the NVE's underlying address and the associated TSI addresses. the NVE SHOULD also change its local ACLs or filter settings to match for the relevant addresses. port info updates will make the local tag for the VN get provisioned on the new port and removed from the old one ✨
3.3. TSI De-Associate and Deactivate 🎀
okay so De-Associate and Deactivate are basically the reverse of Associate and Activate 🔄
going from Activated back to Associated, the External NVE needs to make sure the resources are still reserved, but the addresses on the TSI aren't functioning anymore. no traffic to or from the TSI is expected or allowed through. for example, the NVE needs to tell the NVA to remove the relevant address mapping info from the forwarding and routing tables. ACLs and filter rules for the relevant addresses SHOULD be disabled 🙃
going from Associated or Activated down to Init, the NVE releases ALL the resources for the TSI instances. the NVE SHOULD also tell the NVA to remove the relevant entries from the mapping table. ACLs or filter rules for the relevant addresses SHOULD be removed. the local tag provisioning on the connecting port on the NVE SHOULD be cleared 🧹
a VM suspension event (Section 2 💅) MAY cause the relevant TSI instance(s) on the NVE to go from Activated to Associated.
a VM pause event normally does NOT affect the state of the relevant TSI instance(s) on the NVE, since the VM is expected to run again soon 🍓
a VM shutdown event will normally make the relevant TSI instance(s) on the NVE go from Activated to Init. all resources SHOULD be released. ✨
a VM migration makes the TSI instance on the source NVE leave the Activated state. when a VM migrates to another hypervisor that connects to the SAME NVE (so the source and destination NVE are the same), the NVE SHOULD use the TSI_ID and the incoming port to tell the two TSI instances apart 🦋
okay this part is so cool 🤓. the triggering messages for the Figure 6 transitions don't show the difference between a VM creation/shutdown event and a VM migration arrival/departure event, BUT the External NVE can still make optimizations if it's told. for example, if the NVE knows the incoming Activate message came from a migration instead of VM creation, it MAY trigger some mechanisms to make sure the dynamic configurations or provisionings on the destination NVE match the ones on the source NVE for the migrated VM. like, an IGMP query [RFC2236] can be triggered by the destination External NVE to the migrated VM so the VM is forced to send an IGMP report to the multicast router. then the multicast router can correctly route multicast traffic to the new External NVE for the multicast groups the VM joined before the migration 🌷
4. Hypervisor-to-NVE Control-Plane Protocol Requirements 🧚
okay bestie, here are the requirements, and these are where the BCP 14 words live, so pay attention 🫶
Req-1: The protocol MUST support a bridged network connecting end devices to the External NVE.
Req-2: The protocol MUST support multiple end devices sharing the same External NVE via the same physical port across a bridged network.
Req-3: The protocol MAY support an end device using multiple External NVEs simultaneously, but only one External NVE for each VN (active-standby External NVE case for a VN).
Req-4: The protocol MAY support an end device using multiple External NVEs simultaneously for the same VN (active-active External NVE case for a VN).
Req-5: The protocol MUST allow the end device to initiate a request to its associated External NVE to be connected/disconnected to a given VN.
Req-6: The protocol MUST allow an External NVE initiating a request to its connected end devices to be disconnected from a given VN.
Req-7: When a Tenant System attaches to a VN, the protocol MUST allow for an end device and its External NVE to negotiate one or more locally significant tags for carrying traffic associated with a specific VN (e.g., tags per [IEEE802.1Q]).
Req-8: The protocol MUST allow an end device initiating a request to associate/de-associate and/or activate/deactivate some or all addresses of a TSI instance to a VN on an NVE port.
Req-9: The protocol MUST allow the External NVE initiating a request to de-associate and/or deactivate some or all addresses of a TSI instance to a VN on an NVE port.
Req-10: The protocol MUST allow an end device initiating a request to add, remove, or update address(es) associated with a TSI instance on the External NVE. Addresses can be expressed in different formats -- for example, MAC, IP, or IP-MAC pair.
Req-11: The protocol MUST allow the External NVE and the connected end device to authenticate each other.
Req-12: The protocol MUST be able to run over Layer 2 links between the end device and its External NVE.
Req-13: The protocol SHOULD support an end device that indicates that an Associate or Activate request from the end device is the result of a VM hot migration event.
5. VDP Applicability and Enhancement Needs ✨
okay so VDP (the Virtual Station Interface Discovery and Configuration Protocol) from [IEEE802.1Q] can be the control-plane protocol running between the hypervisor and the External NVE 🫶 Appendix A has informative VDP illustrations if you want to see it in action.
VDP handles the automatic discovery and configuration of Edge Virtual Bridging (EVB) stations and EVB bridges 🌸 An EVB station is usually an end station running a bunch of VMs, so in this doc we treat it as conceptually the same thing as a hypervisor. An EVB bridge is conceptually the External NVE. Iconic pairing, honestly.
VDP can pre-associate, associate, and de-associate a VSI on an EVB station with a port on the EVB bridge 🦋 In this document, a VSI is kind of like a virtual port that a VM uses to plug into the hypervisor. The EVB station and the EVB bridge can also agree on the VLAN ID(s) assigned to a VSI through a VDP message exchange, and other configuration parameters can get swapped over VDP too. VDP rides on the Edge Control Protocol (ECP) [IEEE802.1Q], which gives reliable transport over a Layer 2 network. Reliable, Layer 2, and no drama 💅
Now, VDP needs some upgrades to hit the requirements listed in Section 4 of this document. Table 1 shows the extensions and clarifications needed in the NVO3 context 💜
+------+-----------+-----------------------------------------------+
| Req | Supported | Remarks |
| | by VDP? | |
+------+-----------+-----------------------------------------------+
| Req-1| | |
+------+ |Needs extension. Must be able to send to a |
| Req-2| |specific unicast MAC, and should be able to |
+------+ Partially |send to a non-reserved well-known multicast |
| Req-3| |address other than the nearest customer bridge |
+------+ |address. |
| Req-4| | |
+------+-----------+-----------------------------------------------+
| Req-5| Yes |The VN is indicated by GroupID. |
+------+-----------+-----------------------------------------------+
| Req-6| Yes |The bridge sends a De-Associate. |
+------+-----------+------------------------+----------------------+
| | |VID==NULL in the request. The bridge returns |
| | |the assigned VLAN ID (VID) value in the |
| Req-7| Yes |response. GroupID, which is optionally present|
| | |in the request, is equivalent to the VN ID in |
| | |the context of NVO3. Multiple VLANs per group |
| | |are allowed. |
+------+-----------+------------------------+----------------------+
| | | Requirements | VDP Equivalent |
| | +------------------------+----------------------+
| Req-8| Partially | Associate/De-Associate |Pre-Assoc/De-Associate|
| | | Activate/Deactivate |Associate/De-Associate|
| | +------------------------+----------------------|
| | |Needs extension to allow Associate->Pre-Assoc. |
+------+-----------+------------------------+----------------------+
| Req-9| Yes |The VDP bridge initiates a De-Associate. |
+------+-----------+-----------------------------------------------+
|Req-10| Partially |Needs extension for an IPv4/IPv6 address. |
| | |Add a new "filter information format" type. |
+------+-----------+-----------------------------------------------+
| | |An out-of-band mechanism is preferred, e.g., |
| | |MACsec or 802.1X. Implicit authentication |
|Req-11| No |based on control of physical connectivity |
| | |exists in VDP when the External NVE connects to|
| | |the end device directly and is reachable with |
| | |the nearest customer bridge address. |
+------+-----------+-----------------------------------------------+
|Req-12| Yes |VDP naturally runs on the Layer 2 protocol. |
+------+-----------+-----------------------------------------------+
| | |A migration event may cause the M-bit to be set|
| | |to 1 in the VDP request to the migration |
| | |destination hypervisor and the S-bit to be set |
| | |to 1 in the VDP request to the migration source|
| | |hypervisor. However, a setting of M-bit = 0 or|
|Req-13| Partially |S-bit = 0 can indicate that no information is |
| | |available regarding migration or that the |
| | |events in question are not caused by migration.|
| | |To fully meet the requirement, this ambiguity |
| | |would need to be fixed so that migration or no |
| | |migration could be safely inferred from the |
| | |M-bit or S-bit settings. |
+------+-----------+-----------------------------------------------+
Table 1: Comparison of Split-NVE Requirements and VDP CapabilitiesHonestly the big takeaway: if VDP can carry Layer 3 addresses as described in Req-10, it covers most of the hypervisor-to-NVE control-plane functionality we need 🎀✨
6. Security Considerations 🔐
okay this part is serious bestie, so read it properly 😭 External NVEs must make sure that only properly authorized Tenant Systems are allowed to join and become part of any particular VN. Sometimes the tNVE wants to connect to the nNVE for provisioning. That can mean the tNVE needs to authenticate the nNVE, in addition to the nNVE authenticating the tNVE. If a secure channel is needed between the tNVE and the nNVE to carry the encrypted Split-NVE control-plane protocol, then existing mechanisms like MACsec [IEEE802.1AE] can be used. In some deployments, authentication can be implicit, based on control of physical connectivity. For example, that works if the nNVE sits in the bridge directly connected to the server holding the tNVE. The use of the "nearest customer bridge address" in VDP [IEEE802.1Q] is an example of where this kind of implicit authentication is possible, although explicit authentication also applies in that case.
Since the control-plane protocol causes configuration changes on both the tNVE and the nNVE, implementations of both should log all state changes, including those described in Section 3. Implementations should also log significant protocol events, like the establishment or loss of control-plane protocol connectivity between the tNVE and the nNVE, plus authentication results 📝
External NVEs also need proper mechanisms to make sure any hypervisor that wants to use an NVE's services is actually authorized to do so 🧚 One design point is whether the hypervisor should:
- supply the External NVE with necessary information (e.g., VM addresses, VN information, or other parameters) that the External NVE uses directly, or
- only supply a VN ID and an identifier for the associated VM (e.g., its MAC address), with the External NVE using that information to obtain the information it needs to validate the hypervisor-provided parameters or obtain related parameters in a secure manner.
The first approach works in a trusted environment, so the External NVE can use everything it pulls from the hypervisor directly for local configuration. That saves the External NVE the work of retrieving and/or validating information. The second approach gives more reliable information, because the External NVE has to fetch it from a management-system database. Some network-related parameters, like VLAN IDs, can even be passed back to the hypervisor as a more authoritative form of provisioning. But sometimes it's hard or inefficient for an External NVE to get rights to access or query those management systems. In that case, the External NVE has to get the information from the hypervisor instead 💀
7. IANA Considerations 🌈
This document has no IANA actions. Nothing to register, nothing to update, just vibes 🦋
8. References 📚
8.1. Normative References
[IEEE802.1Q]IEEE, "IEEE Standard for Local and metropolitan area networks--Bridges and Bridged Networks", IEEE Standard 802.1Q-2014, DOI 10.1109/IEEESTD.2014.6991462.[RFC2119]Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>.[RFC7365]Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. Rekhter, "Framework for Data Center (DC) Network Virtualization", RFC 7365, DOI 10.17487/RFC7365, October 2014, <https://www.rfc-editor.org/info/rfc7365>.[RFC7666]Asai, H., MacFaden, M., Schoenwaelder, J., Shima, K., and T. Tsou, "Management Information Base for Virtual Machines Controlled by a Hypervisor", RFC 7666, DOI 10.17487/RFC7666, October 2015, <https://www.rfc-editor.org/info/rfc7666>.[RFC8014]Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T. Narten, "An Architecture for Data-Center Network Virtualization over Layer 3 (NVO3)", RFC 8014, DOI 10.17487/RFC8014, December 2016, <https://www.rfc-editor.org/info/rfc8014>.[RFC8174]Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>.
8.2. Informative References ✨
Okay bestie, here's the "extra reading" list 📚🩷 These are the docs the spec mentions but doesn't depend on to work:
- [IEEE802.1AE] IEEE, "IEEE Standard for Local and Metropolitan Area Networks: Media Access Control (MAC) Security", IEEE Standard 802.1AE-2006, DOI 10.1109/IEEESTD.2006.245590.
- [NVO3-HYPERVISOR-NVE-CP] Kreeger, L., Narten, T., and D. Black, "Network Virtualization Hypervisor-to-NVE Overlay Control Protocol Requirements", Work in Progress, draft-kreeger-nvo3-hypervisor-nve-cp-01, February 2013.
- [NVO3-TES-NVE] Yingjie, G. and L. Yizhou, "The mechanism and signalling between TES and NVE", Work in Progress, draft-gu-nvo3-tes-nve-mechanism-01, October 2012.
- [NVO3-VM-NVE] Kompella, K., Rekhter, Y., Morin, T., and D. Black, "Signaling Virtual Machine Activity to the Network Virtualization Edge", Work in Progress, draft-kompella-nvo3-server2nve-02, April 2013.
- [RFC2236] Fenner, W., "Internet Group Management Protocol, Version 2", RFC 2236, DOI 10.17487/RFC2236, November 1997,
<https://www.rfc-editor.org/info/rfc2236>. - [RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally Unique IDentifier (UUID) URN Namespace", RFC 4122, DOI 10.17487/RFC4122, July 2005,
<https://www.rfc-editor.org/info/rfc4122>. - [RFC7364] Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L., Kreeger, L., and M. Napierala, "Problem Statement: Overlays for Network Virtualization", RFC 7364, DOI 10.17487/RFC7364, October 2014,
<https://www.rfc-editor.org/info/rfc7364>.
Appendix A. VDP Illustrations (per IEEE 802.1Q) (for Information Only)
Time for the VDP tea! ☕🦋 VDP (the VSI Discovery and Discovery and Configuration Protocol; see Clause 41 of [IEEE802.1Q]) can be thought of as a controlling protocol running between the hypervisor and the external bridge. The VDP association TLV is laid out like this, per Figure 7:
+--------+--------+------+-----+--------+------+------+------+------+
|TLV Type|TLV Info|Status|VSI |VSI Type|VSI ID|VSI ID|Filter|Filter|
| |String | |Type |Version |Format| |Info |Info |
| |Length | |ID | | | |Format| |
+--------+--------+------+-----+--------+------+------+------+------+
| | |<--VSI Type and instance--->|<--Filter--->|
| | |<-------------VSI attributes------------->|
|<--TLV header--->|<-----------TLV information string ------------->|Figure 7: VDP Association TLV
Okay so there are basically four TLV types 💅✨
- Pre-Associate: The Pre-Associate is used to Pre-Associate a VSI instance with a bridge port. The bridge validates the request and returns a failure status in the case of errors. A successful Pre-Associate does not imply that the indicated VSI Type or provisioning will be applied to any traffic flowing through the VSI. By allowing the bridge to obtain the VSI Type prior to an association, the Pre-Associate enables faster response to an Associate.
- Pre-Associate with Resource Reservation: The Pre-Associate with Resource Reservation involves the same steps as those for the Pre-Associate, but on success it also reserves resources in the bridge to prepare for a subsequent Associate request.
- Associate: The Associate request creates and activates an association between a VSI instance and a bridge port. A bridge allocates any required bridge resources for the referenced VSI. The bridge activates the configuration for the VSI Type ID. This association is then applied to the traffic flow to/from the VSI instance.
- De-Associate: The De-Associate is used to remove an association between a VSI instance and a bridge port. Pre-associated and associated VSIs can be de-associated. The De-Associate releases any resources that were reserved as a result of prior Associate or Pre-Associate operations for that VSI instance.
The De-Associate can be initiated by either side, and the other types can only be initiated by the server side. 🙈
Some important flag values in the VDP Status field are as follows 🌸
- M-bit (Bit 5): M-bit = 1 indicates that the user of the VSI (e.g., the VM) is migrating. M-bit = 0 means there is no indication of whether the VSI user is migrating. The M-bit is used as an indicator relative to the VSI to which the user is migrating.
- S-bit (Bit 6): S-bit = 1 indicates that the VSI user (e.g., the VM) is suspended. S-bit = 0 means there is no indication of whether the VSI user is suspended. A keep-alive Associate request with S-bit = 1 can be sent when the VSI user is suspended. The S-bit is used as an indicator relative to the VSI from which the user is migrating.
Heads up, the S-bit and M-bit are both used as indicators relative to the VSI, but they point at different things: M-bit is about where the user is going, S-bit is about where the user is coming from. Don't mix them up 😭
The filter information format currently defines four types 💖 Details for each one are in Figures 8 through 11. "PCP" stands for Priority Code Point [IEEE802.1Q]. The PCP value, if specified, is used by the EVB station as the default PCP value associated with the VSI and VID. The filter information contains a PCP Significant (PS) bit associated with each PCP field, indicating whether the PCP field carries a PCP value (binary 1) or does not carry a PCP value (binary 0).
+----------+-------+--------+--0------+
| # of | PS | PCP | VID |
|entries |(1 bit)|(3 bits)|(12 bits)|
|(2 octets)| | | |
+----------+-------+--------+---------+
|<---Repeated per entry--->|Figure 8: VID Filter Information Format
+----------+--------------+-------+--------+---------+
| # of | MAC address | PS | PCP | VID |
|entries | (6 octets) |(1 bit)|(3 bits)|(12 bits)|
|(2 octets)| | | | |
+----------+--------------+-------+--------+---------+
|<----------Repeated per entry----------->|Figure 9: MAC/VID Filter Information Format
+----------+--------------+-------+--------+---------+
| # of | GroupID | PS | PCP | VID |
|entries | (4 octets) |(1 bit)|(3 bits)|(12 bits)|
|(2 octets)| | | | |
+----------+--------------+-------+--------+---------+
|<----------Repeated per entry----------->|Figure 10: GroupID/VID Filter Information Format
+----------+----------+-------------+-------+--------+---------+
| # of | GroupID | MAC address | PS | PCP | VID |
|entries |(4 octets)| (6 octets) |(1 bit)|(3 bits)|(12 bits)|
|(2 octets)| | | | | |
+----------+----------+-------------+-------+--------+---------+
|<---------------Repeated per entry---------------->|Figure 11: GroupID/MAC/VID Filter Information Format
The null VID can be used in the VDP Request sent from the station to the external bridge. The null VID indicates that the set of VID values associated with the VSI is expected to be supplied by the bridge. The set of VID values is returned to the station via the VDP Response. The returned VID values can be locally significant values. When GroupID is used, it is equivalent to the VN ID in NVO3. GroupID will be provided by the station to the bridge. The bridge maps GroupID to a locally significant VLAN ID. 🧚
The VSI ID in the VDP association TLV that identifies a VM can be in one of the following formats: IPv4 address, IPv6 address, MAC address, Universally Unique Identifier (UUID) [RFC4122], or locally defined. 🌈
Acknowledgements
This document was initiated based on the merger of the following documents: [NVO3-HYPERVISOR-NVE-CP], [NVO3-TES-NVE], and [NVO3-VM-NVE]. A huge thank-you to all the coauthors and contributing members of those documents 🫶
The authors would like to specially thank Lucy Yong and Jon Hudson for their generous help in improving this document. 👑
Authors' Addresses
Yizhou Li
Huawei Technologies
101 Software Avenue
Nanjing 210012
China
Phone: +86-25-56625409
Email: liyizhou@huawei.com
Donald Eastlake 3rd
Huawei R&D USA
155 Beaver Street
Milford, MA 01757
United States of America
Phone: +1-508-333-2270
Email: d3e3e3@gmail.com
Lawrence Kreeger
Arrcus, Inc.
Email: lkreeger@gmail.com
Thomas Narten
IBM
Email: narten@us.ibm.com
David Black
Dell EMC
176 South Street
Hopkinton, MA 01748
United States of America
Email: david.black@dell.com📦 where this page comes from
- ietf: rfc8394 2018-05
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms