RFC 9732

A Framework for NRP-Based Enhanced Virtual Private Networks

📅 published March 2025, from ietf

RFC 9732: A Framework for NRP-Based Enhanced Virtual Private Networks ✨

Okay bestie, this is the first part of the RFC 9732 glow-up 🦋 It's an Informational doc from the IETF, dated March 2025, and it's all about how to make VPNs (Virtual Private Networks) way more special. Let's get into the title block first 💅

  • Authors: J. Dong (Huawei), S. Bryant (University of Surrey), Z. Li (China Mobile), T. Miyasaka (KDDI Corporation), and Y. Lee (Samsung) 🫶
  • Category: Informational, so it's not a Standards Track spec 🌸
  • Stream: IETF, with ISSN 2070-1721 🎀
  • Date: March 2025 💖
  • Obsoletes / Updates: The title block doesn't list anything it obsoletes or updates, so there's nothing to report there ✨

The vibe here: this doc is the framework, not a rulebook. It describes how to pair VPN (the overlay) with NRP (the underlay) to get enhanced stuff like low latency and bounded jitter. 🧚

Abstract

Okay so the abstract says this framework is built for apps that need specific traffic performance, like low latency and bounded jitter 🍓 NRP-based enhanced VPNs mix VPN and Traffic Engineering (TE) tech and add the extra features that certain services need, beyond what regular VPNs can do 💜 Usually this underpins network slicing, but it's also useful on its own for enhanced connectivity between customer sites 🌈 The doc also gives an overview of tech across different network layers and flags some areas for future work 👑

Status of This Memo

This document is not an Internet Standards Track specification. It's published for informational purposes 🦋

It's an IETF product that represents community consensus. It got public review and was approved by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 for that ✨

More info about the current status, errata, and how to give feedback lives at https://www.rfc-editor.org/info/rfc9732 💅

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. 🎀

This doc is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as of the publication date. Please read them carefully because they spell out your rights and restrictions 🙈 Code Components pulled from this doc must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Revised BSD License 💜

Table of Contents

1.  Introduction
2.  Terminology
3.  Overview of the Requirements
  3.1.  Performance Guarantees
  3.2.  Interaction Between Enhanced VPN Services
    3.2.1.  Requirements on Traffic Isolation
    3.2.2.  Limited Interaction with Other Services
    3.2.3.  Realization of Limited Interaction with Enhanced VPN
            Services
  3.3.  Integration with Network Resources and Service Functions
    3.3.1.  Abstraction
  3.4.  Dynamic Changes
  3.5.  Customized Control
  3.6.  Applicability to Overlay Technologies
  3.7.  Inter-Domain and Inter-Layer Network
4.  The Architecture of NRP-Based Enhanced VPNs
  4.1.  Layered Architecture
  4.2.  Connectivity Types
  4.3.  Application-Specific Data Types
  4.4.  Scalable Service Mapping
5.  Candidate Technologies
  5.1.  Underlay Forwarding Resource Partitioning
    5.1.1.  Flex Ethernet
    5.1.2.  Dedicated Queues
    5.1.3.  Time-Sensitive Networking
  5.2.  Network Layer Encapsulation and Forwarding
    5.2.1.  Deterministic Networking (DetNet)
    5.2.2.  MPLS Traffic Engineering (MPLS-TE)
    5.2.3.  Segment Routing
    5.2.4.  New Encapsulation Extensions
  5.3.  Non-Packet Data Plane
  5.4.  Control Plane
  5.5.  Management Plane
  5.6.  Applicability of Service Data Models to Enhanced VPNs
6.  Applicability in Network Slice Realization
  6.1.  NRP Planning
  6.2.  NRP Creation
  6.3.  Network Slice Service Provisioning
  6.4.  Network Slice Traffic Steering and Forwarding
7.  Scalability Considerations
  7.1.  Maximum Stack Depth of SR
  7.2.  RSVP-TE Scalability
  7.3.  SDN Scaling
8.  Enhanced Resiliency
9.  Manageability Considerations
  9.1.  OAM Considerations
  9.2.  Telemetry Considerations
10. Operational Considerations
11. Security Considerations
12. IANA Considerations
13. References
  13.1.  Normative References
  13.2.  Informative References
Acknowledgements
Contributors
Authors' Addresses

1. Introduction 🌸

Okay so VPNs have been a total icon for the industry 💅 They give different groups of users logically isolated connectivity over a shared network. That shared base network is usually called the "underlay", and the VPN on top of it is called the "overlay" 🦋

Customers might want fancy connectivity like low-latency guarantees, bounded jitter, or isolation from other services or customers. That way, stuff happening elsewhere (network load changes, congestion, outages) has no effect, or only acceptable effects, on the throughput or latency the customer sees 🙈 These are called "enhanced VPNs". They're like regular VPNs, giving the customer the connectivity they need, but with extra glow ✨

This doc is a framework for delivering VPN services with enhanced characteristics, like guaranteed resources, latency, and jitter 💖 That list isn't exhaustive. Other enhanced features will probably get added over time, and this framework is meant to support those with changes or enhancements across network layers and planes (data plane, control plane, and management plane) 🧚

Network slicing has been blowing up, mostly because of 5G 🚀 See [NGMN-NS-Concept], [TS23501], and [TS28530]. According to [TS28530], a 5G end-to-end network slice has three major segments: Radio Access Network (RAN), Transport Network (TN), and mobile Core Network (CN). The transport network connects the entities in the RAN and CN segments, with specific performance commitments 💜

[RFC9543] covers the general framework, components, and interfaces for requesting and operating network slices using IETF technologies. Those slices might be called "RFC 9543 Network Slices," but since this doc is only about IETF tech, we just say "network slice" 🎀 A network slice service links a set of Service Demarcation Points (SDPs) with specific Service Level Objectives (SLOs) and Service Level Expectations (SLEs) over a shared underlay. A slice can be a logical network connecting a bunch of endpoints, tied to shared or dedicated resources that satisfy the SLO and SLE requirements. A network slice counts as one target use case of enhanced VPNs 🌈

[RFC9543] also brings in the NRP concept. An NRP is a subset of the buffer, queuing, and scheduling resources plus associated policies on each link in a connected set of underlay links 🍓 An NRP can be tied to a dedicated or shared network topology to pick out which links and nodes are involved ✨

Enhanced VPN services can't just be handled by overlay networks alone. They need tighter coordination and integration between the overlay and the underlay 👑

On the overlay side, the VPN is the network construct that gives different services or customers their required connectivity, and there are multiple VPN flavors to build that construct [RFC4026] 🦋 On the underlay side, the NRP represents a subset of network resources and policies, again tied to a dedicated or shared topology to select the links and nodes 💅

An enhanced VPN service comes together by integrating a VPN in the overlay with an NRP in the underlay. That's the "NRP-based enhanced VPN" 💖 This gets you enhanced properties, like guaranteed resources and assured or predictable performance 🫶 An enhanced VPN may also involve a set of service functions (see Section 1.4 of [RFC7665] for the definition) 🧚 The techniques for delivering an NRP-based enhanced VPN can instantiate a network slice service (covered in Section 6), and they're also handy for general enhanced connectivity between customer sites or service endpoints 🌸

This doc describes a framework that uses existing, modified, and potential new technologies as components for NRP-based enhanced VPN services 🎀 Specifically, it provides:

  • The functional requirements and service characteristics of an enhanced VPN service 💜
  • The design of the data plane for NRP-based enhanced VPNs 🌈
  • The necessary control and management protocols in both the underlay and overlay of enhanced VPNs 🍓
  • The mechanisms for integrating the overlay network with the underlay network ✨
  • The necessary Operation, Administration, and Maintenance (OAM) methods to instrument an enhanced VPN, making sure the Service Level Agreement (SLA) between the customer and network operator is met, and to take corrective action (like switching traffic to an alternate path) to avoid SLA violations 👑

One possible layered structure for achieving all this is shown in Section 4.1 🦋

Enhanced VPN services aren't meant to replace conventional VPN services 💅 Regular VPNs keep running on existing mechanisms and can coexist with enhanced ones 💖 Whether enhanced features get added to an active VPN service is up to the deployment 🫶

Part 1 of 9. More parts coming, bestie, so stay tuned 💜🧚

2. Terminology 💅

okay so bestie, let's lock in the vocab first 🎀 these four terms are literally the whole plot: "VPN", "enhanced VPN", "NRP", and "Network Slice". Here's how they connect:

  • A Virtual Private Network (VPN) is the overlay network service that gives connectivity between different customer sites and keeps each customer's traffic separate from everyone else's 🔒 Examples of technologies that provide VPN services: IP VPN [RFC2764] [RFC4364], L2VPN [RFC4664], and EVPN [RFC7432].
  • An enhanced VPN service is an evolved VPN that comes with extra service-specific commitments ✨ An NRP-based enhanced VPN is made by combining a VPN with a set of network resources allocated in the underlay network (aka an NRP) 🦋
  • An NRP, as defined in [RFC9543], is a subset of the buffer/queuing/scheduling resources and associated policies on each of a connected set of links in the underlay network 🌈 An NRP can be tied to a dedicated or shared network topology to pick out which links and nodes are involved. It's designed to meet the network resources and performance characteristics that enhanced VPN services need 💜
  • A network slice service could be delivered by provisioning one or more NRP-based enhanced VPNs in the network 🧚 Other mechanisms for realizing network slices may exist, but they're not in the scope of this document.

The term "tenant" is used in this document to mean a customer of the enhanced VPN services 👑

Next up, some terms that are defined in other documents but still show up here:

  • SLA: Service Level Agreement (see [RFC9543])
  • SLO: Service Level Objective (see [RFC9543])
  • SLE: Service Level Expectation (see [RFC9543])
  • ACTN: Abstraction and Control of TE Networks (see [RFC8453])
  • DetNet: Deterministic Networking (see [RFC8655])
  • FlexE: Flex Ethernet (see [FLEXE])
  • TSN: Time-Sensitive Networking (see [TSN])
  • VN: Virtual Network (see [RFC8453])

3. Overview of the Requirements 🎀

this section gives you the overview of what an enhanced VPN service is actually expected to do 🫶

3.1. Performance Guarantees

okay, performance guarantees are the promises network operators make to their customers about the services they deliver 💖 They're usually written down in SLAs as a set of SLOs 📝

There are several kinds: guaranteed maximum packet loss, guaranteed maximum delay, and guaranteed delay variation 🍓 Quick but important note: these guarantees only apply to conformance traffic. Out-of-profile traffic gets handled under a separate agreement with the customer (see, for example, Section 3.6 of [RFC7297]) 💅

Guaranteed maximum packet loss is usually handled by setting packet priorities, queue sizes, and discard policies 🌸 It gets harder when you also have latency requirements. The extreme case is zero congestion loss, and that's the goal of DetNet [RFC8655] and Time-Sensitive Networking (TSN) [TSN] 😭 In modern optical networks, loss from transmission errors already approaches zero, but the interface or the fiber itself could fail. That kind of fault can be handled with some form of signal duplication and transmission over diverse paths 🔥

Guaranteed maximum latency is needed by a bunch of applications, especially real-time control apps and some types of augmented reality and virtual reality (AR/VR) apps 🌈 DetNet techniques may be considered [RFC8655]. However, additional methods of enhancing the underlay to better support delay guarantees may be needed too, and those methods will need to be integrated with the overall service provisioning mechanisms 🧚

Guaranteed maximum delay variation is another performance guarantee that may also be needed 💜 [RFC8578] calls out a number of cases that need this guarantee, for example in electrical utilities ⚡ Time transfer is an example service that needs it. It's in the nature of time, though, that the service might be delivered by the underlay as a shared service and not through different enhanced VPNs. Alternatively, a dedicated enhanced VPN might be used to provide time transfer as a shared service 🦋

So the takeaway is that a whole spectrum of service guarantees needs to be considered when designing and deploying an enhanced VPN 👑 For illustration, and without claiming to be exhaustive, four types of services are considered:

  • Best effort
  • Assured bandwidth
  • Guaranteed latency
  • Enhanced delivery

Some services may have mixed requirements from this list, e.g., both assured bandwidth and guaranteed latency can be required 💖

The best-effort service is the basic connectivity service that current VPNs can provide 🌸

An assured bandwidth service is a connectivity service in which the bandwidth over some period of time is assured 🍓 This could be achieved either simply through a best-effort service with over-capacity provisioning, or through MPLS TE Label Switching Paths (TE-LSPs) with bandwidth reservations. Depending on the technique used, though, the bandwidth is not necessarily assured at any instant 😭 Providing assured bandwidth to VPNs, for example by using per-VPN TE-LSPs, isn't widely deployed, at least partly because of scalability concerns 💀 The more common approach of aggregating multiple VPNs onto common TE-LSPs results in shared bandwidth, so it may reduce the bandwidth assurance for any one service. Enhanced VPNs aim to provide a more scalable approach for these services ✨

A guaranteed latency service has an upper bound on edge-to-edge latency ⏱️ Assuring that upper bound is sometimes more important than minimizing latency itself 🎀 A few new technologies help with this guarantee:

  • the IEEE TSN project [TSN] introduces the concept of scheduling delay-sensitive and loss-sensitive packets 🌈
  • FlexE [FLEXE] is useful for helping to provide a guaranteed upper bound on latency 🦋
  • DetNet is relevant for assuring an upper bound on end-to-end packet latency at the network layer 🧚

Using these technologies to deliver enhanced VPN services needs to be considered when a guaranteed latency service is required 💜

An enhanced delivery service is a connectivity service in which the underlay network (at Layer 3) needs to ensure that packet loss is eliminated or minimized when equipment or media fails 🚨 This may be achieved by delivering a copy of the packet through multiple paths. Such a mechanism may need to be used for enhanced VPN services 🫶

3.2. Interaction Between Enhanced VPN Services

there's a fine distinction between how a customer asks for limits on interaction between an enhanced VPN service and other services (whether those are other enhanced VPN services or any other network service) and how the service provider actually delivers that 🙈 This section looks at the requirements and the realization of limited interaction between an enhanced VPN service and other services 💅

3.2.1. Requirements on Traffic Isolation

"Traffic isolation" is a generic term for the requirements around separating the services of different customers, or different service types, in the network 🌸 In the context of network slicing, traffic isolation is defined as an SLE of the network slice service (Section 8.1 of [RFC9543]), which is one element of the SLA 🍓 A customer may care about disruption caused by other services, contamination by other traffic, or their traffic being delivered to the wrong destinations 😱

A customer may want to specify (and therefore pay for) the traffic isolation the service provider gives them 💰 Some customers (banking, for example) may have strict requirements on how their flows are handled when delivered over a shared network 🏦 Some professional services rely on specific certifications and audits to make sure a network complies with traffic-isolation requirements and, in particular, to prevent data leaks 🔐

With traffic isolation, a customer expects that their service traffic cannot be received by other customers in the same network 👀 [RFC4176] mentions traffic isolation as one of the requirements of VPN customers, and traffic isolation is also described in Section 3.8 of [RFC7297] 🦋

There can be different expectations of traffic isolation 🌈 For example, a customer may further protect their traffic by requesting specific encryption schemes at the enhanced VPN access, and also when it's transported between Provider Edge (PE) nodes 🔒

An enhanced VPN service customer may request traffic isolation together with other operator-defined service characteristics 💖 The exact details of the expected behavior need to be specified in the service request, so that meaningful service assurance and fulfillment feedback can be exposed to the customer 📋 Elaborating the service-modeling considerations is out of the scope of this document 🧚

3.2.2. Limited Interaction with Other Services

[RFC2211] describes the controlled-load service 🎀 In that document, the end-to-end behavior that a series of network elements providing controlled-load service gives an application is described as closely approximating the behavior that applications see with best-effort service, when those network elements aren't carrying substantial traffic from other services 🌸

So a consumer of a controlled-load service may assume that:

  • A very high percentage of transmitted packets will be successfully delivered by the network to the receiving end nodes ✅
  • The transit delay experienced by a very high percentage of the delivered packets will not greatly exceed the minimum transmit delay experienced by any successfully delivered packet ⏱️

An enhanced VPN customer may request a controlled-load service in one of two ways 💅

  1. They may configure a set of SLOs (for example, for delay and loss) so that the delivered enhanced VPN meets the customer's behavioral objectives 🎯
  2. As described in [RFC2211], a customer may request the controlled-load service without referring to or specifying target values for control parameters such as delay or loss 🧁 Instead, accepting a controlled-load service request is defined to imply a commitment by the network element to give the requestor service closely equivalent to what uncontrolled (best-effort) traffic gets under lightly loaded conditions 🫧 This way of requesting the service is an SLE ✨

Limited interaction between enhanced VPN services does not cover service degradation caused by non-interaction-related causes, such as link errors ⚡

3.2.3. Realization of Limited Interaction with Enhanced VPN Services 🎀

okay so bestie, a service provider can turn those limited-interaction requirements into real engineering rules in its network 🔧. Honoring the service requirement can mean tweaking a bunch of QoS, TE, security, and planning tools, and traffic isolation means configuring routing and authorization stuff properly 💅.

Concretely, there are loads of existing techniques for traffic isolation, like IP and MPLS VPNs or other multi-tenant virtual network techniques 🦋. Controlled-load services can be realized as described in [RFC2211] ✨. Then there's a whole toolkit of resource management and reservation tricks: network capacity planning, allocating dedicated network resources, traffic policing or shaping, prioritizing use of shared network resources, etc. 🧰. The point is that a subset of bandwidth, buffers, and queueing resources can be available in the underlay network to support the enhanced VPN services, and honestly it's giving very organized 🍓.

To get the required traffic isolation, or to cut down on interaction with other enhanced VPN services, network resources may need to be reserved in the data plane of the underlay network and dedicated to traffic from a specific enhanced VPN service or a specific group of them 🌸. ngl this can raise scalability concerns in the implementation, since each enhanced VPN may need to be tracked in the network 😭. It can also raise scalability concerns in deployment, like how many resources need to be reserved and how the services get mapped to those resources (Section 4.4) 🙈. So there's a trade-off to think about if you want traffic isolation and limited interaction between an enhanced VPN service and other services ⚖️.

A dedicated physical network can meet stricter SLO and SLE requests, but you're allocating resources on a long-term and end-to-end basis, so it's a big commitment 💜. On the flip side, where adequate traffic isolation and limited interaction can be achieved at the packet layer, the resources can be shared across a group of services and only dedicated to a service on a temporary basis ✨. By combining conventional VPNs with TE, QoS, and security techniques, an enhanced VPN offers a variety of ways to honor the customer's requirements 👑.

3.3. Integration with Network Resources and Service Functions 🧚

so the way an enhanced VPN service gets things like guaranteed or predictable performance is by integrating the overlay VPN with a particular set of underlay network resources that are allocated to meet the service requirements 💖. This has to be done in a flexible and scalable way so it can be widely deployed in operators' networks and support a good number of enhanced VPN services 🌈.

Mobile networks, and 5G especially, make integrating the network with service functions likely a requirement 📱. The IETF's work on Service Function Chaining (SFC) [RFC7665] gives us a foundation for this 🔗. Service functions in the underlay network can be considered part of the enhanced VPN services, which means they may need to be an integral part of the corresponding NRP 🫶. The details of how service functions and enhanced VPNs integrate are out of the scope of this document, so that's a whole other episode 🎬.

3.3.1. Abstraction 🪞

integrating the overlay VPN with underlay network resources and service functions doesn't always have to be a direct mapping 🙌. As described in [RFC7926], abstraction is the process of applying policy to a set of information about a traffic-engineered network to produce selective information that represents the potential ability to connect across the network 🧠. The process presents the connectivity graph in a way that doesn't depend on the underlying network technologies, capabilities, and topology, so the graph can be used to plan and deliver network services in a uniform way 🌟.

Using the abstraction approach, an enhanced VPN may be built on top of an abstracted topology that represents the connectivity capabilities of the underlay TE-based network, as described in the framework for Abstraction and Control of TE Networks (ACTN) [RFC8453] and discussed further in Section 5.5 💅✨.

3.4. Dynamic Changes 🦋

Enhanced VPNs need to be created, modified, and removed from the network according to service demands, including scheduled requests 📅. An enhanced VPN that requires limited interaction with other services (Section 3.2.2) must not be disrupted by the instantiation or modification of another enhanced VPN service 🚫. As discussed in Section 3.1 of [RFC4176], assessing traffic isolation is part of managing a VPN service 🔍. Figuring out whether changing an enhanced VPN could disrupt that enhanced VPN, and whether the traffic in flight will get disrupted, can be a genuinely difficult problem 😩.

Dynamic changes, both to the enhanced VPN and to the underlay network, need to be managed so they don't disrupt services that are sensitive to changes in network performance 🌸.

Besides managing the network without disruption during changes like adding a new enhanced VPN service endpoint or changing a link, enhanced VPN traffic might need to be moved because traffic patterns and volume changed 🌊. So during the lifetime of an enhanced VPN service, closed-loop optimization is needed so the delivered service always matches the ordered SLA 🔁💜.

The data plane side of this problem is discussed further in Sections 5.1, 5.2, and 5.3 📦.

The control plane side is discussed further in Section 5.4 🎛️.

The management plane side is discussed further in Section 5.5 🗂️.

3.5. Customized Control 🎀

In lots of cases, enhanced VPN services get delivered to customers without any info about the underlying NRPs 🙈. But depending on the agreement between the operator and the customer, the customer may also get some information about the underlying NRPs 📝. That info can be filtered or aggregated according to the operator's policy 🔍. This lets the customer have some visibility and even control over how the underlying topology and resources of the NRP are used 👀. For example, the customer may be able to specify the path or path constraints within the NRP for specific traffic flows of their enhanced VPN service 🛣️. Depending on the requirements, an enhanced VPN customer may have their own network controller, which may be given an interface to the control or management system run by the network operator 🖥️. Just note that this control stays within the scope of the customer's enhanced VPN service. Any changes beyond that would need some intervention by the network operator ⚠️.

The control plane aspects of this are discussed further in Section 5.4, and the management plane aspects of this feature can be found in Section 5.5 🧩.

3.6. Applicability to Overlay Technologies 🧁

The concept of an enhanced VPN can be applied to any existing and future multi-tenancy overlay technologies, including but not limited to: 🌈

  • Layer 2 point-to-point (P2P) services, such as pseudowires (see [RFC3985]) 🔌
  • Layer 2 VPNs (see [RFC4664]) 💫
  • Ethernet VPNs (see [RFC7209] and [RFC7432]) 🌊
  • Layer 3 VPNs (see [RFC4364] and [RFC2764]) 🦄

Where these VPN service types need enhanced isolation and delivery characteristics, the technologies described in Section 5 can be used to tweak the underlay so it provides the required enhanced performance 🚀💅.

3.7. Inter-Domain and Inter-Layer Network 🌐

In some scenarios, an enhanced VPN service may span multiple network domains 🗺️. A domain is any collection of network elements under the responsibility of the same administrative entity, for example, an Autonomous System (AS) 🏛️. In some domains, the network operator may manage a multi-layered network, like a packet network over an optical network 🧅. When enhanced VPN services get provisioned in these scenarios, the technologies used in different network planes (the data plane, control plane, and management plane) need to provide mechanisms to support multi-domain and multi-layer coordination and integration 🤝. This is what gives the different enhanced VPN services their required service characteristics and improves network efficiency and operational simplicity 📈✨. The mechanisms for multi-domain VPNs (see [RFC4364]) may be reused, and some enhancement may be needed to meet the additional requirements of enhanced VPN services 💖.

4. The Architecture of NRP-Based Enhanced VPNs ✨

okay so bestie, this is where the whole picture comes together 🧚 Multiple NRP-based enhanced VPN services can totally share the same common network infrastructure, and that's the vibe we want 💅 Each NRP-based enhanced VPN service gets provisioned with an overlay VPN and mapped onto a matching NRP. That NRP has its own specific set of network resources and service functions carved out in the underlay so it can meet what the customer actually needs 🫶 One NRP can support one or more NRP-based enhanced VPN services, which is so efficient 🔥 The integration between the overlay connectivity and the underlay resources is what keeps different enhanced VPN services properly isolated from each other and gives different customers their guaranteed performance. Iconic, honestly 👑

The NRP-based enhanced VPN architecture needs to be designed with consideration given to:

  • An enhanced data plane. 🦋
  • A control plane to create enhanced VPNs and NRPs, making use of the data plane isolation and performance guarantee techniques.
  • A management plane to manage enhanced VPN service life cycles.
  • The OAM mechanisms for enhanced VPNs and the underlying NRPs.
  • Telemetry mechanisms for enhanced VPNs and the underlying NRPs.

These topics are expanded below, so let's go 🌸

  • The enhanced data plane provides:
    • The required packet-latency and jitter characteristics.
    • The required packet-loss characteristics.
    • The required resource-isolation capability, e.g., bandwidth guarantee.
    • The mechanism to associate a packet with the set of resources allocated to an NRP to which the enhanced VPN service packet is mapped.
  • The control plane:
    • Collects information about the underlying network topology and network resources and exports this to network nodes and/or a centralized controller as required.
    • Creates NRPs with the network resource and topology properties needed by the enhanced VPN services.
    • Distributes the attributes of NRPs to network nodes that participate in the NRPs and/or a centralized controller.
    • Computes and sets up network paths in each NRP.
    • Maps enhanced VPN services to an appropriate NRP.
    • Determines the risk of SLA violation and takes appropriate avoidance/correction actions.
    • Considers the right balance of per-packet and per-node state according to the needs of the enhanced VPN services to scale to the required size. 📈
  • The management plane includes management interfaces, the OAM and telemetry mechanisms. More specifically, it provides:
    • An interface between the enhanced VPN service provider (e.g., the operator's network management system) and the enhanced VPN customer (e.g., an organization or service with an enhanced VPN requirement) such that the requests for specific operations and the related parameters can be exchanged without the awareness of other enhanced VPN customers. 🙈
    • An interface between the enhanced VPN service provider and the enhanced VPN customers to expose the network capability information toward the customer.
    • The service life-cycle management and operation of enhanced VPN services (e.g., creation, modification, assurance/monitoring, and decommissioning).
    • The OAM tools to verify whether the underlay network resources (i.e., NRPs) are correctly allocated and operating properly.
    • The OAM tools to verify the connectivity and monitor the performance of the enhanced VPN service.
    • Telemetry of information in the underlay network for overall performance evaluation and the planning of the enhanced VPN services.
    • Telemetry of information of enhanced VPN services for monitoring and analytics of the characteristics and SLA fulfillment of the enhanced VPN services.

4.1. Layered Architecture 🎀

okay so the layered architecture of NRP-based enhanced VPNs is shown in Figure 1 below 🦋

At the bottom of everything is the physical network infrastructure layer, which provides the underlying resources used to provision the separate NRPs. This layer is responsible for partitioning link and/or node resources for different NRPs. Each subset of a link or node resource can be considered a virtual link or virtual node that gets used to build the NRPs 💜

                               /\
                               ||
                     +-------------------+       Centralized
                     | Network Controller|   Control & Management
                     +-------------------+
                               ||
                               \/
                 o---------------------------o   Enhanced VPN #1
                               /-------------o
                 o____________/______________o   Enhanced VPN #2
                            _________________o
                      _____/
                 o___/     \_________________o   Enhanced VPN #3
                     \_______________________o
                            ......                  ...
                 o-----------\ /-------------o
                 o____________X______________o   Enhanced VPN #n

                    __________________________
                   /       o----o-----o      /
                  /       /          /      /       NRP-1
                 / o-----o-----o----o----o /
                /_________________________/
                    __________________________
                   /       o----o            /
                  /       /    / \          /       NRP-2
                 / o-----o----o---o------o /
                /_________________________/
                          ......                     ...
                   ___________________________
                  /             o----o       /
                 /             /    /       /       NRP-m
                /  o-----o----o----o-----o /
               /__________________________/


                  ++++   ++++   ++++
                  +--+===+--+===+--+
                  +--+===+--+===+--+
                  ++++   +++\\  ++++
                   ||     || \\  ||                Physical
                   ||     ||  \\ ||                Network
           ++++   ++++   ++++  \\+++   ++++     Infrastructure
           +--+===+--+===+--+===+--+===+--+
           +--+===+--+===+--+===+--+===+--+
           ++++   ++++   ++++   ++++   ++++

      o    Virtual Node     ++++
                            +--+  Physical Node with resource partition
      --   Virtual Link     +--+
                            ++++
      ==  Physical Link with resource partition

            Figure 1: The Layered Architecture of Enhanced VPNs

Various components and techniques discussed in Section 5 can be used to enable resource partitioning of the physical network infrastructure, such as FlexE, TSN, dedicated queues, etc. These partitions may be physical or virtual, so long as the SLA required by the higher layers is met ✨

Based on the set of NRPs provided by the physical network infrastructure, multiple NRPs can be created. Each of these NRPs:

  • has a set of dedicated or shared network resources allocated from the physical underlay network,
  • can be associated with a customized logical network topology, and
  • meets the requirements of a specific enhanced VPN service or a specific group of enhanced VPN services.

According to the associated logical network topology, each NRP needs to be instantiated on a set of network nodes and links that are involved in the logical topology. On each node or link, each NRP is associated with a set of local resources that are allocated for the processing of traffic in the NRP. The NRP provides the integration between the logical network topology and the required underlying network resources 🔥

According to the service requirements of connectivity, performance, isolation, etc., enhanced VPN services can be mapped to the appropriate NRPs in the network. Different enhanced VPN services can be mapped to different NRPs, and it's also possible that multiple enhanced VPN services are mapped to the same NRP. Thus, the NRP is an essential scaling technique, because it has the potential to eliminate per-service per-path state from the network 🧚 In addition, when a group of enhanced VPN services is mapped to a single NRP, only the network state of the single NRP needs to be maintained in the network (see Section 4.4 for more information).

The network controller is responsible for creating an NRP, instructing the involved network nodes to allocate network resources to the NRP, and provisioning the enhanced VPN services on the NRP. A distributed control plane may be used for distributing the NRP resource and topology attributes among nodes in the NRP. Extensions to distributed control protocols (if any) are out of the scope of this document 🫶

The process used to create NRPs and to allocate network resources for use by the NRPs needs to take a holistic view of the needs of all of the service provider's customers and partition the resources accordingly. However, within an NRP, these resources can be managed via a dynamic control plane if required. This provides the required scalability and isolation with some flexibility 💖

4.2. Connectivity Types 🌈

At the VPN service level, the required connectivity for a Multipoint-to-Multipoint (MP2MP) VPN service is usually full or partial mesh. To support such VPN services, the corresponding NRP also needs to provide MP2MP connectivity among the endpoints.

Other service requirements may be expressed at different granularities. Some of them can apply to the whole service, while others may only apply to some pairs of endpoints. For example, when a particular level of performance guarantee is required, the point-to-point path through the underlying NRP of the enhanced VPN service may need to be specifically engineered to meet the required performance guarantee 🍓

4.3. Application-Specific Data Types 💅

Although a lot of the traffic carried over enhanced VPN will likely be IP based, the design must be capable of carrying other traffic types, in particular Ethernet traffic. This is easily accomplished through various pseudowire (PW) techniques [RFC3985].

Where the underlay is MPLS, Ethernet traffic can be carried over an enhanced VPN encapsulated according to the method specified in [RFC4448]. Where the underlay is IP, L2 Tunneling Protocol - Version 3 (L2TPv3) [RFC3931] can be used, with Ethernet traffic carried according to [RFC4719]. Encapsulations have been defined for most of the common L2 types for both PW over MPLS and for L2TPv3 🦋

4.4. Scalable Service Mapping ✨

okay so VPNs are basically built as overlays sitting on top of an operator's network, and then the operator sells them to its customers 💖. The big perk of overlays is that they can deliver services without putting per-service state in the core of the underlay network 🦋.

An enhanced VPN might need to install some extra state inside the network to get the features it wants 👀. Solutions have to think about how big that state gets, and deployments should limit how many enhanced VPN services they run so the extra state stays acceptable and under control 💅. It's expected that there will only be a few enhanced VPN services at the start 🌸. Even later on there will be fewer of them than conventional VPNs, because existing VPN techniques are already good enough for most VPN-type services 🙈.

In general, the state in the network doesn't have to be 1:1 with the enhanced VPN services 🧚. Usually you can aggregate a set or group of enhanced VPN services so they share the same NRP and the same set of network resources (kinda like how current VPNs get aggregated over transport tunnels) 🎀. That way, services that need the same behavior from the network (resource reservation, latency bounds, resiliency, etc.) can be grouped together 💜. This is a huge deal for keeping NRP-based enhanced VPN deployments scaling nicely ⭐.

[NRP-SCALABILITY] has more details on scalability considerations for the NRPs used to instantiate NRPs, and Section 7 goes deeper into scalability considerations overall 📚.

5. Candidate Technologies 🎀

okay so a VPN is created by applying a demultiplexing technique to the underlying network (the underlay) so the traffic of one VPN can be told apart from another's 🧩. The connections of a VPN are supported by a set of underlay paths 🛤️. Any path other than the shortest path through the underlay normally needs state to specify that path 👀.

That state can be applied to the underlay through the RSVP-TE signaling protocol, or directly through a Software-Defined Networking (SDN) controller 🎛️. Based on Segment Routing (SR), the state could be kept at the ingress node of the path and carried in the data packet 📦. Other techniques may show up as this problem gets studied 🔬. The catch is that this state gets harder to manage as the number of paths goes up 😩. And as we increase the coupling between the underlay and the overlay to support the enhanced VPN service, this state is likely to grow even more 📈.

This is where NRP comes in 👑. A subset of underlay network resources can be either dedicated to one enhanced VPN service or shared among a group of them 🫶. A group of underlay paths can then be set up using the common set of network resources of the NRP ✨.

This section looks at the candidate technologies and checks them against the different network planes that may be used together to build NRPs 🌈. Section 5.1 covers the L2 packet-based or frame-based forwarding-plane mechanisms for resource partitioning 🧱. Section 5.2 covers the matching encapsulation and forwarding mechanisms in the network layer 📬. Non-packet data plane mechanisms get a quick mention in Section 5.3 🫧. The control plane and management plane mechanisms are discussed in Sections 5.4 and 5.5, respectively 🗂️.

5.1. Underlay Forwarding Resource Partitioning 🍰

Several candidate L2 packet-based or frame-based forwarding-plane mechanisms can provide the traffic isolation and performance guarantees we need, and they're described in the following sections 💖.

5.1.1. Flex Ethernet

okay FlexE [FLEXE] is so cute 🦋. It lets you multiplex channels over an Ethernet link to create point-to-point fixed-bandwidth connections, and it keeps enhanced VPN services separated from each other 🎀. FlexE can also bond multiple low-capacity links together to make a bigger link 🔗.

BUT (sorry, it's giving plot twist) FlexE is only a link-level technology 😭. When packets get to the downstream node, they still need to be processed in a way that preserves traffic isolation 📦. That means you need a queuing and forwarding implementation that keeps enhanced VPNs separated end to end 🧱.

Also, if different FlexE channels are used for different services, then no sharing is possible between those channels 🙅‍♀️. So it can be hard to dynamically hand unused bandwidth over to lower-priority services in another FlexE channel 💸. If one FlexE channel belongs to one customer, that customer can use some methods to manage the relative priority of their own traffic inside the channel 💅.

5.1.2. Dedicated Queues

Diffserv-based queuing systems are described in [RFC2475] and [RFC4594] 📚. Honestly though, this approach isn't enough to separate enhanced VPN services 🙃. Diffserv doesn't have enough markers to tell apart the traffic of a large number of enhanced VPN services 🏷️. It also doesn't offer the full range of service classes that each enhanced VPN service needs to give its tenants 💔. This problem is especially bad with an MPLS underlay, because MPLS only provides eight traffic classes 😵.

And on top of that, Diffserv, as it's currently implemented, mainly does per-hop priority-based scheduling 🚦. It's hard to use it to get quantitative resource reservation for different enhanced VPN services 📉.

So to fix these problems and cut down on the potential interactions between enhanced VPN services, traffic would need to be steered to dedicated input and output queues per enhanced VPN service, or per group of them 🧁. Some routers have a large number of queues and sophisticated queuing systems that can handle this 👑, while other routers may struggle to give the granularity and level of separation that enhanced VPN applications need 🥺.

5.1.3. Time-Sensitive Networking

[TSN] is an IEEE project that provides a method for carrying time-sensitive information over Ethernet ⏰. It brings in the concept of packet scheduling, where a packet stream can be given a time slot that guarantees it experiences no queuing delay and no increase in latency beyond a very small scheduling delay 🕑. The mechanisms defined in TSN can meet the requirements of time-sensitive traffic flows of an enhanced VPN service 🫶.

Ethernet can be emulated over an L3 network using an IP or MPLS pseudowire 🧵. But here's the tea: a TSN Ethernet payload would be opaque to the underlay 🫥. So it wouldn't be treated specifically as time-sensitive data 🙈. The preferred method for carrying TSN over an L3 network is DetNet, which is explained in Section 5.2.1 💖.

5.2. Network Layer Encapsulation and Forwarding 📬

This section looks at the problem of enhanced VPN service differentiation and how underlying network resources get represented in the network layer 🔍. More specifically, it describes the possible data plane mechanisms for working out the network resources and the logical network topology, or the paths, associated with an NRP 🗺️.

5.2.1. Deterministic Networking (DetNet)

DetNet [RFC8655] is a technique being developed in the IETF to help L3 networks deliver packets more reliably, with greater control over delay 🎯. The design can't use retransmission techniques like TCP, because that can blow past the delay the applications can tolerate 😬. Instead, DetNet preemptively sends copies of the packet over various paths, so there's less chance of every copy getting lost 🫧. It also tries to set an upper bound on latency, but the goal isn't to minimize latency 🙅‍♀️. DetNet can be realized over the IP data plane [RFC8939] or the MPLS data plane [RFC8964], and it may be used to provide deterministic paths for enhanced VPN services 🌟.

5.2.2. MPLS Traffic Engineering (MPLS-TE)

MPLS-TE (see [RFC2702] and [RFC3209]) brings in the concept of reserving end-to-end bandwidth for a TE-LSP 🛣️. That can be used to provide a set of point-to-point, resource-reserved paths across the underlay network to support VPN services 🎀. VPN traffic can be carried over dedicated TE-LSPs to get guaranteed bandwidth for each specific connection in a VPN 💪, and VPNs with similar behavior requirements may be multiplexed onto the same TE-LSPs 🫂.

Some network operators have concerns about the scalability and management overhead of MPLS-TE systems, especially the ones that use an active control plane 😩. That's led them to look at other solutions for TE in their networks 🔄.

5.2.3. Segment Routing

SR [RFC8402] is a method that prepends instructions to packets at the headend of a path 📜. Those instructions specify the nodes and links to be traversed, and they let packets take paths other than the shortest path 🧭. By encoding the state in the packet, per-path state gets moved out of the network 🪄. SR can be instantiated using the MPLS data plane (SR-MPLS) (see [RFC8660]) or the IPv6 data plane (SRv6) (see [RFC8986]) 💜.

An SR traffic-engineered path works at the granularity of a link 🔗. Hints about priority are provided using the Traffic Class (TC) field in the packet header 🏷️. But to get the performance and isolation that enhanced VPN customers are after, packets need to be steered through specific virtual links and/or queues on the same link, and told to use specific resources 🎯. With SR, that kind of fine-grained packet steering is possible by specifying the queues and their associated resources through an SR instruction list 🧁. One approach to doing this is described in [RESOURCE-AWARE-SEGMENTS] 📖.

Note that the concept of a queue is a useful abstraction for the different types of underlay mechanisms that may be used to provide enhanced isolation and performance support 🧩. How the queue meets the requirement is implementation specific, and it's transparent to the L3 data plane and control plane mechanisms in use 🫥.

With SR, the SR instruction list could be used to build a P2P SR path 🛤️. A group of SR Segment Identifiers (SIDs) could also be used to represent an MP2MP network 🕸️. So the SR-based mechanism could provide both resource-reserved paths and NRPs for enhanced VPN services 👑✨.

5.2.4. New Encapsulation Extensions

Instead of reusing an existing data plane for enhanced VPN, another option is to introduce new encapsulations, or extensions to an existing data plane 🧬. These would allow dedicated identifiers for the underlay network resources of an enhanced VPN, and for the logical network topology or paths associated with it 🪪. Yes, this may mean more protocol work 😅, but the potential benefits are that it can reduce the impact on existing network operation and improve the scalability of enhanced VPN 📈. More details about the encapsulation extensions and the scalability concerns are in [NRP-SCALABILITY] 💖.

5.3. Non-Packet Data Plane ✨

okay so non-packet underlay data planes, like the optical ones 🌈, often have TE properties and behaviors that are sooo good. they hit a lot of the key requirements, especially:

  • bandwidth guarantees 💰
  • traffic isolation (and physical isolation is usually a built-in part of the tech 🔒)
  • super predictable latency and jitter 🎯
  • measurable loss characteristics 📏
  • easy identification of flows 🔍

the catch is that the resources get allocated on a long-term, end-to-end basis 🥺. that means the full cost of those resources has to be paid by the client they're assigned to. but when one NRP built on this data plane supports multiple enhanced VPN services, the cost can be split across that group of services. bestie, that's giving sharing the bill 💅

5.4. Control Plane 🎀

the control plane for NRP-based enhanced VPNs is likely to be a hybrid setup 🧚. it takes advantage of a logically centralized controller for on-demand provisioning and global optimization, but it still leans on a distributed control plane for scalability, high reliability, fast reaction, automatic failure recovery, and more. to support the enhanced properties of an NRP-based enhanced VPN, both the centralized and distributed control planes need extending and optimizing 🔧.

as described in Section 4, the enhanced VPN control plane needs to handle these functions:

  • collecting information about the underlying network topology and network resources, and exporting it to network nodes and/or a centralized controller as needed 🗺️
  • creating NRPs with the network resource and topology properties that NRP-based enhanced VPN services need 🏗️
  • distributing NRP attributes to the network nodes that take part in the NRPs and/or to the centralized controller 📣
  • mapping enhanced VPN services to an appropriate NRP 🎯
  • computing and setting up service paths in each NRP so they meet enhanced VPN service requirements 🛤️

underlying topology and resource info can be collected with mechanisms based on the existing IGP and Border Gateway Protocol - Link State (BGP-LS) [RFC9552] 🙌. creating NRPs and distributing NRP attributes may need further control protocol extensions. computing service paths based on the NRP's attributes and constraints can be done either by the headend node of the path or by a centralized Path Computation Element (PCE) [RFC4655] 🧠.

there are two candidate control plane mechanisms for path setup in the NRP: RSVP-TE and SR 👀.

  • RSVP-TE, described in [RFC3209], provides the signaling mechanism for establishing a TE-LSP in an MPLS network with end-to-end resource reservation. it can be seen as a way to get resource-reserved paths that could bind the VPN to a specific set of network resources allocated within the underlay. however, there are still scalability concerns, as mentioned in Section 5.2.2 😬.
  • the SR control plane, described in [RFC8665], [RFC8667], and [RFC9085], can't signal resource reservations along the path 🙅. on the flip side, SR gives a potential way to bind underlay network resources to NRPs without needing per-path state to be kept in the network ✨. a centralized controller can do resource planning and reservation for NRPs, and it needs to tell the network nodes to make sure resources get allocated correctly for the NRP. the controller could provision the SR paths to the headend nodes of those paths using the mechanism in [RFC9256] 🚀.

depending on the service requirements for connectivity, performance, and isolation, one enhanced VPN service may be mapped to a dedicated NRP, or a group of enhanced VPN services may be mapped to the same NRP 🫶. mapping enhanced VPN services to an NRP can be done with existing control mechanisms, possibly with extensions. the mapping can be based on either the characteristics of the data packet or the attributes of the VPN service routes 💖.

5.5. Management Plane 🦋

the management plane is the interface between the enhanced VPN service provider and the customers for the life-cycle management of the enhanced VPN service. that covers creation, modification, assurance/monitoring, and decommissioning 🔄. it relies on a set of service data models to describe the information and operations needed on that interface 📋.

as an example, in the context of 5G end-to-end network slicing [TS28530], the transport network segment of a 5G end-to-end network slice can be managed with the enhanced VPN's management plane 📱. the 3GPP management system may give the transport network connectivity and performance-related parameters as requirements for the management plane. it may also ask the transport network to expose the capabilities and status of the network slice 👑. so coordinating 5G end-to-end network slice management needs an interface between the enhanced VPN management plane and the 5G network slice management system, plus relevant service data models 🤝.

the management plane interface and data models for enhanced VPN services can be based on the service models described in Section 5.6 🌸.

life-cycle management should support in-place modification of enhanced VPN services. that means it should be possible to add and remove endpoints, and to change the requested characteristics of the service being delivered 🔧. the management system needs to be able to assess revised enhanced VPN requests and work out whether they can be served by the existing NRPs, or whether changes have to be made. it will also need to figure out whether those NRP changes are possible at all. if they're not, the customer's modification request may be rejected 🙅‍♀️.

when modifying an enhanced VPN service is possible, the management system must make every effort to make the changes in a non-disruptive way 🫧. in other words, changing the enhanced VPN service or the underlying NRP must not disturb traffic on the enhanced VPN service in a way that drops the service level below the agreed levels. changes to one enhanced VPN service also should not disrupt other enhanced VPN services 💅.

the underlay network operator (i.e., the provider of the enhanced VPN service) may delegate some operational aspects of the overlay VPN and the underlying NRP to the customer. this way, the enhanced VPN is presented to the customer as a virtual network, and the customer decides how to use it 🌐. some mechanisms in the operator's network are needed so that:

  • a customer cannot exceed the capabilities of the virtual links and nodes, but
  • the customer can decide how to load traffic onto the network. for example, they can assign different metrics to the virtual links so they control how traffic is routed through the virtual network 🗺️

this approach needs a management system for the virtual network. it doesn't necessarily need any coordination between the virtual network's management systems and the physical network's, except that the virtual network management system might notice when the NRP is close to capacity or heavily under-used, and then automatically request changes in the service the underlay network provides 📈.

5.6. Applicability of Service Data Models to Enhanced VPNs 📚

this section covers how existing and in-progress service data models apply to enhanced VPNs 🧐. [RFC8309] describes the scope and purpose of service models and shows where a service model might fit into an SDN-based network management architecture. new service models may also be introduced for some of the required management functions 🆕.

service data models are used to represent, monitor, and manage the virtual networks and services that enhanced VPNs enable 🌟. the VPN customer service models, such as the L3VPN Service Model (L3SM) in [RFC8299], the L2VPN Service Model (L2SM) in [RFC8466], or the ACTN VN model in [RFC9731], can give the customer's view of the enhanced VPN service 👀. the L3VPN Network Model (L3NM) from [RFC9182] and the L2VPN Network Model (L2NM) from [RFC9291] give the operator's view of the managed infrastructure as a set of virtual networks and their associated resources 🏢. the Service Attachment Points (SAPs) model in [RFC9408] gives an abstract view of the SAPs to various network services in the provider network, and enhanced VPN could be one of those service types 📍. [RFC9375] provides the data model for performance monitoring of network and VPN services 📊. augmentations to these service models may be needed to provide enhanced VPN services 🧩. the NRP model in [NRP-YANG] goes further, managing the NRP topology and resources both in the controller and in the network devices, so the NRPs needed for enhanced VPN services can be instantiated 🏗️.

6. Applicability in Network Slice Realization 🍓

this section describes how NRP-based enhanced VPNs apply to network slice realization 🌈.

to provide network slice services to customers, a technology-agnostic network slice service model [NETWORK-SLICE-YANG] is needed so customers can communicate their network slice requirements (SDPs, connectivity, SLOs, and SLEs) 📝. these requirements may be realized with the technology specified in this document, which instructs the network to deliver an enhanced VPN service that meets the needs of the network slice customers 🎯. depending on where the SDPs for the network slice service are located (see Section 5.2 of [RFC9543]), an SDP can be mapped to a Customer Edge (CE), a PE, a port on a CE, or a customer-facing port on a PE. any of these can be correlated with the endpoint of the enhanced VPN service 🔗. the detailed approach for SDP mapping is described in [NETWORK-SLICE-YANG] 💫.

6.1. NRP Planning 🗓️

an NRP is used to support the SLOs and SLEs required by network slice services 💜. depending on the network operator's network resource planning policy, or on the requirements of one customer or a group of customers or services, an NRP may need to be created to meet network slice service requirements 🏗️. one basic requirement for an NRP is to provide a set of dedicated network resources, so it's protected from unexpected interference by other services in the same network 🛡️. other possible requirements include the required topology and connectivity, bandwidth, latency, reliability, and so on ✨.

a centralized network controller can be responsible for calculating a subset of the underlay network topology (called a logical topology) that supports the NRP requirements 🧮. on the network nodes and links within that logical topology, the controller can also determine the set of network resources to allocate to the NRP 🎀. this calculation normally needs to take into account the underlay network connectivity information and the available resource information of the underlay network 📡. the network controller may also consider the status of existing NRPs when planning and calculating a new NRP 🙌.

6.2. NRP Creation ✨

okay so once the NRP planning is done, the network nodes and links that are part of the NRP's logical topology get told to go allocate the network resources the NRP needs 🫶 one or more of the mechanisms from Section 5.1 can be used to partition the forwarding-plane resources and hand different subsets to different NRPs 💅 on top of that, the data plane identifiers that point at the set of resources allocated to the NRP also have to be provisioned on the network nodes 🎀 depending on which data plane tech is in use, the NRP's resources can be identified with things like resource-aware SR segments (see [RESOURCE-AWARE-SEGMENTS] and [SR-ENHANCED-VPN]), or you can introduce a dedicated Resource ID (see [IPv6-NRP-OPTION]) 🦋 the network nodes involved in an NRP may also distribute the logical topology info, the NRP-specific resource info, and the NRP's Resource ID over the control plane 🌸 that info can then be used by the controller and the nodes to compute TE or shortest paths within the NRP and install the NRP-specific forwarding entries on the nodes 🧚

6.3. Network Slice Service Provisioning 🎀

bestie, this part is giving clean architecture 😭 according to the connectivity requirements of a network slice service, an overlay VPN can be created using existing or future multi-tenancy overlay technologies, as described in Section 3.6 💜

then, according to the SLO and SLE requirements of the network slice service, the slice service gets mapped to an appropriate NRP as the virtual underlay 🌈 putting the overlay VPN and the underlay NRP together is what gives you a network slice service ✨ iconic, honestly 👑

6.4. Network Slice Traffic Steering and Forwarding 🦋

at the edge of the operator's network, network slice traffic can be classified based on the rules in the operator's policy 🍓 the whole point is that traffic matching the rules for a specific network slice service gets mapped to the matching NRP 💖 so packets belonging to that slice get processed and forwarded by network nodes based on either:

  • the traffic-engineered paths, or
  • the shortest paths in the associated network topology

and they use the set of network resources of the corresponding NRP 💅🔥

7. Scalability Considerations 🌈

okay real talk, NRP-based enhanced VPNs give you performance-guaranteed services in packet networks 🫶 BUT this comes with the potential cost of extra state in the network 😭 there are at least three ways that extra state can get added 💀

  • by putting the complete state into the packet, like SR does 🎀 this lets the controller specify the detailed series of forwarding and processing instructions for the packet as it transits the network 🦋 the cost is a bigger packet header 🙈 another cost is that systems have to provide NRP-specific segments in case a service calls on them 💜 that's a type of latent state, and it grows as the segments and resources exclusively available to the enhanced VPN service get specified more precisely 🌸
  • by introducing the state into the network 🧚 this is usually done by creating a path with signaling like RSVP-TE 💅 it could be extended to include any element that needs specifying along the path, like explicitly specifying queuing policy ✨ it's also possible to use other methods to introduce path state, like an SDN controller or possibly modifying a routing protocol 🔥 with this approach there's state per path, a per-path characteristic that has to be maintained over the life of the path 💀 it's more network state than SR needs, but the packets are usually shorter 🍓
  • by providing a hybrid approach 🌸 one example uses binding SIDs (see [RFC8402]) to represent path fragments and bind them together with SR 💖 dynamic creation of a VPN service path using SR needs less state maintenance in the network core, at the expense of larger packet headers 🙈 the packet size can be lower if a form of loose source routing is used (with a few nodal SIDs), and it'll be lower still if no specific functions or resources on the routers are specified 👑 for Segment Routing over IPv6 (SRv6), the packet size may also be reduced using the compression techniques in [SRv6-SRH-COMPRESSION] 🫶

okay so reducing state in the network matters a lot for deploying enhanced VPNs 😭 they need the overlay to be way more tightly integrated with the underlay than conventional VPNs 💅 that tighter coupling would normally mean more state gets created and maintained in the network, since state about fine-granularity processing has to be loaded and kept in the routers 🥺 aggregation is a well-established way to cut down state and improve scaling, and NRP is considered the network construct that aggregates the states of enhanced VPN services ✨ plus, an SR approach lets much of the state get spread across the network ingress nodes and carried transiently in the packets as SIDs 🦋

the following subsections describe some of the scalability concerns you need to consider 🎀 more discussion of the scalability considerations of the underlying network constructs of NRP-based enhanced VPNs can be found in [NRP-SCALABILITY] 💜

7.1. Maximum Stack Depth of SR 🧚

one of the challenges with SR is the stack depth that nodes are able to impose on packets [RFC8491] 😭 this creates a tricky balance between:

  • adding state to the network and minimizing stack depth, and
  • minimizing state and increasing the stack depth 💀

7.2. RSVP-TE Scalability 🌸

the established way to create a resource-reserved path through an MPLS network is the RSVP-TE protocol 💅 but there have been concerns that this requires significant continuous state maintenance in the network 🙈 work to improve the scalability of RSVP-TE LSPs in the control plane can be found in [RFC8370] ✨

there's also concern about the scalability of the forwarder footprint of RSVP-TE as the number of paths through a Label Switching Router (LSR) grows 🔥 [RFC8577] addresses this by employing SR within a tunnel established by RSVP-TE 👑

7.3. SDN Scaling 💜

the centralized SDN-based approach requires control plane state to be stored in the network, but it can reduce the overhead of control channels that need maintaining 🫶 each individual network node may need a control channel with an SDN controller, which is considered more scalable than maintaining control channels with a set of neighbor nodes 🌈

BUT SDN may move some scalability concerns from the network to a centralized controller 😬 in particular, there may be a heavy processing burden at the controller and a heavy load in the network surrounding the controller 💀 a centralized controller may also be a single point of failure in the network 🙈

8. Enhanced Resiliency 🦋

each enhanced VPN service has a life cycle and may need modification during deployment as its tenant's needs change (see Section 5.5) 🎀 also, as the network evolves, garbage collection may need to happen to consolidate resources into usable quanta 🧹✨

systems where the path is imposed, like SR or some form of explicit routing, tend to do well here because it's possible to do an atomic transition from one path to another 💅 that means a single action by the headend changes the path without needing coordinated action from the routers along the path 🔥 however, implementations and the monitoring protocols need to make sure the new path is operational and meets the required SLA before traffic moves to it 😭 it's possible for deadlocks to happen as the network gets fragmented over time, so it becomes impossible to create a new path or modify an existing one without impacting the SLA of other paths 💀 the global concurrent optimization mechanisms in [RFC5557], discussed in [RFC7399], may help 🌸 but full resolution of this is as much a commercial issue as a technical one 💅

there are two manifestations of the latency problem that are for further study in any of these approaches 🙈

  • packets overtaking one another if path latency reduces during a transition.
  • transient variation in latency in either direction as a path migrates.

there's also the matter of what happens when the underlay infrastructure fails 😱 fast reroute is one approach, but it still produces a transient loss, with a normal goal of fixing that within 50 ms [RFC5654] 🫶 an alternative is some form of N+1 delivery, like what's been used for years to protect against service disruption 💜 this can be taken to another level using the techniques of DetNet with multiple in-network replications and the culling of later packets [RFC8655] ✨

on top of the approach used to protect high-priority packets, you should also think about how best-effort traffic affects high-priority packets during a transition 👑 specifically, if a conventional re-convergence process is used, micro-loops will inevitably happen 😭 some form of explicit routing will protect the high-priority traffic, but lower-priority traffic on best-effort shortest paths will micro-loop without a loop-prevention technology 💀 so to give high-priority traffic the best quality of service, either that traffic has to be shielded from the micro-loops, or micro-loops have to be prevented completely 🔥

9. Manageability Considerations 🧚

this section covers the OAM and telemetry considerations for supporting the verification, monitoring, and optimization of the characteristics and SLA fulfillment of NRP-based enhanced VPN services 🎀 read it alongside Section 5.5, which covers the management plane techniques you can use to build NRPs 💖

9.1. OAM Considerations 💅

okay these are the requirements you need to consider when designing OAM for enhanced VPN services 🫶

  • instrumentation of the NRP (the virtual underlay), so the network operator can be sure the resources committed to a customer are operating correctly and delivering the required performance 🌸 it's important that OAM packets follow the same path and set of resources as the service packets mapped to the NRP 🦋
  • instrumentation of the overlay by the customer 💜 this is likely to be transparent to the network operator and use existing methods 🙈 particular attention needs to go to verifying the various committed performance characteristics 👑
  • instrumentation of the overlay by the service provider, to proactively demonstrate that the committed performance is being delivered 🔥 this needs to be done in a non-intrusive manner, especially when the tenant is deploying a performance-sensitive application 🥺

a study of OAM in SR networks is documented in [RFC8403] ✨

9.2. Telemetry Considerations ✨

Okay so, network visibility is basically everything for network operation, bestie 🔭💖 Network telemetry is considered the ideal way to get that visibility, with better flexibility, scalability, accuracy, coverage, and performance than the conventional OAM technologies. Obsessed, honestly 😭

As defined in [RFC9232], the goal of network telemetry is to acquire network data remotely for network monitoring and operation. It's a general umbrella term for a big family of network visibility techniques and protocols 🧚 It addresses the current network operation issues and helps the network glow up toward intent-driven autonomous networks ✨ Telemetry can be applied on the forwarding plane, the control plane, and the management plane, so it's giving full coverage 🌈 For enhanced VPN service, these are the telemetry requirements you need to keep in mind:

  • Collecting data of NRPs for overall performance evaluation and the planning of the enhanced VPN services 📊
  • Collecting data of each enhanced VPN service for monitoring and analytics of the service characteristics and SLA fulfillment 💅

How the telemetry mechanisms could be used or extended for enhanced VPN services is out of the scope of this document. That's a different story for a different doc, bestie 🎀

10. Operational Considerations 🍓

It's expected that NRP-based enhanced VPN services will be introduced into networks that already have conventional VPN services deployed 🦋 Depending on service requirements, the tenants or the operator may choose to use a VPN or an enhanced VPN to fulfill a service requirement. The information and parameters that help make that call need to be supplied on the management interface between the tenant and the operator 🫶 The management interface and data models (as described in Section 5.6) can be used for the life-cycle management of enhanced VPN services, such as service creation, modification, performance monitoring, and decommissioning. Full life-cycle glow-up, iykyk 💜

11. Security Considerations 🔒

Okay, this section is serious, so let's get into it 😭 All types of virtual networks require special consideration when it comes to isolating traffic belonging to different tenants. That is, traffic belonging to one VPN must not be delivered to endpoints outside that VPN. In this regard, the enhanced VPN neither introduces nor experiences greater security risks than other VPNs ✨

However, in an enhanced VPN service, the additional service requirements need to be considered 💀 For example, if a service requires a specific upper bound to latency, it can be damaged by simply delaying the packets through the activities of another tenant. Basically, by introducing bursts of traffic for other services. In some respects, this makes the enhanced VPN more susceptible to attacks since the SLA may be broken. Another view is that the operator must, in any case, perform monitoring of the enhanced VPN to ensure the SLA is met. So the operator may be more likely to spot the early onset of a security attack and be able to take preemptive protective action 🛡️

The measures to address these dynamic security risks must be specified as part of the specific solution to the isolation requirements of an enhanced VPN service 🔐

While an enhanced VPN service may be sold as offering encryption and other security features as part of the service, customers would be well advised to take responsibility for their security requirements themselves 🫶 Possibly by encrypting traffic before handing it off to the service provider.

The privacy of enhanced VPN service customers must be preserved 🙈 It should not be possible for one customer to discover the existence of another customer, nor should the sites that are members of an enhanced VPN be externally visible. Main character privacy energy, periodt 👑

An enhanced VPN service (even one with traffic isolation requirements or with limited interaction with other enhanced VPNs) does not provide any additional guarantees of privacy for customer traffic compared to regular VPNs. The traffic within the network may be intercepted and errors may lead to mis-delivery 😭 Users who wish to ensure the privacy of their traffic must take their own precautions, including end-to-end encryption. Not me telling you to encrypt everything, bestie 🔐

12. IANA Considerations 📝

This document has no IANA actions 🌸

13. References 📚

13.1. Normative References

13.2. Informative References

Okay, the citations keep going, bestie 📚✨ Here's the rest of the reference list 💜

Acknowledgements 🫶

The authors wanna give a huge shoutout to Charlie Perkins, James N. Guichard, John E. Drake, Shunsuke Homma, Luis M. Contreras, and Joel Halpern for reviewing the doc and dropping super valuable comments 💖

This work was also supported in part by the European Commission funded H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). Thank you, funding friends 🌈

Contributors 💅

  • Daniel King, Email: daniel@olddog.co.uk
  • Adrian Farrel, Email: adrian@olddog.co.uk
  • Jeff Tantsura, Email: jefftant.ietf@gmail.com
  • Zhenbin Li, Email: lizhenbin@huawei.com
  • Qin Wu, Email: bill.wu@huawei.com
  • Bo Wu, Email: lana.wubo@huawei.com
  • Daniele Ceccarelli, Email: daniele.ietf@gmail.com
  • Mohamed Boucadair, Email: mohamed.boucadair@orange.com
  • Sergio Belotti, Email: sergio.belotti@nokia.com
  • Haomian Zheng, Email: zhenghaomian@huawei.com

Authors' Addresses 🦋

  • Jie Dong, Huawei, Email: jie.dong@huawei.com
  • Stewart Bryant, University of Surrey, Email: stewart.bryant@gmail.com
  • Zhenqiang Li, China Mobile, Email: lizhenqiang@chinamobile.com
  • Takuya Miyasaka, KDDI Corporation, Email: ta-miyasaka@kddi.com
  • Young Lee, Samsung, Email: younglee.tx@gmail.com
📦 where this page comes from