RFC 4110
A Framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs)
RFC 4110: A Framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs) 🎀
Okay bestie, this is RFC 4110, published in July 2005 as an Informational doc 🌸 The authors are R. Callon from Juniper Networks and M. Suzuki from NTT Corporation. The title block doesn't list anything it obsoletes or updates, so there's nothing to chase down there ✨
Status of This Memo 📝
This memo gives information to the Internet community. It is not a standard of any kind, so don't treat it like a spec you have to implement 💅 Distribution of this memo is unlimited, so share it with everyone 🦋
Copyright Notice ©️
Copyright (C) The Internet Society (2005).
Abstract ✨
Okay so this doc gives you a framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs) 🫶 The whole point is to help the standards people build protocols and mechanisms for layer 3 PPVPNs. It's meant to be a coherent description of the big technical issues that matter when you're designing layer 3 PPVPN solutions. Picking specific approaches, making engineering tradeoffs, and writing the detailed protocol specs are all outside the scope of this framework doc 🧚
Table of Contents 🗂️
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Objectives of the Document . . . . . . . . . . . . . . . 3
1.2. Overview of Virtual Private Networks . . . . . . . . . . 4
1.3. Types of VPNs. . . . . . . . . . . . . . . . . . . . . . 7
1.3.1. CE- vs PE-based VPNs . . . . . . . . . . . . . . 8
1.3.2. Types of PE-based VPNs . . . . . . . . . . . . . 9
1.3.3. Layer 3 PE-based VPNs. . . . . . . . . . . . . . 10
1.4. Scope of the Document. . . . . . . . . . . . . . . . . . 10
1.5. Terminology. . . . . . . . . . . . . . . . . . . . . . . 11
1.6. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . 13
2. Reference Models . . . . . . . . . . . . . . . . . . . . . . . 14
2.1. Reference Model for Layer 3 PE-based VPN . . . . . . . . 14
2.1.1. Entities in the Reference Model. . . . . . . . . 16
2.1.2. Relationship Between CE and PE . . . . . . . . . 18
2.1.3. Interworking Model . . . . . . . . . . . . . . . 19
2.2. Reference Model for Layer 3 Provider-Provisioned
CE-based VPN . . . . . . . . . . . . . . . . . . . . . . 21
2.2.1. Entities in the Reference Model. . . . . . . . . 22
3. Customer Interface . . . . . . . . . . . . . . . . . . . . . . 23
3.1. VPN Establishment at the Customer Interface. . . . . . . 23
3.1.1. Layer 3 PE-based VPN . . . . . . . . . . . . . . 23
3.1.1.1. Static Binding . . . . . . . . . . . . 24
3.1.1.2. Dynamic Binding. . . . . . . . . . . . 24
3.1.2. Layer 3 Provider-Provisioned CE-based VPN. . . . 25
3.2. Data Exchange at the Customer Interface. . . . . . . . . 25
3.2.1. Layer 3 PE-based VPN . . . . . . . . . . . . . . 25
3.2.2. Layer 3 Provider-Provisioned CE-based VPN. . . . 26
3.3. Customer Visible Routing . . . . . . . . . . . . . . . . 26
3.3.1. Customer View of Routing for Layer 3 PE-based
VPNs . . . . . . . . . . . . . . . . . . . . . . 26
3.3.1.1. Routing for Intranets . . . . . . . . 27
3.3.1.2. Routing for Extranets . . . . . . . . 28
3.3.1.3. CE and PE Devices for Layer 3
PE-based VPNs. . . . . . . . . . . . . 29
3.3.2. Customer View of Routing for Layer 3 Provider-
Provisioned CE-based VPNs. . . . . . . . . . . . 29
3.3.3. Options for Customer Visible Routing . . . . . . 30
4. Network Interface and SP Support of VPNs . . . . . . . . . . . 32
4.1. Functional Components of a VPN . . . . . . . . . . . . . 32
4.2. VPN Establishment and Maintenance. . . . . . . . . . . . 34
4.2.1. VPN Discovery . . . . . . . . . . . . . . . . . 35
4.2.1.1. Network Management for Membership
Information. . . . . . . . . . . . . . 35
4.2.1.2. Directory Servers. . . . . . . . . . . 36
4.2.1.3. Augmented Routing for Membership
Information. . . . . . . . . . . . . . 36
4.2.1.4. VPN Discovery for Inter-SP VPNs. . . . 37
4.2.2. Constraining Distribution of VPN Routing
Information . . . . . . . . . . . . . . . . . . 38
4.2.3. Controlling VPN Topology . . . . . . . . . . . . 38
4.3. VPN Tunneling . . . . . . . . . . . . . . . . . . . . . 40
4.3.1. Tunnel Encapsulations. . . . . . . . . . . . . . 40
4.3.2. Tunnel Multiplexing. . . . . . . . . . . . . . . 41
4.3.3. Tunnel Establishment . . . . . . . . . . . . . . 42
4.3.4. Scaling and Hierarchical Tunnels . . . . . . . . 43
4.3.5. Tunnel Maintenance . . . . . . . . . . . . . . . 45
4.3.6. Survey of Tunneling Techniques . . . . . . . . . 46
4.3.6.1. GRE . . . . . . . . . . . . . . . . . 46
4.3.6.2. IP-in-IP Encapsulation . . . . . . . . 47
4.3.6.3. IPsec. . . . . . . . . . . . . . . . . 48
4.3.6.4. MPLS . . . . . . . . . . . . . . . . . 49
4.4. PE-PE Distribution of VPN Routing Information. . . . . . 51
4.4.1. Options for VPN Routing in the SP. . . . . . . . 52
4.4.2. VPN Forwarding Instances . . . . . . . . . . . . 52
4.4.3. Per-VPN Routing . . . . . . . . . . . . . . . . 53
4.4.4. Aggregated Routing Model . . . . . . . . . . . . 54
4.4.4.1. Aggregated Routing with OSPF or IS-IS. 55
4.4.4.2. Aggregated Routing with BGP. . . . . . 56
4.4.5. Scalability and Stability of Routing with Layer
3 PE-based VPNs. . . . . . . . . . . . . . . . . 59
4.5. Quality of Service, SLAs, and IP Differentiated Services 61
4.5.1. IntServ/RSVP . . . . . . . . . . . . . . . . . . 61
4.5.2. DiffServ . . . . . . . . . . . . . . . . . . . . 62
4.6. Concurrent Access to VPNs and the Internet . . . . . . . 62
4.7. Network and Customer Management of VPNs. . . . . . . . . 63
4.7.1. Network and Customer Management. . . . . . . . . 63
4.7.2. Segregated Access of VPN Information . . . . . . 64
5. Interworking Interface . . . . . . . . . . . . . . . . . . . . 66
5.1. Interworking Function. . . . . . . . . . . . . . . . . . 66
5.2. Interworking Interface . . . . . . . . . . . . . . . . . 66
5.2.1. Tunnels at the Interworking Interface. . . . . . 67
5.3. Support of Additional Services . . . . . . . . . . . . . 68
5.4. Scalability Discussion . . . . . . . . . . . . . . . . . 69
6. Security Considerations. . . . . . . . . . . . . . . . . . . . 69
6.1. System Security. . . . . . . . . . . . . . . . . . . . . 70
6.2. Access Control . . . . . . . . . . . . . . . . . . . . . 70
6.3. Endpoint Authentication . . . . . . . . . . . . . . . . 70
6.4. Data Integrity . . . . . . . . . . . . . . . . . . . . . 71
6.5. Confidentiality. . . . . . . . . . . . . . . . . . . . . 71
6.6. User Data and Control Data . . . . . . . . . . . . . . . 72
6.7. Security Considerations for Inter-SP VPNs . . . . . . . 72
Appendix A: Optimizations for Tunnel Forwarding. . . . . . . . . . 73
A.1. Header Lookups in the VFIs . . . . . . . . . . . . . . . 73
A.2. Penultimate Hop Popping for MPLS . . . . . . . . . . . . 73
A.3. Demultiplexing to Eliminate the Tunnel Egress VFI Lookup 74
Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . 75
Normative References . . . . . . . . . . . . . . . . . . . . . . . 76
Informative References . . . . . . . . . . . . . . . . . . . . . . 76
Contributors' Addresses. . . . . . . . . . . . . . . . . . . . . . 801. Introduction 🌈
1.1. Objectives of the Document 🎯
Okay so the main goal here is a framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs) 💜 It's meant to help standardize protocols and mechanisms that make layer 3 PPVPNs work interoperably, which is the big vibe 🫶
"Provider-provisioned VPNs" means VPNs where the Service Provider (SP) takes part in managing and provisioning the VPN, as defined in section 1.3. There's more than one way a provider can get involved, so there are lots of different kinds of PPVPNs 🦋 This doc talks about layer 3 VPNs (as defined in section 1.3).
First up, it gives you a reference model for layer 3 PPVPNs. Then it covers the technical side of layer 3 PPVPN operation, starting from the customer's point of view and then the provider's point of view 👑 It digs into the technical issues that matter when you're designing standards and mechanisms for running and supporting layer 3 PPVPNs. It also clarifies the technical aspects of layer 3 PPVPN interworking, and finally it addresses the security issues for layer 3 PPVPNs 🔥
This doc takes a "horizontal description" approach, which is giving you multiple approaches for every technical issue 🍓 If you want a specific PPVPN strategy, you have to pick one way to solve each problem. This doc doesn't make those choices or pick any particular approach to support VPNs, so don't look for a winner here 🙈
The "vertical description" approach lives in other documents, namely the ones that describe particular PPVPN solutions 💅 Any specific solution has to make choices based on SP requirements, customer needs, implementation cost, and engineering tradeoffs. Solutions also need to choose between flexibility (supporting multiple options) and conciseness (selecting specific options to keep implementation and deployment simpler) ✨ A framework doc can talk about the issues and criteria that feed into those choices, but actually selecting a solution is outside the scope of a framework doc 🧚
1.2. Overview of Virtual Private Networks 🎀
Okay so a "Virtual Private Network" (VPN) is a group of sites that talk to each other, where (a) communication between sites outside the group and sites inside it is restricted, but (b) the sites inside still send their traffic over a network infrastructure that other sites also use. The sharing is the whole vibe 🫶 The fact that the infrastructure is shared by multiple VPNs (and possibly by non-VPN traffic too) is what makes a VPN different from a private network. We're calling that shared network infrastructure the "VPN Backbone" 🦋
The logical structure of the VPN, meaning addressing, topology, connectivity, reachability, and access control, is equivalent to part of or all of a conventional private network that uses private facilities [RFC2764] [VPN-2547BIS]. So it gives full private network energy, just over shared roads 💅
In this document, we only care about the case where the shared network infrastructure (VPN backbone) is an IP and/or MPLS network. We also only care about the case where the Service Provider's edge devices, whether at the provider edge (PE) or at the Customer Edge (CE), decide how to route VPN traffic by looking at the IP and/or MPLS headers of the packets they receive from the customer's edge devices. That's the signature move of Layer 3 VPNs 👑
In some cases, one SP may offer VPN services to another SP. The former SP is known as a carrier of carriers, and the service it offers is known as "carrier of carriers" service. 🧚 In this document, when the customer could be either an enterprise or an SP network, we'll use the term "customer" to mean the user of the VPN services. Similarly, we'll use the term "customer network" to mean the user's network.
VPNs may be intranets, where the multiple sites are under the control of a single customer administration, like multiple sites of one company. Or VPNs may be extranets, where the multiple sites are controlled by administrations of different customers, like sites belonging to a company, its suppliers, and its customers. Both are lowkey iconic in their own way 💖
Figure 1.1. illustrates an example network that we'll be using in the discussions below. PE1 and PE2 are Provider Edge devices within an SP network. CE1, CE2, and CE3 are Customer Edge devices within a customer network. Routers r3, r4, r5, and r6 are IP routers internal to the customer sites. Here's the picture 👀
............ ................. ............
. . . . . .
. +---+ +-------+ +-------+ +---+ .
. r3---| | | | | |----|CE2|---r5 .
. | | | | | | +---+ .
. |CE1|----| PE1 | | PE2 | : .
. | | | | | | +---+ .
. r4---| | | | | |----|CE3|---r6 .
. +---+ +-------+ +-------+ +---+ .
. Customer . . Service . . Customer .
. site 1 . . provider(s) . . site 2 .
............ ................. ............Figure 1.1.: VPN interconnecting two sites.
In many cases, Provider Edge (PE) and Customer Edge (CE) devices may be either routers or LSRs. Mix and match, bestie 🌸
In this document, the Service Providers' network is an IP or MPLS network, and we want to interconnect the customer network sites over the Service Providers' network. Some VPN solutions require that the VPN service be provided either over a single SP network, or over a small set of closely cooperating SP networks. Other VPN solutions are meant to let VPN service run over an arbitrary set of minimally cooperating SP networks (i.e., over the public Internet). Wild range, no cap 🌈
In many cases, customer networks use private IP addresses [RFC1918] or other non-unique IP addresses (i.e., unregistered addresses). There's no guarantee that the IP addresses used in a customer network are globally unique 😭 The addresses used in one customer's network may overlap with the addresses used in others. Still, a single PE device can provide VPN service to multiple customer networks, even when those customer networks have overlapping addresses. In PE-based layer 3 VPNs, the PE devices may route VPN traffic based on the customer addresses found in the IP headers. This means the PE devices need to maintain a level of isolation between packets from different customer networks, and it's a big deal 🔥 In CE-based layer 3 VPNs, the PEs don't make routing decisions based on the customer's private addresses, so this issue doesn't come up. For either PE-based or CE-based VPNs, the fact that VPNs don't necessarily use globally unique address spaces also means that IP packets from a customer network cannot be transmitted over the SP network in their native form. Instead, some form of encapsulation/tunneling must be used. Tunnels era, literally 🚇
Tunneling is also important for other reasons, such as providing isolation between different customer networks and allowing a wide range of protocols to be carried over an SP network, etc. Different QoS and security characteristics may be associated with different tunnels. The vibes are different for each one ✨
1.3. Types of VPNs 🧚
This section describes multiple types of VPNs and some of the engineering tradeoffs between them. It is not up to this document to decide between different types of VPNs. Different types of VPNs may be appropriate in different situations, and that's totally valid 💜
There is a wide spectrum of possible VPNs, and it's hard to split them into clearly distinguished categories. It's a whole continuum, no cap 🌈
As an example, picture a company using a private network, with several sites interconnected via leased lines. All routing is done via routers that are internal to the private network. Very classic, very safe 🍓
At some point, the administrator of the private network might decide to replace the leased lines with ATM links, using an ATM service from an SP. Here again, all IP-level routing is done between customer premises routers and managed by the private network administrator. Same energy, new links 💅
To reduce the network management burden on the private network, the company may decide to use a provider-provisioned CE device [VPN-CE]. Here the operation of the network might be unchanged, except that the CE devices would be provided and managed by an SP. The SP handles the chores now, bestie 🫶
The SP might decide that it's too difficult to manually configure each CE-CE link. This might lead the SP to replace the ATM links with a layer 2 VPN service between CE devices [VPN-L2]. Auto-discovery might be used to simplify configuration of links between CE devices, and an MPLS service might be used between CE devices instead of an ATM service (for example, to take advantage of the provider's high speed IP or MPLS backbone). Upgrade arc unlocked 🚀
After a while, the SP might decide that it's too much trouble to manage a large number of devices at customers' premises, and might instead physically move these routers onto the provider premises. Each edge router at the provider premises might nonetheless be dedicated to a single VPN. The operation might remain unchanged, except that links from the edge routers to other routers in the private network become MAN links instead of LAN links, and the link from the edge routers to provider core routers becomes a LAN link instead of a MAN link. The routers in question can now be considered provider edge routers, and the service the SP provides has essentially become a layer 3 VPN service. Glow-up complete 🦋
To minimize the cost of equipment, the provider might decide to replace several dedicated PE devices with a single physical router that can run virtual routers (VR) [VPN-VR]. Protocol operation may remain unchanged. In this case, the provider is offering a layer 3 VPN service that makes use of a VR capability. Note that autodiscovery might be used in a manner very similar to how it was done in the layer 2 VPN case described above (for example, BGP might be used between VRs to discover other VRs supporting the same VPN). Efficiency queen 👑
Finally, to simplify operation of routing protocols for the private network over the SP network, the provider might decide to aggregate multiple routing instances into a single BGP instance [VPN-2547BIS]. Big brain move 🧠
In practice, it's highly unlikely that any one network would actually evolve through all of these approaches at different points in time. However, this example shows that there's a continuum of possible approaches, and each approach is relatively similar to at least some of the others for supporting VPN services. Some techniques (such as auto-discovery of VPN sites) may be common across multiple approaches. It's all connected, babe 💖
1.3.1. CE- vs PE-based VPNs 🌸
The term "CE-based VPN" (or Customer Edge-based Virtual Private Network) refers to an approach where the PE devices don't know anything about the routing or addressing of the customer networks. The PE devices offer a simple IP service and expect to receive IP packets whose headers contain only globally unique IP addresses. What makes a CE-based VPN a Provider-Provisioned VPN is that the SP takes on the job of managing and provisioning the CE devices [VPN-CE].
In CE-based VPNs, the backbone of the customer network is a set of tunnels whose endpoints are the CE devices. Various kinds of tunnels may be used (e.g., GRE, IP-in-IP, IPsec, L2TP, MPLS). The only overall requirement is that sending a packet through the tunnel means encapsulating it with a new IP header whose addresses are globally unique. Simple rule, big impact 🎀
For customer-provisioned CE-based VPNs, provisioning and management of the tunnels is the responsibility of the customer network administration. Typically, this involves manual configuration of the tunnels. In this case, the customer is also responsible for operating the routing protocol between CE devices. (Note that discussion of customer-provisioned CE-based VPNs is out of scope for this document.) Not our tea today 🙈
For provider-provisioned CE-based VPNs, provisioning and management of the tunnels is the responsibility of the SP. In this case, the provider may also configure routing protocols on the CE devices. This means routing in the private network is partially under the control of the customer, and partially under the control of the SP. Shared custody, very cute 🫶
For CE-based VPNs (whether customer- or provider-provisioned), routing in the customer network treats the tunnels as layer 2 links. Tunnels acting like wires, iconic 💅
In a PE-based VPN (or Provider Edge-based Virtual Private Network), customer packets are carried through the SP networks in tunnels, just as they are in CE-based VPNs. However, in a PE-based VPN, the tunnel endpoints are the PE devices, and the PE devices must know how to route the customer packets based on the IP addresses they carry. In this case, the CE devices themselves don't need any special VPN capabilities, and they don't even have to know that they're part of a VPN 🙌
In this document, we'll use the generic term "VPN Edge Device" to refer to the device, attached to both the customer network and the VPN backbone, that performs the VPN-specific functions. In the case of CE-based VPNs, the VPN Edge Device is a CE device. In the case of PE-based VPNs, the VPN Edge Device is a PE device. And that's the tea ☕🌈
1.3.2. Types of PE-based VPNs ✨
okay so PE-based VPNs come in different flavours, and you tell them apart by the service they offer 💅
- Layer 3 service: when a PE gets a packet from a CE, it works out how to forward that packet by looking at two things: the packet's incoming link, and the layer 3 information in the packet's header 🦋
- Layer 2 service: when a PE gets a frame from a CE, it works out how to forward the packet by looking at the packet's incoming link and the layer 2 information in the frame header (like an FR, ATM, or MAC header) 🌸 (ngl, layer 2 service is out of scope for this doc, so we're not going deep on it 🙈)
1.3.3. Layer 3 PE-based VPNs 💜
A layer 3 PE-based VPN is one where the SP joins in on IP-level forwarding based on the customer network's IP address space. The customer network is likely to use private and/or non-unique IP addresses 👀 which means at least some devices in the provider network need to understand the IP address space the customer uses. Usually that knowledge only lives in the PE devices that are directly attached to the customer.
The provider also has to help with some management and provisioning, like making sure the PE devices are set up to support the right VPNs. Which means layer 3 PE-based VPNs are, by definition, provider-provisioned VPNs 🫶
The big perk is that they take some VPN management off the customer's plate. From the customer's point of view it just looks like a normal network, because the VPN stuff is hidden from them 🎀 Scaling of the customer's routing can get better too, since some layer 3 PE-based VPN approaches mean the customer's routing algorithm doesn't have to see "N squared" (actually N*(N-1)/2) point to point duplex links between N customer sites 🧚
But there's a catch, bestie 💀 The PE devices have to know a bunch about the routing, addressing, and layer 3 protocols of the customer networks they attach to. One result is that the layer 3 protocols the VPN can support are limited to the ones the PE supports (which in practice means just IP). Another result is that the PE devices have more to do, and the SP has more per-customer management to do too 😭
An SP can offer a whole range of layer 3 PE-based VPN services. At the simplest end, the service just provides connectivity (optionally with QoS support) between specific customer network sites, and this is called "Network Connectivity Service" 💅 Then there's a whole spectrum of other possibilities, like firewalls, user or site of origin authentication, and address assignment (e.g., using Radius or DHCP) ✨
1.4. Scope of the Document 🌈
This framework document is going to talk about methods for providing layer 3 PE-based VPNs and layer 3 provider-provisioned CE-based VPNs. That can include mechanisms which will can be used to constrain connectivity between sites, including the use and placement of firewalls, based on administrative requirements [PPVPN-REQ] [L3VPN-REQ]. The use and placement of NAT functionality gets discussed too 🦋 But this framework document won't cover methods for extra services like firewall administration and address assignment. Specific firewall mechanisms and policies, plus detailed NAT functionality, are outside the scope of this document 🙈
This document doesn't discuss forms of VPNs that are outside the scope of the IETF Provider-Provisioned VPN working group. Specifically, it leaves out PPVPNs that use VPN native (non-IP, non-MPLS) protocols as the base technology for the VPN service (e.g., native ATM service provided using ATM switches with ATM signaling) 💖 Still, that doesn't mean multiprotocol access to the PPVPN by customers is excluded ✨
1.5. Terminology 📖
Backdoor Links: Links between CE devices that are provided by the end customer rather than the SP. They may be used to interconnect CE devices in multiple-homing arrangements 🔗
CE-based VPN: An approach where all the VPN-specific procedures happen in the CE devices, and the PE devices aren't aware in any way that some of the traffic they're processing is VPN traffic 🫣
Customer: A single organization, corporation, or enterprise that administratively controls a set of sites belonging to a VPN 👑
Customer Edge (CE) Device: The equipment on the customer side of the SP-customer boundary (the customer interface) 🏠
IP Router: A device which forwards IP packets and runs associated IP routing protocols (such as OSPF, IS-IS, RIP, BGP, or similar protocols). An IP router might optionally also be an LSR. The term "IP router" is often abbreviated as "router" 🛣️
Label Switching Router: A device which forwards MPLS packets and runs associated IP routing and signaling protocols (such as LDP, RSVP-TE, CR-LDP, OSPF, IS-IS, or similar protocols). A label switching router is also an IP router 🏷️
PE-Based VPNs: The PE devices know that certain traffic is VPN traffic. They forward the traffic (through tunnels) based on the destination IP address of the packet, and optionally on based on other information in the IP header of the packet. The PE devices are themselves the tunnel endpoints. The tunnels may make use of various encapsulations to send traffic over the SP network (such as, but not restricted to, GRE, IP-in-IP, IPsec, or MPLS tunnels) 🌸
Private Network: A network which allows communication between a restricted set of sites, over an IP backbone that is used only to carry traffic to and from those sites 🔒
Provider Edge (PE) Device: The equipment on the SP side of the SP-customer boundary (the customer interface) 🏢
Provider-Provisioned VPNs (PPVPNs): VPNs, whether CE-based or PE-based, that are actively managed by the SP rather than by the end customer 🧑💼
Route Reflectors: An SP-owned network element that is used to distribute BGP routes to the SP's BGP-enabled routers 🪞
Virtual Private Network (VPN): Restricted communication between a set of sites, making use of an IP backbone which is shared by traffic that is not going to or coming from those sites 🌐
Virtual Router (VR): An instance of one of a number of logical routers located within a single physical router. Each logical router emulates a physical router using existing mechanisms and tools for configuration, operation, accounting, and maintenance 🧩
VPN Forwarding Instance (VFI): A logical entity that resides in a PE and includes the router information base and forwarding information base for a VPN 📦
VPN Backbone: IP and/or MPLS network which is used to carry VPN traffic between the customer sites of a particular VPN 🦴
VPN Edge Device: Device, attached to both the VPN backbone and the customer network, which performs VPN-specific functions. For PE-based VPNs, this is the PE device; for CE-based VPNs, this is the CE device 🚪
VPN Routing: Routing that is specific to a particular VPN 🗺️
VPN Tunnel: A logical link between two PE or two CE entities, used to carry VPN traffic, and implemented by encapsulating packets that are transmitted between those two entities 🛸
1.6. Acronyms 🔤
ATM Asynchronous Transfer Mode
BGP Border Gateway Protocol
CE Customer Edge
CLI Command Line Interface
CR-LDP Constraint-based Routing Label Distribution Protocol
EBGP External Border Gateway Protocol
FR Frame Relay
GRE Generic Routing Encapsulation
IBGP Internal Border Gateway Protocol
IKE Internet Key Exchange
IGP Interior Gateway Protocol
(e.g., RIP, IS-IS and OSPF are all IGPs)
IP Internet Protocol (same as IPv4)
IPsec Internet Protocol Security protocol
IPv4 Internet Protocol version 4 (same as IP)
IPv6 Internet Protocol version 6
IS-IS Intermediate System to Intermediate System routing
protocol
L2TP Layer 2 Tunneling Protocol
LAN Local Area Network
LDAP Lightweight Directory Access Protocol
LDP Label Distribution Protocol
LSP Label Switched Path
LSR Label Switching Router
MIB Management Information Base
MPLS Multi Protocol Label Switching
NBMA Non-Broadcast Multi-Access
NMS Network Management System
OSPF Open Shortest Path First routing protocol
P Provider equipment
PE Provider Edge
PPVPN Provider-Provisioned VPN
QoS Quality of Service
RFC Request For Comments
RIP Routing Information Protocol
RSVP Resource Reservation Protocol
RSVP-TE Resource Reservation Protocol with Traffic
Engineering Extensions
SNMP Simple Network Management Protocol
SP Service Provider
VFI VPN Forwarding Instance
VPN Virtual Private Network
VR Virtual Router2. Reference Models 🧸
This section describes PPVPN reference models. The point of discussing reference models is to clarify the common components and pieces that are needed to build and deploy a PPVPN 🛠️ Two types of VPNs, layer 3 PE-based VPN and layer 3 provider-provisioned CE-based VPN, are covered in separated sections below 💕
2.1. Reference Model for Layer 3 PE-based VPN 🎀
okay bestie, this is where we get into how a layer 3 PE-based VPN is actually put together 💅 it's all about the functional pieces and how they relate to each other ✨
Figure 2.1 is the reference model for layer 3 PE-based VPNs, and Figures 2.2 and 2.3 show how the entities in that model connect 🦋
so in Figure 2.1, the customer interface is the one between CE and PE devices, and the network interface is the one between a pair of PE devices 💜
Figure 2.2 shows one logical tunnel between each pair of VFIs that support the same VPN. other setups are possible too 🌈 for example, one tunnel could run between two PEs, with a bunch of per-VFI tunnels multiplexed over that PE-to-PE tunnel. there can also be several tunnels between two VFIs, like to optimize forwarding inside the VFI. we'll get to the other possibilities later in this framework document ⭐
+---------+ +------------------------------------+ +---------+
| | | | | |
| | | +------+ +------+ : +------+
+------+ : | | | | | | : | CE |
| CE | : | | | P | | PE | : |device|
|device| : +------+ VPN tunnel : |router| |device| : | of |
| of |-:--| |================:===============| |--:-|VPN A|
|VPN A| : | | : +------+ +------+ : +------+
+------+ : | PE | : | | : |
+------+ : |device| Network interface | | : |
| CE | : | | : +------+ : +------+
|device|-:--| |================:===============| |--:-| CE |
| of | : +------+ : VPN tunnel | PE | : |device|
|VPN B| : | | |device| : | of |
+------+ : | | +------------+ +------------+ | | : |VPN B|
| : | | | Customer | | Network | +------+ : +------+
|Customer | | | management | | management | | | : |
|interface| | | function | | function | | |Customer |
| | | +------------+ +------------+ | |interface|
| | | | | |
+---------+ +------------------------------------+ +---------+
| Access | |<---------- SP network(s) --------->| | Access |
| network | | single or multiple SP domains | | network |
Figure 2.1: Reference model for layer 3 PE-based VPN.
+----------+ +----------+
+-----+ |PE device | |PE device | +-----+
| CE | | | | | | CE |
| dev | Access | +------+ | | +------+ | Access | dev |
| of | conn. | |VFI of| | VPN tunnel | |VFI of| | conn. | of |
|VPN A|----------|VPN A |======================|VPN A |----------|VPN A|
+-----+ | +------+ | | +------+ | +-----+
| | | |
+-----+ Access | +------+ | | +------+ | Access +-----+
| CE | conn. | |VFI of| | VPN tunnel | |VFI of| | conn. | CE |
| dev |----------|VPN B |======================|VPN B |----------| dev |
| of | | +------+ | | +------+ | | of |
|VPN B| | | | | |VPN B|
+-----+ +----------+ +----------+ +-----+
Figure 2.2: Relationship between entities in reference model (1).
+----------+ +----------+
+-----+ |PE device | |PE device | +-----+
| CE | | | | | | CE |
| dev | Access | +------+ | | +------+ | Access | dev |
| of | conn. | |VFI of| | | |VFI of| | conn. | of |
|VPN A|----------|VPN A | | | |VPN A |----------|VPN A|
+-----+ | +------+\| Tunnel |/+------+ | +-----+
| >==================< |
+-----+ Access | +------+/| |\+------+ | Access +-----+
| CE | conn. | |VFI of| | | |VFI of| | conn. | CE |
| dev |----------|VPN B | | | |VPN B |----------| dev |
| of | | +------+ | | +------+ | | of |
|VPN B| | | | | |VPN B|
+-----+ +----------+ +----------+ +-----+Figure 2.2: Relationship between entities in reference model (1).
Figure 2.3: Relationship between entities in reference model (2).
2.1.1. Entities in the Reference Model 💖
okay so here's the cast of characters in the reference model 🧚 each one has its own job, so let's go through them 💅
- Customer edge (CE) device 🌸 In the context of layer 3 provider-provisioned PE-based VPNs, a CE device may be a router, LSR, or host that has no VPN-specific functionality. It is attached via an access connection to a PE device.
- P router 🦋 A router within a provider network which is used to interconnect PE devices, but which does not have any VPN state and does not have any direct attachment to CE devices.
- Provider edge (PE) device 👑 In the context of layer 3 provider-provisioned PE-based VPNs, a PE device implements one or more VFIs and maintains per-VPN state for the support of one or more VPNs. It may be a router, LSR, or other device that includes VFIs and provider edge VPN functionality such as provisioning, management, and traffic classification and separation. (Note that access connections are terminated by VFIs from the functional point of view). A PE device is attached via an access connection to one or more CE devices.
- Customer site 🍓 A customer site is a set of users that have mutual IP reachability without use of a VPN backbone that goes beyond the site.
- SP networks 🌈 An SP network is an IP or MPLS network administered by a single service provider.
- Access connection ✨ An access connection represents an isolated layer 2 connectivity between a CE device and a PE device. Access connections can be, e.g., dedicated physical circuits, logical circuits (such as FR, ATM, and MAC), or IP tunnels (e.g., using IPsec, L2TP, or MPLS).
- Access network 🫶 An access network provides access connections between CE and PE devices. It may be a TDM network, layer 2 network (e.g., FR, ATM, and Ethernet), or IP network over which access is tunneled (e.g., using L2TP [RFC2661] or MPLS).
- VPN tunnel 🔥 A VPN tunnel is a logical link between two VPN edge devices. A VPN packet is carried on a tunnel by encapsulating it before transmitting it over the VPN backbone.
Multiple VPN tunnels at one level may be hierarchically multiplexed into a single tunnel at another level. For example, multiple per- VPN tunnels may be multiplexed into a single PE to PE tunnel (e.g., GRE, IP-in-IP, IPsec, or MPLS tunnel). This is illustrated in Figure 2.3. See section 4.3 for details.
- VPN forwarding instance (VFI) 🧚 A single PE device is likely to be connected to a number of CE devices. The CE devices are unlikely to all be in the same VPN. The PE device must therefore maintain a separate forwarding instances for each VPN to which it is connected. A VFI is a logical entity, residing in a PE, that contains the router information base and forwarding information base for a VPN. The interaction between routing and VFIs is discussed in section 4.4.2.
- Customer management function 💜 The customer management function supports the provisioning of customer specific attributes, such as customer ID, personal information (e.g., name, address, phone number, credit card number, and etc.), subscription services and parameters, access control policy information, billing and statistical information, and etc.
The customer management function may use a combination of SNMP manager, directory service (e.g., LDAP [RFC3377]), or proprietary network management system.
- Network management function 🎀 The network management function supports the provisioning and monitoring of PE or CE device attributes and their relationships.
The network management function may use a combination of SNMP manager, directory service (e.g., LDAP [RFC3377]), or proprietary network management system.
2.1.2. Relationship Between CE and PE 🌸
okay this part is all about multi-homing, which is basically when a CE device hooks up to more than one PE device 🫶 that's giving robustness ✨ for robustness, a CE device may be connected to more than one PE device, resulting in a multi-homing arrangement. Four distinct types of multi-homing arrangements, shown in Figure 2.4, may be supported 💅
+---------------- +---------------
| |
+------+ +------+
+---------| PE | +---------| PE |
| |device| | |device| SP network
| +------+ | +------+
+------+ | +------+ |
| CE | | | CE | +---------------
|device| | SP network |device| +---------------
+------+ | +------+ |
| +------+ | +------+
| | PE | | | PE |
+---------|device| +---------|device| SP network
+------+ +------+
| |
+---------------- +---------------
This type includes a CE device connected
to a PE device via two access connections.
(a) (b)
+---------------- +---------------
| |
+------+ +------+ +------+ +------+
| CE |-----| PE | | CE |-----| PE |
|device| |device| |device| |device| SP network
+------+ +------+ +------+ +------+
| | | |
| Backdoor | | Backdoor +---------------
| link | SP network | link +---------------
| | | |
+------+ +------+ +------+ +------+
| CE | | PE | | CE | | PE |
|device|-----|device| |device|-----|device| SP network
+------+ +------+ +------+ +------+
| |
+---------------- +---------------
(c) (d)
Figure 2.4: Four types of double-homing arrangements.2.1.3. Interworking Model 🌈
Okay so it's super natural to assume you'll end up with a bunch of different layer 3 VPN approaches running around, especially when the VPN backbone spans more than one SP network 🌸. For example, (1) each SP picks one or more layer 3 PE-based VPN approaches from multiple vendors' implementations, which means different SPs might pick different approaches, and (2) an SP might run multiple networks of layer 3 PE-based VPNs (like an old network and a new one). So it's really important to allow interworking between layer 3 PE-based VPNs that use different approaches 🫶
There are three scenarios that make this interworking happen 💖
- Interworking function 🧚
This scenario uses a PE located at one or more points that sit logically between VPNs based on different layer 3 VPN approaches. For example, this PE might be on the boundary between SP networks that use different layer 3 VPN approaches [VPN-DISC]. A PE at one of these points is called an interworking function (IWF), and an example configuration is shown in Figure 2.5 ✨
+------------------+ +------------------+
| | | |
+------+ VPN tunnel +------+ VPN tunnel +------+
| |==============| |==============| |
| | | | | |
| PE | | PE | | PE |
| | |device| | |
|device| |(IWF) | |device|
| | VPN tunnel | | VPN tunnel | |
| |==============| |==============| |
+------+ +------+ +------+
| | | |
+------------------+ +------------------+
|<-VPN approach 1->| |<-VPN approach 2->|
Figure 2.5: Interworking function.- Interworking interface 🦋
This scenario uses tunnels between PEs that support different layer 3 VPN approaches. As shown in Figure 2.6, the interworking interface is the interface that exists between a pair of PEs and connects two SP networks built with different approaches. It's kind of like the customer interface between PE and CE, but the interworking interface is supported by tunnels that identify VPNs, while the customer interface is supported by access connections 💅
+------------------+ +------------------+
| | : | |
+------+ VPN tunnel +------+Tunnel: +------+ VPN tunnel +------+
| |============| |======:======| |============| |
| | | | : | | | |
| PE | | PE | : | PE | | PE |
| | | | : | | | |
|device| |device| : |device| |device|
| | VPN tunnel | |Tunnel: | | VPN tunnel | |
| |============| |======:======| |============| |
+------+ +------+ : +------+ +------+
| | : | |
+------------------+ Interworking +------------------+
|<-VPN approach 1->| interface |<-VPN approach 2->|
Figure 2.6: Interworking interface.- Customer-based interworking 👑
If a customer site has a CE attached to one kind of VPN and a CE attached to another kind, communication between the two kinds of VPN happens automatically 🎀
2.2. Reference Model for Layer 3 Provider-Provisioned CE-based VPN 💜
This subsection walks through the functional components and how they relate to each other when you're implementing layer 3 provider-provisioned CE-based VPN 🌷
Figure 2.7 shows the reference model for layer 3 provider-provisioned CE-based VPN. As you can see in Figure 2.7, the customer interface is the interface that exists between CE and PE devices 💕
In this model, a CE device maintains one or more VPN tunnel endpoints, and a PE device has no VPN-specific functionality. Because of that, the interworking issues from section 2.1.3 don't come up at all 🙌
+---------+ +------------------------------------+ +---------+
| | | | | |
| | | +------+ +------+ : +------+
+------+ : | | | | | | : | CE |
| CE | : | | | P | | PE | : |device|
|device| : +------+ VPN tunnel |router| |device| : | of |
| of |=:====================================================:=|VPN A|
|VPN A| : | | +------+ +------+ : +------+
+------+ : | PE | | | : |
+------+ : |device| | | : |
| CE | : | | VPN tunnel +------+ : +------+
|device|=:====================================================:=| CE |
| of | : +------+ | PE | : |device|
|VPN B| : | | |device| : | of |
+------+ : | | +------------+ +------------+ | | : |VPN B|
| : | | | Customer | | Network | +------+ : +------+
|Customer | | | management | | management | | | : |
|interface| | | function | | function | | |Customer |
| | | +------------+ +------------+ | |interface|
| | | | | |
+---------+ +------------------------------------+ +---------+
| Access | |<---------- SP network(s) --------->| | Access |
| network | | | | network |
Figure 2.7: Reference model for layer 3
provider-provisioned CE-based VPN.2.2.1. Entities in the Reference Model 🧩
Here's the rundown on each entity in the reference model 📝
- Customer edge (CE) device 🏠
In the context of layer 3 provider-provisioned CE-based VPNs, a CE device provides layer 3 connectivity to the customer site. It may be a router, LSR, or host that maintains one or more VPN tunnel endpoints. A CE device is attached via an access connection to a PE device and is usually located at the edge of a customer site or co-located on an SP premises 🌟
- P router (see section 2.1.1) 🛤️
- Provider edge (PE) device 🌐
In the context of layer 3 provider-provisioned CE-based VPNs, a PE device may be a router, LSR, or other device that has no VPN-specific functionality. It is attached via an access connection to one or more CE devices 💅
- Customer Site (see section 2.1.1) 🏡
- SP networks 📡
An SP network is a network administrated by a single service provider. It is an IP or MPLS network. In the context of layer 3 provider-provisioned CE-based VPNs, the SP network consists of the SP's network and the SP's management functions, which manage both its own network and the customer's VPN functions on the CE device 💜
- Access connection (see section 2.1.1) 🔌
- Access network (see section 2.1.1) 🕸️
- VPN tunnel 🦄
A VPN tunnel is a logical link between two entities. It's created by encapsulating packets within an encapsulating header for the purpose of transmitting between those two entities, in support of VPNs. In the context of layer 3 provider-provisioned CE-based VPNs, a VPN tunnel is an IP tunnel (for example, using GRE, IP-in-IP, IPsec, or L2TP) or an MPLS tunnel between two CE devices over the SP's network 🫧
- Customer management function (see section 2.1.1) 🗂️
- Network management function 🛠️
The network management function supports the provisioning and monitoring of PE or CE device attributes and their relationships. It covers the PE and CE devices that define the VPN connectivity of the customer VPNs 📊
The network management function may use a combination of SNMP manager, directory service (e.g., LDAP [RFC3377]), or proprietary network management system 🌈
3. Customer Interface 🎀
3.1. VPN Establishment at the Customer Interface 💖
3.1.1. Layer 3 PE-based VPN 🌸
It's necessary for each PE device to know which CEs it's attached to, and which VPNs each CE is associated with 🧠
VPN membership refers to the association of VPNs, CEs, and PEs. A given CE belongs to one or more VPNs. Each PE is therefore associated with a set of VPNs, and a given VPN has a set of associated PEs that support that VPN. If a PE has at least one attached CE belonging to a given VPN, then state information for that VPN (e.g., the VPN routes) must exist on that PE. The set of VPNs that exist on a PE may change over time as customer sites are added to or removed from the VPNs 🔄
In some layer 3 PE-based PPVPN schemes, VPN membership information (i.e., information about which PEs are attached to which VPNs) is explicitly distributed. In others, the membership information is inferred from other information that is distributed. Different schemes use the membership information in different ways, e.g., some to determine what set of tunnels to set up, and some to constrain the distribution of VPN routing information 🤓
A VPN site may be added or deleted as a result of a provisioning operation carried out by the network administrator, or it may be dynamically added or deleted as a result of a subscriber-initiated operation. So VPN membership information may be either static or dynamic, as discussed below ✨
3.1.1.1. Static Binding 🔗
Static binding happens when a provisioning action binds a particular PE-CE access link to a particular VPN. For example, a network administrator may set up a dedicated link layer connection, such as an ATM VCC or a FR DLCI, between a PE device and a CE device. In this case the binding between a PE-CE access connection and a particular VPN is fixed at provisioning time, and it stays the same until another provisioning action changes it 💜
3.1.1.2. Dynamic Binding ✨
Dynamic binding is when some real-time protocol interaction temporarily hooks a particular PE-CE access link up to a particular VPN 🦋. Like, a mobile user dials into the provider network, goes through user authentication and VPN selection, and the PE they end up on isn't permanently tied to them 💅. It's usually whichever PE is geographically close to wherever they happen to be 📍. Another example is a permanent access connection between a PE and a CE at a public place like a hotel or conference center 🏨, where lots of users take turns on the same link and each one might want a different VPN 🫶.
To support dynamically connected users, PPP and RADIUS are commonly used, because these protocols handle user identification, authentication and VPN selection 🔐. Other mechanisms are also possible 🌈. For example, a user's HTTP traffic might first get intercepted by a PE and diverted to a provider hosted web server. After a dialogue that includes user authentication and VPN selection, the user can then be connected to the required VPN. This is sometimes called a "captive portal" 🍓.
No matter which mechanisms handle the authentication and VPN selection, dynamic binding means a user for a given VPN can show up at any PE at any time 😭. So VPN membership can change at any moment because of user-initiated actions, rather than network provisioning actions. This suggests there needs to be a way to distribute membership information rapidly and reliably when these user-initiated actions take place 🚀.
3.1.2. Layer 3 Provider-Provisioned CE-based VPN 👑
In layer 3 provider-provisioned CE-based VPNs, the PE devices have no knowledge of the VPNs 🙈. A PE device attached to a particular VPN has no knowledge of the addressing or routing information of that specific VPN.
CE devices have IP or MPLS connectivity via a connection to a PE device, which just provides ordinary connectivity to the global IP address space or to an address space which is unique in a particular SPs network. The IP connectivity may be via a static binding, or via some kind of dynamic binding.
The establishment of the VPNs is done at each CE device, making use of the IP or MPLS connectivity to the others. Therefore, it is necessary for a given CE device to know which other CE devices belong to the same VPN. In this context, VPN membership refers to the association of VPNs and CE devices 💖.
3.2. Data Exchange at the Customer Interface 📦
3.2.1. Layer 3 PE-based VPN
For layer 3 PE-based VPNs, the exchange is normal IP packets, transmitted in the same form that's available for interconnecting routers in general. For example, IP packets may be exchanged over Ethernet, SONET, T1, T3, dial-up lines, and any other link layer available to the router. It is important to note that those link layers are strictly local to the interface for the purpose of carrying IP packets, and are terminated at each end of the customer interface. The IP packets may contain addresses which, while unique within the VPN, are not unique on the VPN backbone. Optionally, the data exchange may use MPLS to carry the IP packets.
3.2.2. Layer 3 Provider-Provisioned CE-based VPN
The data exchanged at the customer interface are always normal IP packets that are routable on the VPN backbone, and whose addresses are unique on the VPN backbone. Optionally, MPLS frames can be used, if the appropriate label-switched paths exist across the VPN backbone. The PE device doesn't know whether these packets are VPN packets or not. At the current time, MPLS is not commonly offered as a customer-visible service, so CE-based VPNs most commonly make use of IP services 🌸.
3.3. Customer Visible Routing 🗺️
Once VPN tunnels are set up between pairs of VPN edge devices, it's necessary to set up mechanisms which ensure that packets from the customer network get sent through the proper tunnels. This routing function must be performed by the VPN edge device.
3.3.1. Customer View of Routing for Layer 3 PE-based VPNs
There is a PE-CE routing interaction which enables a PE to obtain those addresses, from the customer network, that are reachable via the CE. The PE-CE routing interaction also enables a CE device to obtain those addresses, from the customer network, which are reachable via the PE; these will generally be addresses that are at other sites in the customer network.
The PE-CE routing interaction can make use of static routing, an IGP (such as RIP, OSPF, IS-IS, etc.), or BGP.
If the PE-CE interaction is done via an IGP, the PE will generally maintain at least several independent IGP instances; one for the backbone routing, and one for each VPN. Thus the PE participates in the IGP of the customer VPNs, but the CE does not participate in the backbone's IGP.
If the PE-CE interaction is done via BGP, the PE MAY support one instance of BGP for each VPN, as well as an additional instance of BGP for the public Internet routes. Alternatively, the PE might support a single instance of BGP, using, e.g., different BGP Address Families to distinguish the public Internet routes from the VPN routes.
Routing information which a PE learns from a CE in a particular VPN must be forwarded to the other PEs that are attached to the same VPN. Those other PEs must then forward the information in turn to the other CEs of that VPN 🔁.
The PE-PE routing distribution can be done as part of the same routing instance to which the PE-CE interface belongs. Alternatively, it can be done via a different routing instance, possibly using a different routing algorithm. In this case, the PE must redistribute VPN routes from one routing instance to another.
Note that VPN routing information is never distributed to the P routers. VPN routing information is known at the edge of the VPN backbone, but not in the core 🤫.
If the VPN's IGP is different than the routing algorithm running on the CE-PE link, then the CE must support two routing instances, and must redistribute the VPN's routes from one instance to the other (e.g., [VPN-BGP-OSPF]).
In the case of layer 3 PE-based VPNs, a single PE device is likely to provide service for several different VPNs. Since different VPNs may have address spaces which are not mutually unique, a PE device must have several forwarding tables, in general one for each VPN to which it is attached. These will be referred to as VPN Forwarding Instances (VFIs). Each VFI is a logical entity internal to the PE device. VFIs are defined in section 2.1.1, and discussed in more detail in section 4.4.2 🧚.
The scaling and management of the customer network (as well as the operation of the VPN) will depend upon the implementation approach and the manner in which routing is done.
3.3.1.1. Routing for Intranets 🏠
In the intranet case, all of the sites to be interconnected belong to the same administration (for example, the same company). The options for routing within a single customer network include:
- A single IGP area (using OSPF, IS-IS, or RIP)
- Multiple areas within a single IGP
- A separate IGP within each site, with routes redistributed from each site to backbone routing (i.e., to a backbone as seen by the customer network).
Note that these options look at routing from the perspective of the overall routing in the customer network. This list does not specify whether PE device is considered to be in a site or not. This issue is discussed below.
A single IGP area (such as a single OSPF area, a single IS-IS area, or a single instance of RIP between routers) may be used. One could have all routers within the customer network (including the PEs, or more precisely, including a VFI within each PE) appear within a single area. Tunnels between the PEs could also appear as normal links.
In some cases the multi-level hierarchy of OSPF or IS-IS may be used. One way to apply this to VPNs would be to have each site be a single OSPF or IS-IS area. The VFIs will participate in routing within each site as part of that area. The VFIs may then be interconnected as the backbone (OSPF area 0 or IS-IS level 2). If OSPF is used, the VFIs therefore appear to the customer network as area border routers. If IS-IS is used, the VFIs therefore participate in level 1 routing within the local area, and appear to the customer network as if they are level 2 routers in the backbone 🪜.
Where an IGP is used across the entire network, it is straightforward for VPN tunnels, access connections, and backdoor links to be mixed in a network. Given that OSPF or IS-IS metrics will be assigned to all links, paths via alternate links can be compared and the shortest cost path will be used regardless of whether it is via VPN tunnels, access connections, or backdoor links. If multiple sites of a VPN do not use a common IGP, or if the backbone does not use the same common IGP as the sites, then special procedures may be needed to ensure that routes to/from other sites are treated as intra-area routes, rather than as external routes (depending upon the VPN approach taken).
Another option is to operate each site as a separate routing domain. For example, each site could operate as a single OSPF area, a single IS-IS area, or a RIP domain. In this case the per-site routing domains will need to redistribute routes into a backbone routing domain (Note: in this context the "backbone routing domain" refers to a backbone as viewed by the customer network). In this case it is optional whether or not the VFIs participate in the routing within each site.
3.3.1.2. Routing for Extranets 🤝
In the extranet case, the sites to be interconnected belong to multiple different administrations. In this case IGPs (such as OSPF, IS-IS, or RIP) are normally not used across the interface between organizations. Either static routes or BGP may be used between sites. If the customer network administration wishes to maintain control of routing between its site and other networks, then either static routing or BGP may be used across the customer interface. If the customer wants to outsource all such control to the provider, then an IGP or static routes may be used at this interface.
The use of BGP between sites allows for policy based routing between sites. This is particularly useful in the extranet case. Note that private IP addresses or non-unique IP address (e.g., unregistered addresses) should not be used for extranet communication 🚫.
3.3.1.3. CE and PE Devices for Layer 3 PE-based VPNs ✨
okay so when you run a single IGP area across an intranet 🌸, the whole customer network hangs out in one area of an IGP. 🫶 In this case, for layer 3 PE-based VPNs, both the CE and PE devices show up as normal routers inside that area. no extra drama 💅
The other options split things into routing within a site and routing between sites. A CE device would normally count as part of whatever site it lives in. BUT there's a choice to make about how the PE devices get treated. 👀
Sometimes, from the point of view of routing inside the customer network, a PE device (or more precisely a VFI within a PE device) can be treated as internal to the same area or routing domain as the site it's attached to. 🦋 This is lowkey great for the customer network admins, since inter-area routing gets handled by the provider. 💖
For example, from the view of routing inside the customer network, the CE devices might be the area border or AS boundary routers of the IGP area. 👑 In that case, static routing, BGP, or whatever routing is used in the backbone may be used across the customer interface. ✨
3.3.2. Customer View of Routing for Layer 3 Provider-Provisioned CE-based VPNs 🎀
For layer 3 provider-provisioned CE-based VPNs, the PE devices aren't aware of the set of addresses that are reachable at particular customer sites. 🙈 The CE and PE devices don't exchange the customer's routing information. not me being dramatic about it, that's just how it works 💅
Customer sites that belong to the same VPN can swap routing information through the CE-CE VPN tunnels, which show up to the customer's IGP as router adjacencies. 🌈 Or, instead of exchanging routing info through the VPN tunnels, the SP's management system can take care of configuring the static route information of one site toward the other sites in the VPN. 🧚
Routing within the customer site can be done in any possible way, using any kind of routing protocols (see section 3.3.3). 🍓
As the CE device receives an IP or MPLS service from the SP, the CE and PE devices may exchange routing information that is meaningful within the SP routing realm. 🔥
Moreover, because forwarding of tunneled customer packets in the SP network is based on global IP forwarding, the routes to the various CE devices must be known across the entire SP's network. 💜
This means a CE device may need to participate in two different routing processes:
- routing in its own private network (VPN routing), within its own site and with the other VPN sites through the VPN tunnels, possibly using private addresses. 🦋
- routing in the SP network (global routing), as such, peering with its PE. 🌸
However, in many scenarios, the use of static/default routes at the CE-PE interface might be all the global routing that's required. ✨
3.3.3. Options for Customer Visible Routing 💖
The following technologies are available for the exchange of routing information. 🫶
- Static routing Routing tables may be configured through a management system. 🛠️
- RIP (Routing Information Protocol) [RFC2453] RIP is an interior gateway protocol used within an autonomous system. 🌈 It sends out routing updates at regular intervals and whenever the network topology changes. Routing information then gets propagated by the adjacent routers to their neighbors, and so on through the entire network. A route from a source to a destination is the path with the least number of routers. This number is called the "hop count," and its maximum value is 15. 💀 This implies that RIP is suitable for small- or medium-sized networks. 🎀
- OSPF (Open Shortest Path First) [RFC2328] OSPF is an interior gateway protocol applied to a single autonomous system. 👑 Each router distributes the state of its interfaces and neighboring routers as a link state advertisement, and maintains a database describing the autonomous system's topology. A link state is advertised every 30 minutes or when the topology is reconfigured. ⭐
Each router maintains an identical topological database, from which it constructs a tree of shortest paths with itself as the root. The algorithm is known as the Shortest Path First, or SPF. The router generates a routing table from the tree of shortest paths. OSPF supports a variable length subnet mask, which enables effective use of the IP address space. 🧠
OSPF allows sets of networks to be grouped together into an area. Each area has its own topological database. The topology of the area is invisible from outside its area. The areas are interconnected via a "backbone" network, which distributes routing information between the areas. The area routing scheme can reduce the routing traffic and compute the shortest path trees, and it's indispensable for larger scale networks. 🔥
Each multi-access network with multiple routers attached has a designated router. The designated router generates a link state advertisement for the multi-access network and synchronizes the topological database with other adjacent routers in the area. The concept of designated router can thus reduce the routing traffic and compute shortest path trees. To achieve high availability, a backup designated router is used. 🫶
- IS-IS (intermediate system to intermediate system) [RFC1195] IS-IS is a routing protocol designed for the OSI (Open Systems Interconnection) protocol suites. Integrated IS-IS is derived from IS-IS in order to support the IP protocol. In the Internet community, IS-IS means integrated IS-IS. 🌸 In this, a link state is advertised over a connectionless network service. IS-IS has the same basic features as OSPF. They include link state advertisement and maintenance of a topological database within an area, calculation of a tree of shortest paths, generation of a routing table from a tree of shortest paths, the area routing scheme, a designated router, and a variable length subnet mask. 💅
- BGP-4 (Border Gateway Protocol version 4) [RFC1771] BGP-4 is an exterior gateway protocol applied to the routing of inter-autonomous systems. 🦋 A BGP speaker establishes a session with other BGP speakers and advertises routing information to them. A session may be an External BGP (EBGP) that connects two BGP speakers within different autonomous systems, or an internal BGP (IBGP) that connects two BGP speakers within a single autonomous system. Routing information is qualified with path attributes, which differentiate routes for the purpose of selecting an appropriate one from possible routes. Also, routes are grouped by the community attribute [RFC1997] [BGP-COM]. 🌈
The IBGP mesh size tends to increase dramatically with the number of BGP speakers in an autonomous system. 😭 BGP can reduce the number of IBGP sessions by dividing the autonomous system into smaller autonomous systems and grouping them into a single confederation [RFC3065]. Route reflection is another way to reduce the number of IBGP sessions [RFC1966]. BGP divides the autonomous system into clusters. Each cluster establishes the IBGP full mesh within itself, and designates one or more BGP speakers as "route reflectors," which communicate with other clusters via their route reflectors. Route reflectors in each cluster maintain path and attribute information across the autonomous system. The autonomous system still functions like a fully meshed autonomous system. On the other hand, confederations provide finer control of routing within the autonomous system by allowing for policy changes across confederation boundaries, while route reflection requires the use of identical policies. 🧚
BGP-4 has been extended to support IPv6, IPX, and others as well as IPv4 [RFC2858]. Multiprotocol BGP-4 carries routes from multiple "address families." 🌟
4. Network Interface and SP Support of VPNs 💜
4.1. Functional Components of a VPN ✨
okay so a VPN has three basic functional components 💅
- A mechanism to acquire VPN membership/capability information 🗂️
- A mechanism to tunnel traffic between VPN sites 🚇
- For layer 3 PE-based VPNs, a means to learn customer routes, distribute them between the PEs, and advertise reachable destinations to customer sites 🛣️
Depending on the implementation, these functions could be handled separately for each VPN, or one common mechanism could cover all of them. For instance, a single process could handle the routing information for every VPN, or a separate process could be created for each VPN. It's giving flexibility 🧚
Logically, setting up a VPN can be thought of as three stages. In the first stage, the VPN edge devices find out about each other. In the second stage, they set up tunnels to each other. In the third stage, they swap routing information 🦋 But ngl, not every VPN solution has to split up this way. In some VPN solutions, tunnels aren't set up after the membership info is learned, and instead pre-existing tunnels get picked and used. Also, in some VPN solutions, the membership info and the routing info are combined 🌸
In the membership/capability discovery stage, the devices need membership and capability info so they can figure out whether two VPN edge devices support any VPNs in common. One way to do this is to exchange the VPN identifiers of the configured VPNs at each VPN edge device. The devices also need to figure out each other's capabilities, so they can agree on a common mechanism for tunneling and/or routing. Like, if site A supports both IPsec and MPLS as tunneling mechanisms and site B only supports MPLS, they can both agree to use MPLS for tunneling 🫶 Sometimes the capability info is determined implicitly. For example, some SPs may implement a single VPN solution, so there's nothing to negotiate. Likewise, the routing information for VPNs can be distributed using the methods discussed in section 4.4 🔥
In the tunnel establishment stage, mechanisms may need to be invoked to actually set up the tunnels. With IPsec, for instance, this could involve using IKE to exchange keys and policies for securing the data traffic 🔐 However, if IP tunneling, e.g., is used, there may not be any need to explicitly set up tunnels. And if MPLS tunnels are used, they may be pre-established as part of normal MPLS functioning 🎀
In the VPN routing stage, routing information for the VPN sites has to be exchanged before data can move between the sites. Depending on the VPN model, this could involve static routes, IGPs such as OSPF/ISIS/RIP, or an EGP such as BGP 🗺️
VPN membership and capability information can be distributed from a central management system, using protocols such as, e.g., LDAP. Or it can be distributed manually. But manual configuration doesn't scale and is error prone, so its use is discouraged 🙈 A third option is to distribute VPN information through protocols that automatically and consistently get the info out in a timely manner, kind of like how routing protocols handle routing information 🚀 This may suggest that the information be carried in the routing protocols themselves, though only if that can be done without negatively impacting the essential routing functions 💜
It's honestly wild how much information has to be exchanged just to establish and maintain a VPN 😭 The scaling and stability consequences need to be analyzed for any VPN approach 📊
Every VPN solution must address the functionality of all three components, but the combinations of mechanisms used to provide that functionality, and the order in which the pieces get carried out, may be different ✨
For layer 3 provider-provisioned CE-based VPNs, the VPN service is offering tunnels between CE devices. IP routing for the VPN is done by the customer network 🏠 With these solutions, the SP is involved in the membership/capability discovery stage and the tunnel establishment stage. The IP routing functional component may be entirely up to the customer network. Or, the SP's management system may be responsible for distributing the reachability information of the VPN sites to the other sites of the same VPN 🧚♀️
4.2. VPN Establishment and Maintenance 🌈
For a layer 3 provider-provisioned VPN, the SP is responsible for establishing and maintaining the VPNs. There are lots of different approaches and schemes for providing layer 3 PPVPNs, but there are some generic problems that any VPN solution has to deal with, including:
- For PE-based VPNs, when a new site is added to a PE, how do the other PEs find out about it? When a PE first gets attached to a given VPN, how does it determine which other PEs are attached to the same VPN? For CE-based VPNs, when a new site is added, how does its CE find out about all the other CEs at other sites of the same VPN? 👀
- For layer 3 PE-based VPNs to scale, all routes for all VPNs can't live on all PEs. So how is the distribution of VPN routing information constrained so it only goes to the devices that need it? 📦
- An administrator may want to provision different topologies for different VPNs (e.g., a full mesh or a hub & spoke topology). How is this achieved? 🕸️
This section looks at some of these generic problems and at some of the mechanisms that can be used to solve them 💖
4.2.1. VPN Discovery 🔍
Mechanisms are needed to acquire the information that allows VPNs to be established and maintained. This may include, for example, information on VPN membership, topology, and VPN device capabilities. This information may be statically configured, or distributed by an automated protocol. Because of how these mechanisms and protocols work, a device can figure out where to set up tunnels, and where to advertise the VPN routes for each VPN 🗝️
With a physical network, the equivalent problem can be solved by controlling the physical interconnection of links, and by having a router run a discovery/hello protocol over its locally connected links. With VPNs, both the routers and the links (tunnels) may be logical entities, so some other mechanisms are needed 🧠
There are a number of different approaches to VPN discovery. One scheme uses the network management system to configure and provision the VPN edge devices. This approach can also be used to distribute VPN discovery information, either using proprietary protocols or using standard management protocols and MIBs 🛠️ Another approach has the VPN edge devices act as clients of a centralized directory or database server that holds the VPN discovery information 📚 Another possibility is piggybacking VPN discovery information onto a routing protocol running between the VPN edge devices [VPN-DISC] 🥰
4.2.1.1. Network Management for Membership Information 🖥️
SPs use network management extensively to configure and monitor the many devices spread throughout their networks. This approach could also be used for distributing VPN-related information. A network management system (either centralized or distributed) could be used by the SP to configure and provision VPNs on the VPN edge devices at various locations. VPN configuration information could be entered into a network management application and distributed to the remote sites through the same means used to distribute other network management information. This approach is most natural when all the devices that must be provisioned are inside a single SP's network, since the SP has access to all VPN edge devices in its domain 🏢 Security and access control are important, and could be achieved using, for example, SNMPv3, SSH, or IPsec tunnels 🔒
4.2.1.2. Directory Servers 📂
An SP typically needs to maintain a database of VPN configuration/membership information, no matter which mechanisms are used to distribute it. LDAPv3 [RFC3377] is a standard directory protocol that makes it possible to use a common mechanism for both storing that information and distributing it 🙌
To make different implementations work together, as well as the management systems of different SPs, a standard schema for representing VPN membership and configuration information would have to be developed 🤝
LDAPv3 supports authentication of messages and associated access control, which can be used to limit access to VPN information to authorized entities 🔐
4.2.1.3. Augmented Routing for Membership Information 🛤️
Extensions to the use of existing BGP mechanisms for distributing VPN membership information are proposed in [VPN-2547BIS]. In that scheme, BGP is used to distribute VPN routes, and each route carries a set of attributes that indicate which VPN (or VPNs) the route belongs to. This lets the VPN discovery information and the routing information be combined in a single protocol. Information needed to set up per-VPN tunnels can also be carried as attributes of the routes. This takes advantage of BGP's ability to carry large amounts of routing information effectively 💪
It's also possible to use BGP to distribute just the membership/capability information, while using a different technique to distribute the routing. A BGP update message would be used to indicate that a PE is attached to a particular VPN, and a BGP withdraw message would be used to indicate that a PE has stopped being attached to a particular VPN. This takes advantage of BGP's ability to dynamically distribute real-time changes in a reliable and fairly rapid manner ⚡ Plus, if a BGP route reflector is used, PEs never have to be provisioned with each other's IP addresses at all 👑 Both cases make use of BGP's mechanisms, such as route filters, for constraining the distribution of information 🎯
Augmented routing may be done in combination with aggregated routing, as discussed in section 4.4.4. Of course, when using BGP to distribute any kind of VPN-specific information, you have to make sure you aren't disrupting the classical use of BGP for distributing public Internet routing information 🌍 For further discussion of this, see the discussion of aggregated routing in section 4.4.4 🦋
4.2.1.4. VPN Discovery for Inter-SP VPNs
okay bestie, this one's about what happens when the two sites of one VPN live on different SP networks 🌐✨ The SPs need a shared way to swap membership and capability info. That could be manual config, or an automated exchange, and automation gets way easier if a few VPN discovery mechanisms are standardized and supported across all the SPs involved 🫶 Inter-SP trust relationships also have to be set up, like deciding exactly which info about the VPNs is allowed to cross between SPs, and how much 🔐
Different SPs might use totally different discovery approaches 🙈 When that happens, multi-SP VPNs need some manual coordination and config. Not ideal, but it is what it is 💀
How much info has to be shared can vary hugely depending on the number or size of the multi-SP VPNs 📈 So the SPs have to figure out together how much membership info to expect, and how often that info changes. They may also need ways to authenticate the VPN membership info 🔑
VPN info should only be sent to the places that actually need it, whether that's inside one provider or across providers 🎯 This is where it's different from inter-provider routing info, which has to be spread all over the Internet 🌍 Also, two SPs jointly supporting a VPN should not force any third SP to keep state for that VPN. Again, compare that to inter-provider routing, where sending traffic from one SP to another can totally require routing state in a third SP 😭
Here's one possible example. Say SP A and SP C want to back the same VPN, and they're connected through SP B 🦋 B has to know how to route traffic between A and C, so it needs some info about A and C, like enough routing info to forward IP traffic, and/or to connect MPLS LSPs between PEs or route reflectors in A and C 🛣️ For scaling, though, it's really desirable that B does not need to know the VPN-specific stuff about the VPNs that A and C are running 🧚
4.2.2. Constraining Distribution of VPN Routing Information
okay so in layer 3 provider-provisioned CE-based VPNs, the VPN tunnels connect the CE devices 💖 That means IP routing info gets shared between the CE devices on the customer sites, and no extra constraints on distributing VPN routing info are needed here ✨
Layer 3 PE-based VPNs are a different story 👑 The PE devices have to know the VPN routing info for every VPN they're attached to. For scalability, you do not want a setup where every PE holds every route for every VPN 😱 Only the PEs attached to sites in a given VPN should hold that VPN's routing info. So the distribution of VPN routing info between PE devices has to be constrained 🎀
Membership can change on the fly, so there needs to be a mechanism that lets VPN route info reach any PE that has an attached user for that VPN, and also lets that info get removed once it's no longer needed 🌸
In the Virtual Router scheme, per-VPN tunnels have to be set up before any routes for a VPN get distributed, and then the routes travel through those tunnels 🚇 So by building the right set of tunnels, you're implicitly controlling how per-VPN routing info gets distributed, which is iconic 🔥 In this scheme, membership info is just the set of VPNs that exist on each PE, plus info about the desired topology. That lets a PE figure out which remote PEs it needs to build tunnels to for a particular VPN 🧠
In the aggregated routing scheme (see section 4.4.4), VPN routing info is constrained with route filtering instead 🧹 When VPN membership changes on a PE, the route filters between that PE and its peers can get adjusted. Each peer then adjusts the filters with its own peers in turn, so the changes ripple across the whole network 🌊 When BGP is used, this filtering may happen at route reflectors, as discussed in section 4.4.4 💜
4.2.3. Controlling VPN Topology
a VPN's topology is a set of nodes connected by tunnels 🕸️ It can be a full mesh, a hub and spoke, or an arbitrary topology. The set of nodes includes every VPN edge device that has attached sites for that VPN. No matter the topology, all VPN sites can reach each other. The topology just controls the path traffic takes between them 🗺️ For example, in one topology, traffic from site A to site B goes straight across the VPN backbone. In another, traffic from site A to site B has to go through site C first before it reaches site B 🚦
The simplest topology is a full mesh, where there's a tunnel between every pair of VPN edge devices 💅 If we assume point-to-point tunnels (rather than multipoint-to-point), a full mesh has N*(N-1)/2 duplex tunnels or N*(N-1) simplex tunnels for N VPN edge devices. Each tunnel uses some resources at a VPN edge device. Depending on the tunnel type, it may or may not use resources in intermediate routers or LSRs 🧮 One reason to pick a partial mesh is to cut down on the number of tunnels a VPN edge device, and/or the network, has to support 📉 Another reason is when an admin needs all traffic from certain sites to pass through one particular site for policy or control reasons, like forcing traffic through a firewall, or for monitoring or accounting 🛡️ Also, the topologies for each VPN are separate, so the same VPN edge device can be in a full mesh for one VPN and a partial mesh for another 🦄
A partial mesh might be a great fit for a VPN with lots of telecommuters and only a few corporate sites 🏠💼 Most traffic goes between telecommuters and the corporate sites, not between pairs of telecommuters. A hub and spoke topology matches that real traffic flow, with telecommuters on spoke VPN edge devices and corporate sites on hub VPN edge devices 🥰 Telecommuter-to-telecommuter traffic is still supported, it just goes through a hub VPN edge device 🌈
Picking a topology is an administrative choice, but it's useful to look at protocol mechanisms that automate building the topology you want, so there's less config to do 🤖 It's handy for a VPN edge device to advertise per-VPN topology info to other VPN edge devices. It's probably simplest to advertise that at the same time as membership info, using the same mechanisms 📣
The simple scheme is where a VPN edge device advertises itself as either a hub or a spoke, for each VPN it has 🎯 When other VPN edge devices receive that, they can use it to decide whether to set up a tunnel. A more comprehensive scheme lets a VPN edge device advertise a set of topology groups, with tunnels set up between a pair of VPN edge devices if they have a group in common 🫧
4.3. VPN Tunneling
okay, tunneling time 🚇✨ VPN solutions use tunneling to carry VPN packets across the VPN backbone, from one VPN edge device to another. There are different types of tunneling protocols, different ways to set up and maintain tunnels, and different ways to link tunnels to VPNs, like shared versus dedicated per-VPN tunnels 🎀 Sections 4.3.1 through 4.3.5 discuss some common traits shared by all forms of tunneling, plus some common problems that tunnels solve 💡 Section 4.3.6 surveys the available tunneling techniques 🦋 Also, tunneling protocol issues are generally independent of the mechanisms used for VPN membership and VPN routing, so you can think about them separately 🫶
One big reason for tunneling is that the packet addressing inside a VPN may have nothing to do with the packet addressing used between the VPN edge devices 🙃 For example, customer VPN traffic could use non-unique or private IP addressing [RFC1918]. An IPv6 VPN could also run across an IPv4 provider backbone 🌍 So the forwarding between VPN edge devices has to rely on info other than what's inside the VPN packets themselves 🧩 A tunneling protocol adds extra info, like an extra header or label, to a VPN packet, and that extra info is what gets used to forward the packet between VPN edge devices 📦
Tunneling can also optionally provide isolation between different VPN traffic flows 🔒 Different flows may have different QoS and security requirements, and those can be met by using different tunnels with the right characteristics ⭐ This lets a provider offer different service characteristics for traffic in different VPNs, or for subsets of traffic flows within a single VPN 💅
The specific tunneling protocols covered in this section are GRE, IP-in-IP, IPsec, and MPLS, because they're the most suitable for carrying VPN traffic across the VPN backbone 🏆 Other tunneling protocols, like L2TP [RFC2661], may be used as access tunnels, carrying traffic between a PE and a CE 🧁 Backbone tunneling is independent of, and orthogonal to, access tunneling, so the protocols for access tunneling aren't discussed here 🌸
4.3.1. Tunnel Encapsulations 🎀
okay so every tunneling protocol slaps an encapsulation on the packet, and that extra info is what the VPN backbone uses to forward it 🦋 Examples are in section 4.3.6, so keep that bookmarked 🔖
one big thing to clock is whether the SP network needs per-tunnel state to forward these packets 👀 For the IP tunneling schemes (GRE, IP-in-IP, and IPsec), the per-tunnel state is totally confined to the VPN edge devices ✨ Everyone else in the network is unaware of the tunnels and just forwards by the IP header 💅 MPLS is the odd one out here, since intermediate LSRs have to examine and swap the top label in the label stack, so it needs per-tunnel state 😭 You can cut down that state with hierarchical multiplexing and multi-point to point tunnels, which we'll get into below 🌸
another vibe check is tunneling overhead 🫶 IPsec can be pretty heavy, since it may include an ESP header, an ESP trailer, and an extra IP header 💀 The other mechanisms are lighter, and MPLS is the most lightweight of the bunch ✨ Any tunneling overhead can cause extra IP packet fragmentation if the resulting packet is too big for the underlying link layer 🙈 So it's super important to report reduced MTU sizes through things like path MTU discovery, so fragmentation gets avoided wherever possible 💖
and then there's transparency to the Internet 🧚 IP-based encapsulation can carry a packet anywhere in the Internet, but MPLS encapsulation can only carry a packet over IP networks that support MPLS 😤 If an MPLS-encapsulated packet has to cross the networks of multiple SPs, the adjacent SPs must have bilateral agreements to accept MPLS packets from each other 🤝 If only part of the backbone path lacks MPLS support, an MPLS-in-IP encapsulation can move the MPLS packets across that part of the backbone 🌈 It does add complexity though, ngl 😬 On the flip side, MPLS has efficiency advantages, especially where encapsulations may need to be nested 👑
transparency to the Internet is sometimes a requirement and sometimes not 💅 It totally depends on the sort of service the SP is offering its customer 🍓
4.3.2. Tunnel Multiplexing 🦋
when a tunneled packet shows up at the tunnel egress, it has to be possible to figure out which VPN it belongs to from its encapsulation header ✨ For MPLS, you infer that from the packet's label stack 🌸 For IP-based encapsulations, you can infer it from some combo of the IP source address, the IP destination address, and a "multiplexing field" in the encapsulation header 💜 That multiplexing field might have been explicitly designed for multiplexing, or it might be a field that wasn't originally meant for this but got pushed into service 🙈 Here's how it breaks down:
- GRE: Packets are associated to a VPN by source IP address, destination IP address, and Key field, even though the Key field was originally intended for authentication 🔑
- IP-in-IP: Packets are associated to a VPN by the IP destination address in the outer header 📮
- IPsec: Packets are associated to a VPN by IP source address, IP destination address, and SPI field 🎟️
- MPLS: Packets are associated to a VPN by label stack 🏷️
okay here's the tea ☕ IP-in-IP doesn't have a real multiplexing field, so every VPN supported by a given PE needs its own, different IP destination address 😱 With the other IP-based encapsulations, a PE only needs a single IP address, and the multiplexing field does the work of telling the different VPNs apart 💖 So IP-in-IP has a significant disadvantage: it needs a potentially huge number of IP addresses to be allocated and assigned, and every single one has to be reachable through backbone routing 🥺
for the rest of this doc, "multiplexing field" means whichever field in the encapsulation header has to be used to tell different VPNs apart at a given PE 🫶 For IP-in-IP that's the destination IP address field, and for the other encapsulations it's a true multiplexing field ✨
4.3.3. Tunnel Establishment 🌈
when tunnels get set up, the tunnel endpoints have to agree on the multiplexing field values that will indicate that particular packets are in particular VPNs 🤝 Using "well known" or explicitly provisioned values wouldn't scale well as the number of VPNs grows 😵 So you need some kind of protocol interaction where the endpoints agree on those values 💅
for some tunneling protocols, setting up a tunnel needs an explicit exchange of signaling messages 📨 Usually the multiplexing field values get agreed on as part of that exchange ✨ For example, with IPsec encapsulation, the SPI field plays the multiplexing role and IKE signaling distributes the SPI values 🔐 With MPLS encapsulation, LDP, CR-LDP, or RSVP-TE can distribute the MPLS label value used as the multiplexing field 🏷️ The identity of the VPN the tunnel is associated with also has to be exchanged as part of the signaling protocol (for instance, a VPN-ID can be carried in the signaling protocol) 🆔 The big upside is that per-tunnel security, QoS, and other characteristics can be negotiable through the signaling protocol 🔥 The downside is that signaling imposes overhead, which can lead to scalability considerations we'll talk about further down 😩
for other tunneling protocols, there's no explicit protocol interaction that sets up the tunnel, so the multiplexing field values have to be exchanged some other way 🧩 For MPLS tunnels, for example, MPLS labels can be piggybacked on the protocols used to distribute VPN routes or VPN membership information 🎀 GRE and IP-in-IP have no associated signaling protocol, so the multiplexing values have to be distributed some other mechanism, like configuration, a control protocol, or being piggybacked in some manner on a VPN membership protocol 🦋
the resources used by the different tunnel establishment mechanisms can vary 📊 With a full mesh VPN topology and explicit signaling, each VPN edge device has to establish a tunnel to all the other VPN edge devices in each VPN 🕸️ That can eat up a lot of resources on a VPN edge device 😮💨 Recovery time after a device failure also matters, since the time to recover includes the time needed to reestablish a large number of tunnels ⏳
4.3.4. Scaling and Hierarchical Tunnels 🪜
if tunnels need state maintained in the core of the network, it may not be feasible to set up per-VPN tunnels between every pair of devices that are adjacent in some VPN topology 🚫 That would violate the principle that there is no per-VPN state in the core, and the core would scale poorly as the number of VPNs grows 📉 For example, MPLS tunnels require core network devices to maintain state for the topmost label in the label stack 🏷️ If every core router had to maintain one or more labels for every VPN, scaling would be really bad 💀
there are also scaling considerations tied to explicit signaling for tunnel establishment 📡 Even if the tunneling protocol doesn't maintain per-tunnel state in the core, a single VPN edge device may still need to handle a lot of tunnels, since that number grows with the number of VPNs and the number of neighbors per VPN 🫠 One way to reduce the number of tunnels is to use a VPN topology other than a full mesh 🕸️ That might not always be desirable though, and even with hub and spoke topologies, the hubs' VPN edge devices may still have to handle large numbers of tunnels 👑
if the core routers need to maintain any per-tunnel state at all, scaling can be greatly improved with hierarchical tunnels 🧁 You establish one tunnel between each pair of VPN edge devices, and then multiple VPN-specific tunnels get carried through that single "outer" tunnel 🥹 Now the amount of state only depends on the number of VPN edge devices, not the number of VPNs 🎉 Scaling can be improved even further by making the outer tunnels multipoint-to-point "merging" tunnels 🌟 Then the amount of state in the core is on the order of the number of VPN edge devices, not on the order of the square of that number 🤯 So the amount of tunnel state is roughly equivalent to the state needed to maintain IP routes to the VPN edge devices 💖 This is almost (if not quite) as good as using tunnels that don't require any state in the core 🥇
hierarchical tunnels can also reduce the amount of state kept in the VPN edge devices, especially when maintaining the outer tunnels takes more state than maintaining the per-VPN tunnels running inside them 🌸
other factors matter when deciding how many VPN edge to VPN edge "outer" tunnels to use 🤔 A single outer tunnel has the best scaling properties, but using more than one may let different traffic flows (from the same or different VPNs) get different QoS capabilities or different security characteristics 🛡️
when tunnels are used hierarchically, the tunnels in the hierarchy may all be the same type (for example, an MPLS label stack), or they may be different types (for example, a GRE tunnel carried inside an IPsec tunnel) 🪆
one example of hierarchical tunnels is setting up a number of different IPsec security associations that provide different levels of security between a given pair of VPN edge devices 🔐 Per-VPN GRE tunnels can then be grouped together and carried over the appropriate IPsec tunnel, instead of having a separate IPsec tunnel per VPN 📦 Another example is an MPLS label stack, where a single PE-PE LSP carries all the per-VPN LSPs 🏷️ The mechanisms used for label establishment are typically different: the PE-PE LSP could be established using LDP as part of normal backbone operation, while the per-VPN LSP labels get established by piggybacking on VPN routing (for example, using BGP), as discussed in sections 3.3.1.3 and 4.1 🎀✨
4.3.5. Tunnel Maintenance 🔧
Okay bestie, once a tunnel is up you still need to know it's actually working 👀 So we need ways to notice when a tunnel breaks and then respond so service comes back ✨
There's a sneaky problem with failures spreading when tunnels are multiplexed hierarchically 😭 Say a bunch of VPN-specific tunnels are multiplexed inside one PE to PE tunnel. If routing for each VPN runs over those VPN-specific tunnels (which can happen with CE-based and VR approaches), and the PE to PE tunnel fails, then multiple VPN-specific tunnels can fail at once. Layer 3 routing might react at the same time for every VPN that was using it. When the PE to PE tunnel comes back, you've got multiple VPN-specific tunnels and multiple routing protocol instances that all need to recover, and each one could need some control traffic exchanged. Total chaos 💀
So when a tunnel fails but can be restored quickly, it might be better to bring it back without the higher levels (like the other tunnels multiplexed inside the failed one) reacting at all 🫶 If higher levels wait before responding to a lower-level failed tunnel, that can limit the control traffic needed to fully restore correct service. But if the failed tunnel can't be restored quickly, then the tunnels or routing instances multiplexed over it do need to respond. It's preferable for them to respond quickly and without explicit action by network operators 👑
With most layer 3 provider-provisioned CE-based VPNs and the VR scheme, a per-VPN routing instance runs over the tunnel. That means any loss of connectivity between the tunnel endpoints gets picked up by the VPN routing instance, so tunnel failure gets detected fast 🏃♀️ Careful timer adjustment might be needed to avoid failure propagation, as discussed above 🙈 With the aggregated routing scheme, there isn't a per-VPN routing instance running over the tunnel. So if the tunnel can't be restored rapidly, some other scheme is needed to detect loss of connectivity 🦋
Detecting tunnel failure can be really hard to do reliably 😩 Mechanisms that can detect it include:
- loss of the underlying connectivity to the remote endpoint (as indicated by, e.g., "no IP route to host" or no MPLS label)
- timeout of higher layer "hello" mechanisms (e.g., IGP hellos, when the tunnel is an adjacency in some IGP)
- timeout of keep alive mechanisms in the tunnel establishment protocols (if any)
But none of these techniques gives completely reliable detection of all failure modes. Additional monitoring techniques may also be necessary 🌸
With hierarchical tunnels, it may be enough to only monitor the outermost tunnel for loss of connectivity. However, there may be failure modes in a device where the outermost tunnel is up but one of the inner tunnels is down 😱
4.3.6. Survey of Tunneling Techniques 🌈
Tunneling mechanisms give isolated communication between two CE-PE devices 💜 The available ones include (but aren't limited to): GRE [RFC2784] [RFC2890], IP-in-IP encapsulation [RFC2003] [RFC2473], IPsec [RFC2401] [RFC2402], and MPLS [RFC3031] [RFC3035].
Heads up, the subsections below cover tunnel overhead so the fragmentation risk is clear 📏 Some SP networks have layer 2 switches that enforce the standard/default MTU of 1500 bytes. In that case, any encapsulation at all creates a significant risk of fragmentation 😬 But layer 2 switch vendors are generally aware of IP tunneling and stacked VLAN overhead, so many switches practically allow an MTU of approximately 1512 bytes now. Then up to 12 bytes of encapsulation can be used before there's any risk of fragmentation. Plus, to improve TCP and NFS performance, switches that support 9K bytes "jumbo frames" are on the market too. With those, there's no risk of fragmentation 🚀
4.3.6.1. GRE [RFC2784] [RFC2890]
Generic Routing Encapsulation (GRE) is a protocol for encapsulating an arbitrary payload protocol over an arbitrary delivery protocol [RFC2784] ✨ It works when both the payload and the delivery protocol are IP, which is the case in layer 3 VPNs. A GRE tunnel is a tunnel whose packets are encapsulated by GRE 🎀
- Multiplexing: The GRE spec [RFC2784] doesn't explicitly support multiplexing 🙃 But the key field extension to GRE is specified in [RFC2890], and it may be used as a multiplexing field.
- QoS/SLA: GRE doesn't have intrinsic QoS/SLA capabilities, but it inherits whatever capabilities exist in the delivery protocol (IP). Additional mechanisms, like Diffserv or RSVP extensions [RFC2746], can be applied.
- Tunnel setup and maintenance: There is no standard signaling protocol for setting up and maintaining GRE tunnels 🤷♀️
- Large MTUs and minimization of tunnel overhead: The resulting packet is a delivery protocol header, followed by a GRE header, followed by the payload packet. When the delivery protocol is IPv4 and the
keyfield isn't present, GRE adds at least 28 bytes of overhead (36 bytes if the key field extension is used.) - Security: GRE doesn't provide any significant security 💀 The optional
keyfield can be used as a clear text password to help detect misconfigurations, but it doesn't provide integrity or authentication. An SP network that supports VPNs must do extensive IP address filtering at its borders to stop spoofed packets from penetrating the VPNs. If multi-provider VPNs are being supported, it may be difficult to set up these filters.
4.3.6.2. IP-in-IP Encapsulation [RFC2003] [RFC2473]
IP-in-IP is the format and procedures for IP-in-IP encapsulation 📦 It lets one IP datagram get encapsulated inside another. So the packet is an outer IP header, followed immediately by the payload packet. There's no middle header like GRE has. [RFC2003] and [RFC2473] specify the IPv4 and IPv6 encapsulations, respectively. Once the encapsulated datagram reaches the intermediate destination (named in the outer IP header), it gets decapsulated, giving back the original IP datagram. That then goes to the destination in the original destination address field 🌈
- Multiplexing: The IP-in-IP specs don't explicitly support multiplexing. But if a different IP address is used for every VPN, then the IP address field can be used for that purpose (see section 4.3.2 for detail).
- QoS/SLA: IP-in-IP doesn't have intrinsic QoS/SLA capabilities, but it inherits whatever capabilities exist for IP. Additional mechanisms, like RSVP extensions [RFC2764] or DiffServ extensions [RFC2983], may be used with it.
- Tunnel setup and maintenance: There is no standard setup and maintenance protocol for IP-in-IP 🤷♀️
- Large MTUs and minimization of tunnel overhead: When the delivery protocol is IPv4, IP-in-IP adds at least 20 bytes of overhead.
- Security: IP-in-IP encapsulation doesn't provide any significant security 💀 An SP network that supports VPNs must do extensive IP address filtering at its borders to prevent spoofed packets from penetrating the VPNs. If multi-provider VPNs are being supported, it may be difficult to set up these filters.
4.3.6.3. IPsec [RFC2401] [RFC2402] [RFC2406] [RFC2409]
IP Security (IPsec) provides security services at the IP layer [RFC2401] 🔐 It's made up of the authentication header (AH) protocol [RFC2402], the encapsulating security payload (ESP) protocol [RFC2406], and the Internet key exchange (IKE) protocol [RFC2409]. AH gives data integrity, data origin authentication, and an anti-replay service. ESP gives data confidentiality and limited traffic flow confidentiality. It may also give data integrity, data origin authentication, and an anti-replay service. AH and ESP may be used in combination 🫶
IPsec can be used in transport mode or tunnel mode. In transport mode, either an AH or ESP header is inserted immediately after the payload packet's IP header. In tunnel mode, an IP packet is encapsulated with an outer IP packet header, and either an AH or ESP header goes between them. AH and ESP establish a unidirectional secure communication path between two endpoints, which is called a security association. In tunnel mode, a PE-PE tunnel (or a CE-CE tunnel) consists of a pair of unidirectional security associations. The IPsec and IKE protocols are used for setting up IPsec tunnels 🔑
- Multiplexing: The
SPIfield of AH and ESP is used to multiplex security associations (or tunnels) between two peer devices 🧚 - QoS/SLA: IPsec doesn't have intrinsic QoS/SLA capabilities, but it inherits whatever mechanisms exist for IP. Other mechanisms, such as "RSVP Extensions for IPsec Data Flows" [RFC2207] or DiffServ extensions [RFC2983], may be used with it.
- Tunnel setup and maintenance: The IPsec and IKE protocols are used for setup and maintenance of tunnels ✨
- Large MTUs and minimization of tunnel overhead: IPsec transport mode adds at least 8 bytes of overhead. IPsec tunnel mode adds at least 28 bytes of overhead. IPsec transport mode adds minimal overhead. In PE-based PPVPNs, the processing overhead of IPsec (due to its cryptography) may limit the PE's performance, especially if privacy is being provided. This is not generally an issue in CE-based PPVPNs 🍓
- Security: When IPsec tunneling is used in conjunction with IPsec's cryptographic capabilities, excellent authentication and integrity functions can be provided 🔥 Privacy can also be optionally provided 💖
4.3.6.4. MPLS [RFC3031] [RFC3032] [RFC3035] 🦋
Okay bestie, MPLS (Multiprotocol Label Switching) is a way of forwarding packets through a network 🚀 Routers at the edge slap simple labels onto packets. A label can get inserted between the data link and network headers, or it can ride in the data link header (like the VPI/VCI field in an ATM header) ✨ Then the routers inside the network just switch packets based on those labels, with barely any lookup overhead 💅 A path, or a tunnel in the PPVPN, is called a "label switched path (LSP)" 🌸
- Multiplexing
- LSPs may be multiplexed within other LSPs. 🧚
- QoS/SLA
- MPLS doesn't have intrinsic QoS or SLA management mechanisms 🙈 BUT bandwidth can be allocated to LSPs, and their routing can be explicitly controlled 💖 Other techniques like DiffServ and DiffServ aware traffic engineering may be used with it [RFC3270] [MPLS-DIFF-TE]. QoS capabilities from IP may be inherited too 👑
- Tunnel setup and maintenance
- LSPs are set up and maintained by LDP (Label Distribution Protocol), RSVP (Resource Reservation Protocol) [RFC3209], or BGP 🔥
- Large MTUs and minimization of tunnel overhead.
- MPLS encapsulation adds four bytes per label. VPN-2547BIS's [VPN-2547BIS] approach uses at least two labels for encapsulation and still adds minimal overhead 🍓
- Encapsulation
- MPLS packets may optionally be encapsulated in IP or GRE, for cases where it's desirable to carry MPLS packets over an IP-only infrastructure 🌈
- Security
- > MPLS encapsulation does not provide any significant security. 💀 An SP that's providing VPN service can refuse to accept MPLS packets from outside its borders. That gives the same level of assurance as IP address filtering does when IP-based encapsulations are used.
- If a VPN is jointly provided by multiple SPs, care should be taken to make sure a labeled packet is accepted from a neighboring router in another SP only if its top label is one that was actually distributed to that router 😭
- Applicability
- MPLS is the only one of the encapsulation techniques that can't be guaranteed to run over any IP network 🥺 So it wouldn't be applicable when transparency to the Internet is a requirement.
- If the VPN backbone is made of several cooperating SP networks that support MPLS, then the adjacent networks may support MPLS at their interconnects 🫶 If two cooperating SP networks that support MPLS are separated by a third one that doesn't support MPLS, then MPLS-in-IP or MPLS-in-IPsec tunneling may be done between them ✨
4.4. PE-PE Distribution of VPN Routing Information 🧭
Okay so in layer 3 PE-based VPNs, PE devices look at the IP headers of packets they get from the customer networks 👀 Forwarding is based on routing information received from the customer network, which means the PE devices need to participate somehow in routing for that customer network 💅 Section 3.3 covered how routing works in the customer network, including the customer interface. In this section we talk about how routing information from a particular VPN gets passed over the shared VPN backbone, among the set of PEs attached to that VPN 🦋
The PEs need to distribute two types of routing information to each other:
- (i) Public Routing: routing information that specifies how to reach addresses on the VPN backbone (i.e., "public addresses"). Let's call this "public routing information" 🌸
- (ii) VPN Routing: routing information obtained from the CEs, which specifies how to reach addresses ("private addresses") that are in the VPNs ✨
How the first category gets distributed is outside the scope of this document 🙅♀️ We're only going to talk about distributing the second category. Of course, one requirement for distributing VPN routing information is that it be kept separate and distinct from the public information 💜 Another requirement is that distributing VPN routing information must not destabilize or otherwise interfere with distributing public routing information 😤
Similarly, distributing VPN routing information for one VPN shouldn't destabilize or otherwise interfere with other VPNs 🌈 These requirements matter, for example, when a private network is suffering from instability or other internal routing problems that could get propagated into the VPN used to support that private network 😬
This issue doesn't come up in CE-based VPNs, though 💖 In CE-based VPNs, the PE devices don't see packets from the VPN until after the packets have been encapsulated in an outer header that has only public addresses 🫶
4.4.1. Options for VPN Routing in the SP 🛠️
The following technologies can be used for exchanging the VPN routing information discussed in sections 3.3.1.3 and 4.1:
4.4.2. VPN Forwarding Instances (VFIs) 🧚
In layer 3 PE-based VPNs, PE devices receive unencapsulated IP packets from the CE devices, and they use the IP destination addresses in those packets to help make forwarding decisions 🍓 To do this properly, PE devices must obtain routing information from the customer networks. This means the PE device participates somehow in the customer network's routing 💅
In layer 3 PE-based VPNs, a single PE device can be connected to several CE devices that are in the same VPN, and it may also be connected to CE devices of different VPNs 🌸 The route the PE chooses for a given IP destination address in a given packet depends on the VPN the packet was received from 👑 So a PE device must have a separate forwarding table for each VPN it's attached to 🔥 We call these forwarding tables "VPN Forwarding Instances" (VFIs), as defined in section 2.1 ✨
A VFI contains routes to locally attached VPN sites, as well as routes to remote VPN sites 🦋 Section 4.4 discusses how routes to remote sites are obtained.
Routes to local sites can be obtained in several ways 💫 One way is to explicitly configure static routes into the VFI. That can be useful in simple deployments, but it requires that one or more devices in the customer's network be configured with static routes (perhaps just a default route), so traffic gets directed from the site to the PE device 🙈
Another way is to have the PE device be a routing peer of the CE device, using a routing algorithm such as RIP, OSPF, or BGP 🫶 Depending on the deployment scenario, the PE might need to advertise a large number of routes to each CE (e.g., all the routes the PE obtained from remote sites in the CE's VPN), or it might just need to advertise a single default route to the CE 🌈
A PE device uses some resources in proportion to the number of VFIs it has, especially if a distinct dynamic routing protocol instance is associated with each VFI 😩 A PE device also uses some resources in proportion to the total number of routes it supports. That total includes all the routes in all its VFIs, plus all the public routes 💅 These scaling factors will limit the number of VPNs a single PE device can support 🥲
When dynamic routing is used between a PE and a CE, each VFI isn't necessarily associated with a single routing protocol instance 🌀 A single routing protocol instance may provide routing information for multiple VFIs, and/or multiple routing protocol instances might provide information for a single VFI. See sections 4.4.3, 4.4.4, 3.3.1, and 3.3.1.3 for details 📚
There are several options for how VPN routes are carried between the PEs, as discussed below 🎀
4.4.3. Per-VPN Routing 🌷
One option is to run separate instances of routing protocols between the PEs, one instance for each VPN 💜 When this is done, routing protocol packets for each customer network need to be tunneled between PEs. This uses the same tunneling method, and optionally the same tunnels, as is used for transporting VPN user data traffic between PEs ✨
With per-VPN routing, a distinct routing instance for each VPN exists within the corresponding PE device 🏰 VPN-specific tunnels are set up between PE devices (using the control mechanisms discussed in sections 3 and 4). Logically, these tunnels run between the VFIs inside the PE devices 🦄 The tunnels are then used as if they were normal links between normal routers. Routing protocols for each VPN operate between VFIs and the routers within the customer network 💅
This approach establishes, for each VPN, a distinct "control plane" operating across the VPN backbone 🌈 No two VPNs share a control plane, and the VPN routing doesn't share a control plane with the public routing either 🙌 With this approach, each PE device can logically be thought of as consisting of multiple independent routers 🤯
The multiple routing instances within a PE device may be separate processes, or they may be in the same process with different data structures 🧩 Similarly, there may be mechanisms internal to the PE devices to partition memory and other resources between routing instances. The mechanisms for implementing multiple routing instances within a single physical PE are outside the scope of this framework document, and they're also outside the scope of other standards documents 📜
This approach tends to minimize the explicit interactions between different VPNs, as well as between VPN routing and public routing 💖 However, as long as the independent logical routers share the same hardware, there's some sharing of resources, so interactions are still possible 👀 Also, each independent control plane has its own overheads, and this can raise scale issues 😵💫 For example, the PE device must run a potentially large number of independent routing "decision processes," and must also maintain a potentially very large number of routing adjacencies 🥹
4.4.4. Aggregated Routing Model 🦋
okay so, another option is to use one single instance of a routing protocol to carry VPN routing info between the PEs 💅 Basically, the routing info for a bunch of different VPNs gets squished together into one routing protocol. Iconic.
This is lowkey a huge win because the PEs don't have to keep up nearly as many routing adjacencies 🫶 There's no need to maintain more than one of those adjacencies between any given pair of PEs. And if the single routing protocol supports a hierarchical route distribution mechanism (like BGP's "route reflectors" ✨), the PE-PE adjacencies can be completely eliminated! The number of backbone adjacencies can become a small constant that doesn't depend on how many PE devices there are. The scaling is literally so good 😭
Extra routing instances might still be needed for swapping routing info between the PE and its locally attached CEs. These can be eliminated (scalability glow-up 🌈) by using static routing on the PE-CE interfaces, or possibly by having the PE-CE routing interaction use the same protocol instance that distributes VPN routes across the VPN backbone (see section 4.4.4.2 for a way to do this).
So with this approach, the number of routing protocol instances in a PE device does not depend on how many CEs the PE supports, as long as the routing between PE and CE devices is static or BGP-4. 🎀 BUT, if CE and PE devices inside a VPN exchange route info using a routing protocol other than BGP-4, then the number of routing protocol entities in a PE device depends on the number of CEs it supports. Not me having to explain this twice 💀
In principle, routing can be aggregated using either BGP or an IGP. 🧚
4.4.4.1. Aggregated Routing with OSPF or IS-IS 🌸
When you're supporting VPNs, there can be a large number of them within any given SP network. Generally only a small number of PE devices care about the operation of any one VPN, though. So even though the total routing info for all those customer networks is huge, any single PE only needs to know about a small number of them. Main character energy for each PE, honestly 👑
SP networks usually use OSPF or IS-IS for interior routing within the SP network. There are very good reasons for that choice, but they're outside the scope of this document, so we're not getting into it here 🙈
Both OSPF and IS-IS are link state routing protocols. In link state routing, routing info gets distributed through a flooding protocol. The set of routing peers is generally not fully meshed, but there's a path from any router in the set to any other. Flooding makes sure routing info from any one router reaches all the others. The catch? This requires all routers in the set to maintain the same routing info. You can't withhold any routing info from a particular peer unless you know that none of the peers further downstream will need it, and in general you can't know that. It's giving no privacy, lol 💅
So if someone tried to do aggregated routing with OSPF, with all the PEs in the set of routing peers, every single PE would end up with the exact same routing info. There's no way to constrain distribution of routing info to a subset of the PEs. And given how massive the total routing info for a large number of VPNs could be, that would have really unfortunate scaling implications 😭🔥
Also, some VPNs may span multiple areas within a provider, or even span multiple providers. If VPN routing info were aggregated into the IGP used within the provider, some method would be needed to extend the reach of IGP routing info between areas and between SPs. Okay, that's a whole plot twist 🍓
4.4.4.2. Aggregated Routing with BGP 💜
Okay, so to use BGP for aggregated routing, the VPN routing info has to be clearly distinguished from the public Internet routing info ✨ This is typically done by using BGP's ability to handle multiple address families, and treating VPN routes as being in a different address family than public Internet routes. A VPN route also typically carries attributes that depend on the particular VPN or VPNs that route belongs to. Very organized, very slay 🎀
When BGP carries VPN info, the total amount of information in BGP (Internet routes and VPN routes included) can get pretty big. As noted above, there may be a large number of VPNs supported by any particular provider, so the total routing info for all VPNs can be quite large too. But any one PE will generally only need to be aware of a small number of VPNs 🦋 So where VPN routing info gets aggregated into BGP, it's really desirable to be able to limit which VPN info gets distributed to which PEs.
In "Interior BGP" (IBGP), routing info isn't flooded. It's sent directly, over a TCP connection, to peer routers (or to a route reflector). Those peer routers (unless they're route reflectors) aren't even allowed to redistribute the info to each other. BGP also has a comprehensive set of mechanisms for constraining what routing info any one peer sends to another, based on policies set by network administration. So IBGP satisfies one of the requirements for aggregated routing within a single SP network: it makes it possible to ensure that routing info relevant to a particular VPN is processed only by the PE devices attached to that VPN. The only thing needed is that each VPN route gets distributed with one or more attributes that identify the distribution policies. Then distribution can be constrained by filtering against those attributes. Literally so clean 🧚
In "Exterior BGP" (EBGP), routing peers do redistribute routing info to each other. However, it's very common to constrain distribution of particular routing info so it only goes to exterior peers who have a "need to know," though this does require knowing in advance which paths may validly lead to which addresses 🤓 For VPN routing, if a VPN is provided by a small set of cooperating SPs, those constraints can be applied so the routing info relevant to that VPN doesn't get distributed anywhere it doesn't need to be. To the extent a VPN is supported by a small number of cooperating SPs with private peering arrangements, this is especially straightforward, because the set of EBGP neighbors that need the routing info from that VPN is easier to figure out ✨
BGP also has mechanisms (like "Outbound Route Filtering," ORF) that let the right set of VPN routing distribution constraints be dynamically distributed. That cuts down the management burden of setting up the constraints, which improves scalability. Less admin, more slay 💅
Within a single routing domain (in the layer 3 VPN context, this usually means within a single SP's network), it's common for IBGP routers to peer directly with one or two route reflectors, instead of peering directly with each other. This hugely reduces the number of IBGP adjacencies any one router has to support. And a route reflector doesn't just redistribute routing info, it "digests" it first by running its own decision processes. Only routes that survive the decision process get redistributed. Chef's kiss 👑
So when route reflectors are used, the amount of routing info carried around the network, and especially the amount any given router must receive and process, gets greatly reduced 🌈 That massively boosts the scalability of the routing distribution system.
It was already mentioned that a given PE only has VPN routing info for the PEs it's directly attached to. It's similarly important, for scalability, that no single route reflector has to hold all the routing info for all VPNs 🥺 It's totally possible for the total number of VPN routes (across all VPNs an SP supports) to exceed what a single route reflector can handle. So the VPN routes may be partitioned, with some route reflectors carrying one subset and other route reflectors carrying a different subset. Route reflectors that carry public Internet routes can also be completely separate from the ones that carry VPN routes 🫶
Outbound route filters let any one PE and any one route reflector exchange info about only those VPNs that both of them are interested in. That makes sure each PE and each route reflector only receives routing info about the VPNs it's directly supporting. So large SPs supporting lots of VPNs can partition the info needed to support them. Big brain move 🧠
Generally, a PE device is restricted in the total number of routes it can support, whether those are public Internet routes or VPN routes. So a PE might be able to attach to a larger number of VPNs if it doesn't also need to support Internet routes 🎀
How VPN routes get partitioned among PEs and/or route reflectors is a deployment issue. With suitable deployment procedures, the limited capacity of these devices won't limit how many VPNs can be supported 🦋
Same deal with whether a given PE and/or route reflector holds Internet routes as well as VPN routes. That's also a deployment issue. If the customer networks served by a particular PE don't need Internet access, that PE doesn't need to be aware of the Internet routes. If some or all of the VPNs served by a PE do need Internet access, but the PE doesn't hold Internet routes, then the PE can maintain a default route that sends all of that PE's Internet traffic to a different router within the SP network, where that other router holds the full Internet routing table. With this approach the PE only needs a single default route for all the Internet routes. Simple and iconic 💖
For all the reasons above, BGP seems like a reasonable protocol for distributing VPN routing info. Additional reasons to use BGP are:
- BGP has been proven useful for distributing very large amounts of routing info; there isn't any routing distribution protocol known to scale any better. 🚀
- The same BGP instance used for PE-PE distribution of VPN routes can also be used for PE-CE route distribution, if CE-PE routing is static or BGP. PEs and CEs are really parts of distinct Autonomous Systems, and BGP is especially well-suited for carrying routing info between Autonomous Systems. 🧚
On the other hand, BGP is also used for distributing public Internet routes, and it is crucially important that VPN route distribution not compromise the distribution of public Internet routes in any way. 😭 This issue is discussed in the following section. 🙈
4.4.5. Scalability and Stability of Routing with Layer 3 PE-based VPNs ✨
okay so for layer 3 PE-based VPNs, it's pretty likely that a service provider will also offer Internet access over the same link that carries the VPN service 🌐. That means one CE-to-PE link can carry both private network IP packets (going between sites of the private network over the VPN) and public Internet traffic (going from the private site out to the Internet, and coming back in). This section looks at how scalable and stable routing is in that setup. It's worth noting that this kind of issue can show up with per-VPN routing too, not only with aggregated routing 💅.
For layer 3 PE-based VPNs, the PE devices need to be able to forward IP packets using the address spaces of the supported private networks, and also using the full Internet address space. So PE devices might sometimes end up taking part in routing for the private networks as well as for the public Internet 🧚.
In some cases the routing load on the PE is low enough, and the PE is powerful enough, that it's totally reasonable for the PE to fully participate in routing for both the private networks and the public Internet 👑. For example, the PE device might take part in normal BGP operation as part of the global Internet. The PE device might also run routing protocols (or, in some cases, use static routing) to exchange routes with CE devices 🎀.
For large installations, or when PE capabilities are more limited, it may not be ideal for the PE to fully participate in routing for both the VPNs and the public Internet 😭. For example, suppose the total volume of routes and routing instances supported by one PE across multiple VPNs is very large. Now suppose that one or more of the private networks has routing instabilities, which could result in a large number of routing updates being sent to the PE device. In that case it is important to stop that routing from causing any instability in the routing used on the global Internet 🛑.
So in these cases it may be necessary to partition routing, so the PE doesn't need to hold as big a collection of routes, and so the PE isn't able to adversely effect Internet routing 🙅♀️. Also, since the total number of route prefixes and routing instances the PE has to maintain could be really big, it may be a good idea to limit how much a PE takes part in Internet routing if it's supporting lots of VPNs or very large VPNs 🫶.
Let's look at a case where a PE is supporting a VERY large number of VPNs, some of which have a lot of sites. To pick a VERY large example, let's say 1000 VPNs, with an average of 100 sites each, plus 10 prefixes per site on average. The PE also needs to be able to route traffic to the Internet in general. In this example the PE might need to support approximately 1,000,000 prefixes for the VPNs, plus more than 100,000 prefixes for the Internet. If augmented and aggregated routing is used, that means a large number of routes which may be advertised in a single routing protocol (most likely BGP) 😳. If the VR approach is used, there are also 100,000 neighbor adjacencies spread across the various per-VPN routing protocol instances. In some cases that many routing prefixes and/or that many adjacencies might be really hard to support in one device 💀.
In this case, an alternate approach is to limit the PE's participation in Internet routing to the absolute minimum required 🍓. Specifically, the PE will need to know which Internet address prefixes are reachable via directly attached CE devices. All other Internet routes may be summarized into a single default route pointing to one or more P routers. In lots of cases, the P routers the default route points to may be the same P routers the PE is directly attached to (which are the ones it needs to use for forwarding most Internet traffic). So if there are M CE devices directly connected to the PE, and those M CE devices are the next hop for a total of N globally addressable Internet address prefixes, then the PE device would maintain N+1 routes corresponding to globally routable Internet addresses 🧮.
In this example, the PE devices that provide VPN service run routing to compute routes for the VPNs, but they don't operate Internet routing. Instead they just use a default route to send traffic to all Internet destinations (not counting the addresses reachable via directly attached CE devices) 🌸. The P routers need to maintain Internet routes, so they take part in Internet routing protocols. However, the P routers don't know anything about the VPN routes 🦋.
In some cases, the maximum number of routes and/or routing instances a single PE device can support may limit how many VPNs that PE can support. For example, this might require two different PE devices to be used to support VPN services for a set of multiple CEs, even if one PE would have had enough throughput to handle the data traffic from the full set of CEs 😤. Similarly, the amount of resources any one VPN is allowed to use in a single PE might be restricted 🔒.
There will be cases where it isn't necessary to partition the routing, since the PEs can maintain all VPN routes and all Internet routes without any problem 🥰. Still, it's important that VPN approaches allow partitioning to be used where needed, so future scaling problems get prevented. Again, making the system scalable is a matter of proper deployment 🧁.
It may be wondered whether it is ever desirable to have both Internet routing and VPN routing running in a single PE device or route reflector 🤔. In fact, if even a single system is running both Internet routing and VPN routing, doesn't that raise the possibility that a disruption in the VPN routing system could cause a disruption in the Internet routing system? 😱
Certainly that possibility exists in theory. To minimize it, BGP implementations that support multiple address families should be organized so that the processing and distribution of one address family affects the processing and distribution of another as little as possible 🛠️. This could be done, for example, by suitable partitioning of resources. That partitioning may help protect Internet routing from VPN routing, and also protect well-behaved VPN customers from "mis-behaving" VPNs 🛡️. Or one could try to protect the Internet routing system from the VPN routing system by giving preference to Internet routing. Such implementation issues are outside the scope of this document. If someone has inadequate confidence in an implementation, deployment procedures can be used, as explained above, to separate Internet routing from VPN routing 💜.
4.5. Quality of Service, SLAs, and IP Differentiated Services 🌈
The following technologies for QoS/SLA may be applicable to PPVPNs 💖.
4.5.1. IntServ/RSVP [RFC2205] [RFC2208] [RFC2210] [RFC2211] [RFC2212]
Integrated services, or IntServ for short, is a mechanism for providing QoS/SLA through admission control 🚦. RSVP is used to reserve network resources. The network needs to maintain state for each reservation, so the number of states in the network grows in proportion to the number of concurrent reservations 📈.
In some cases, IntServ at the edge of a network (for example, over the customer interface) may be mapped to DiffServ in the SP network 🔁.
4.5.2. DiffServ [RFC2474] [RFC2475]
IP differentiated service, or DiffServ for short, is a mechanism for providing QoS/SLA by differentiating traffic 🎀. Traffic entering a network is classified into several behavior aggregates at the network edge, and each one is assigned a corresponding DiffServ codepoint. Inside the network, traffic is treated according to its DiffServ codepoint. Some behavior aggregates have already been defined. Expedited forwarding behavior [RFC3246] guarantees the QoS, whereas assured forwarding behavior [RFC2597] differentiates traffic packet precedence values ✨.
When DiffServ is used, network provisioning is done on a per-traffic-class basis. This makes sure a specific class of service can be achieved for a class (as long as the traffic load is controlled) 🧠. All packets within a class are then treated equally within an SP network. Policing is done at input so no single user can exceed their allocation and wreck the provisioning for the class as a whole 🙅♀️. If a user goes over their traffic contract, the excess packets may optionally be discarded, or they may be marked as "over contract" 🏷️. Routers throughout the network can then preferentially discard over contract packets when there's congestion, so those packets don't defeat the service guarantees meant for in-contract traffic 💅🔥.
4.6. Concurrent Access to VPNs and the Internet ✨
okay so sometimes customers need to be on their VPN and the public Internet at the same time 🌐💖 which sounds simple but it's giving two big problems: private addresses and security threats 😭
- The use of private addresses
The IP addresses used in the customer's sites might belong to a private routing realm, which means they can't be used on the public Internet 🙈 So a network address translation function (e.g., NAT) will need to be implemented so VPN customers can reach the Public Internet 🦋
For layer 3 PE-based VPNs, this translation function will be implemented in the PE that the CE device is connected to 💅 For layer 3 provider-provisioned CE-based VPNs, this translation function will be implemented on the CE device itself 🎀
- Potential security threat
Some of the traffic flowing to and from the public Internet isn't necessarily under the SP's or the customer's control, so some traffic analyzing function (e.g., a firewall function) will be implemented to control the traffic entering and leaving the VPN 🔥 very necessary, bestie
For layer 3 PE-based VPNs, this traffic analyzing function will be implemented in the PE device (or in the VFI supporting a specific VPN) 🧚 For layer 3 provider provisioned CE-based VPNs, this function will be implemented in the CE device 💜
- Handling of a customer IP packet destined for the Internet
For layer 3 PE-based VPNs, an IP packet coming from a customer site will be handled in the corresponding VFI ✨ If the IP destination address in the packet's IP header belongs to the Internet, there are multiple scenarios depending on the adapted policy 👑 In the first possibility, when Internet access is not allowed, the packet will be dropped 💀 In the second possibility, when (controlled) Internet access is allowed, the IP packet will go through the translation function and eventually through the traffic analyzing function before further processing in the PE's global Internet forwarding table 🌸
Note that different implementation choices are possible 🌈 One option is to implement the translation and/or the traffic analyzing function in every VFI (or CE device in the context of layer 3 provider-provisioned CE-based VPNs), or alternatively in a subset or even in only one VPN network element. That single-element option means the traffic to and from the Internet from any VPN site needs to be routed through that one element (this is what happens in a hub and spoke topology, for example) 🍓
4.7. Network and Customer Management of VPNs 🧚
4.7.1. Network and Customer Management
Network and customer management systems that manage VPN networks have several challenges, depending on the type of VPN network or networks they're required to manage 💅
For any type of provider-provisioned VPN, it's useful to have one place where the VPN can be viewed and optionally managed as a whole 👀 The NMS may therefore be a place where the collective instances of a VPN are brought together into a cohesive picture to form a VPN. To be more precise, the instances of a VPN on their own do not form the VPN. Rather, the collection of disparate VPN sites together forms the VPN ✨ This is important because VPNs are typically configured at the edges of the network (i.e., PEs), either through manual configuration or auto-configuration. This results in no state information being kept within the "core" of the network 🫶 Sometimes little or no information about other PEs is configured at any particular PE, which is kinda wild ngl 😭
Support of any one VPN may span a wide range of network equipment, potentially including equipment from multiple implementors. Allowing a unified network management view of the VPN is therefore simplified through the use of standard management interfaces and models 🌈 This will also facilitate customer self-managed (monitored) network devices or systems 🎀
In cases where significant configuration is required whenever a new service is provisioned, it is important for scalability reasons that the NMS provide a largely automated mechanism for this operation 🔥 Manual configuration of VPN services (i.e., new sites, or re-provisioning existing ones) could lead to scalability issues, and should be avoided 🙅♀️ So it's important for network operators to maintain visibility of the complete picture of the VPN through the NMS system 👑 This must be achieved using standard protocols such as SNMP, XML, or LDAP. Proprietary command-line interfaces have the disadvantage that they don't lend themselves to standard representations of managed objects, which is a big mood-killer 💀
To achieve the goals outlined above for network and customer management, device implementors should employ standard management interfaces to expose the information required to manage VPNs 🦋 To this end, devices should use standards-based mechanisms such as SNMP, XML, or LDAP to achieve this goal 💜
4.7.2. Segregated Access of VPN Information
Segregated access to VPN information matters because customers sometimes need to access information in several different ways 🌸 First, it's important for some customers (or operators) to access PEs, CEs or P devices within the context of a particular VPN, on a per-VPN basis, to get statistics, configuration or status information 🍓 This can happen under the guise of general management, operator-initiated provisioning, or SLA verification (SP, customer or operator) 🧚
Where users outside of the SP have access to information from PE or P devices, managed objects within the managed devices must be accessible on a per-VPN basis, so the customer, the SP or the third party SLA verification agent gets a high degree of security and convenience 🔒✨
Security may require authentication or encryption of network management commands and information 🫶 Information hiding may use encryption, or it may isolate information through a mechanism that provides per-VPN access. Authentication or encryption of both requests and responses for managed objects within a device may be employed 💅 Examples of how this can be achieved include IPsec tunnels, SNMPv3 encryption for SNMP-based management, or encrypted telnet sessions for CLI-based management 🌈
In the case of information isolation, any one customer should only be able to view information pertaining to its own VPN or VPNs 👀 Information isolation can also be used to partition the space of managed objects on a device in a way that makes it more convenient for the SP to manage the device. In certain deployments it's also important for the SP to have access to information pertaining to all VPNs, so it may be important for the SP to create virtual VPNs within the management domain that overlap across existing VPNs 👑
If the user is allowed to change the configuration of their VPN, customers may sometimes make unanticipated changes or even mistakes, which can cause their VPN to mis-behave 😭 This in turn may require an audit trail, so it can be figured out what went wrong, and some way to inform the carrier of the cause 🔍
Segregation and security access of information on a per-VPN basis is also important when the carrier of carrier's paradigm is employed 🦋 In this case it may be desirable for customers (i.e., sub-carriers or VPN wholesalers) to manage and provision services within their VPNs on their respective devices, which reduces the management overhead cost to the carrier of carrier's SP 💖 In this case, it's important to observe the guidelines detailed above regarding information hiding, isolation and encryption. There may be many flavors of information hiding and isolation employed by the carrier of carrier's SP 🌸 If the carrier of carriers SP does not want to grant the sub-carrier open access to all of the managed objects within their PEs or P routers, it is necessary for devices to provide network operators with secure and scalable per-VPN network management access to their devices 🔥 For the reasons outlined above, it's therefore desirable to provide standard mechanisms for achieving these goals ✨
5. Interworking Interface 🦋
This section describes interworking between different layer 3 VPN approaches. This may happen either within a single SP network, or at an interface between SP networks 🌐
5.1. Interworking Function 🧚
Figure 2.5 (see section 2.1.3) illustrates a case where one or more PE devices sit at the logical interface between two different layer 3 VPN approaches 💅 With this approach, the interworking function happens at a PE device that participates in two or more layer 3 VPN approaches 🎀 This might be physically located at the boundary between service providers, or it might occur at the logical interface between different approaches within a service provider 🌈
With layer 3 VPNs, the PE devices are in general layer 3 routers, and they're able to forward layer 3 packets on behalf of one or more private networks ✨ For example, it may be common for a PE device supporting layer 3 VPNs to contain multiple logical VFIs (sections 1, 2, 3.3.1, 4.4.2), each of which supports forwarding and routing for a private network 👑
The PE which implements an interworking function needs to participate in the normal manner in the operation of multiple approaches for supporting layer 3 VPNs 💖 This involves the functions discussed elsewhere in this document, such as VPN establishment and maintenance, VPN tunneling, routing for the VPNs, and QoS maintenance 🫶
VPN establishment and maintenance information, as well as VPN routing information, will need to be passed between VPN approaches 🍓 This might involve passing information between approaches as part of the interworking function. Optionally, this might involve manual configuration so that, for example, all of the participants in the VPN on one side of the interworking function consider the PE performing the interworking function to be the point to use to contact a large number of systems (comprising all systems supported by the VPN located on the other side of the interworking function) 🌸
5.2. Interworking Interface 🌈
Figure 2.6 (see section 2.1.3) illustrates a case where interworking is performed by use of tunnels between PE devices 🦋 In this case, each PE device participates in the operation of one layer 3 VPN approach. Interworking between approaches makes use of per-VPN tunnels set up between PEs. Each PE operates as if it is a normal PE, and considers each tunnel to be associated with a particular VPN 💅
Information can then be transmitted over the interworking interface in the same manner that it is transmitted over a CE to PE interface ✨
In some cases, establishment of the interworking interfaces may require manual configuration, for example to allow each PE to determine which tunnels should be set up, and which private network is associated with each tunnel 🔥
5.2.1. Tunnels at the Interworking Interface 🎀
Okay so, if two SP networks want to interwork to support one or more PPVPN that spans both of them, they need a way to swap customer data AND the control data that goes with it (like routing data) 💅. That part is non-negotiable, bestie.
Since the PEs of the SP networks being interworked can only talk to each other across a network cloud, an appropriate tunnel through that cloud gets used to exchange the data for the PPVPN that the interworked SP networks are building together 🦋.
So each interworking tunnel gets assigned to its own layer 3 PE-based VPN. Basically, a tunnel is terminated by a VFI (the one tied to the PPVPN) in a PE device 💜. This gives you traffic isolation for the PPVPNs running over the Interworking Interface and spanning multiple SP networks ✨. Each SP network is totally free to use whatever tech it wants to provide the PPVPN, so both sides can use different technologies 🌈. How each tunnel gets assigned to a PE-based VPN depends on the implementation technology the SP network uses at the PE device where it connects to the tunnel 🔥.
The identifier of the layer 3 PE-based VPN at each end only means something inside that specific SP network's technology. The other SP network you're interworking with doesn't need to understand it 🫶.
Here are the tunneling mechanisms that may be used at the interworking interface. The list includes (but is not limited to): GRE, IP-in-IP, IP over ATM, IP over FR, IPsec, and MPLS 🍓.
- GRE The tunnels at the interworking interface may be provided by GRE [RFC2784] with key and sequence number extensions [RFC2890] 🌸.
- IP-in-IP The tunnels at the interworking interface may be provided by IP-in-IP [RFC2003] [RFC2473] 🧚.
- IP over ATM AAL5 The tunnels at the interworking interface may be provided by IP over ATM AAL5 [RFC2684] [RFC2685] 💫.
- IP over FR The tunnels at the interworking interface may be provided by IP over FR 🦋.
- IPsec The tunnels at the interworking interface may be provided by IPsec [RFC2401] [RFC2402] 🔐.
- MPLS The tunnels at the interworking interface may be provided by MPLS [RFC3031] [RFC3035] 👑.
5.3. Support of Additional Services ✨
This subsection covers the extra stuff you can layer on top for QoS/SLA, customer visible routing, and customer visible multicast routing. These are all services of layer 3 PE-based VPNs that span multiple SP networks 💖.
- QoS/SLA QoS/SLA management mechanisms for GRE, IP-in-IP, IPsec, and MPLS tunnels were discussed in sections 4.3.6 and 4.5. Go check those sections for the details 🎀. FR and ATM are capable of QoS guarantee, so QoS/SLA may also be supported at the interworking interface 🔥.
- Customer visible routing As described in section 3.3, customer visible routing lets customer sites swap unicast routing information using a routing protocol like OSPF, IS-IS, RIP, or BGP-4 🛣️. At the interworking interface, routing packets (such as OSPF packets) travel through a tunnel tied to a layer 3 PE-based VPN in exactly the same way as user data packets inside the VPN 🌈.
- Customer visible multicast routing Customer visible multicast routing lets customer sites share multicast routing information using a routing protocol like DVMRP or PIM 💜. At the interworking interface, multicast routing packets go through a tunnel tied to a layer 3 PE-based VPN, same as user data packets inside the VPN. This is what makes a multicast tree construction possible inside the layer 3 PE-based VPN 🦋.
5.4. Scalability Discussion 📈
This subsection talks about the scalability side of the interworking scenario 💅.
- Number of routing protocol instances In the interworking scenario covered in this section, the number of routing protocol instances and the number of layer 3 PE-based VPNs are the same 🧚. BUT, the number of layer 3 PE-based VPNs in a PE device is limited by the PE device's resource amount and performance 😭. Also, each tunnel is expected to need some bandwidth, and the total bandwidth is capped by the capacity of a PE device, so the number of tunnels is limited by what the PE can handle 💀. Honestly, this limit is not a critical drawback ✨.
- Performance of packet transmission The interworking scenario in this section doesn't put any extra burden on the tunneling technologies used at the interworking interface 🌸. Packet transmission performance depends on the tunneling technology you pick, so you should pick carefully when you provision interworking 🍓. For example, IPsec puts computational requirements on encryption and decryption 🔐.
6. Security Considerations 🛡️
Security is one of the biggest deals for VPNs 💖. In network environments, the word "security" covers a lot of different aspects, and the most important ones from a networking perspective are briefly discussed below 👑.
Also, the Provider-Provisioned VPN requirements document goes into more detail on the different security requirements for Provider-Provisioned VPNs, so go read that too 🌈.
6.1. System Security 🔒
Like in every network environment, system security is the most important aspect and must be enforced 😤. Care must be taken so that no unauthorized party can get access to the network elements that control VPN functionality (for example, PE and CE devices) 🚫.
Since VPN customers are using the shared SP's backbone, the SP must ensure the system security of its network elements and management systems 🫶.
6.2. Access Control 🚪
When a network, or part of one, is private, one of the requirements is that access to it must be restricted to a limited number of well-defined customers 💅. To make this happen, the responsible authority has to control every possible access point to the network 🔑.
For PE-based VPNs, the access points to a VPN must be limited to the interfaces that the SP knows about 💜.
6.3. Endpoint Authentication 🪪
When you receive data from an entity, you want to be sure who sent it 👀. You want to be sure the sending entity really is who it claims to be, and that it's authorized to reach a particular destination ✨.
For layer 3 PE-based VPNs, both the data the PEs receive from customer sites via the SP network and the data destined for a customer site should be authenticated 🦋.
Different authentication methods exist, and note that. In some situations, identifying incoming packets by their specific customer interfaces might be enough 🌸. In other situations (for example, temporary access or dial-in scenarios), a preliminary authentication phase might be requested, like when PPP is used 📞. Or, an authentication process might need to be present in every data packet transmitted (for example, in remote access via IPsec) 🔐.
For layer 3 PE-based VPNs, VPN traffic is tunneled from PE to PE, and the VPN tunnel endpoint checks the origin of the transmitted packet 🎯. When MPLS is used for VPN tunneling, the tunnel endpoint checks whether the correct labels are used 🏷️. When IPsec is used for VPN tunneling, the tunnel endpoint can make use of the IPsec authentication mechanisms 🔐.
For layer 3 provider-provisioned CE-based VPNs, endpoint authentication is enforced by the CE devices 🧚.
6.4. Data Integrity 🧩
When information is exchanged over part of a network, you want to be sure that what the receiving party gets is identical to what the sending party sent 💯.
For layer 3 PE-based VPNs, the SP assures data integrity by making sure every network element has system security 🛡️. Alternatively, explicit mechanisms may be implemented in the tunneling technique that's used (for example, IPsec) 🔐.
For layer 3 provider-provisioned CE-based VPNs, the underlying network that tunnels the encapsulated packets won't always be trusted ⚠️. So the CE devices responsible for tunneling also ensure data integrity, for example by using the IPsec architecture 🔐.
6.5. Confidentiality 🤫
You want the information sent from one party to another to not be received or readable by other parties 🙈. With traffic flow confidentiality, you even want the characteristics of the information sent to be hidden from third parties 👻. Data privacy is the confidentiality of the user data 💜.
In the context of PPVPNs, confidentiality is often seen as the basic service offered, because the functions of a private network are provided over a shared infrastructure 🌈.
For layer 3 PE-based VPNs, the SP network (and more precisely, the PE devices) takes part in routing and forwarding the customer VPN data, so it's the SP's responsibility to ensure confidentiality 👑. The technique used in PE-based VPN solutions is to ensure PE-to-PE data separation. By implementing VFIs in the PE devices and tunneling VPN packets through the shared network infrastructure between PE devices, the VPN data always stays in a separate context and is kept apart from other data 🦋.
Sometimes, this data separation isn't enough 😬. When the VPN tunnel goes through network parts that aren't only trusted and SP-controlled, you need stronger confidentiality measures, like cryptographic data encryption 🔐. This is the case in certain inter-SP VPN scenarios, or when the SP itself is a client of a third party network provider 🫠.
For layer 3 provider-provisioned CE-based VPNs, the SP network doesn't bear responsibility for confidentiality, since the SP just offers IP connectivity 💁. Confidentiality is enforced at the CE, and it lies in the tunneling (data separation) or in cryptographic encryption (for example, using IPsec) done by the CE device 🔐.
Note that for very sensitive user data (for example, data used in banking operations), the VPN customer may not outsource data privacy enforcement to a trusted SP 🏦. In those situations, PE-to-PE confidentiality won't be enough, and the VPN customer has to implement end-to-end cryptographic encryption on its own private equipment (for example, using CE-based VPN technologies or cryptographic encryption over the provided VPN connectivity) 🔐💥.
6.6. User Data and Control Data 🎛️
This is a really important point: BOTH the user data and the VPN control data must be protected 🚨.
The previous subsections focused on protecting user data, but all the control data (for example, data used to set up VPN tunnels, to configure the VFIs, or to configure the CE devices in the context of layer 3 provider-provisioned CE-based VPNs) also needs to be secured by the SP 🛡️. That's how you prevent deliberate misconfiguration of provider-provisioned VPNs 💅✨.
6.7. Security Considerations for Inter-SP VPNs ✨
okay so sometimes a single VPN has to cross multiple SPs 🌍
The edge-to-edge part of the data path isn't controlled by the same entity, and that can have security implications, for example for endpoint authentication 🔐
Another thing: the SPs involved have to work really closely together so they don't end up with conflicting configuration on the VPN network elements (like VFIs, PEs, and CE devices) that connect to each other's networks. Main character energy for coordination 💅
Appendix A: Optimizations for Tunnel Forwarding 🎀
A.1. Header Lookups in the VFIs
okay so if layer 3 PE-based VPNs are built in the most straightforward way, PE devices might have to do several header lookups just to forward one data packet 🤓 This section walks through an example of how those multiple lookups could happen in that simple version. The optimizations that can cut down on the lookups are covered in the sections after this one.
Here's the example: a lot of the time, a tunnel gets set up between VFIs inside PEs to support a particular VPN. When a packet shows up at the egress PE, the PE may need to do a lookup on the outer header to figure out which VFI the packet belongs to. Then it may need a second lookup on the packet that was encapsulated across the VPN tunnel, using the forwarding table that's specific to that VPN, before it can forward the packet 🧚
For scaling reasons, it might be wanted in some cases to set up VPN tunnels and then multiplex multiple VPN-specific tunnels inside those VPN tunnels 🦋
So in the most straightforward implementation, one PE device might need three header lookups 😭 The first lookup may tell it that this is the end of the VPN tunnel, which means the associated header needs to be stripped off. The second lookup may tell it that this is the end of the VPN-specific tunnel. That lookup strips off the second encapsulating header and identifies the VFI context for the final lookup. The last lookup uses the IP address space associated with the VPN, and it's what gets the packet forwarded to the correct CE within the correct VPN 💖
A.2. Penultimate Hop Popping for MPLS
Penultimate hop popping is an optimization described in the MPLS architecture document [RFC3031]. It's giving efficiency 🍓
Picture the egress node of any MPLS LSP. It looks at the label, figures out that it's the last node, strips off the label header, and then looks at the next header in the packet. That next header might be an IP header, or it might be another MPLS header if hierarchical nesting of LSPs is being used. For the last node on the LSP, the outer MPLS header doesn't really convey any useful information, except in one situation we'll talk about below 🙈
So the MPLS standards let the egress node ask the penultimate node to strip the MPLS header. If it asks, the penultimate node doesn't have a valid label for the LSP anymore and has to strip the MPLS header itself. The egress node then gets the packet with that header already gone, and it can forward the packet properly without having to strip the header for the LSP that ends at that egress node ✨
There IS one case where the MPLS header does convey useful information: a VPN-specific LSP terminating at a PE device. Here, the label value tells the PE which LSP the packet arrived on. That in turn determines which VFI is used for the packet, meaning which VPN-specific forwarding table should be used to forward it 👑
Now consider the case where multiple VPN-specific LSPs are multiplexed inside one PE-to-PE LSP. Let's also say that the egress PE has picked all the incoming labels (for all the LSPs) to be unique within that PE. That means the label associated with the PE-to-PE LSP isn't needed by the egress node. It can work out which VFI to use based on the VPN-specific LSP instead. In this situation, the egress PE can ask the penultimate LSR to do penultimate label popping for the PE-to-PE LSP, and that gets rid of one header lookup in the egress LSR 🫶
Note that penultimate node label popping only applies to VPN standards that use multiple levels of LSPs. Even then, it's only done when the egress node specifically asks the penultimate node for it 🙈
A.3. Demultiplexing to Eliminate the Tunnel Egress VFI Lookup
Let's say a VPN standard uses MPLS as the tunneling mechanism. Any standard for encapsulating VPN traffic inside LSPs needs to say how granular the assignment of user data traffic to LSPs can be 🧩 In other words, for any given LSP, the ingress or egress PE device needs to know which LSPs need to be set up, and the ingress PE needs to know which set of VPN packets is allowed to be mapped to each particular LSP.
Now suppose a VPN standard allows some flexibility in how packets get mapped to LSPs, and that it lets the egress node decide the granularity. In that case, the egress node would need some way to tell the ingress node what granularity it's using, so the ingress node knows which packets can be mapped to each LSP 📬
Here's where it gets fun: the egress node might decide to map packets to LSPs in a way that makes the header lookup at the egress node simpler 🧠 For example, the egress node could pick the set of packets it's going to forward to a particular neighbor CE device. It can then specify that the set of IP packets using a particular LSP corresponds to that specific set. For packets arriving on that LSP, the egress node doesn't have to do a header lookup on the VPN's customer address space. It can just pop the MPLS header and forward the packet to the right CE device. If all the LSPs are set up this way, the egress node doesn't need to do any lookup for VPN traffic arriving on LSPs from other PEs. In other words, the PE device won't need a second lookup in its role as an egress node 🦄
Note that PE devices will most likely also be ingress routers for traffic going the other direction. In that role, the PE device needs to do an address lookup in the customer network's address space. But in this direction, the PE still only needs to do a single header lookup 💅
When this optional optimization is used with MPLS tunnels, it reduces the need for header lookups. The tradeoff is that it might increase the number of label values that need to be assigned, since one label would need to be assigned for each next-hop CE device, rather than just one label for every VFI 🌈
The same approach is also possible when other encapsulations are used, such as GRE [RFC2784] [RFC2890], IP-in-IP [RFC2003] [RFC2473], or IPsec [RFC2401] [RFC2402]. This requires that distinct values are used for the multiplexing field in the tunneling protocol. See section 4.3.2 for detail 🎀
Acknowledgments 🫶
This document is the output of the framework document design team of the PPVPN WG. The members of the design team are listed in the "contributors" and "author's addresses" sections below.
The sources of this document are based on various inputs from colleagues of the authors and contributors. We'd like to thank Junichi Sumimoto, Kosei Suzuki, Hiroshi Kurakami, Takafumi Hamano, Naoto Makinae, Kenichi Kitami, Rajesh Balay, Anoop Ghanwani, Harpreet Chadha, Samir Jain, Lianghwa Jou, Vijay Srinivasan, and Abbie Matthews. Thank you all, you're iconic 🌸
We'd also like to thank Yakov Rekhter, Scott Bradner, Dave McDysan, Marco Carugi, Pascal Menezes, Thomas Nadeau, and Alex Zinin for their valuable comments and suggestions 💜
Normative References 📚
- [PPVPN-REQ] Nagarajan, A., Ed., "Generic Requirements for Provider Provisioned Virtual Private Networks (PPVPN)", RFC 3809, June 2004.
- [L3VPN-REQ] Carugi, M., Ed. and D. McDysan, Ed., "Service Requirements for Layer 3 Provider Provisioned Virtual Private Networks (PPVPNs)", RFC 4031, April 2005.
Informative References 📚✨
okay bestie, this is the reading list 💅 these are the works the spec leans on for context, not the stuff you're strictly required to follow. they're still worth a peek 🧚
[BGP-COM]Sangli, S., et al., "BGP Extended Communities Attribute", Work In Progress, February 2005.[MPLS-DIFF-TE]Le Faucheur, F., Ed., "Protocol extensions for support of Differentiated-Service-aware MPLS Traffic Engineering", Work In Progress, December 2004.[VPN-2547BIS]Rosen, E., et al., "BGP/MPLS VPNs", Work In Progress.[VPN-BGP-OSPF]Rosen, E., et al., "OSPF as the Provider/Customer Edge Protocol for BGP/MPLS IP VPNs", Work In Progress, May 2005.[VPN-CE]De Clercq, J., et al., "An Architecture for Provider Provisioned CE-based Virtual Private Networks using IPsec", Work In Progress.[VPN-DISC]Ould-Brahim, H., et al., "Using BGP as an Auto-Discovery Mechanism for Layer-3 and Layer-2 VPNs," Work In Progress.[VPN-L2]Andersson, L. and E. Rosen, Eds., "Framework for Layer 2 Virtual Private Networks (L2VPNs)", Work In Progress.[VPN-VR]Knight, P., et al., "Network based IP VPN Architecture Using Virtual Routers", Work In Progress, July 2002.[RFC1195]Callon, R., "Use of OSI IS-IS for Routing in TCP/IP and Dual Environments", RFC 1195, December 1990.[RFC1771]Rekhter, Y. and T. Li, "A Border Gateway Protocol 4 (BGP-4)", RFC 1771, March 1995.[RFC1918]Rekhter, Y., Moskowitz, B., Karrenberg, D., de Groot, G., and E. Lear, "Address Allocation for Private Internets", BCP 5, RFC 1918, February 1996.[RFC1966]Bates, T., "BGP Route Reflection: An alternative to full mesh IBGP", RFC 1966, June 1996.[RFC1997]Chandra, R., Traina, P., and T. Li, "BGP Communities Attribute", RFC 1997, February 2001.[RFC2003]Perkins, C., "IP Encapsulation within IP", RFC 2003, October 1996.[RFC2205]Braden, R., Zhang, L., Berson, S., Herzog, S., and S. Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", RFC 2205, September 1997.[RFC2208]Mankin, A., Ed., Baker, F., Braden, B., Bradner, S., O'Dell, M., Romanow, A., Weinrib, A., and L. Zhang, "Resource ReSerVation Protocol (RSVP) Version 1 Applicability Statement Some Guidelines on Deployment", RFC 2208, September 1997.[RFC2210]Wroclawski, J., "The Use of RSVP with IETF Integrated Services", RFC 2210, September 1997.[RFC2211]Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, September 1997.[RFC2212]Shenker, S., Partridge, C., and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, September 1997.[RFC2207]Berger, L. and T. O'Malley, "RSVP Extensions for IPSEC Data Flows", RFC 2207, September 1997.[RFC2328]Moy, J., "OSPF Version 2", STD 54, RFC 2328, April 1998.[RFC2401]Kent, S. and R. Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.[RFC2402]Kent, S. and R. Atkinson, "IP Authentication Header", RFC 2402, November 1998.[RFC2406]Kent, S. and R. Atkinson, "IP Encapsulating Security Payload (ESP)", RFC 2406, November 1998.[RFC2409]Harkins, D. and D. Carrel, "The Internet Key Exchange (IKE)", RFC 2409, November 1998.[RFC2453]Malkin, G., "RIP Version 2", STD 56, RFC 2453, November 1994.[RFC2473]Conta, A. and S. Deering, "Generic Packet Tunneling in IPv6 Specification", RFC 2473, December 1998.[RFC2474]Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998.[RFC2475]Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An architecture for Differentiated Services", RFC 2475, December 1998.[RFC2597]Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, "Assured Forwarding PHB Group", RFC 2597, June 1999.[RFC2661]Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G., and B. Palter, "Layer Two Tunneling Protocol 'L2TP'", RFC 2661, August 1999.[RFC2684]Grossman, D. and J. Heinanen, "Multiprotocol Encapsulation Over ATM Adaptation Layer 5", RFC 2684, September 1999.[RFC2685]Fox B. and B. Gleeson, "Virtual Private Networks Identifier," RFC 2685, September 1999.[RFC2746]Terzis, A., Krawczyk, J., Wroclawski, J., and L. Zhang, "RSVP Operation Over IP Tunnels", RFC 2746, January 2000.[RFC2764]Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. Malis, "A Framework for IP Based Virtual Private Networks", RFC 2764, February 2000.[RFC2784]Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, March 2000.[RFC2890]Dommety, G., "Key and Sequence Number Extensions to GRE", RFC 2890, September 2000.[RFC2858]Bates, T., Rekhter, Y., Chandra, R., and D. Katz, "Multiprotocol Extensions for BGP-4", RFC 2858, June 2000.[RFC2983]Black, D., "Differentiated Services and Tunnels", RFC 2983, October 2000.[RFC3031]Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, January 2001.[RFC3032]Rosen E., Tappan, D., Fedorkow, G., Rekhter, Y., Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Encoding", RFC 3032, January 2001.[RFC3035]Davie, B., Lawrence, J., McCloghrie, K., Rosen, E., Swallow, G., Rekhter, Y., and P. Doolan, "MPLS using LDP and ATM VC Switching", RFC 3035, January 2001.[RFC3065]Traina, P., McPherson, D., and J. Scudder, "Autonomous System Confederations for BGP", RFC 3065, June 1996.[RFC3209]Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, December 2001.[RFC3246]Davie, B., Charny, A., Bennet, J.C.R., Benson, K., Le Boudec, J.Y., Courtney, W., Davari, S., Firoiu, V., and D. Stiliadis, "An Expedited Forwarding PHB (Per-Hop Behavior)", RFC 3246, March 2002.[RFC3270]Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, May 2002.[RFC3377]Hodges, J. and R. Morgan, "Lightweight Directory Access Protocol (v3): Technical Specification", RFC 3377, September 2002.
Contributors' Addresses 💖
these are the amazing humans who helped build this spec, so here's where to find them 🫶
Jeremy De Clercq
Alcatel
Fr. Wellesplein 1,
2018 Antwerpen, Belgium
EMail: jeremy.de_clercq@alcatel.be
Bryan Gleeson
Nokia
313 Fairchild Drive,
Mountain View, CA 94043 USA.
EMail: bryan.gleeson@nokia.com
Andrew G. Malis
Tellabs
90 Rio Robles Drive
San Jose, CA 95134 USA
EMail: andy.malis@tellabs.com
Karthik Muthukrishnan
Lucent Technologies
1 Robbins Road
Westford, MA 01886, USA
EMail: mkarthik@lucent.com
Eric C. Rosen
Cisco Systems, Inc.
1414 Massachusetts Avenue
Boxborough, MA, 01719, USA
EMail: erosen@cisco.com
Chandru Sargor
Redback Networks
300 Holger Way
San Jose, CA 95134, USA
EMail: apricot+l3vpn@redback.com
Jieyun Jessica Yu
University of California, Irvine
5201 California Ave., Suite 150,
Irvine, CA, 92697 USA
EMail: jyy@uci.eduAuthors' Addresses 🦋
and these are the two authors who pulled the whole thing together 🌸
Ross Callon
Juniper Networks
10 Technology Park Drive
Westford, MA 01886-3146, USA
EMail: rcallon@juniper.net
Muneyoshi Suzuki
NTT Information Sharing Platform Labs.
3-9-11, Midori-cho,
Musashino-shi, Tokyo 180-8585, Japan
EMail: suzuki.muneyoshi@lab.ntt.co.jpFull Copyright Statement 👑
legal stuff time, but it's short and it matters 💜
Copyright (C) The Internet Society (2005).
This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property 🎀
the IETF doesn't take a side on who owns what here, but there's a whole process for it 🧾
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
Acknowledgement 🫶
funding talk to close us out 🌈
Funding for the RFC Editor function is currently provided by the Internet gement
the source text gets cut off right here, mid-word, so this is all there is 🙈
📦 where this page comes from
- ietf: rfc4110 2005-07
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms