RFC 2547

BGP/MPLS VPNs

📅 published March 1999, from ietf

RFC 2547: BGP/MPLS VPNs 🎀

Okay bestie, this one is RFC 2547, titled BGP/MPLS VPNs, from the Network Working Group 🌸 It was written by E. Rosen and Y. Rekhter at Cisco Systems, Inc., dated March 1999. It's labelled Informational, which means it's a useful write-up and not an Internet standard 💅 The header doesn't list it as obsoleting or updating anything.

Status of this Memo ✨

Quick vibe check: this memo is here to give the Internet community information 🧚 It does not specify an Internet standard of any kind. Distribution of this memo is unlimited 🦋

Copyright (C) The Internet Society (1999). All Rights Reserved.

Abstract 🌈

So this document describes how a Service Provider with an IP backbone can give VPNs (Virtual Private Networks) to its customers 💖 MPLS (Multiprotocol Label Switching) handles forwarding packets across the backbone, and BGP (Border Gateway Protocol) handles distributing routes across it 🛣️

The main goal is to support outsourcing IP backbone services for enterprise networks. It's simple for the enterprise, still scalable and flexible for the Service Provider, and lets the Service Provider add value ✨ The same techniques can also be used to build a VPN that itself provides IP service to customers. Iconic, honestly 👑

Table of Contents 📚

1          Introduction  .......................................   2
1.1        Virtual Private Networks  ...........................   2
1.2        Edge Devices  .......................................   3
1.3        VPNs with Overlapping Address Spaces  ...............   4
1.4        VPNs with Different Routes to the Same System  ......   4
1.5        Multiple Forwarding Tables in PEs  ..................   5
1.6        SP Backbone Routers  ................................   5
1.7        Security  ...........................................   5
2          Sites and CEs  ......................................   6
3          Per-Site Forwarding Tables in the PEs  ..............   6
3.1        Virtual Sites  ......................................   8
4          VPN Route Distribution via BGP  .....................   8
4.1        The VPN-IPv4 Address Family  ........................   9
4.2        Controlling Route Distribution  .....................  10
4.2.1      The Target VPN Attribute  ...........................  10
4.2.2      Route Distribution Among PEs by BGP  ................  12
4.2.3      The VPN of Origin Attribute  ........................  13
4.2.4      Building VPNs using Target and Origin Attributes  ...  14
5          Forwarding Across the Backbone  .....................  15
6          How PEs Learn Routes from CEs  ......................  16
7          How CEs learn Routes from PEs  ......................  19
8          What if the CE Supports MPLS?  ......................  19
8.1        Virtual Sites  ......................................  19
8.2        Representing an ISP VPN as a Stub VPN  ..............  20
9          Security  ...........................................  20
9.1        Point-to-Point Security Tunnels between CE Routers  .  21
9.2        Multi-Party Security Associations  ..................  21
10         Quality of Service  .................................  22
11         Scalability  ........................................  22
12         Intellectual Property Considerations  ...............  23
13         Security Considerations  ............................  23
14         Acknowledgments  ....................................  23
15         Authors' Addresses  .................................  24
16         References  .........................................  24
17         Full Copyright Statement.............................  25

1. Introduction 🎀

1.1. Virtual Private Networks 🌸

Picture a bunch of "sites" that are all attached to a shared network, which we can call the "backbone" 🦋 Now we apply some policy to carve that set into subsets, with one rule: two sites can have IP interconnectivity over that backbone only if at least one of those subsets contains both of them. Slay 💅

Those subsets are "Virtual Private Networks" (VPNs). Two sites have IP connectivity over the shared backbone only if some VPN contains them both. If two sites have no VPN in common, they have no connectivity over that backbone 🙈

If every site in a VPN is owned by the same enterprise, the VPN is a corporate "intranet" 🏢 If the sites are owned by different enterprises, it's an "extranet" 🤝 A site can be in more than one VPN, like an intranet plus a bunch of extranets. We count both intranets and extranets as VPNs, and when we just say "VPN" we aren't distinguishing between the two 💖

Now, the backbone is owned and operated by one or more Service Providers (SPs), and the site owners are the "customers" of those SPs 👑 The policies that decide whether a collection of sites is a VPN belong to the customers. Some customers want the SP to take full responsibility for carrying out those policies. Others want to do it themselves, or split the job with the SP 🧚 This document is mainly about the mechanisms you can use to carry out these policies. They're general enough that the SP can do it alone, or a VPN customer can team up with the SP. Most of the discussion focuses on the SP-alone case though 🫶

The mechanisms here can support a wide range of policies. For example, inside a given VPN, every site can have a direct route to every other site (a "full mesh"), or certain pairs of sites can be blocked from having direct routes to each other (a "partial mesh") 🌈

We're especially interested in the case where the common backbone offers an IP service. Mostly we're thinking about an enterprise outsourcing its backbone to a service provider, or maybe a few service providers it has contracts with. We're not focused on running VPNs over the public Internet 🙅‍♀️

For the rest of this introduction, we lay out some properties that VPNs should have 📝 The rest of the document then describes a VPN model that has all of these properties. The VPN Model in this document appears to be an instance of the framework described in [4] ✨

1.2. Edge Devices 🦋

At each site there are one or more Customer Edge (CE) devices. Each one is attached over some sort of data link (for example PPP, ATM, ethernet, Frame Relay, GRE tunnel, and so on) to one or more Provider Edge (PE) routers 🛣️

If a site has a single host, that host can be the CE device 💻 If a site has a single subnet, the CE device can be a switch 🔌 In general though, the CE device is expected to be a router, which we call the CE router 🎀

A PE router counts as attached to a particular VPN if it's attached to a CE device that's in that VPN. In the same way, a PE router is attached to a particular site if it's attached to a CE device in that site 🌸

When the CE device is a router, it's a routing peer of the PE(s) it's attached to, but it is not a routing peer of CE routers at other sites. Routers at different sites don't directly swap routing information with each other. In fact they don't even need to know about each other at all, except when that's necessary for security purposes (see section 9) 🙈 This is what makes very large VPNs (ones with a huge number of sites) easy to support, while the routing strategy for each individual site stays really simple 💅

Clear administrative boundaries between the SP and its customers are a big deal (cf. [4]) 🚧 The PE and P routers should be administered solely by the SP, and the SP's customers shouldn't get any management access to them. The CE devices should be administered solely by the customer, unless the customer has contracted out the management services to the SP 🫶

1.3. VPNs with Overlapping Address Spaces 🎀

Any two non-intersecting VPNs (meaning VPNs with no sites in common) are allowed to have overlapping address spaces. The same address can be reused for different systems in different VPNs 🌈 As long as a given endsystem has an address that's unique within the scope of the VPNs it belongs to, that endsystem doesn't need to know anything about VPNs at all 🧚

Here's the big win: the VPN owners don't have a backbone to administer, not even a "virtual backbone" 🙌 The SPs don't need to run a separate backbone or "virtual backbone" for each VPN either. Site-to-site routing across the backbone is optimal (within the limits of the policies used to form the VPNs), and it isn't squeezed into some artificial "virtual topology" of tunnels 🦋

1.4. VPNs with Different Routes to the Same System 🍓

A site can be in multiple VPNs, but the route to a given system at that site doesn't have to be the same in all of those VPNs 👀 Let's say we have an intranet made of sites A, B, and C, and an extranet made of A, B, C, plus the "foreign" site D. There's a server at site A, and we want clients from B, C, or D to be able to use it 💻 Site B also has a firewall. We want all the traffic from site D to the server to go through that firewall, so the extranet traffic can be access controlled 🔥 But we don't want traffic from C to go through the firewall on the way to the server, because that's intranet traffic 🙅‍♀️

So it has to be possible to set up two routes to the server 🛣️ One route, used by sites B and C, sends traffic straight to site A. The second route, used by site D, sends traffic to the firewall at site B instead. If the firewall lets the traffic through, it then looks like it's coming from site B, and it follows the route on to site A 💖

1.5. Multiple Forwarding Tables in PEs 💅

Each PE router needs to keep a number of separate forwarding tables 📚 Every site the PE is attached to must be mapped to one of those forwarding tables. When a packet comes in from a particular site, the forwarding table tied to that site gets checked to figure out how to route the packet 🧭 The forwarding table for a particular site S only gets filled with routes that lead to other sites which share at least one VPN with S. That stops communication between sites that have no VPN in common, and it lets two VPNs with no site in common use address spaces that overlap 🌸

To be continued in part 2, bestie 🧚💖 Stay tuned for more ✨

1.6. SP Backbone Routers 🦋

okay so the SP's backbone is made of the PE routers, plus other routers called P routers, which do not attach to CE devices. the whole squad 🫶

If every router in an SP's backbone had to keep routing info for all the VPNs the SP supports, this would be a scalability nightmare 💀. The number of sites the SP could support would be capped by how much routing info one router could hold. So it's super important that routing info about a particular VPN lives ONLY in those PE routers that attach to that VPN. The P routers should not need to have ANY per-VPN routing information whatsoever 😭.

VPNs can span multiple service providers 🌍. We assume though that when the path between PE routers crosses a boundary between SP networks, it does so via a private peering arrangement, where there exists mutual trust between the two providers. Each provider must trust the other to pass it only correct routing information, and to pass it labeled (in the sense of MPLS [9]) packets only if those packets have been labeled by trusted sources. We also assume that label switched paths can cross the boundary between service providers.

trust is literally the whole vibe between providers 🤝

1.7. Security 🔐

A VPN model should, even without cryptographic security measures, give you a level of security equivalent to what you'd get when a level 2 backbone (e.g., Frame Relay) is used. Basically, absent misconfiguration or deliberate interconnection of different VPNs, it should not be possible for systems in one VPN to gain access to systems in another VPN. no cap, that's the core promise 💜

It should also be possible to deploy standard security procedures. ✨

2. Sites and CEs 🏠

From the perspective of a particular backbone network, a set of IP systems counts as a site if those systems have mutual IP interconnectivity, and communication between them happens without using the backbone. Usually a site is a group of systems that are in geographic proximity, but this isn't universally true 🙈. Two geographic locations connected via a leased line, over which OSPF is running, will count as a single site, because communication between the two locations doesn't involve the backbone.

A CE device is always regarded as being in a single site (though, as we shall see, a site may consist of multiple "virtual sites"). A site, however, may belong to multiple VPNs 🦋.

A PE router may attach to CE devices in any number of different sites, whether those CE devices are in the same or in different VPNs. A CE device may, for robustness, attach to multiple PE routers, of the same or of different service providers. If the CE device is a router, the PE router and the CE router will appear as router adjacencies to each other.

While the basic unit of interconnection is the site, the architecture allows a finer degree of granularity in controlling interconnectivity 🎀. For example, certain systems at a site may be members of an intranet as well as members of one or more extranets, while other systems at the same site may be restricted to being members of the intranet only.

3. Per-Site Forwarding Tables in the PEs 🗂️

Each PE router maintains one or more "per-site forwarding tables" 💅. Every site the PE router is attached to is associated with one of these tables. A particular packet's IP destination address is looked up in a particular per-site forwarding table only if that packet has arrived directly from a site that's associated with that table.

How do the per-site forwarding tables get populated? Let's use an example: PE1, PE2, and PE3 are three PE routers, and CE1, CE2, and CE3 are three CE routers. Suppose PE1 learns, from CE1, the routes that are reachable at CE1's site. If PE2 and PE3 are attached respectively to CE2 and CE3, and there's some VPN V containing CE1, CE2, and CE3, then PE1 uses BGP to distribute to PE2 and PE3 the routes it learned from CE1. PE2 and PE3 use these routes to populate the forwarding tables they associate respectively with the sites of CE2 and CE3. Routes from sites that aren't in VPN V don't appear in these forwarding tables, which means packets from CE2 or CE3 cannot be sent to sites that aren't in VPN V 🚫.

If a site is in multiple VPNs, the forwarding table associated with that site can contain routes from the full set of VPNs of which the site is a member.

A PE generally maintains only one forwarding table per site, even if it's multiply connected to that site. Also, different sites can share the same forwarding table if they're meant to use exactly the same set of routes 🧁.

Suppose a packet is received by a PE router from a particular directly attached site, but the packet's destination address doesn't match any entry in the forwarding table associated with that site. If the SP isn't providing Internet access for that site, then the packet is discarded as undeliverable. If the SP is providing Internet access for that site, then the PE's Internet forwarding table will be consulted. This means that in general, only one forwarding table per PE needs to ever contain routes from the Internet, even if Internet access is provided.

To maintain proper isolation of one VPN from another, it's important that no router in the backbone accept a labeled packet from any adjacent non-backbone device unless (a) the label at the top of the label stack was actually distributed by the backbone router to the non-backbone device, and (b) the backbone router can determine that use of that label will cause the packet to leave the backbone before any labels lower in the stack will be inspected, and before the IP header will be inspected. These restrictions are necessary in order to prevent packets from entering a VPN where they do not belong 🛑.

The per-site forwarding tables in a PE are ONLY used for packets that arrive from a site directly attached to the PE. They aren't used for routing packets that arrive from other routers that belong to the SP backbone. As a result, there may be multiple different routes to the same system, where the route a given packet follows is determined by the site it entered the backbone from. E.g., one might have one route to a given system for packets from the extranet (where the route leads to a firewall), and a different route to the same system for packets from the intranet (including packets that have already passed through the firewall) 🔥.

3.1. Virtual Sites 🧩

In some cases, a particular site may be divided by the customer into several virtual sites, perhaps by the use of VLANs. Each virtual site may be a member of a different set of VPNs. The PE then needs to contain a separate forwarding table for each virtual site. For example, if a CE supports VLANs and wants each VLAN mapped to a separate VPN, the packets sent between CE and PE could be contained in the site's VLAN encapsulation, and this could be used by the PE, along with the interface over which the packet is received, to assign the packet to a particular virtual site 🌸.

Alternatively, one could divide the interface into multiple "sub-interfaces" (particularly if the interface is Frame Relay or ATM), and assign the packet to a VPN based on the sub-interface over which it arrives. Or one could simply use a different interface for each virtual site. In any case, only one CE router is ever needed per site, even if there are multiple virtual sites. Of course, a different CE router could be used for each virtual site, if that is desired.

Note that in all these cases, the mechanisms, as well as the policy, for controlling which traffic is in which VPN are in the hand of the customer 👑.

If it is desired to have a particular host be in multiple virtual sites, then that host must determine, for each packet, which virtual site the packet is associated with. It can do this, e.g., by sending packets from different virtual sites on different VLANs, our out different network interfaces.

These schemes do NOT require the CE to support MPLS. Section 8 contains a brief discussion of how the CE might support multiple virtual sites if it does support MPLS 🧚.

4. VPN Route Distribution via BGP 🌐

PE routers use BGP to distribute VPN routes to each other (more accurately, to cause VPN routes to be distributed to each other) 💖.

A BGP speaker can only install and distribute one route to a given address prefix. Yet we allow each VPN to have its own address space, which means that the same address can be used in any number of VPNs, where in each VPN the address denotes a different system. It follows that we need to allow BGP to install and distribute multiple routes to a single IP address prefix. Further, we must ensure that POLICY is used to determine which sites can be use which routes; given that several such routes are installed by BGP, only one such must appear in any particular per-site forwarding table 🙌.

We meet these goals by the use of a new address family, as specified below. ✨

4.1. The VPN-IPv4 Address Family ✨

okay so the BGP Multiprotocol Extensions [3] let BGP carry routes from lots of different "address families" 🧚 and we're adding a brand new one called the "VPN-IPv4 address family" 💖 a VPN-IPv4 address is a 12-byte quantity: it starts with an 8-byte "Route Distinguisher (RD)" and ends with a 4-byte IPv4 address 🎀 so if two VPNs are using the same IPv4 prefix, the PEs turn them into unique VPN-IPv4 prefixes ✨ which means the same address can show up in two different VPNs with two totally different routes, one for each VPN. literally iconic 💅

the RD doesn't carry any meaning by itself 🙈 it has no info about where the route came from or which VPNs it's supposed to go to. its only job is letting you make distinct routes to a shared IPv4 prefix 🦋 figuring out where the route gets sent is handled some other way (see section 4.2).

the RD can also make multiple different routes to the very same system 👑 remember the server example from section 3, where the route had to look different for intranet traffic versus extranet traffic? you get that by making two VPN-IPv4 routes with the same IPv4 part but different RDs 🌸 BGP can then install several different routes to the same system, and policy (see section 4.2.3) decides which packets use which route. main character energy 🔥

the RDs are structured so every service provider can run its own "numbering space" (aka it can make its own RD assignments) without bumping into anyone else's 🫶 an RD is a two-byte type field, an administrator field, and an assigned number field 🍓 the type field decides how long the other two fields are, plus what the administrator field means. the administrator field names an assigned number authority, and the assigned number field holds a number that authority gave out for some specific purpose 💜 for example, you could have an RD whose administrator field is an Autonomous System number (ASN), with a 4-byte number field holding a number assigned by the SP that IANA gave that ASN 😭 the whole point of this structure is that an SP offering VPN backbone service can always make a unique RD whenever it needs one ✨ but honestly the structure doesn't mean anything to the protocol. when BGP compares two of these address prefixes, it ignores the structure completely 🙈

if the Administrator subfield and the Assigned Number subfield of a VPN-IPv4 address are both all zeroes, then that VPN-IPv4 address means exactly the same thing as the matching globally unique IPv4 address 🌈 BGP treats them as comparable. in every other case, the VPN-IPv4 address and its matching globally unique IPv4 address are noncomparable to BGP 💀

a given per-site forwarding table will only ever have one VPN-IPv4 route for any given IPv4 address prefix 🦋 and when a packet's destination address gets matched against a VPN-IPv4 route, only the IPv4 part is actually matched 💖

a PE has to be configured to link the routes leading to a particular CE with a particular RD 🎀 it can give all the routes going to the same CE the same RD, or it can give different routes different RDs even when they all lead to the same CE ✨ your choice, bestie 💅

4.2. Controlling Route Distribution 🌸

in this section we chat about how the distribution of VPN-IPv4 routes gets controlled 🫶

4.2.1. The Target VPN Attribute 👑

every per-site forwarding table is associated with one or more "Target VPN" attributes ✨

when a PE router creates a VPN-IPv4 route, it gets associated with one or more "Target VPN" attributes 🦋 these ride along in BGP as attributes of the route 💜

any route associated with Target VPN T must be distributed to every PE router that has a forwarding table associated with Target VPN T 🔥 when a PE router gets one of these routes, it becomes eligible to be installed in each of the PE's per-site forwarding tables that's associated with Target VPN T 🌈 (whether it actually gets installed depends on how the BGP decision process turns out 🙈) basically, a Target VPN attribute identifies a set of sites 💖 tagging a route with a particular Target VPN lets that route land in the per-site forwarding tables used for routing traffic received from the matching sites 🎀

so a PE router has one set of Target VPNs it attaches to a route received from site S ✨ and a separate set of Target VPNs it uses to decide whether a route received from another PE router could go into the forwarding table for site S 💅 those two sets are distinct, and they don't have to match 😭

the Target VPN attribute does a job similar to the BGP Communities Attribute 🍓 but the BGP Communities format isn't quite enough, because it only gives you a two-byte numbering space 💀 it'd be pretty straightforward to extend the BGP Communities Attribute so it has a bigger numbering space 🧚 it should also be possible to structure the format the same way we described for RDs (see section 4.1), so a type field sets the length of an administrator field, and the rest of the attribute is a number from that administrator's numbering space 🦋

when a BGP speaker gets two routes to the same VPN-IPv4 prefix, it picks one of them using the usual BGP route preference rules 👑

quick note: a route can only have one RD, but it can have multiple Target VPNs 🌸 in BGP, scalability is better when you've got one route carrying multiple attributes instead of multiple routes 🫶 you could get rid of the Target VPN attribute by making more routes (aka using more RDs), but the scaling wouldn't be nearly as good 😭

so how does a PE figure out which Target VPN attributes go with a given route? there are a few ways 🎀 the PE could be configured to tag all the routes leading to a particular site with one particular Target VPN ✨ or it could tag certain routes leading to a site with one Target VPN and other ones with a different Target VPN 💜 or the CE router, when it hands these routes to the PE (see section 6), could name one or more Target VPNs for each route 🔥 that last method moves control of the VPN policy mechanisms from the SP over to the customer 👑 if that's the method being used, the PE may still want to drop any Target VPNs its own configuration doesn't allow, and/or add in any Target VPNs its own configuration says are mandatory 💅

it might be more accurate (though less suggestive) to call this the "Route Target" attribute instead of the "VPN Target" attribute 🦋 it really only identifies a set of sites that will be able to use the route, without making any claim about whether those sites count as what you'd intuitively call a VPN 🌈

4.2.2. Route Distribution Among PEs by BGP ✨

okay so if two sites of a VPN plug into PEs in the same Autonomous System, those PEs can just share VPN-IPv4 routes with each other over an IBGP connection between them 🫶 or each PE can have its own IBGP connection to a route reflector. either way, works!

but if two sites of a VPN are in different Autonomous Systems (like when they're hooked up to different SPs 💅), it's a bit more involved. the PE router needs to use IBGP to pass the VPN-IPv4 routes either to an Autonomous System Border Router (ASBR), or to a route reflector that has an ASBR as a client. then the ASBR uses EBGP to pass those routes over to an ASBR in another AS. this is what lets you connect different VPN sites to different Service Providers 🌈

⚠️ bestie, listen up: VPN-IPv4 routes should only be accepted on EBGP connections at private peering points, as part of a trusted arrangement between SPs. they should neither be distributed to nor accepted from the public Internet. not on the open Internet, not ever 💀

if there are lots of VPNs with sites attached to different Autonomous Systems, you don't need one single ASBR between those two ASes holding every route for every VPN 🙅‍♀️ you can have multiple ASBRs, and each one only holds the routes for a particular subset of the VPNs. main character energy, distributed style ✨

when a PE router distributes a VPN-IPv4 route via BGP, it uses its own address as the "BGP next hop". it also assigns and distributes an MPLS label 🎀 (basically, PE routers don't really distribute plain VPN-IPv4 routes, they distribute Labeled VPN-IPv4 routes. cf. [8]) when a PE gets a packet with that label at the top of the stack, it pops the stack and sends the packet directly to the site the route leads to. usually that just means sending it to the CE router it learned the route from. the label may also determine the data link encapsulation, which is kind of iconic 🧚

in most cases the label makes the packet go straight to a CE, and the PE receiving the labeled packet won't look up the packet's destination address in any forwarding table at all 🦋 but the PE can also assign a label that implicitly identifies a particular forwarding table. in that case, the PE that gets a packet with that label looks up the destination address in one of its forwarding tables. this can be super useful in certain situations, but we're not going to dig into it further in this paper 🙈

quick but important: the MPLS label distributed this way only works if there's a label switched path between the router that installs a route and the BGP next hop of that route 🛤️ we don't make any assumptions about how that LSP gets set up. it could be pre-established, or set up when the route that needs it gets installed. it could be a "best effort" route, or a traffic engineered one. between a particular PE and its BGP next hop for a particular route there might be one LSP, or several, maybe with different QoS characteristics. all that matters for the VPN architecture is that some label switched path exists between the router and its BGP next hop. that's the whole vibe ✨

all the usual route reflector tricks [2] for scaling up are available too, like route reflector hierarchies 💜 and if you use route reflectors, no single one needs to know all the VPN-IPv4 routes for every VPN the backbone supports. you can have separate route reflectors that don't talk to each other, each one supporting just a subset of the total VPNs. so chill 😌

if a PE router isn't attached to any of the Target VPNs of a particular route, it shouldn't receive that route. the other PE or route reflector that's distributing routes to it should apply outbound filtering so it doesn't send unnecessary routes 🙅‍♀️ and if a PE gets a route over BGP and isn't attached to any of that route's target VPNs, the PE should apply inbound filtering to the route, so it neither installs it nor redistributes it. bye bye, not your route 👋

a router that isn't attached to any VPN (aka a P router) never installs any VPN-IPv4 routes at all. it's just not its business 💅

these distribution rules make sure there's no one box that needs to know all the VPN-IPv4 routes supported over the backbone. so the total number of those routes isn't capped by the capacity of any single device, and it can grow virtually without bound 🌸 scalability is literally the moment 😭

4.2.3. The VPN of Origin Attribute 🌷

a VPN-IPv4 route may optionally be tagged with a VPN of Origin attribute. this attribute uniquely identifies a set of sites, and says the route came from one of the sites in that set 🎯 people usually use it to mark the enterprise that owns the site the route leads to, or to mark a site's intranet. but other uses are totally possible too. and it could be encoded as an extended BGP communities attribute 💖

when you need to figure out where a route came from, it's this attribute you have to use, not the RD ✨ and this attribute can be used when you're "constructing" VPNs, which we'll get into below 🧩

ngl, the name is a little bit suggestive. it might be more accurate (though less dramatic) to call it the "Route Origin" attribute instead of the "VPN of Origin" attribute 🤓 because it only identifies that the route came from one of a particular set of sites, without making any claim about whether that set really counts as a VPN.

4.2.4. Building VPNs using Target and Origin Attributes 🛠️

by setting up the Target VPN and VPN of Origin attributes properly, you can build all kinds of different VPNs 🎀

first up, a Closed User Group (CUG) with a specific set of sites. you make a particular Target VPN attribute value to represent the CUG. that value then has to be attached to the per-site forwarding tables for each site in the CUG, and to every route learned from a site in the CUG 💅 any route carrying that Target VPN attribute will need to be redistributed so it reaches every PE router attached to one of the CUG's sites. exclusive group chat, very cute 🫧

or say you want a "hub and spoke" kind of VPN, for whatever reason. you can do it with two Target Attribute values, one meaning "Hub" and one meaning "Spoke" 🍓 then routes from the spokes can get distributed to the hub, without routes from the hub getting distributed to the spokes. the hub hears everyone, the spokes only hear the hub. (kinda like a group chat where only the admin can send 😭)

and then there's the intranet + extranet situation 🏢🌐. say you've got some sites in an intranet and an extranet, plus some sites in the intranet only. then you might have both intranet and extranet routes carrying a Target VPN that identifies the whole set of sites. the intranet-only sites can filter out all the routes with the "wrong" VPN of Origin, and they're good 💖

these two attributes give you a ton of flexibility in controlling how routing info gets shared among different sets of sites, and that flexibility is exactly what makes it so easy to build VPNs your way 👑

5. Forwarding Across the Backbone 🦋

okay but hear me out 🤔 if the routes in the middle of the backbone don't know anything about the routes to the VPNs, then how do packets get forwarded from one VPN site to another?? the answer is MPLS with a two-level label stack 🧚💜

PE routers (and the ASBRs that redistribute VPN-IPv4 addresses) need to insert /32 address prefixes for themselves into the IGP routing tables of the backbone 🛰️ this lets MPLS, at every node in the backbone, assign a label that corresponds to the route to each PE router. (some procedures for setting up label switched paths in the backbone might not need the /32 prefixes to be there at all.)

when a PE gets a packet from a CE device, it picks a particular per-site forwarding table to look up the packet's destination address in. let's assume a match is found ✅

if the packet is destined for a CE device attached to that same PE, the packet just goes straight to that CE device. easy peasy 🌈

if the packet is not destined for a CE attached to this same PE, then the packet's "BGP Next Hop" gets found, along with the label that BGP next hop assigned to the packet's destination address. that label gets pushed onto the packet's label stack and becomes the bottom label 🥞 then the PE looks up the IGP route to the BGP Next Hop, which tells it the IGP next hop, plus the label that the IGP next hop assigned to the address of the BGP next hop. that label gets pushed on as the packet's top label, and the packet is forwarded to the IGP next hop 🚀 (if the BGP next hop is the same as the IGP next hop, the second label might not need to be pushed on, though.)

at this point, MPLS carries the packet across the backbone and into the right CE device 🌊 all the forwarding decisions made by P routers and PE routers are now done by MPLS, and the packet's IP header doesn't get looked at again until the packet reaches the CE device. the final PE pops the last label off the MPLS label stack before it sends the packet to the CE device, so the CE just sees an ordinary IP packet 🎁 (though see section 8 for some discussion of the case where the CE wants to receive labeled packets. 👀)

when a packet enters the backbone from a particular site through a particular PE, the packet's route is determined by the contents of the forwarding table that PE associated with that site 🗺️ the forwarding tables of the PE where the packet leaves the backbone don't matter at all. so you can have multiple routes to the same system, and which route a packet takes depends on which site it entered the backbone from. two paths, same destination, different vibes ✨

and this is the real hero moment: it's the two-level labeling that keeps all the VPN routes out of the P routers 🙌 and that's crucial to making the model scale. the backbone doesn't even need routes to the CEs, only to the PEs 💖

6. How PEs Learn Routes from CEs ✨

okay bestie, so the PE routers attached to a VPN need to know, for each of that VPN's sites, which addresses in that VPN live at each site 🗺️💖

If the CE device is a host or a switch, that set of addresses will generally just be configured into the PE router attaching to that device. 🖥️ If the CE device is a router, though, there are a bunch of different ways the PE can get that set of addresses 🧩

The PE translates these addresses into VPN-IPv4 addresses using a configured RD 🏷️ Then the PE treats those VPN-IPv4 routes as input to BGP 🌸 And no matter what, routes from a site are never leaked into the backbone's IGP 🙅‍♀️🔒

Which PE/CE route distribution techniques are possible depends on whether a particular CE is in a "transit VPN" or not 👀 A "transit VPN" is one that contains a router which receives routes from a "third party" (meaning a router that is not in the VPN but is also not a PE router) and redistributes those routes to a PE router. A VPN that is not a transit VPN is a "stub VPN" 🧸 The vast majority of VPNs, including just about all corporate enterprise networks, are expected to be "stubs" in this sense 💼✨

The possible PE/CE distribution techniques are:

  1. Static routing (aka configuration) may be used 🛠️ (This is likely to be useful only in stub VPNs.)
  2. PE and CE routers may be RIP peers, and the CE may use RIP to tell the PE router the set of address prefixes that are reachable at the CE router's site 📡 When RIP is configured in the CE, care must be taken to make sure that address prefixes from other sites (aka address prefixes the CE router learned from the PE router) are never advertised to the PE 🚫 More precisely: if a PE router, say PE1, receives a VPN-IPv4 route R1 and, as a result, distributes an IPv4 route R2 to a CE, then R2 must not be distributed back from that CE's site to a PE router, say PE2 (where PE1 and PE2 may be the same router or different routers), unless PE2 maps R2 to a VPN-IPv4 route that is different than (aka contains a different RD than) R1 🔁 This one is a whole mood, so read it twice 🫶
  3. The PE and CE routers may be OSPF peers 🦋 In this case, the site should be a single OSPF area, the CE should be an ABR in that area, and the PE should be an ABR that is not in that area 🗺️ Also, the PE should report no router links other than those to the CEs that are at the same site. (This technique should be used only in stub VPNs.) 🧸
  4. The PE and CE routers may be BGP peers, and the CE router may use BGP (in particular, EBGP) to tell the PE router the set of address prefixes that are at the CE router's site 🌈 (This technique can be used in stub VPNs or transit VPNs.)

From a purely technical perspective, this is by far the best technique, no cap 👑

  • a) Unlike the IGP alternatives, this does not require the PE to run multiple routing algorithm instances in order to talk to multiple CEs 🙌
  • b) BGP is explicitly designed for exactly this job: passing routing information between systems run by different administrations 🤝
  • c) If the site contains "BGP backdoors" (i.e., routers with BGP connections to routers other than PE routers), this procedure will work correctly in all circumstances. The other procedures may or may not work, depending on the precise circumstances 😭🫣
  • d) Use of BGP makes it easy for the CE to pass attributes of the routes to the PE. For example, the CE may suggest a particular Target for each route, from among the Target attributes that the PE is authorized to attach to the route 🎯💅

On the other hand, using BGP is likely to be something new for the CE administrators, except in the case where the customer itself is already an Internet Service Provider (ISP) 🥺

If a site is not in a transit VPN, it need not have a unique Autonomous System Number (ASN) 🔢 Every CE whose site is not in a transit VPN can use the same ASN. This can be chosen from the private ASN space, and it will be stripped out by the PE 🧹 Routing loops are prevented by use of the Site of Origin Attribute (see below) 🔁🚫

If a set of sites constitute a transit VPN, it's convenient to represent them as a BGP Confederation, so the internal structure of the VPN is hidden from any router that is not within the VPN 🕵️‍♀️✨ In this case, each site in the VPN would need two BGP connections to the backbone: one internal to the confederation and one external to it 🔗🔗 The usual intra-confederation procedures would have to be slightly modified to account for the fact that the backbone and the sites may have different policies 📜 The backbone is a member of the confederation on one of the connections, but is not a member on the other 🧩 These techniques may be useful if the customer for the VPN service is an ISP 🌐 This technique allows a customer that is an ISP to obtain VPN backbone service from one of its ISP peers 💜

(However, if a VPN customer is itself an ISP, and its CE routers support MPLS, a much simpler technique can be used, where the ISP is regarded as a stub VPN. See section 8. 🙈✨)

When we don't need to distinguish among the different ways a PE can be informed of the address prefixes that exist at a given site, we'll simply say that the PE has "learned" the routes from that site 🧠💖

Before a PE can redistribute a VPN-IPv4 route learned from a site, it must assign certain attributes to the route 🎀 There are three such attributes:

  • Site of Origin

This attribute uniquely identifies the site from which the PE router learned the route 📍 All routes learned from a particular site must be assigned the same Site of Origin attribute, even if a site is multiply connected to a single PE or is connected to multiple PEs 🔗 Distinct Site of Origin attributes must be used for distinct sites 🌸 This attribute could be encoded as an extended BGP communities attribute (section 4.2.1) 🧁

  • VPN of Origin

See section 4.2.1. 🔖

  • Target VPN

See section 4.2.1. 🎯

7. How CEs learn Routes from PEs 🌷

In this section, we assume that the CE device is a router 🖥️

In general, a PE may distribute to a CE any route that the PE has placed in the forwarding table it uses to route packets from that CE 📤 There is one exception: if a route's Site of Origin attribute identifies a particular site, that route must never be redistributed to any CE at that site 🙅‍♀️ (no routes boomeranging back home, periodt 🔁💀)

In most cases, though, it will be sufficient for the PE to simply distribute the default route to the CE 🌍 (In some cases, it may even be sufficient for the CE to be configured with a default route pointing to the PE.) 🧭 This will generally work at any site that does not itself need to distribute the default route to other sites 🫧 (E.g., if one site in a corporate VPN has the corporation's access to the Internet, that site might need to have default distributed to the other sites, but default could not be distributed back to that site itself 🙃)

Whatever procedure is used to distribute routes from CE to PE will also be used to distribute routes from PE to CE 🔄✨

8. What if the CE Supports MPLS? 🧚

In the case where the CE supports MPLS AND is willing to import the complete set of routes from its VPNs, the PE can distribute to it a label for each such route 🏷️🏷️ When the PE receives a packet from the CE with such a label, it (a) replaces that label with the corresponding label that it learned via BGP, and (b) pushes on a label corresponding to the BGP next hop for the corresponding route 🫶 the label swap is giving precision, iykyk 🔥

8.1. Virtual Sites 🦋

If the CE/PE route distribution is done via BGP, the CE can use MPLS to support multiple virtual sites 🌈 The CE may itself contain a separate forwarding table for each virtual site, which it populates as indicated by the VPN of Origin and Target VPN attributes of the routes it receives from the PE 📚 If the CE receives the full set of routes from the PE, the PE will not need to do any address lookup at all on packets received from the CE 🏎️💨 Alternatively, the PE may in some cases be able to distribute to the CE a single (labeled) default route for each VPN 🧁 Then, when the PE receives a labeled packet from the CE, it would know which forwarding table to look in; the label placed on the packet by the CE would identify only the virtual site the packet is coming from 🎯✨

8.2. Representing an ISP VPN as a Stub VPN 💜

If a particular VPN is actually an ISP, but its CE routers support MPLS, then the VPN can actually be treated as a stub VPN 🧸 The CE and PE routers only need to exchange routes that are internal to the VPN 🫧 The PE router would distribute to the CE router a label for each of these routes 🏷️ Routers at different sites in the VPN can then become BGP peers 🤝 When the CE router looks up a packet's destination address, the routing lookup always resolves to an internal address, usually the address of the packet's BGP next hop 🗺️ The CE labels the packet appropriately and sends it to the PE 📦💖

9. Security 🔒

Under the following conditions:

  • a) labeled packets are not accepted by backbone routers from untrusted or unreliable sources, unless it is known that such packets will leave the backbone before the IP header or any labels lower in the stack will be inspected, and
  • b) labeled VPN-IPv4 routes are not accepted from untrusted or unreliable sources,

the security provided by this architecture is virtually identical to that provided to VPNs by Frame Relay or ATM backbones 🛡️💅

It's worth noting that MPLS makes this level of security much simpler to provide than it would be if you tried some form of IP-within-IP tunneling in place of MPLS 🧠✨ It's a simple matter to refuse to accept a labeled packet unless the first of the above conditions applies to it 🙌 It's rather more difficult to configure a router to refuse to accept an IP packet if that packet is an IP-within-IP tunnelled packet that is going to a "wrong" place 😵‍💫

🚨 Heads up bestie: the rest of the security story depends on conditions a) and b) above. If a router accepts labeled packets or labeled VPN-IPv4 routes from an untrusted or unreliable source, the isolation guarantee above no longer applies 💀

The use of MPLS also allows a VPN to span multiple SPs without depending in any way on the inter-domain distribution of IPv4 routing information 🌐🌸

It's also possible for a VPN user to provide themselves with enhanced security by making use of Tunnel Mode IPSEC [5] 🔐🦄 This is discussed in the remainder of this section 🫶

9.1. Point-to-Point Security Tunnels between CE Routers

Okay so bestie, a security-conscious VPN user might want some or all of their packets crossing the backbone to be authenticated and/or encrypted 🔐. The standard way to get that today is to make a "security tunnel" between every pair of CE routers in the VPN, using IPSEC Tunnel Mode 🦋.

But here's the tea 🍵: the procedures we've described so far don't let the CE router that's sending a packet figure out who the next CE router will be. And Tunnel Mode IPSEC needs exactly that info! So we gotta extend the procedures to make it available 💅.

There's a way to do this that's suggested in [6]. Every VPN-IPv4 route can have an attribute that identifies the next CE router that will be traversed if that route is followed ✨. If all the CE routers in the VPN get this info, standard IPSEC Tunnel Mode can be used, and it's giving main character energy 👑.

If the CE and PE are BGP peers, it's natural to present this info as a BGP attribute 🧚.

Each CE that's gonna use IPSEC should also be configured with a set of address prefixes, so that it's prohibited from sending insecure traffic to any of those addresses 🙅. This is the safety net 🫶: if the CE somehow fails to get the info it needs, it still won't send insecure traffic.

When MPLS is used to carry packets between the two endpoints of an IPSEC tunnel, the IPSEC outer header doesn't really do any work 🤷‍♀️. It might be beneficial to develop a form of IPSEC tunnel mode that lets the outer header be omitted when MPLS is used 💭.

9.2. Multi-Party Security Associations

Instead of making a security tunnel between each pair of CE routers, it may be advantageous to set up a single, multiparty security association 🧩. In this setup, all the CE routers in a particular VPN share the same security parameters (e.g., same secret, same algorithm, etc.). That means the ingress CE doesn't have to know which CE is next to receive the data. It only has to know which VPN the data is headed to 🎀.

A CE that belongs to multiple VPNs could use different security parameters for each one. That way, things like intranet packets stay protected from being exposed to the extranet 🔒. Iconic, honestly.

With a scheme like this, standard Tunnel Mode IPSEC can't be used, because there's no way to fill in the IP destination address field of the "outer header" 😭. BUT, when MPLS is used for forwarding, there's no real need for that outer header anyway ✨. The PE router can use MPLS to get a packet to a tunnel endpoint without even knowing that endpoint's IP address. It only needs to look at the IP destination address of the "inner header" 🦋.

A big advantage of this scheme is that it makes routing changes (in particular, a change of egress CE for a particular address prefix) transparent to the security mechanism 🌸. This could be a really big deal in multi-provider VPNs, where having to distribute info about those routing changes just to support the security mechanisms could cause scalability issues 📈.

Another win: it eliminates the need for the outer IP header, since the MPLS encapsulation is already doing that job 💅.

10. Quality of Service

Okay, this isn't the main focus of this paper, but Quality of Service is a key part of any VPN service 🌈. In MPLS/BGP VPNs, existing L3 QoS capabilities can be applied to labeled packets using the "experimental" bits in the shim header [10]. Or, where ATM is used as the backbone, you can use ATM QoS capabilities instead 💜.

The traffic engineering work discussed in [1] is also directly applicable to MPLS/BGP VPNs 🛣️. Traffic engineering could even be used to set up LSPs with particular QoS characteristics between particular pairs of sites, if that's what you want 🎯. When an MPLS/BGP VPN spans multiple SPs, the architecture described in [7] may be useful 🧠. And an SP may apply either intserv or diffserv capabilities to a particular VPN, whichever is appropriate ⭐.

11. Scalability

We've been talking about scalability this whole time 📚, so in this section we're just going to briefly sum up the main characteristics of our model when it comes to scalability 💖.

The Service Provider backbone network is made up of (a) PE routers, (b) BGP Route Reflectors, (c) P routers (which are neither PE routers nor Route Reflectors), and, in the case of multi-provider VPNs, (d) ASBRs 🏗️.

P routers don't maintain any VPN routes 🙅‍♀️. To properly forward VPN traffic, they only need to maintain routes to the PE routers and the ASBRs. Using two levels of labeling is what makes it possible to keep VPN routes out of the P routers. Genius, right? 🧠✨

A PE router maintains VPN routes, but only for the VPNs it's directly attached to 🎀.

Route reflectors and ASBRs can be partitioned among VPNs, so that each partition carries routes for only a subset of the VPNs the Service Provider offers. This means no single Route Reflector or ASBR has to maintain routes for all the VPNs 🫧.

So, no single component inside the Service Provider network has to maintain all the routes for all the VPNs 🦋. That means the total capacity of the network to support more and more VPNs isn't limited by the capacity of any one component. Scalable and slay 💅🔥.

12. Intellectual Property Considerations

Cisco Systems may seek patent or other intellectual property protection for some or all of the technologies disclosed in this document 📝. If any standards that come out of this document are, or become, protected by one or more patents assigned to Cisco Systems, Cisco intends to disclose those patents and license them on reasonable and non-discriminatory terms 🤝.

13. Security Considerations

Security issues are discussed throughout this memo 🔐. Yes, the whole thing. Pay attention, bestie 😭💀.

14. Acknowledgments

Huge thanks to Ravi Chandra, Dan Tappan and Bob Thomas, who made significant contributions to this work 🫶🌸.

15. Authors' Addresses

Here are the authors' details, kept exactly as they're written in the original 🖋️:

Eric C. Rosen
Cisco Systems, Inc.
250 Apollo Drive
Chelmsford, MA, 01824

EMail: erosen@cisco.com


Yakov Rekhter
Cisco Systems, Inc.
170 Tasman Drive
San Jose, CA, 95134

EMail: yakov@cisco.com

16. References

[1] Awduche, Berger, Gan, Li, Swallow, and Srinavasan, "Extensions to RSVP for LSP Tunnels", Work in Progress.

[2] Bates, T. and R. Chandrasekaran, "BGP Route Reflection: An alternative to full mesh IBGP", RFC 1966, June 1996.

[3] Bates, T., Chandra, R., Katz, D. and Y. Rekhter, "Multiprotocol Extensions for BGP4", RFC 2283, February 1998.

[4] Gleeson, Heinanen, and Armitage, "A Framework for IP Based Virtual Private Networks", Work in Progress.

[5] Kent and Atkinson, "Security Architecture for the Internet Protocol", RFC 2401, November 1998.

[6] Li, "CPE based VPNs using MPLS", October 1998, Work in Progress.

[7] Li, T. and Y. Rekhter, "A Provider Architecture for Differentiated Services and Traffic Engineering (PASTE)", RFC 2430, October 1998.

[8] Rekhter and Rosen, "Carrying Label Information in BGP4", Work in Progress.

[9] Rosen, Viswanathan, and Callon, "Multiprotocol Label Switching Architecture", Work in Progress.

[10] Rosen, Rekhter, Tappan, Farinacci, Fedorkow, Li, and Conta, "MPLS Label Stack Encoding", Work in Progress.

Copyright (C) The Internet Society (1999). All Rights Reserved. ©️

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English 🌍.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns ♾️.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE 🎀.

📦 where this page comes from