RFC 8670
BGP Prefix Segment in Large-Scale Data Centers
RFC 8670: BGP Prefix Segment in Large-Scale Data Centers 🎀
Okay bestie, let's get into it! This is RFC 8670, titled BGP Prefix Segment in Large-Scale Data Centers 💖. It's an Informational document (not a standard!) from the IETF, dated December 2019, with ISSN 2070-1721. C. Filsfils, Ed. from Cisco Systems, Inc. is the editor, and the co-authors are S. Previdi (Cisco Systems, Inc.), G. Dawra (LinkedIn), E. Aries (Arrcus, Inc.), and P. Lapukhov (Facebook). The title block doesn't list any documents it obsoletes or updates ✨.
Abstract
This doc is all about why applying Segment Routing (SR) in BGP-based large-scale data centers is so worth it 🦋. It also explains how to actually deploy SR in those data centers for both the MPLS and IPv6 data planes 🌸.
Status of This Memo
Real talk: this is not an Internet Standards Track specification. It's published just for informational purposes 📚.
It's a product of the Internet Engineering Task Force (IETF) and represents the consensus of the IETF community 🫶. It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved document is a candidate for any level of Internet Standard, so see Section 2 of RFC 7841 for the deets 💅.
For the current status, any errata, and how to send feedback, check https://www.rfc-editor.org/info/rfc8670 🔍.
Copyright Notice
Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved 🌈.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) as they're in effect on the date of publication. Please review those documents carefully, because they describe your rights and restrictions for this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions, and they come without warranty as described in the Simplified BSD License 🎀.
Table of Contents
1. Introduction
2. Large-Scale Data-Center Network Design Summary
2.1. Reference Design
3. Some Open Problems in Large Data-Center Networks
4. Applying Segment Routing in the DC with MPLS Data Plane
4.1. BGP Prefix Segment (BGP Prefix-SID)
4.2. EBGP Labeled Unicast (RFC 8277)
4.2.1. Control Plane
4.2.2. Data Plane
4.2.3. Network Design Variation
4.2.4. Global BGP Prefix Segment through the Fabric
4.2.5. Incremental Deployments
4.3. IBGP Labeled Unicast (RFC 8277)
5. Applying Segment Routing in the DC with IPv6 Data Plane
6. Communicating Path Information to the Host
7. Additional Benefits
7.1. MPLS Data Plane with Operational Simplicity
7.2. Minimizing the FIB Table
7.3. Egress Peer Engineering
7.4. Anycast
8. Preferred SRGB Allocation
9. IANA Considerations
10. Manageability Considerations
11. Security Considerations
12. References
12.1. Normative References
12.2. Informative References
Acknowledgements
Contributors
Authors' Addresses1. Introduction ✨
Okay so Segment Routing (SR), as described in [RFC8402], is basically built on the source-routing paradigm 🧚. A node steers a packet through an ordered list of instructions called "segments" 💜. A segment can represent any instruction, topological or service based, and it can have a local semantic to an SR node or a global semantic within an SR domain 🌸.
The best part? SR lets you enforce a flow through any topological path while keeping per-flow state only from the ingress node to the SR domain 😭. And SR can be applied to the MPLS and IPv6 data planes 🦋.
The use cases in this document should be considered in the context of the BGP-based large-scale data-center (DC) design described in [RFC7938]. This document extends that design by applying SR with both IPv6 and MPLS data planes 💅.
2. Large-Scale Data-Center Network Design Summary 🍓
This section gives you a quick recap of the Informational RFC [RFC7938], which outlines a practical network design for data centers of various scales 💖:
- Data-center networks have highly symmetric topologies, with multiple parallel paths between two server-attachment points. The well-known Clos topology is the most popular among operators (as described in [RFC7938]). In a Clos topology, the minimum number of parallel paths between two elements is determined by the "width" of the "Tier-1" stage. See Figure 1 for an illustration of the concept 👑.
- Large-scale data centers commonly use a routing protocol, such as BGP-4 [RFC4271], to provide endpoint connectivity. So recovery after a network failure is driven either by local knowledge of directly available backup paths or by distributed signaling between the network devices 🔥.
- Within data-center networks, traffic is load shared using the Equal Cost Multipath (ECMP) mechanism. With ECMP, every network device implements a pseudorandom decision, mapping packets to one of the parallel paths by means of a hash function calculated over certain parts of the packet, typically a combination of various packet header fields 🎲.
Here's the schematic of a five-stage Clos topology with four devices in the "Tier-1" stage 🦋. Notice that the number of paths between Node1 and Node12 equals four, and those paths have to cross all of the Tier-1 devices 🌈. Meanwhile, the number of paths between Node1 and Node2 equals two, and those paths only cross Tier-2 devices 🧚.
Other topologies are possible, but for simplicity, only the topologies that have a single path from Tier-1 to Tier-3 are considered below. The rest could be treated similarly, with a few modifications to the logic 💅.
2.1. Reference Design 🌸
Tier-1
+-----+
|NODE |
+->| 5 |--+
| +-----+ |
Tier-2 | | Tier-2
+-----+ | +-----+ | +-----+
+------------>|NODE |--+->|NODE |--+--|NODE |-------------+
| +-----| 3 |--+ | 6 | +--| 9 |-----+ |
| | +-----+ +-----+ +-----+ | |
| | | |
| | +-----+ +-----+ +-----+ | |
| +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ |
| | | +---| 4 |--+->| 7 |--+--| 10 |---+ | | |
| | | | +-----+ | +-----+ | +-----+ | | | |
| | | | | | | | | |
+-----+ +-----+ | +-----+ | +-----+ +-----+
|NODE | |NODE | Tier-3 +->|NODE |--+ Tier-3 |NODE | |NODE |
| 1 | | 2 | | 8 | | 11 | | 12 |
+-----+ +-----+ +-----+ +-----+ +-----+
| | | | | | | |
A O B O <- Servers -> Z O O O
Figure 1: 5-Stage Clos TopologyIn the reference topology from Figure 1, here's what we're assuming 🎀:
- Each node is its own autonomous system (AS). Node X has AS X. 4-byte AS numbers are recommended ([RFC6793]) 🌈.
- For simple and efficient route propagation filtering, Node5, Node6, Node7, and Node8 use the same AS; Node3 and Node4 use the same AS; and Node9 and Node10 use the same AS 🧚.
- If 2-byte autonomous system numbers are used for efficient usage of the scarce 2-byte Private Use AS pool, different Tier-3 nodes might use the same AS 💜.
- Without loss of generality, these details will be simplified in this document. It is to be assumed that each node has its own AS 🌸.
- Each node peers with its neighbors with a BGP session. If not specified, external BGP (EBGP) is assumed. In a specific use case, internal BGP (IBGP) will be used, but this will be called out explicitly in that case 💅.
- Each node originates the IPv4 address of its loopback interface into BGP and announces it to its neighbors 🦋.
- The loopback of Node X is
192.0.2.x/32✨.
- The loopback of Node X is
In this document, the Tier-1, Tier-2, and Tier-3 nodes are referred to as "Spine", "Leaf", and "ToR" (top of rack) nodes, respectively 🍓. When a ToR node acts as a gateway to the "outside world", it's called a "border node" 🌍.
3. Some Open Problems in Large Data-Center Networks 😭
The data-center-network design we just summarized gives you ways to move traffic between hosts with reasonable efficiency 👑. But there are a few open performance and reliability problems that pop up in this design 🙈:
- ECMP is most commonly realized per flow. That means large, long-lived "elephant" flows can hurt the performance of smaller, short-lived "mouse" flows, and they can reduce the efficiency of per-flow load sharing 🐘. In other words, per-flow ECMP doesn't perform efficiently when the flow-lifetime distribution is heavy tailed 💀. Plus, because of hash-function inefficiencies, you can get frequent flow collisions where more flows land on one path than on the others 🌪️.
- Shortest-path routing with ECMP is an oblivious routing model. It isn't aware of network imbalances 🙃. If the network symmetry gets broken, for example by link failures, utilization hotspots may appear 🔥. For example, if a link fails between Tier-1 and Tier-2 devices (e.g., Node5 and Node9), Tier-3 devices Node1 and Node2 won't be aware of it, since there are other paths available from the perspective of Node3 🫠. They'll keep sending roughly equal traffic to Node3 and Node4 as if the failure didn't exist, which may cause a traffic hotspot 💥.
- Isolating faults is nontrivial with multiple parallel paths and ECMP-based routing, because the routing lacks determinism 🕵️♀️. Specifically, connections from HostA to HostB may take a different path every time a new connection is formed, which makes consistently reproducing a failure much harder 😩. This complexity scales linearly with the number of parallel paths in the network, and it stems from the random nature of path selection by the network devices 🎲.
4. Applying Segment Routing in the DC with MPLS Data Plane 🌈✨
4.1. BGP Prefix Segment (BGP Prefix-SID) ✨
Okay so a BGP Prefix Segment is a segment tied to a BGP prefix 🦋 It's a network-wide instruction that tells packets to take the ECMP-aware best path to the related prefix. Main character energy, no cap 💅
The BGP Prefix Segment is defined as the BGP Prefix-SID Attribute in [RFC8669], and that attribute carries an index 🧚 For the rest of this doc we just call it the "BGP Prefix-SID", and the encoded index is the label index.
Now the design call 🎀 In this document, we're assuming every node is allocated the same SRGB (Segment Routing Global Block), like [16000, 23999]. This makes operations way simpler (more on that in Section 8 🙈), but it's not a requirement, so your network can do its own thing if it wants.
For illustration, when we're looking at an MPLS data plane, say the label index for prefix 192.0.2.x/32 is X. Then every node throughout the DC fabric allocates a local label of (16000+x) for that prefix 💜
When we're looking at the IPv6 data plane, Node X is allocated the IPv6 address (segment) 2001:DB8::X 🌸
4.2. EBGP Labeled Unicast (RFC 8277)
Referring to Figure 1 and [RFC7938], these design modifications are introduced 🌈
- Each node peers with its neighbors via an EBGP session with the extensions defined in [RFC8277] (we're calling it "EBGP8277" all throughout this doc 💅), and with the BGP Prefix-SID attribute extension as defined in [RFC8669].
- The forwarding plane at Tier-2 and Tier-1 is MPLS.
- The forwarding plane at Tier-3 is either IP2MPLS (if the host sends IP traffic) or MPLS2MPLS (if the host sends MPLS-encapsulated traffic).
Figure 2 zooms into a path from ServerA to ServerZ inside the Figure 1 topology 🦋
+-----+ +-----+ +-----+
+---------->|NODE | |NODE | |NODE |
| | 4 |--+->| 7 |--+--| 10 |---+
| +-----+ +-----+ +-----+ |
| |
+-----+ +-----+
|NODE | |NODE |
| 1 | | 11 |
+-----+ +-----+
| |
A <- Servers -> ZFigure 2: Path from A to Z via Nodes 1, 4, 7, 10, and 11
Referring to Figures 1 and 2, and assuming the IP address with the AS and label-index allocation we just went over, the next sections dig into the control-plane operation and the data-plane states for the prefix 192.0.2.11/32 (the loopback of Node11) 👑
4.2.1. Control Plane
Node11 originates 192.0.2.11/32 in BGP and allocates it a BGP Prefix-SID with label-index: index11 [RFC8669].
Node11 sends this EBGP8277 update to Node10:
IP Prefix: 192.0.2.11/32
Label: Implicit NULL
Next hop: Node11's interface address on the link to Node10
AS Path: {11}
BGP Prefix-SID: Label-Index 11Node10 receives the update above. Since it's SR capable, Node10 can interpret the BGP Prefix-SID, so it knows to allocate the label from its own SRGB, offset by the label index it got (16000+11, hence 16011), for the NLRI. That's way better than allocating a nondeterministic label out of a dynamically allocated part of the local label space 🙌 The implicit NULL label in the NLRI tells Node10 that it's the penultimate hop and that it must pop the top label on the stack before forwarding traffic for this prefix to Node11.
Then Node10 sends this EBGP8277 update to Node7:
IP Prefix: 192.0.2.11/32
Label: 16011
Next hop: Node10's interface address on the link to Node7
AS Path: {10, 11}
BGP Prefix-SID: Label-Index 11Node7 receives the update above. Since it's SR capable, Node7 can interpret the BGP Prefix-SID, so it allocates the local (incoming) label 16011 (16000 + 11) to the NLRI (instead of pulling a "dynamic" local label from its label manager). Node7 uses the label in the received EBGP8277 NLRI as the outgoing label. The index is only used to derive the local/incoming label 🧠
Node7 sends this EBGP8277 update to Node4:
IP Prefix: 192.0.2.11/32
Label: 16011
Next hop: Node7's interface address on the link to Node4
AS Path: {7, 10, 11}
BGP Prefix-SID: Label-Index 11Node4 receives the update above. Since it's SR capable, Node4 can interpret the BGP Prefix-SID, so it allocates the local (incoming) label 16011 to the NLRI (instead of a "dynamic" local label from its label manager). Node4 uses the label in the received EBGP8277 NLRI as an outgoing label. Again, the index is only used to derive the local/incoming label ✨
Node4 sends this EBGP8277 update to Node1:
IP Prefix: 192.0.2.11/32
Label: 16011
Next hop: Node4's interface address on the link to Node1
AS Path: {4, 7, 10, 11}
BGP Prefix-SID: Label-Index 11Node1 receives the update above. Since it's SR capable, Node1 can interpret the BGP Prefix-SID, so it allocates the local (incoming) label 16011 to the NLRI (instead of a "dynamic" local label from its label manager). Node1 uses the label in the received EBGP8277 NLRI as an outgoing label. The index is only used to derive the local/incoming label 💖
4.2.2. Data Plane
Referring to Figure 1, and assuming all nodes apply the same advertisement rules described above and all nodes have the same SRGB (16000-23999), here are the IP/MPLS forwarding tables for prefix 192.0.2.11/32 at Node1, Node4, Node7, and Node10 📊
+----------------------------------+----------------+------------+
| Incoming Label or IP Destination | Outgoing Label | Outgoing |
| | | Interface |
+----------------------------------+----------------+------------+
| 16011 | 16011 | ECMP{3, 4} |
+----------------------------------+----------------+------------+
| 192.0.2.11/32 | 16011 | ECMP{3, 4} |
+----------------------------------+----------------+------------+Table 1: Node1 Forwarding Table
+----------------------------------+----------------+------------+
| Incoming Label or IP Destination | Outgoing Label | Outgoing |
| | | Interface |
+----------------------------------+----------------+------------+
| 16011 | 16011 | ECMP{7, 8} |
+----------------------------------+----------------+------------+
| 192.0.2.11/32 | 16011 | ECMP{7, 8} |
+----------------------------------+----------------+------------+Table 2: Node4 Forwarding Table
+----------------------------------+----------------+-----------+
| Incoming Label or IP Destination | Outgoing Label | Outgoing |
| | | Interface |
+----------------------------------+----------------+-----------+
| 16011 | 16011 | 10 |
+----------------------------------+----------------+-----------+
| 192.0.2.11/32 | 16011 | 10 |
+----------------------------------+----------------+-----------+Table 3: Node7 Forwarding Table
+----------------------------------+----------------+-----------+
| Incoming Label or IP Destination | Outgoing Label | Outgoing |
| | | Interface |
+----------------------------------+----------------+-----------+
| 16011 | POP | 11 |
+----------------------------------+----------------+-----------+
| 192.0.2.11/32 | N/A | 11 |
+----------------------------------+----------------+-----------+Table 4: Node10 Forwarding Table
4.2.3. Network Design Variation
Here's a design choice you could make 🎀: switch all the traffic through Tier-1 and Tier-2 as MPLS traffic. In that case, you could filter away the IP entries at Node4, Node7, and Node10. This might be beneficial for optimizing the forwarding table size 🌸
Another design choice is letting the hosts send MPLS-encapsulated traffic, based on the Egress Peer Engineering (EPE) use case defined in [SR-CENTRAL-EPE]. For example, applications at HostA would send their Z-destined traffic to Node1 with an MPLS label stack where the top label is 16011 and the next label is an EPE peer segment ([SR-CENTRAL-EPE]) at Node11 directing the traffic to Z 🦋
4.2.4. Global BGP Prefix Segment through the Fabric
When the previous design is deployed, the operator gets global BGP Prefix-SID and label allocation throughout the DC fabric 😭🫶
A few examples follow:
- Normal forwarding to Node11: A packet with top label 16011 received by any node in the fabric will be forwarded along the ECMP-aware BGP best path towards Node11, and the label 16011 is penultimate popped at Node10 (or at Node 9).
- Traffic-engineered path to Node11: An application on a host behind Node1 might want to restrict its traffic to paths via the Spine node Node5. The application achieves this by sending its packets with a label stack of
{16005, 16011}. BGP Prefix-SID 16005 directs the packet up to Node5 along the path (Node1, Node3, Node5). BGP Prefix-SID 16011 then directs the packet down to Node11 along the path (Node5, Node9, Node11) 👑
4.2.5. Incremental Deployments 🚀
okay so the design we just went through can be rolled out bit by bit, which is so important for real networks 💅 let's pretend Node7 doesn't speak the BGP Prefix-SID yet and see if the fabric connectivity survives 🫶
signaling-wise, nothing changes at all ✨ even though Node7 doesn't support the BGP Prefix-SID, it still passes the attribute along unmodified to its neighbors 🦋
label allocation is the only part that's different 🎀 Node7 just hands out a dynamic (random) label for 192.0.2.11/32 (e.g., 123456) instead of using the "hinted" label the BGP Prefix-SID would've asked for. Its neighbors adapt automatically, because they always use the label from the BGP8277 NLRI as their outgoing label 💜
Node4 does understand the BGP Prefix-SID, so it allocates the indexed label in the SRGB (16011) for 192.0.2.11/32 🌸
the result? every data-plane entry across the network stays the same, except the ones at Node7 and its neighbor Node4, shown in the tables below 💖
the big takeaway is that the end-to-end Label Switched Path (LSP) is preserved 😭 the outgoing label always comes from the label received in the BGP8277 NLRI. The index in the BGP Prefix-SID is only a hint for picking the local label (the incoming label), and it is never used for the outgoing label 🧚
+----------------------------------+----------------+-----------+
| Incoming Label or IP Destination | Outgoing Label | Outgoing |
| | | Interface |
+----------------------------------+----------------+-----------+
| 12345 | 16011 | 10 |
+----------------------------------+----------------+-----------+Table 5: Node7 Forwarding Table
+----------------------------------+----------------+-----------+
| Incoming Label or IP Destination | Outgoing Label | Outgoing |
| | | Interface |
+----------------------------------+----------------+-----------+
| 16011 | 12345 | 7 |
+----------------------------------+----------------+-----------+Table 6: Node4 Forwarding Table
so yes, the BGP Prefix-SID can be deployed incrementally, one node at a time 🎉
and when you run the same SRGB across the whole fabric (homogeneous SRGB), the operator gets the global prefix segment benefits more and more as the rollout moves through the network 👑
4.3. IBGP Labeled Unicast (RFC 8277) 🌈
this is the exact same design as EBGP8277, with a few tweaks 🧁
- All nodes use the same AS number.
- Each node peers with its neighbors via an internal BGP session (IBGP) with extensions defined in [RFC8277] (named "IBGP8277" throughout this document).
- Each node acts as a route reflector for each of its neighbors and uses the next-hop-self option. Next-hop-self is a well-known operational feature that rewrites the next hop of a BGP update before sending it to the neighbor. Normally, people apply next-hop-self toward IBGP peers for EBGP-learned routes. In this design, next-hop-self is also applied to IBGP-learned routes 💅
Cluster-1
+-----------+
| Tier-1 |
| +-----+ |
| |NODE | |
| | 5 | |
Cluster-2 | +-----+ | Cluster-3
+---------+ | | +---------+
| Tier-2 | | | | Tier-2 |
| +-----+ | | +-----+ | | +-----+ |
| |NODE | | | |NODE | | | |NODE | |
| | 3 | | | | 6 | | | | 9 | |
| +-----+ | | +-----+ | | +-----+ |
| | | | | |
| | | | | |
| +-----+ | | +-----+ | | +-----+ |
| |NODE | | | |NODE | | | |NODE | |
| | 4 | | | | 7 | | | | 10 | |
| +-----+ | | +-----+ | | +-----+ |
+---------+ | | +---------+
| |
| +-----+ |
| |NODE | |
Tier-3 | | 8 | | Tier-3
+-----+ +-----+ | +-----+ | +-----+ +-----+
|NODE | |NODE | +-----------+ |NODE | |NODE |
| 1 | | 2 | | 11 | | 12 |
+-----+ +-----+ +-----+ +-----+Figure 3: IBGP Sessions with Reflection and Next-Hop-Self
- For simple and efficient route propagation filtering, as shown in Figure 3:
- Node5, Node6, Node7, and Node8 use the same Cluster ID (Cluster-1).
- Node3 and Node4 use the same Cluster ID (Cluster-2).
- Node9 and Node10 use the same Cluster ID (Cluster-3).
- The control-plane behavior is mostly the same as in the previous section. The only difference is that the EBGP8277 path propagation gets swapped out for IBGP8277 path reflection, with the next hop changed to self ✨
- The data-plane tables are exactly the same 🫶
5. Applying Segment Routing in the DC with IPv6 Data Plane 💜
the design in [RFC7938] gets reused with just one modification 🔥 we'll look at it using reachability to Node11 through Spine node Node5 🦋
Node5 originates 2001:DB8::5/128 with the attached BGP Prefix-SID for IPv6 packets destined to segment 2001:DB8::5 ([RFC8402]) 🌸
Node11 originates 2001:DB8::11/128 with the attached BGP Prefix-SID, advertising support for the Segment Routing Header (SRH) for IPv6 packets destined to segment 2001:DB8::11 🎀
the control-plane and data-plane processing of every other node in the fabric stays unchanged 💅 specifically, the routes to 2001:DB8::5 and 2001:DB8::11 get installed in the FIB along the EBGP best path to Node5 (Spine node) and Node11 (ToR node) respectively ✨
say an application on HostA needs to send traffic to HostZ through only Node5 (Spine node) 👑 it can do that by sending IPv6 packets with a Segment Routing Header (SRH, [IPv6-SRH]). The destination address and active segment are set to 2001:DB8::5, and the next and last segment is set to 2001:DB8::11 🧚
the application must only use IPv6 addresses that have been advertised as capable of SRv6 segment processing (e.g., ones for which the BGP Prefix Segment capability has been advertised). How applications learn this (e.g., through a centralized controller and orchestration) is outside the scope of this document 🙈
6. Communicating Path Information to the Host 📬
there are two general ways to tell end-hosts about path information: "proactive" and "reactive," aka "push" and "pull" models 💖 each of these has multiple implementations. One option is a centralized controller, which either tells the hosts the prefix-to-path mappings ahead of time and updates them as needed (network event driven push), or answers host requests for a path to a specific destination (host event driven pull) 🫶 It's also possible to go hybrid: push some state from the controller in response to particular network events, while the host pulls other state on demand 🌈
also, when you share network data with end-hosts, there's a trade-off between how much information you send and how much visibility into the network state the hosts get 🍓 This applies to both push and pull models. In the extreme case, the host requests path information for every flow and keeps no local state at all 💀 On the other end of the spectrum, information for every prefix in the network, along with the available paths, could be pushed and continuously updated on all hosts 😭
7. Additional Benefits 🎀
7.1. MPLS Data Plane with Operational Simplicity ✨
as required by [RFC7938], no new signaling protocol gets introduced 🦋 The BGP Prefix-SID is a lightweight extension to BGP Labeled Unicast [RFC8277], and it applies to both EBGP- and IBGP-based designs 💜
specifically, LDP and RSVP-TE are not used 🙅♀️ These protocols would seriously complicate the operations of the data center and wouldn't scale. This lines up with the requirements in [RFC7938] 🫶
provided the same SRGB is configured on all nodes, all nodes use the same MPLS label for a given IP prefix 🌸 That's simpler operationally, as discussed in Section 8 ✨
7.2. Minimizing the FIB Table 🧚
the designer may decide to switch all the traffic at Tier-1 and Tier-2 over to MPLS, which drastically shrinks the IP table size at those nodes 👑
this is easy to do by encapsulating the traffic either directly at the host or at the source ToR node 💅 The encapsulation pushes the BGP Prefix-SID of the destination ToR for intra-DC traffic, or pushes the BGP Prefix-SID of the border node for inter-DC or DC-to-outside-world traffic 🔥
7.3. Egress Peer Engineering 🌈
it's totally straightforward to combine this design with the Egress Peer Engineering (EPE) use case described in [SR-CENTRAL-EPE] 🦋
in that case, the operator can engineer its outbound traffic on a per-host-flow basis, without adding any extra state at intermediate points in the DC fabric 💖
for example, the controller only needs to inject per-flow state on HostA to force its traffic destined to a specific Internet destination D out through a chosen border node (say Node12 in Figure 1, instead of another border node, Node11) and a specific egress peer of Node12 (say peer AS 9999, via local PeerNode segment 9999 at Node12, instead of any other peer that provides a path to destination D) 🎀 Any packet matching this state at HostA gets encapsulated with the SR segment list (label stack) {16012, 9999}. 16012 steers the flow through the DC fabric, using any ECMP along the best path to border node Node12 🧚 Once the flow reaches border node Node12, the active segment is 9999 (because of Penultimate Hop Popping (PHP) on the upstream neighbor of Node12). This EPE PeerNode segment forces border node Node12 to forward the packet to peer AS 9999 without any IP lookup at the border node 💅 There's no per-flow state for this engineered flow in the DC fabric. A benefit of SR is that the per-flow state is only required at the source 🫶
on top of full traffic-engineering control, this design also saves FIB table space 🍓 The Internet-scale FIB at border node Node12 isn't needed if all FIB lookups are avoided there by using EPE 👑
7.4. Anycast 📡
bestie, this part is so pretty ✨ The design here keeps the availability and load-balancing perks of the base design from [RFC8402]. It's giving reliable, no drama 💅
Say you give an anycast loopback 192.0.2.20/32 and attach segment index 20 to it on the border nodes Node11 and Node12, on top of their node-specific loopbacks. Now the EPE controller can say "go to the Internet via any border node" as just the segment list {16020} 🌐 From any host in the DC fabric or from any ToR node, 16020 steers the packet toward Node11 or Node12, and ECMP gets used wherever it's available along the best paths to them. Iconic. 🦋
8. Preferred SRGB Allocation 🌸
okay so the MPLS case recommends using the same SRGB on every node. That's the move 💖
Different SRGBs per node make things way messier, both for the operators and for the controller. 🙈
Operationally, it's so much simpler when the same global label means the same destination everywhere. Troubleshooting MPLS then feels like troubleshooting IPv6, where that global property is just a given 🧚
For the controller, one shared SRGB means you can write simple policies that work across the whole fabric 👑
Let's look at an example. App A is hooked up to Node1 and app B is hooked up to Node2, both ToR nodes. A has two flows, FA1 and FA2, going to Z. B also has two flows, FB1 and FB2, going to Z. The controller wants FA1 and FB1 load shared across the fabric, while FA2 and FB2 have to be steered through Node5 and Node8 respectively. 🎀
If the SRGB is consistent and unique across the fabric, as this document describes, the controller just tells A and B to use {16011} for FA1 and FB1, and to use {16005 16011} and {16008 16011} for FA2 and FB2. Easy peasy 🍓
Now imagine each node has a different SRGB, where the SRGB of Node K starts at K*1000 and has a length of 1000. So Node1's SRGB is [1000, 1999], Node2's SRGB is [2000, 2999], and so on. Each node's SRGB is advertised using the Originator SRGB TLV of the BGP Prefix-SID, as defined in [RFC8669]. 📣
In that world, the controller has to collect and store every one of those SRGBs, for example through the Originator SRGB TLV of the BGP Prefix-SID. It also has to adapt the policy for each host 😭 It would tell A to use {1011} for FA1, but B would need {2011} for FB1. With one shared SRGB, both of those are just {16011}.
It gets even worse. The controller would tell A to use {1005, 5011} for FA1, and B to use {2011, 8011} for FB1. With one shared SRGB, the second segment is 16011 in both policies. When you combine segments into a policy, you have to carefully update the label of every single segment. That's way more error prone, more complex, and harder to troubleshoot. Not me being dramatic, it's just true 💀
9. IANA Considerations 🌈
This document has no IANA actions. Nothing to see here for the registry gremlins 🧚
10. Manageability Considerations 💜
The design and deployment guidelines in this document are based on the network design described in [RFC7938].
The deployment model assumed here is a single domain, where the interconnected DCs are part of the same administrative domain. That domain is split into different autonomous systems, but the operator has full control of the whole thing. The usual operational and management mechanisms and procedures are used to stop any information about internal prefixes and topology from leaking outside the domain 🔒
As recommended in [RFC8402], the same SRGB should be allocated on all nodes, so the design, deployment, and operations of the domain stay simple 🫶
When EPE ([SR-CENTRAL-EPE]) is used, as explained in Section 7.3, the same operational model is assumed. EPE information is originated and propagated throughout the domain toward an internal server. Unless the operator explicitly configures it, no EPE information leaks outside the domain boundaries 🛡️
11. Security Considerations 🔥
okay this section is serious, so listen up bestie 😤
This document proposes applying SR to a well-known scalability requirement expressed in [RFC7938], using the BGP Prefix-SID as defined in [RFC8669].
As noted in Section 10, the design illustrated in [RFC7938] and in this document refers to a deployment model where all nodes are under the same administration. In this context, the operator is assumed not to want any information about internal prefixes and topology leaking outside the domain. That internal information includes Prefix-SID and EPE information. To prevent such leaks, the standard BGP mechanisms (filters) are applied on the boundary of the domain 🚧
Heads up: the security and confidentiality of prefix and topology information depends on outbound filters at each peering point of the domain, as described in Section 10. If those filters slip, the internal info can leak. Guard them like your life depends on it 💀
So the solution in this document doesn't add any security concerns beyond what [RFC7938] and [RFC8669] already cover. It's assumed that outbound filters at each peering point keep the prefix and topology information secure and confidential, as Section 10 describes 🔐
12. References 📚
12.1. Normative References 💖
- [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI 10.17487/RFC4271, January 2006,
https://www.rfc-editor.org/info/rfc4271. - [RFC7938] Lapukhov, P., Premji, A., and J. Mitchell, Ed., "Use of BGP for Routing in Large-Scale Data Centers", RFC 7938, DOI 10.17487/RFC7938, August 2016,
https://www.rfc-editor.org/info/rfc7938. - [RFC8277] Rosen, E., "Using BGP to Bind MPLS Labels to Address Prefixes", RFC 8277, DOI 10.17487/RFC8277, October 2017,
https://www.rfc-editor.org/info/rfc8277. - [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, July 2018,
https://www.rfc-editor.org/info/rfc8402. - [RFC8669] Previdi, S., Filsfils, C., Lindem, A., Ed., Sreekantiah, A., and H. Gredler, "Segment Routing Prefix Segment Identifier Extensions for BGP", RFC 8669, DOI 10.17487/RFC8669, December 2019,
https://www.rfc-editor.org/info/rfc8669.
12.2. Informative References 🦋
- [IPv6-SRH] Filsfils, C., Dukes, D., Previdi, S., Leddy, J., Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header (SRH)", Work in Progress, Internet-Draft, draft-ietf-6man-segment-routing-header-26, 22 October 2019,
https://tools.ietf.org/html/draft-ietf-6man-segment-routing-header-26. - [RFC6793] Vohra, Q. and E. Chen, "BGP Support for Four-Octet Autonomous System (AS) Number Space", RFC 6793, DOI 10.17487/RFC6793, December 2012,
https://www.rfc-editor.org/info/rfc6793. - [SR-CENTRAL-EPE] Filsfils, C., Previdi, S., Dawra, G., Aries, E., and D. Afanasiev, "Segment Routing Centralized BGP Egress Peer Engineering", Work in Progress, Internet-Draft, draft-ietf-spring-segment-routing-central-epe-10, 21 December 2017,
https://tools.ietf.org/html/draft-ietf-spring-segment-routing-central-epe-10.
Acknowledgements 🫶
The authors want to thank Benjamin Black, Arjun Sreekantiah, Keyur Patel, Acee Lindem, and Anoop Ghanwani for their comments and review of this document. Huge thanks, you're all amazing 💖
Contributors 💜
Gaya Nagarajan Facebook United States of America Email: gaya@fb.com
Gaurav Dawra Cisco Systems United States of America Email: gdawra.ietf@gmail.com
Dmitry Afanasiev Yandex Russian Federation Email: fl0w@yandex-team.ru
Tim Laberge Cisco United States of America Email: tlaberge@cisco.com
Edet Nkposong Salesforce.com Inc. United States of America Email: enkposong@salesforce.com
Mohan Nanduri Microsoft United States of America Email: mohan.nanduri@oracle.com
James Uttaro ATT United States of America Email: ju1738@att.com
Saikat Ray Unaffiliated United States of America Email: raysaikat@gmail.com
Jon Mitchell Unaffiliated United States of America Email: jrmitche@puck.nether.net
Authors' Addresses 🦋
Clarence Filsfils (editor) Cisco Systems, Inc. Brussels Belgium Email: cfilsfil@cisco.com
Stefano Previdi Cisco Systems, Inc. Italy Email: stefano@previdi.net
Gaurav Dawra LinkedIn United States of America Email: gdawra.ietf@gmail.com
Ebben Aries Arrcus, Inc. 2077 Gateway Place, Suite #400 San Jose, CA 95119 United States of America Email: exa@arrcus.com
Petr Lapukhov Facebook United States of America Email: petr@fb.com
That's the whole RFC, bestie. We made it 🌈✨💅
📦 where this page comes from
- ietf: rfc8670 2019-12
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms