RFC 7938

Use of BGP for Routing in Large-Scale Data Centers

📅 published August 2016, from ietf

Okay bestie, I'm literally obsessed with this one 🦋 Part 1 of RFC 7938 coming in hot, so buckle up 🎀

Quick heads up: I can't keep the original document's exact wording everywhere in every single line, so I'll stick to the facts, numbers, section titles and requirement keywords while giving it my own bubbly spin ✨

RFC 7938 Title Block 💖

Okay so, the title is "Use of BGP for Routing in Large-Scale Data Centers" 🌸 It's an Informational RFC (so it's sharing knowledge, not a standard you have to follow 🫶), published in August 2016 by the IETF, and it's labelled ISSN 2070-1721. The main authors are P. Lapukhov from Facebook, A. Premji from Arista Networks, and J. Mitchell (the editor) 💅 It doesn't obsolete or update any other RFCs, which is kind of iconic because it's standing on its own ✨

Abstract 🧚

Some network operators build data centers that support over one hundred thousand servers 😭 The RFC calls these "large-scale" data centers so they can be told apart from smaller setups. At that scale you need a totally different set of network requirements, with a big focus on operational simplicity and network stability 🔥 This doc sums up what people have learned from designing and running these huge data centers using BGP as the only routing protocol 💜 The whole goal is to share a proven, stable routing design that other folks in the industry can actually use 🎀

Status of This Memo 🦋

This document is not an Internet Standards Track specification 💀 It's published just for information.

It's a product of the Internet Engineering Task Force (IETF) and reflects the consensus of the IETF community 🌈 It got public review and was approved for publication by the Internet Engineering Steering Group (IESG). Not every IESG-approved doc is a candidate for any level of Internet Standard, so check Section 2 of RFC 7841 if you want the details 🙈

Info about the current status, any errata, and how to send feedback is at http://www.rfc-editor.org/info/rfc7938 🍓

Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved ✨

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stood on the publication date. Please read those carefully, because they spell out your rights and restrictions. Any Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and come without warranty as described in that license 🫶

Table of Contents 🌸

1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
2.  Network Design Requirements . . . . . . . . . . . . . . . . .   4
  2.1.  Bandwidth and Traffic Patterns  . . . . . . . . . . . . .   4
  2.2.  CAPEX Minimization  . . . . . . . . . . . . . . . . . . .   4
  2.3.  OPEX Minimization . . . . . . . . . . . . . . . . . . . .   5
  2.4.  Traffic Engineering . . . . . . . . . . . . . . . . . . .   5
  2.5.  Summarized Requirements . . . . . . . . . . . . . . . . .   6
3.  Data Center Topologies Overview . . . . . . . . . . . . . . .   6
  3.1.  Traditional DC Topology . . . . . . . . . . . . . . . . .   6
  3.2.  Clos Network Topology . . . . . . . . . . . . . . . . . .   7
    3.2.1.  Overview  . . . . . . . . . . . . . . . . . . . . . .   7
    3.2.2.  Clos Topology Properties  . . . . . . . . . . . . . .   8
    3.2.3.  Scaling the Clos Topology . . . . . . . . . . . . . .   9
    3.2.4.  Managing the Size of Clos Topology Tiers  . . . . . .  10
4.  Data Center Routing Overview  . . . . . . . . . . . . . . . .  11
  4.1.  L2-Only Designs . . . . . . . . . . . . . . . . . . . . .  11
  4.2.  Hybrid L2/L3 Designs  . . . . . . . . . . . . . . . . . .  12
  4.3.  L3-Only Designs . . . . . . . . . . . . . . . . . . . . .  12
5.  Routing Protocol Design . . . . . . . . . . . . . . . . . . .  13
  5.1.  Choosing EBGP as the Routing Protocol . . . . . . . . . .  13
  5.2.  EBGP Configuration for Clos Topology  . . . . . . . . . .  15
    5.2.1.  EBGP Configuration Guidelines and Example ASN Scheme   15
    5.2.2.  Private Use ASNs  . . . . . . . . . . . . . . . . . .  16
    5.2.3.  Prefix Advertisement  . . . . . . . . . . . . . . . .  17
    5.2.4.  External Connectivity . . . . . . . . . . . . . . . .  18
    5.2.5.  Route Summarization at the Edge . . . . . . . . . . .  19
6.  ECMP Considerations . . . . . . . . . . . . . . . . . . . . .  20
  6.1.  Basic ECMP  . . . . . . . . . . . . . . . . . . . . . . .  20
  6.2.  BGP ECMP over Multiple ASNs . . . . . . . . . . . . . . .  21
  6.3.  Weighted ECMP . . . . . . . . . . . . . . . . . . . . . .  21
  6.4.  Consistent Hashing  . . . . . . . . . . . . . . . . . . .  22
7.  Routing Convergence Properties  . . . . . . . . . . . . . . .  22
  7.1.  Fault Detection Timing  . . . . . . . . . . . . . . . . .  22
  7.2.  Event Propagation Timing  . . . . . . . . . . . . . . . .  23
  7.3.  Impact of Clos Topology Fan-Outs  . . . . . . . . . . . .  24
  7.4.  Failure Impact Scope  . . . . . . . . . . . . . . . . . .  24
  7.5.  Routing Micro-Loops . . . . . . . . . . . . . . . . . . .  26
8.  Additional Options for Design . . . . . . . . . . . . . . . .  26
  8.1.  Third-Party Route Injection . . . . . . . . . . . . . . .  26
  8.2.  Route Summarization within Clos Topology  . . . . . . . .  27
    8.2.1.  Collapsing Tier 1 Devices Layer . . . . . . . . . . .  27
    8.2.2.  Simple Virtual Aggregation  . . . . . . . . . . . . .  29
  8.3.  ICMP Unreachable Message Masquerading . . . . . . . . . .  29
9.  Security Considerations . . . . . . . . . . . . . . . . . . .  30
10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  30
  10.1.  Normative References . . . . . . . . . . . . . . . . . .  30
  10.2.  Informative References . . . . . . . . . . . . . . . . .  31
Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .  35
Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  35

1. Introduction ✨

1.1. Hmm, what's this doc about? 🎀

Okay so this doc describes a practical routing design for a large-scale data center (DC) 💖 These are also called "hyper-scale" or "warehouse-scale" data centers, and the defining thing is that they support over a hundred thousand servers 😭 To handle networks that big, operators keep revisiting their network designs and platforms 🔥

The design here is based on real operational experience with data centers built for huge distributed software, like a web search engine 🦋 What matters most in that world is operational simplicity and network stability, so that a small team can actually run a pretty big network 🫶

Testing has shown that External BGP (EBGP) [RFC4271] is a great stand-alone routing protocol for these data centers 💜 That's different from traditional DC designs, which tend to use simple tree topologies and stretch Layer 2 (L2) domains across multiple devices 🌸 The doc walks through the requirements that led to this choice, the EBGP design itself, and some ideas for future upgrades ✨

Here's the flow: first, an overview of the network design requirements. Then the traditional hierarchical topologies get compared with horizontally scaled Clos networks [CLOS1953] 🧚 Next comes why EBGP with a Clos topology is the best fit, and then the design gets described in detail 💅 Last, some extra considerations and design options 👑 Heads up: the reader is expected to already know BGP well before deploying this design 🙈

1.2. Okay, but what does that mean? 🍓

Nothing is changed in the meaning here. The only difference is that it's written in a way that's easier to chew on 💖

Part 1 of 8 ends here, bestie. Stay tuned for the next part 🦋💅

2.4. Traffic Engineering ✨

Okay so, in any data center, application load balancing is literally a critical job that network devices do 💅. Traditionally, load balancers are dedicated boxes sitting right in the traffic forwarding path 📦. The tricky part is scaling them when traffic demand keeps climbing 📈. A much better vibe would be scaling the load-balancing layer horizontally, by adding more of the uniform nodes and spreading incoming traffic across them 🦋. In a situation like this, the ideal move is using the network infrastructure itself to spread traffic across a group of load balancers 🫶. The combo of anycast prefix advertisement [RFC4786] and Equal Cost Multipath (ECMP) functionality can get this done 🔥. To get more granular load distribution, it's beneficial for the network to support controlled per-hop traffic engineering 🎀. For example, it's beneficial to directly control the ECMP next-hop set for anycast prefixes at every level of the network hierarchy 👑.

2.5. Summarized Requirements ✨

Recap time 💖. This section sums up the requirements from the previous sections:

  • REQ1: Select a topology that can be scaled "horizontally" by adding more links and network devices of the same type without requiring upgrades to the network elements themselves.
  • REQ2: Define a narrow set of software features/protocols supported by a multitude of networking equipment vendors.
  • REQ3: Choose a routing protocol that has a simple implementation in terms of programming code complexity and ease of operational support.
  • REQ4: Minimize the failure domain of equipment or protocol issues as much as possible.
  • REQ5: Allow for some traffic engineering, preferably via explicit control of the routing prefix next hop using built-in protocol mechanics.

3. Data Center Topologies Overview ✨

This section gives you the tea on two general types of data center designs: hierarchical (also known as "tree-based") and Clos-based network designs 🌳.

3.1. Traditional DC Topology ✨

In the networking industry, a super common data center design looks like an upside-down tree 🌳 with redundant uplinks and three layers of hierarchy: core, aggregation/distribution, and access layers (see Figure 1). To handle bandwidth demands, each higher layer, going from the server toward DC egress or WAN, has higher port density and bandwidth capacity, and the core acts as the "trunk" of the tree-based design. To keep terminology uniform and easy to compare with other designs, this document calls these layers Tier 1, Tier 2 and Tier 3 "tiers" instead of core, aggregation, or access layers 🧚.

          +------+  +------+
          |      |  |      |
          |      |--|      |           Tier 1
          |      |  |      |
          +------+  +------+
            |  |      |  |
  +---------+  |      |  +----------+
  | +-------+--+------+--+-------+  |
  | |       |  |      |  |       |  |
+----+     +----+    +----+     +----+
|    |     |    |    |    |     |    |
|    |-----|    |    |    |-----|    | Tier 2
|    |     |    |    |    |     |    |
+----+     +----+    +----+     +----+
   |         |          |         |
   |         |          |         |
   | +-----+ |          | +-----+ |
   +-|     |-+          +-|     |-+    Tier 3
     +-----+              +-----+
      | | |                | | |
  <- Servers ->        <- Servers ->

                Figure 1: Typical DC Network Topology

Unfortunately, as noted previously, a tree-based design can't scale far enough for large-scale designs 😭. You just can't acquire Tier 1 devices with a big enough port density to sufficiently scale Tier 2. Also, the upper-tier devices need continuous upgrades or replacement as deployment size or bandwidth requirements grow, which is operationally complex 💀. For this reason, REQ1 is in place, which eliminates this type of design from consideration 🙅.

3.2. Clos Network Topology ✨

This section describes a common design for horizontally scalable topology in large-scale data centers, built to meet REQ1 🦋.

3.2.1. Overview ✨

A super common pick for a horizontally scalable topology is a folded Clos topology, sometimes called "fat-tree" (for example, [INTERCON] and [ALFARES2008]). This topology has an odd number of stages (sometimes known as "dimensions") and is usually made of uniform elements, e.g., network switches with the same port count. So choosing folded Clos satisfies REQ1 and facilitates REQ2 💖. Check out Figure 2 below for an example of a folded 3-stage Clos topology (3 stages counting the Tier 2 stage twice when tracing a packet flow) 🎀:

+-------+
|       |----------------------------+
|       |------------------+         |
|       |--------+         |         |
+-------+        |         |         |
+-------+        |         |         |
|       |--------+---------+-------+ |
|       |--------+-------+ |       | |
|       |------+ |       | |       | |
+-------+      | |       | |       | |
+-------+      | |       | |       | |
|       |------+-+-------+-+-----+ | |
|       |------+-+-----+ | |     | | |
|       |----+ | |     | | |     | | |
+-------+    | | |     | | |   ---------> M links
 Tier 1      | | |     | | |     | | |
          +-------+ +-------+ +-------+
          |       | |       | |       |
          |       | |       | |       | Tier 2
          |       | |       | |       |
          +-------+ +-------+ +-------+
            | | |     | | |     | | |
            | | |     | | |   ---------> N Links
            | | |     | | |     | | |
            O O O     O O O     O O O   Servers

              Figure 2: 3-Stage Folded Clos Topology

This topology is often also called a "Leaf and Spine" network, where "Spine" is the name for the middle stage of the Clos topology (Tier 1) and "Leaf" is the input/output stage (Tier 2). For uniformity, this document refers to these layers using the "Tier n" notation 🧚.

3.2.2. Clos Topology Properties ✨

Here are some key properties of the Clos topology 👑:

  • The topology is fully non-blocking, or more accurately non-interfering, if M >= N and oversubscribed by a factor of N/M otherwise. Here M and N is the uplink and downlink port count respectively, for a Tier 2 switch as shown in Figure 2.
  • Utilizing this topology requires control and data-plane support for ECMP with a fan-out of M or more.
  • Tier 1 switches have exactly one path to every server in this topology. This is an important property that makes route summarization dangerous in this topology (see Section 8.2 below).
  • Traffic flowing from server to server is load balanced over all available paths using ECMP.

3.2.3. Scaling the Clos Topology ✨

A Clos topology can be scaled either by increasing network element port density or by adding more stages, e.g., moving to a 5-stage Clos, as shown in Figure 3 below 📈:

                                    Tier 1
                                   +-----+
        Cluster                    |     |
+----------------------------+   +--|     |--+
|                            |   |  +-----+  |
|                    Tier 2  |   |           |   Tier 2
|                   +-----+  |   |  +-----+  |  +-----+
|     +-------------| DEV |------+--|     |--+--|     |-------------+
|     |       +-----|  C  |------+  |     |  +--|     |-----+       |
|     |       |     +-----+  |      +-----+     +-----+     |       |
|     |       |              |                              |       |
|     |       |     +-----+  |      +-----+     +-----+     |       |
|     | +-----------| DEV |------+  |     |  +--|     |-----------+ |
|     | |     | +---|  D  |------+--|     |--+--|     |---+ |     | |
|     | |     | |   +-----+  |   |  +-----+  |  +-----+   | |     | |
|     | |     | |            |   |           |            | |     | |
|   +-----+ +-----+          |   |  +-----+  |          +-----+ +-----+
|   | DEV | | DEV |          |   +--|     |--+          |     | |     |
|   |  A  | |  B  | Tier 3   |      |     |      Tier 3 |     | |     |
|   +-----+ +-----+          |      +-----+             +-----+ +-----+
|     | |     | |            |                            | |     | |
|     O O     O O            |                            O O     O O
|       Servers              |                              Servers
+----------------------------+

                    Figure 3: 5-Stage Clos Topology

The small example topology in Figure 3 is built from devices with a port count of 4. In this document, one set of directly connected Tier 2 and Tier 3 devices along with their attached servers is called a "cluster". For example, DEV A, B, C, D, and the servers that connect to DEV A and B, in Figure 3, form a cluster. A cluster can also be a handy single deployment or maintenance unit, one that can be operated on at a different frequency than the entire topology 🫶.

In practice, Tier 3 of the network, which is typically Top-of-Rack switches (ToRs), is where oversubscription gets introduced. That lets you pack more servers into the data center while still meeting the bandwidth requirements for different types of applications 🍓. The main reason to limit oversubscription at a single layer is to keep application development simple, since otherwise apps would need to account for multiple bandwidth pools: within rack (Tier 3), between racks (Tier 2), and between clusters (Tier 1) 😩. Since oversubscription doesn't have a direct relationship to the routing design, it isn't discussed further in this document ✨.

3.2.4. Managing the Size of Clos Topology Tiers 🎀

Okay, so if a data center is small, you can actually cut the number of switches in Tier 1 or Tier 2 of a Clos topology by a factor of two 💅. Let's use Tier 1 as the example. Every Tier 2 device connects to one single group of Tier 1 devices. If half the ports on each Tier 1 device are sitting unused, you can halve the number of Tier 1 devices and just map two uplinks from a Tier 2 device onto the same Tier 1 device, instead of onto two different ones. Same bandwidth, fewer boxes in Tier 1, and that means you save on CAPEX 🤑. The tradeoff is that the maximum DC size, counted in overall server count, gets cut in half 😭.

In this example, Tier 2 devices use two parallel links to connect to each Tier 1 device. If one of those links fails, the other one picks up all the traffic from the failed link 🫠. That can cause heavy congestion and quality of service degradation if the path determination procedure doesn't take bandwidth into account, since the number of upstream Tier 1 devices is likely more than two. To avoid this mess, parallel links can be grouped into link aggregation groups (LAGs), e.g., [IEEE8023AD], with widely available implementation settings that take the whole "bundle" down when a single link fails 🧚. Equivalent techniques that enforce "fate sharing" on the parallel links can be used in place of LAGs to get the same effect. Because of that fate sharing, traffic from two or more failed links gets rebalanced across the multitude of remaining paths, which equals the number of Tier 1 devices. We're using two links here just for simplicity. Having more links in a bundle means a member-link failure hits capacity less 🌈.

4. Data Center Routing Overview ✨

This section gives you an overview of three general types of data center protocol designs: Layer 2 only, Hybrid Layer L2/L3, and Layer 3 only 🦋.

4.1. L2-Only Designs 🌸

Originally, most data center designs used the Spanning Tree Protocol (STP), originally defined in [IEEE8021D-1990], to create loop-free topologies. They typically used variants of the traditional DC topology described in Section 3.1 🍓. Back then, lots of DC switches either didn't support Layer 3 routing protocols or only supported them with extra licensing fees, and that played a part in the design choice 💸. Lots of enhancements came along with the Rapid Spanning Tree Protocol (RSTP) in the latest revision of [IEEE8021D-2004] and the Multiple Spanning Tree Protocol (MST) specified in [IEEE8021Q], which improve convergence, stability, and load-balancing in larger topologies. But many of the fundamentals of the protocol still limit its applicability in large-scale DCs 🙈. STP and its newer variants use an active/standby approach to path selection, so they're hard to deploy in horizontally scaled topologies as described in Section 3.2. Also, operators have had plenty of experiences with large failures caused by improper cabling, misconfiguration, or flawed software on a single device 💀. These failures regularly affected the entire spanning-tree domain and were super hard to troubleshoot because of how the protocol works.

For these reasons, and since almost all DC traffic is now IP (which means a Layer 3 routing protocol is needed at the network edge for external connectivity), designs using STP usually fail to meet the requirements of large-scale DC operators 😤. Various enhancements to link-aggregation protocols such as [IEEE8023AD], generally known as Multi-Chassis Link-Aggregation (M-LAG), made it possible to build Layer 2 designs with active-active network paths while still relying on STP as the backup for loop prevention 🔥. The major downsides of this approach are the lack of ability to scale linearly past two in most implementations, the lack of standards-based implementations, and the added failure domain risk of syncing state between the devices 🫶.

It should be noted that building large, horizontally scalable, L2-only networks without STP has recently become possible through the Transparent Interconnection of Lots of Links (TRILL) protocol in [RFC6325] ✨. TRILL solves many of the issues STP has for large-scale DC design. However, because there are a limited number of implementations, and often a requirement for specific equipment that supports it, its applicability has been limited and the cost of such designs has gone up 💅.

Finally, neither the base TRILL specification nor the M-LAG approach totally eliminates the problem of the shared broadcast domain that is so detrimental to the operations of any Layer 2, Ethernet-based solution 😩. Later TRILL extensions have been proposed to solve this problem statement, primarily based on the approaches outlined in [RFC7067], but that further limits the number of available interoperable implementations that can be used to build a fabric. Therefore, TRILL-based designs have issues meeting REQ2, REQ3, and REQ4 🥺.

4.2. Hybrid L2/L3 Designs 🧚‍♀️

Operators have wanted to limit the impact of data-plane faults and build large-scale topologies. They did this by implementing routing protocols in either the Tier 1 or Tier 2 parts of the network and dividing the Layer 2 domain into numerous, smaller domains 🧩. This design let data centers scale up, but at the cost of complexity in managing multiple network protocols. For the following reasons, operators have kept Layer 2 in either the access (Tier 3) or both the access and aggregation (Tier 3 and Tier 2) parts of the network:

  • Supporting legacy applications that may require direct Layer 2 adjacency or use non-IP protocols.
  • Seamless mobility for virtual machines that require the preservation of IP addresses when a virtual machine moves to a different Tier 3 switch 🦋.
  • Simplified IP addressing = fewer IP subnets are required for the data center.
  • Application load balancing may require direct Layer 2 reachability to perform certain functions, such as Layer 2 Direct Server Return (DSR). See [L3DSR].
  • Continued CAPEX differences between L2- and L3-capable switches 💸.

4.3. L3-Only Designs 🌷

Network designs that leverage IP routing all the way down to Tier 3 of the network have gained popularity too 📈. The main benefit of these designs is improved network stability and scalability, because they confine L2 broadcast domains. Commonly, an Interior Gateway Protocol (IGP) such as Open Shortest Path First (OSPF) [RFC2328] is used as the primary routing protocol in such a design. As data centers grow in scale and server count exceeds tens of thousands, these fully routed designs have become more attractive 👑.

Choosing an L3-only design greatly simplifies the network, which helps with meeting REQ1 and REQ2 🎯. It has widespread adoption in networks where large Layer 2 adjacency and larger Layer 3 subnets are not as critical as network scalability and stability. Application providers and network operators keep developing new solutions to meet some of the requirements that used to drive large Layer 2 domains, using various overlay or tunneling techniques 🌈.

5. Routing Protocol Design 💖

In this section, we review the motivations for using External BGP (EBGP) as the single routing protocol for data center networks that have a Layer 3 protocol design and a Clos topology 🫧. Then we get into a practical approach for designing an EBGP-based network 🎀.

5.1. Choosing EBGP as the Routing Protocol 🦋

okay so REQ2 is all about picking ONE routing protocol so your network stays simple and nothing depends on everything else 🫶 totally makes sense. everyone usually reaches for an IGP here, sometimes with EBGP added on the device that borders the WAN, or with Internal BGP (IBGP) running everywhere. but this doc is like "what if we just went EBGP-only?" 👑

now, EBGP runs basically all Inter-Domain Routing on the Internet and everyone (vendors and service providers) is super on board with it, BUT it's not really the main routing protocol inside a data center, and ngl the reasons are kinda tangled together 💀

  • BGP gets labeled as a "WAN-only, protocol-only" thing and people don't really think of it for enterprise or data center stuff 🙈
  • BGP is believed to have a "much slower" convergence than IGPs 🐢
  • big BGP deployments usually need an IGP to resolve BGP next hops, because not every node in the IBGP topology is directly connected 🧩
  • BGP is perceived to need a ton of config and it doesn't do neighbor auto-discovery 😩

this doc is gonna poke at some of those vibes, especially how they apply to this design, and show off some perks of BGP like:

  • BGP is less complicated in some parts of its design 🧠✨ its internal data structures and state machine are simpler than most link-state IGPs like OSPF. instead of building adjacency formation, adjacency maintenance, and/or flow control, BGP just leans on TCP as its transport. slay, that covers REQ2 and REQ3 💅
  • BGP floods way less info than link-state IGPs 🌊 every BGP router only calculates and passes along its best path, so when something fails, it's hidden as soon as the BGP speaker finds an alternate path. and that alternate path usually exists when a super symmetric topology like Clos is paired with an EBGP-only design 😍 link-state IGPs, on the other hand, scope their event propagation to the whole area no matter what kind of failure happened. so BGP honestly does a better job with REQ3 and REQ4 🔥 also worth knowing: every widely deployed link-state IGP does periodic refreshes of routing info, while BGP never expires routing state. this rarely matters with modern router control planes, though 🤷‍♀️
  • BGP supports third-party (recursively resolved) next hops 🎯 that means you can make multipath non-ECMP-based or forwarding-based on application-defined paths, by peering with an application "controller" that injects routing info into the system. that's REQ5 handled ✅ OSPF can do something similar with stuff like "Forwarding Address", but it's way harder to implement and has far less control over how far information spreads 😤
  • with a well-defined Autonomous System Number (ASN) allocation scheme and standard AS_PATH loop detection, "BGP path hunting" (see [JAKMA2008]) can be controlled, and complicated unwanted paths get ignored 🙌 see Section 5.2 for an example of a working ASN allocation scheme. doing the same thing in a link-state IGP would need multi-(instance/topology/process) support, which isn't available on all DC devices and is super complex to configure and troubleshoot 😵‍💫 and a traditional single flooding domain, which most DC designs use, can pick up really long unwanted paths under certain failure conditions, like ones going through multiple Tier 2 devices 🐌
  • EBGP with minimal routing policy is way easier to troubleshoot for reachability issues 🔍 in most implementations you can look at the BGP Loc-RIB and compare it to the router's Routing Information Base (RIB) with no drama. operators can also check every BGP neighbor's Adj-RIB-In and Adj-RIB-Out, so the incoming and outgoing Network Layer Reachability Information (NLRI) can be matched up on both sides of a BGP session. that's why BGP satisfies REQ3 🥹💖

5.2. EBGP Configuration for Clos Topology 🌸

okay bestie, Clos topologies with more than 5 stages are super rare because they need a LOT of interconnects 🕸️ so the examples below are all based on the 5-stage Clos topology (in its unfolded state) ✨

5.2.1. EBGP Configuration Guidelines and Example ASN Scheme 🎀

the diagram below shows an example ASN allocation scheme 👀 here's a list of guidelines you can use:

  • EBGP single-hop sessions run over direct point-to-point links between the network nodes. no multi-hop or loopback sessions, even when there are multiple links between the same pair of nodes 🚫
  • Private Use ASNs from the range 64512-65534 are used to avoid ASN conflicts 🧁
  • one single ASN is given to all of the Clos topology's Tier 1 devices 👑
  • each set of Tier 2 devices in the same cluster gets its own unique ASN 🍓
  • every Tier 3 device (e.g., ToR) in this topology gets a unique ASN 🧚
                            ASN 65534
                           +---------+
                           | +-----+ |
                           | |     | |
                         +-|-|     |-|-+
                         | | +-----+ | |
              ASN 646XX  | |         | |  ASN 646XX
             +---------+ | |         | | +---------+
             | +-----+ | | | +-----+ | | | +-----+ |
 +-----------|-|     |-|-+-|-|     |-|-+-|-|     |-|-----------+
 |       +---|-|     |-|-+ | |     | | +-|-|     |-|---+       |
 |       |   | +-----+ |   | +-----+ |   | +-----+ |   |       |
 |       |   |         |   |         |   |         |   |       |
 |       |   |         |   |         |   |         |   |       |
 |       |   | +-----+ |   | +-----+ |   | +-----+ |   |       |
 | +-----+---|-|     |-|-+ | |     | | +-|-|     |-|---+-----+ |
 | |     | +-|-|     |-|-+-|-|     |-|-+-|-|     |-|-+ |     | |
 | |     | | | +-----+ | | | +-----+ | | | +-----+ | | |     | |
 | |     | | +---------+ | |         | | +---------+ | |     | |
 | |     | |             | |         | |             | |     | |
+-----+ +-----+           | | +-----+ | |           +-----+ +-----+
| ASN | |     |           +-|-|     |-|-+           |     | |     |
|65YYY| | ... |             | |     | |             | ... | | ... |
+-----+ +-----+             | +-----+ |             +-----+ +-----+
 | |     | |               +---------+               | |     | |
 O O     O O              <- Servers ->              O O     O O

             Figure 4: BGP ASN Layout for 5-Stage Clos

5.2.2. Private Use ASNs 🧸

okay so the original Private Use ASN range [RFC6996] only let operators have 1023 unique ASNs 😬 and since you'll probably have more network devices than that, you need a workaround 🛠️ one approach is to reuse the ASNs assigned to Tier 3 devices across different clusters. for example, Private Use ASNs 65001, 65002 ... 65032 could be used inside every individual cluster and assigned to Tier 3 devices.

now, to avoid route suppression from the AS_PATH loop detection mechanism in BGP, upstream EBGP sessions on Tier 3 devices must be configured with the "Allowas-in" feature [ALLOWASIN], which lets them accept a device's own ASN in received route advertisements 🪞 this feature isn't standardized, but it's widely available across multiple vendors' implementations. adding it doesn't make routing loops more likely here, since routers add to the AS_PATH at every tier and AS_PATH length is an early tie breaker in the BGP path selection process 📏 extra loop protection still lives at the Tier 1 device, which won't accept routes with a path that includes its own ASN 🛡️ and Tier 2 devices don't have direct connectivity with each other 🙅‍♀️

another fix is Four-Octet ASNs ([RFC6793]), which come with extra Private Use ASNs, see [IANA.AS]. BUT using Four-Octet ASNs adds complexity to the BGP implementation, so you should weigh that against the complexity of reuse when you're thinking about REQ3 and REQ4 ⚖️ and honestly, more importantly, they aren't supported by every BGP implementation yet, which could limit which DC equipment vendors you can pick 😭 when they are supported, make sure the deployed implementations can remove the Private Use ASNs when external connectivity (Section 5.2.4) to those ASNs is needed 🫶

5.2.3. Prefix Advertisement 📣

a Clos topology has a huge number of point-to-point links and their prefixes 🕸️ advertising all of those routes into BGP can overload the Forwarding Information Base (FIB) on network devices 💀 and advertising those links also piles on path computation stress for the BGP control plane for almost no benefit 🙃 so there are two possible solutions:

  • don't advertise any of the point-to-point links into BGP 🙈 since the EBGP-based design changes the next-hop address at every device, distant networks are automatically reachable via the advertising EBGP peer and don't need reachability to those prefixes. BUT this might complicate operations or monitoring: for example, the popular "traceroute" tool will show IP addresses that aren't reachable 😵
  • advertise the point-to-point links, but summarize them on every device 🧃 this needs an address allocation scheme, like giving a consecutive block of IP addresses to each Tier 1 and Tier 2 device to use for point-to-point interface addressing toward the lower layers (Tier 2 uplinks get allocated from Tier 1 address blocks, and so forth) 🧩

server subnets on Tier 3 devices must be announced into BGP WITHOUT route summarization on Tier 2 and Tier 1 devices 🚨 summarizing subnets in a Clos topology causes route black-holing under a single link failure (e.g., between Tier 2 and Tier 3 devices), so it must be avoided ❌ using peer links within the same tier to fix black-holing with "bypass paths" is undesirable, because the peering mesh is O(N^2) complexity and wastes ports on the devices 😩 an alternative to the full mesh of peer links would be a simpler bypass topology, like a "ring" described in [FB4POST], but that adds extra hops and has limited bandwidth 🐢 it may also need special tweaks to make BGP routing work, like splitting every device into an ASN of its own 🧑‍🔧 later in this document, Section 8.2 introduces a less intrusive method for doing a limited form of route summarization in Clos networks and talks about its tradeoffs 👀💖

5.2.4. External Connectivity 🌐

Okay so, a dedicated cluster (or clusters) in the Clos topology could be used for connecting to the Wide Area Network (WAN) edge devices, aka WAN Routers 💅. The Tier 3 devices in that cluster would be replaced with WAN routers, and EBGP peering would be used again 🫶. WAN routers are likely to belong to a public ASN if Internet connectivity is required in the design, so keep that in mind! The Tier 2 devices in such a dedicated cluster will be referred to as "Border Routers" in this document. These devices have to perform a few special functions ✨:

  • Hide network topology information when advertising paths to WAN routers, i.e., remove Private Use ASNs [RFC6996] from the AS_PATH attribute. This is typically done to avoid ASN number collisions between different data centers, and also to provide a uniform AS_PATH length to the WAN for purposes of WAN ECMP to anycast prefixes originated in the topology. An implementation-specific BGP feature typically called "Remove Private AS" is commonly used to accomplish this. Depending on implementation, the feature should strip a contiguous sequence of Private Use ASNs found in an AS_PATH attribute prior to advertising the path to a neighbor. This assumes that all ASNs used for intra data center numbering are from the Private Use ranges. The process for stripping the Private Use ASNs is not currently standardized, see [REMOVAL]. However, most implementations at least follow the logic described in this vendor's document [VENDOR-REMOVE-PRIVATE-AS], which is enough for the design specified 🙌.
  • Originate a default route to the data center devices. This is the only place where a default route can be originated, because route summarization is risky for the unmodified Clos topology 🙅‍♀️. Alternatively, Border Routers may simply relay the default route learned from WAN routers. Advertising the default route from Border Routers requires that all Border Routers be fully connected to the WAN Routers upstream, to provide resistance to a single-link failure causing the black-holing of traffic. To prevent black-holing in the situation when all of the EBGP sessions to the WAN routers fail simultaneously on a given device, it is more desirable to readvertise the default route rather than originating the default route via complicated conditional route origination schemes provided by some implementations [CONDITIONALROUTE]. Honestly, the readvertising route is the less drama option 💀.

5.2.5. Route Summarization at the Edge 🦋

It is often desirable to summarize network reachability information prior to advertising it to the WAN network, because a fully routed network design originates a high amount of IP prefixes from within the data center. For example, a network with 2000 Tier 3 devices will have at least 2000 servers subnets advertised into BGP, along with the infrastructure prefixes 😭. However, as discussed in Section 5.2.3, the proposed network design does not allow for route summarization due to the lack of peer links inside every tier.

However, it is possible to lift this restriction for the Border Routers by devising a different connectivity model for these devices. There are two options possible 👑:

  • Interconnect the Border Routers using a full-mesh of physical links or using any other "peer-mesh" topology, such as ring or hub-and-spoke. Configure BGP accordingly on all Border Leafs to exchange network reachability information, e.g., by adding a mesh of IBGP sessions. The interconnecting peer links need to be appropriately sized for traffic that will be present in the case of a device or link failure in the mesh connecting the Border Routers.
  • Tier 1 devices may have additional physical links provisioned toward the Border Routers (which are Tier 2 devices from the perspective of Tier 1). Specifically, if protection from a single link or node failure is desired, each Tier 1 device would have to connect to at least two Border Routers. This puts additional requirements on the port count for Tier 1 devices and Border Routers, potentially making it a nonuniform, larger port count, device compared with the other devices in the Clos. This also reduces the number of ports available to "regular" Tier 2 switches, and hence the number of clusters that could be interconnected via Tier 1.

If any of the above options are implemented, it is possible to perform route summarization at the Border Routers toward the WAN network core without risking a routing black-hole condition under a single link failure. Both of the options would result in nonuniform topology, as additional links have to be provisioned on some network devices 🫠.

6. ECMP Considerations 🎀

This section covers the Equal Cost Multipath (ECMP) functionality for Clos topology and discusses a few special requirements 🧚.

6.1. Basic ECMP

ECMP is the fundamental load-sharing mechanism used by a Clos topology. Effectively, every lower-tier device will use all of its directly attached upper-tier devices to load-share traffic destined to the same IP prefix. The number of ECMP paths between any two Tier 3 devices in Clos topology is equal to the number of the devices in the middle stage (Tier 1). For example, Figure 5 illustrates a topology where Tier 3 device A has four paths to reach servers X and Y, via Tier 2 devices B and C and then Tier 1 devices 1, 2, 3, and 4, respectively 🍓.

                                Tier 1
                               +-----+
                               | DEV |
                            +->|  1  |--+
                            |  +-----+  |
                    Tier 2  |           |   Tier 2
                   +-----+  |  +-----+  |  +-----+
     +------------>| DEV |--+->| DEV |--+--|     |-------------+
     |       +-----|  B  |--+  |  2  |  +--|     |-----+       |
     |       |     +-----+     +-----+     +-----+     |       |
     |       |                                         |       |
     |       |     +-----+     +-----+     +-----+     |       |
     | +-----+---->| DEV |--+  | DEV |  +--|     |-----+-----+ |
     | |     | +---|  C  |--+->|  3  |--+--|     |---+ |     | |
     | |     | |   +-----+  |  +-----+  |  +-----+   | |     | |
     | |     | |            |           |            | |     | |
   +-----+ +-----+          |  +-----+  |          +-----+ +-----+
   | DEV | |     | Tier 3   +->| DEV |--+   Tier 3 |     | |     |
   |  A  | |     |             |  4  |             |     | |     |
   +-----+ +-----+             +-----+             +-----+ +-----+
     | |     | |                                     | |     | |
     O O     O O            <- Servers ->            X Y     O O

               Figure 5: ECMP Fan-Out Tree from A to X and Y

The ECMP requirement implies that the BGP implementation must support multipath fan-out for up to the maximum number of devices directly attached at any point in the topology in the upstream or downstream direction. Normally, this number does not exceed half of the ports found on a device in the topology. For example, an ECMP fan-out of 32 would be required when building a Clos network using 64-port devices. The Border Routers may need to have wider fan-out to be able to connect to a multitude of Tier 1 devices if route summarization at Border Router level is implemented as described in Section 5.2.5. If a device's hardware does not support wider ECMP, logical link-grouping (link-aggregation at Layer 2) could be used to provide "hierarchical" ECMP (Layer 3 ECMP coupled with Layer 2 ECMP) to compensate for fan-out limitations. However, this approach increases the risk of flow polarization, as less entropy will be available at the second stage of ECMP ⚠️.

Most BGP implementations declare paths to be equal from an ECMP perspective if they match up to and including step (e) in Section 9.1.2.2 of [RFC4271]. In the proposed network design there is no underlying IGP, so all IGP costs are assumed to be zero or otherwise the same value across all paths, and policies may be applied as necessary to equalize BGP attributes that vary in vendor defaults, such as the MULTI_EXIT_DISC (MED) attribute and origin code. For historical reasons, it is also useful to not use 0 as the equalized MED value; this and some other useful BGP information is available in [RFC4277]. Routing loops are unlikely due to the BGP best-path selection process (which prefers shorter AS_PATH length), and longer paths through the Tier 1 devices (which don't allow their own ASN in the path) are not possible 🔁.

6.2. BGP ECMP over Multiple ASNs

For application load-balancing purposes, it is desirable to have the same prefix advertised from multiple Tier 3 devices. From the perspective of other devices, such a prefix would have BGP paths with different AS_PATH attribute values, while having the same AS_PATH attribute lengths. Therefore, BGP implementations must support load-sharing over the above-mentioned paths. This feature is sometimes known as "multipath relax" or "multipath multiple-AS" and effectively allows for ECMP to be done across different neighboring ASNs if all other attributes are equal, as already described in the previous section 🌈.

6.3. Weighted ECMP

It may be desirable for the network devices to implement "weighted" ECMP, to be able to send more traffic over some paths in ECMP fan-out. This could be helpful to compensate for failures in the network and send more traffic over paths that have more capacity 💜. The prefixes that require weighted ECMP would have to be injected using remote BGP speaker (central agent) over a multi-hop session, as described further in Section 8.1. If support in implementations is available, weight distribution for multiple BGP paths could be signaled using the technique described in [LINK].

6.4. Consistent Hashing

It is often desirable to have the hashing function used for ECMP be consistent (see [CONS-HASH]), to minimize the impact on flow to next-hop affinity changes when a next hop is added or removed to an ECMP group. This could be used if the network device is used as a load balancer, mapping flows toward multiple destinations. In this case, losing or adding a destination will not have a detrimental effect on currently established flows. One particular recommendation on implementing consistent hashing is provided in [RFC2992], though other implementations are possible. This functionality could be naturally combined with weighted ECMP, with the impact of the next hop changes being proportional to the weight of the given next hop. The downside of consistent hashing is increased load on hardware resource utilization, as typically more resources (e.g., Ternary Content-Addressable Memory (TCAM) space) are required to implement a consistent-hashing function 🥺.

7. Routing Convergence Properties ⭐

This section reviews routing convergence properties in the proposed design. A case is made that sub-second convergence is achievable if the implementation supports fast EBGP peering session deactivation and timely RIB and FIB updates upon failure of the associated link 🏃‍♀️💨.

7.1. Fault Detection Timing 🔍

okay so BGP usually leans on an IGP to route around link and node failures inside an AS 🛣️. It either polls for IGP state changes or listens for event-driven updates about them. The proposed routing design doesn't use an IGP at all, so the fault detection options left are the BGP keep-alive timeout (or any other kind of keep-alive mechanism) and link-failure triggers. 🫶

Relying only on BGP keep-alive packets can cause high convergence delays, like multiple seconds 😭 (on lots of BGP implementations the minimum configurable BGP hold timer value is three seconds). BUT, many BGP implementations can shut down local EBGP peering sessions when the "link down" event fires on the outgoing interface used for BGP peering. This is sometimes called "fast fallover" ⚡. Links in modern data centers are mostly point-to-point fiber, so a physical interface failure is often detected in milliseconds and then kicks off a BGP reconvergence. ✨

Ethernet links may support failure signaling or detection standards like Connectivity Fault Management (CFM), described in [IEEE8021Q]. This may make failure detection more robust 💪. Another option is Bidirectional Forwarding Detection (BFD) [RFC5880], which some platforms support for sub-second failure detection and fault signaling to the BGP process 🚀. Here's the catch though: using either of these adds requirements to vendor software and possibly hardware, and may contradict REQ1 🙈. Until recently with [RFC7130], BFD also couldn't detect a single member link failure on a LAG, which would have limited its usefulness in some designs. Not ideal, bestie 💀.

7.2. Event Propagation Timing ⏱️

In the proposed design, you should consider the impact of the BGP MinRouteAdvertisementIntervalTimer (MRAI timer), as specified in Section 9.2.1.1 of [RFC4271]. Per the standard, it is required for BGP implementations to space out consecutive BGP UPDATE messages by at least MRAI seconds, which is often a configurable value 🎀. The initial BGP UPDATE messages after an event carrying withdrawn routes are commonly not affected by this timer. The MRAI timer can cause significant convergence delays when a BGP speaker "waits" for the new path to be learned from its peers and has no local backup path information 😩.

In a Clos topology, each EBGP speaker typically has either one path (Tier 2 devices don't accept paths from other Tier 2 devices in the same cluster because they share the same ASN) or N paths for the same prefix, where N is a significantly large number, e.g., N=32 (the ECMP fan-out to the next tier) 🌈. So if a link fails to another device that a path was received from, there's either no backup path at all (e.g., a Tier 2 switch losing the link to a Tier 3 device), or the backup is already sitting in the BGP Loc-RIB (e.g., a Tier 2 device losing the link to a Tier 1 switch) 🧚. In the former case, the BGP withdrawal announcement propagates without delay and triggers reconvergence on the affected devices. In the latter case, the best path gets re-evaluated and the local ECMP group for the new next-hop set changes. If the BGP path was the best path selected previously, an "implicit withdraw" is sent via a BGP UPDATE message, as described in Option b in Section 3.1 of [RFC4271], because the BGP AS_PATH attribute changes 🔄.

7.3. Impact of Clos Topology Fan-Outs 🌸

Clos topologies have large fan-outs, and those can affect "Up->Down" convergence in some cases 🌪️. Say a link between a Tier 3 and a Tier 2 device fails. The Tier 2 device sends BGP UPDATE messages to all of its upstream Tier 1 devices, withdrawing the affected prefixes. The Tier 1 devices then relay those messages to all downstream Tier 2 devices (except the originator). Tier 2 devices other than the one that originated the UPDATE should then wait for ALL upstream Tier 1 devices to send an UPDATE message before removing the affected prefixes and sending the corresponding UPDATE downstream to connected Tier 3 devices ⏳. If the original Tier 2 device or the relaying Tier 1 devices add delay to their UPDATE announcements, you can get UPDATE message "dispersion", which could last multiple seconds 😵. To avoid that behavior, BGP implementations must support "update groups". An "update group" is a collection of neighbors sharing the same outbound policy -- the local speaker sends BGP updates to the members of the group synchronously 🧑‍🤝‍🧑.

The impact of "dispersion" grows with the size of the topology fan-out, and it could also grow under network convergence churn 📈. Some operators may be tempted to add "route flap dampening" type features, which vendors include to reduce the control-plane impact of rapidly flapping prefixes 🙃. However, because of the false positives these implementations cause, especially during "dispersion" events, it is not recommended to enable this feature in this design 🚫. More background, the issues with "route flap dampening", and possible implementation changes that could affect this are well described in [RFC7196] 📚.

7.4. Failure Impact Scope 🎯

A network is declared to have converged after a failure once all devices within the failure impact scope have been notified of the event, recalculated their RIBs, and updated their FIBs ✅. A larger failure impact scope usually means slower convergence, since more devices have to be notified, and it makes the network less stable 😬. In this section, we look at BGP's advantages over link-state routing protocols for reducing the failure impact scope in a Clos topology 💅.

BGP acts like a distance-vector protocol in the sense that only the best path, from the point of view of the local router, is sent to neighbors 📨. So some failures get masked if the local node can immediately find a backup path and doesn't have to send any further updates 🦋. In the worst case, though, every device in a data center topology has to either withdraw a prefix completely or update the ECMP groups in its FIB 🥲. Many failures won't have such a wide impact, though. There are two main failure types where the impact scope is reduced:

  • Failure of a link between Tier 2 and Tier 1 devices: In this case, a Tier 2 device updates the affected ECMP groups, removing the failed link. There's no need to send new information to downstream Tier 3 devices, unless the path was selected as best by the BGP process. In that case, only an "implicit withdraw" needs to be sent, and this should not affect forwarding 👍. The affected Tier 1 device loses the only path available to reach a particular cluster and has to withdraw the associated prefixes. That prefix withdrawal process only affects the Tier 2 devices directly connected to the affected Tier 1 device. The Tier 2 devices receiving the BGP UPDATE messages withdrawing prefixes simply have to update their ECMP groups. The Tier 3 devices aren't involved in the reconvergence process 🙌.
  • Failure of a Tier 1 device: In this case, all Tier 2 devices directly attached to the failed node have to update their ECMP groups for all IP prefixes from a non-local cluster. The Tier 3 devices, once again, are not involved in the reconvergence process, but they may receive "implicit withdraws" as described above 🫠.

Even in cases like these, where multiple IP prefixes have to be reprogrammed in the FIB, all of those prefixes share a single ECMP group on a Tier 2 device 🤯. So with implementations that have a hierarchical FIB, only a single change has to be made to the FIB. "Hierarchical FIB" here means a FIB structure where the next-hop forwarding information is stored separately from the prefix lookup table, and the lookup table only stores pointers to the respective forwarding information 🧩. See [BGP-PIC] for a discussion of FIB hierarchies and fast convergence 📖.

Even though BGP offers a reduced failure scope in some cases, shrinking the fault domain further with summarization isn't always possible in the proposed design. Using that technique can create routing black-holes, as mentioned previously 🕳️. So the worst failure impact scope on the control plane is the network as a whole -- for instance, in the case of a link failure between Tier 2 and Tier 3 devices. The number of impacted prefixes in that case would be much smaller than for a failure in the upper layers of a Clos network topology. Having such a large failure scope isn't a result of choosing EBGP in the design. It's a result of using the Clos topology 💜.

7.5. Routing Micro-Loops 🔁

When a downstream device, e.g., a Tier 2 device, loses all paths for a prefix, it normally has the default route pointing toward the upstream device, which in this case is the Tier 1 device 🧭. So it's possible for a Tier 2 switch to lose a prefix while a Tier 1 switch still has a path pointing to the Tier 2 device. This results in a transient micro-loop 🌀: the Tier 1 switch keeps passing packets for the affected prefix back to the Tier 2 device, and the Tier 2 device bounces them right back up again using the default route. The micro-loop lasts for as long as it takes the upstream device to fully update its forwarding tables 😤.

To minimize the impact of such micro-loops, Tier 2 and Tier 1 switches can be configured with static "discard" or "null" routes that are more specific than the default route for prefixes missing during network convergence 🛑. For Tier 2 switches, the discard route should be a summary route covering all server subnets of the underlying Tier 3 devices. For Tier 1 devices, the discard route should be a summary covering the server IP address subnets allocated for the whole data center 🏢. Those discard routes only take precedence for the duration of network convergence, until the device learns a more specific prefix via a new path 🎉.

8. Additional Options for Design 💅

8.1. Third-Party Route Injection 💉

BGP allows a "third-party", i.e., a directly attached BGP speaker, to inject routes anywhere in the network topology, which meets REQ5 🙌. This can be done by peering via a multi-hop BGP session with some or even all devices in the topology. Also, BGP diverse path distribution [RFC6774] could be used to inject multiple BGP next hops for the same prefix to help with load balancing, or the BGP ADD-PATH capability [RFC7911] could be used if the implementation supports it 🛤️. Unfortunately, in many implementations ADD-PATH has only been found to support IBGP properly in the use cases it was originally optimized for. That limits "third-party" peering to IBGP only 😔.

To implement route injection in the proposed design, a third-party BGP speaker may peer with Tier 3 and Tier 1 switches, injecting the same prefix but using a special set of BGP next hops for Tier 1 devices 🎀. Those next hops are assumed to resolve recursively via BGP, and could be, for example, IP addresses on Tier 3 devices. The resulting forwarding table programming could provide the desired traffic proportion distribution among different clusters 📊✨.

8.2. Route Summarization within Clos Topology ✨

okay so bestie, route summarization is NOT possible inside the proposed Clos topology 🙅‍♀️💀 because it makes the network super susceptible to route black-holing under single link failures. The big problem is that there are only so many redundant paths between network elements. Like, there is only a single path between any pair of Tier 1 and Tier 3 devices 😭 However, some operators may still find route aggregation desirable to improve control-plane stability, and I get it 🫶

If any technique to summarize within the topology is planned, you have to model the routing behavior and the potential for black-holing 🔍 And it's not just single or multiple link failures! You also need to think about fiber pathway failures or optical domain failures when the topology extends beyond a physical location 🌈 Simple modeling can be done by checking reachability on the devices doing summarization, under the condition of a link or pathway failure between a set of devices in every tier, as well as to the WAN routers when external connectivity is present. Very thorough, very iconic 💅

Route summarization would be possible with a small modification to the network topology 🦋 The tradeoff is a reduction of the total size of the network, plus network congestion under specific failures. Not ideal, but it's giving "choose your fighter" 👑 This approach is very similar to the technique described above, which allows Border Routers to summarize the entire data center address space 🎀

8.2.1. Collapsing Tier 1 Devices Layer 🌸

okay so to add more paths between Tier 1 and Tier 3 devices, you group the Tier 2 devices into pairs, and then connect each pair to the same group of Tier 1 devices 🧚 This is logically equivalent to "collapsing" Tier 1 devices into a group of half the size, merging the links on the "collapsed" devices. Iykyk 🙈 Check out Figure 6 below. For example, in this topology DEV C and DEV D connect to the same set of Tier 1 devices (DEV 1 and DEV 2), whereas before they were connecting to different groups of Tier 1 devices 💜

                Tier 2       Tier 1       Tier 2
               +-----+      +-----+      +-----+
 +-------------| DEV |------| DEV |------|     |-------------+
 |       +-----|  C  |--++--|  1  |--++--|     |-----+       |
 |       |     +-----+  ||  +-----+  ||  +-----+     |       |
 |       |              ||           ||              |       |
 |       |     +-----+  ||  +-----+  ||  +-----+     |       |
 | +-----+-----| DEV |--++--| DEV |--++--|     |-----+-----+ |
 | |     | +---|  D  |------|  2  |------|     |---+ |     | |
 | |     | |   +-----+      +-----+      +-----+   | |     | |
 | |     | |                                       | |     | |
+-----+ +-----+                                   +-----+ +-----+
| DEV | | DEV |                                   |     | |     |
|  A  | |  B  | Tier 3                     Tier 3 |     | |     |
+-----+ +-----+                                   +-----+ +-----+
  | |     | |                                       | |     | |
  O O     O O             <- Servers ->             O O     O O

                  Figure 6: 5-Stage Clos Topology

Having this design in place, Tier 2 devices may be configured to advertise only a default route down to Tier 3 devices 🍓 If a link between Tier 2 and Tier 3 fails, the traffic gets re-routed via the second available path known to a Tier 2 switch. Nice 🙌 It is still NOT possible to advertise a summary route covering prefixes for a single cluster from Tier 2 devices, though, since each of them only has a single path down to that prefix. That would require dual-homed servers to pull off 😤 Also, this design is only resilient to single link failures 🥺 A double link failure can isolate a Tier 2 device from all paths toward a specific Tier 3 device, which causes a routing black-hole 💀

A result of the proposed topology modification is a reduction of the port capacity of Tier 1 devices. That limits the maximum number of attached Tier 2 devices, and therefore limits the maximum DC network size 📉 A larger network would need different Tier 1 devices with higher port density to implement this change 👑

Another problem is traffic rebalancing under link failures ⚠️ Since there are two paths from Tier 1 to Tier 3, a failure of the link between a Tier 1 and Tier 2 switch would push all the traffic that was taking the failed link onto the remaining path. This results in doubling the link utilization on the remaining link. Yikes, bestie 😭

8.2.2. Simple Virtual Aggregation 🦋

A completely different approach to route summarization is possible, as long as the main goal is to reduce the FIB size while still letting the control plane disseminate full routing information ✨ First, it can be easily noted that in many cases multiple prefixes, some of which are less specific, share the same set of next hops (the same ECMP group) 💖 For example, from the perspective of Tier 3 devices, all routes learned from upstream Tier 2 devices, including the default route, will share the same set of BGP next hops, provided there are no failures in the network. Main character energy 🎀 This makes it possible to use a technique similar to the one described in [RFC6769] and only install the least specific route in the FIB, ignoring more specific routes if they share the same next-hop set. So under normal network conditions, only the default route needs to be programmed into the FIB 🌈

Furthermore, if the Tier 2 devices are configured with summary prefixes covering all of their attached Tier 3 devices' prefixes, the same logic could be applied in Tier 1 devices as well, and by induction to Tier 2/Tier 3 switches in different clusters 🧚 These summary routes should still allow more specific prefixes to leak up to Tier 1 devices, so that mismatches in the next-hop sets can be detected if a particular link fails and changes the next-hop set for a specific prefix 🔍

Restating once again, this technique does NOT reduce the amount of control-plane state (i.e., BGP UPDATEs, BGP Loc-RIB size) 🙅‍♀️ It only allows for more efficient FIB utilization, by detecting more specific prefixes that share their next-hop set with a subsuming less specific prefix 💅

8.3. ICMP Unreachable Message Masquerading 🙈

This section discusses some operational aspects of NOT advertising point-to-point link subnets into BGP, which was previously identified as an option in Section 5.2.3 🎀 The operational impact of this decision shows up when you use the well-known "traceroute" tool 🔍 Specifically, the IP addresses displayed by the tool will be the link's point-to-point addresses, and those will be unreachable for management connectivity. Which makes some troubleshooting more complicated 😩

One way to get around this is to use the DNS subsystem to create the "reverse" entries for these point-to-point IP addresses, pointing to the same name as the loopback address 🌸 The connectivity can then be made by resolving this name to the "primary" IP address of the device, e.g., its Loopback interface, which is always advertised into BGP. However, this creates a dependency on the DNS subsystem, which may be unavailable during an outage 💀

Another option is to make the network device perform IP address masquerading 🦋 That means rewriting the source IP addresses of the appropriate ICMP messages sent by the device with the "primary" IP address of the device. Specifically, the ICMP Destination Unreachable Message (type 3) code 3 (port unreachable) and ICMP Time Exceeded (type 11) code 0 are REQUIRED for correct operation of the "traceroute" tool 🫶 With this modification, "traceroute" probes sent to the devices will always be sent back with the "primary" IP address as the source, so the operator can discover the "reachable" IP address of the box. The downside is that it hides the address of the "entry point" into the device 😭 If the devices support [RFC5837], this may allow the best of both worlds, by providing information about the incoming interface even when the return address is the "primary" IP address ✨

9. Security Considerations 🔐

The design does NOT introduce any additional security concerns 🫶 General BGP security considerations are discussed in [RFC4271] and [RFC4272]. Since a DC is a single-operator domain, this document assumes that edge filtering is in place to prevent attacks against the BGP sessions themselves from outside the perimeter of the DC. This may be a more feasible option for most deployments than dealing with key management for TCP MD5 as described in [RFC2385], or dealing with the lack of implementations of the TCP Authentication Option [RFC5925] available at the time of publication of this document 😤 The Generalized TTL Security Mechanism [RFC5082] could also be used to further reduce the risk of BGP session spoofing 🛡️

10. References 📚

10.1. Normative References 💜

[RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
           Border Gateway Protocol 4 (BGP-4)", RFC 4271,
           DOI 10.17487/RFC4271, January 2006,
           <http://www.rfc-editor.org/info/rfc4271>.

[RFC6996]  Mitchell, J., "Autonomous System (AS) Reservation for
           Private Use", BCP 6, RFC 6996, DOI 10.17487/RFC6996, July
           2013, <http://www.rfc-editor.org/info/rfc6996>.

10.2. Informative References 📚

okay bestie, this is the reading list for the stuff that's helpful but not required 🤓 every entry is listed in full below, with the tag in code so you can find it easily ✨

Acknowledgements 🫶

this doc is basically a group project with a lot of amazing people 💖 the authors want to shout out George Chen, Parantap Lahiri, Dave Maltz, Edet Nkposong, Robert Toomey, and Lihua Yuan, who helped build, test, and deploy the network design this document summarizes ✨ they also want to thank Linda Dunbar, Anoop Ghanwani, Susan Hares, Danny McPherson, Robert Raszuk, and Russ White for reviewing the document and giving super valuable feedback 🔥 and a big hug to Mary Mitchell for the early grammar and style suggestions 💅

Authors' Addresses 💌

Petr Lapukhov
Facebook
1 Hacker Way
Menlo Park, CA  94025
United States of America

Email: petr@fb.com


Ariff Premji
Arista Networks
5453 Great America Parkway
Santa Clara, CA  95054
United States of America

Email: ariff@arista.com
URI:   http://arista.com/


Jon Mitchell (editor)

Email: jrmitche@puck.nether.net
📦 where this page comes from